class MU::Cloud::Google::VPC

Creation of Virtual Private Clouds and associated artifacts (routes, subnets, etc).

Attributes

cloud_desc_cache[R]
routes[R]

Public Class Methods

can_route_to_master_peer?(source_subnets_key, target_subnets_key, instance_id) click to toggle source

Checks if the MU master has a route to a subnet in a peered VPC. Can be used on any subnets @param source_subnets_key [String]: The subnet/subnets on one side of the peered VPC. @param target_subnets_key [String]: The subnet/subnets on the other side of the peered VPC. @param instance_id [String]: The instance ID in the target subnet/subnets. @return [Boolean]

# File modules/mu/providers/google/vpc.rb, line 526
def self.can_route_to_master_peer?(source_subnets_key, target_subnets_key, instance_id)
end
cleanup(noop: false, deploy_id: MU.deploy_id, ignoremaster: false, credentials: nil, flags: {}) click to toggle source

Remove all VPC resources associated with the currently loaded deployment. @param noop [Boolean]: If true, will only print what would be done @param ignoremaster [Boolean]: If true, will remove resources not flagged as originating from this Mu server @return [void]

# File modules/mu/providers/google/vpc.rb, line 554
        def self.cleanup(noop: false, deploy_id: MU.deploy_id, ignoremaster: false, credentials: nil, flags: {})
          flags["habitat"] ||= MU::Cloud::Google.defaultProject(credentials)
          return if !MU::Cloud.resourceClass("Google", "Habitat").isLive?(flags["habitat"], credentials)
          filter = %Q{(labels.mu-id = "#{MU.deploy_id.downcase}")}
          if !ignoremaster and MU.mu_public_ip
            filter += %Q{ AND (labels.mu-master-ip = "#{MU.mu_public_ip.gsub(/\./, "_")}")}
          end
          MU.log "Placeholder: Google VPC artifacts do not support labels, so ignoremaster cleanup flag has no effect", MU::DEBUG, details: filter

          purge_subnets(noop, project: flags['habitat'], credentials: credentials)
          ["route", "network"].each { |type|
# XXX tagged routes aren't showing up in list, and the networks that own them
# fail to delete silently
            retries = 0

            begin
              MU::Cloud::Google.compute(credentials: credentials).delete(
                type,
                flags["habitat"],
                nil,
                noop
              )
            rescue MU::MuError, ::Google::Apis::ClientError => e
              if retries < 5
                if type == "network"
                  MU.log e.message, MU::WARN
                  if e.message.match(/Failed to delete network (.+)/)
                    network_name = Regexp.last_match[1]
                    fwrules = MU::Cloud.resourceClass("Google", "FirewallRule").find(project: flags['habitat'], credentials: credentials)
                    fwrules.reject! { |_name, desc|
                      !desc.network.match(/.*?\/#{Regexp.quote(network_name)}$/)
                    }
                    fwrules.keys.each { |name|
                      MU.log "Attempting to delete firewall rule #{name} so that VPC #{network_name} can be removed", MU::NOTICE
                      MU::Cloud::Google.compute(credentials: credentials).delete_firewall(flags['habitat'], name)
                    }
                  end
                end
                sleep retries*3
                retries += 1
                retry
              else
                raise e
              end
            end
          }

        end
find(**args) click to toggle source

Locate and return cloud provider descriptors of this resource type which match the provided parameters, or all visible resources if no filters are specified. At minimum, implementations of find must honor credentials and cloud_id arguments. We may optionally support other search methods, such as tag_key and tag_value, or cloud-specific arguments like project. See also {MU::MommaCat.findStray}. @param args [Hash]: Hash of named arguments passed via Ruby's double-splat @return [Hash<String,OpenStruct>]: The cloud provider's complete descriptions of matching resources

# File modules/mu/providers/google/vpc.rb, line 237
def self.find(**args)
  args = MU::Cloud::Google.findLocationArgs(args)

  resp = {}
  if args[:cloud_id] and args[:project]
    begin
      vpc = MU::Cloud::Google.compute(credentials: args[:credentials]).get_network(
      args[:project],
      args[:cloud_id].to_s.sub(/^.*?\/([^\/]+)$/, '\1')
    )
    resp[args[:cloud_id]] = vpc if !vpc.nil?
    rescue ::Google::Apis::ClientError
      MU.log "VPC #{args[:cloud_id]} in project #{args[:project]} does not exist, or I do not have permission to view it", MU::WARN
    end
  else # XXX other criteria
    vpcs = begin
      MU::Cloud::Google.compute(credentials: args[:credentials]).list_networks(
        args[:project]
      )
    rescue ::Google::Apis::ClientError => e
      raise e if !e.message.match(/^(?:notFound|forbidden): /)
    end

    if vpcs and vpcs.items
      vpcs.items.each { |v|
        resp[v.name] = v
      }
    end
  end

  resp
end
getInstanceSubnets(instance_id: nil, instance: nil, region: MU.curRegion) click to toggle source

Get the subnets associated with an instance. @param instance_id [String]: The cloud identifier of the instance @param instance [String]: A cloud descriptor for the instance, to save us an API call if we already have it @param region [String]: The cloud provider region of the target instance @return [Array<String>]

# File modules/mu/providers/google/vpc.rb, line 484
def self.getInstanceSubnets(instance_id: nil, instance: nil, region: MU.curRegion)
end
get_route_tables(subnet_ids: [], vpc_ids: [], region: MU.curRegion) click to toggle source

Retrieves the route tables of used by subnets @param subnet_ids [Array]: The cloud identifier of the subnets to retrieve the route tables for. @param vpc_ids [Array]: The cloud identifier of the VPCs to retrieve route tables for. @param region [String]: The cloud provider region of the target subnet. @return [Array<OpenStruct>]: The cloud provider's complete descriptions of the route tables

# File modules/mu/providers/google/vpc.rb, line 534
def self.get_route_tables(subnet_ids: [], vpc_ids: [], region: MU.curRegion)
end
haveRouteToInstance?(target_instance, credentials: nil) click to toggle source

Check whether we (the Mu Master) have a direct route to a particular instance. Useful for skipping hops through bastion hosts to get directly at child nodes in peered VPCs, the public internet, and the like. @param target_instance [OpenStruct]: The cloud descriptor of the instance to check. @return [Boolean]

# File modules/mu/providers/google/vpc.rb, line 496
        def self.haveRouteToInstance?(target_instance, credentials: nil)
          project ||= MU::Cloud::Google.defaultProject(credentials)
          return false if MU.myCloud != "Google"
# XXX see if we reside in the same Network and overlap subnets
# XXX see if we peer with the target's Network
          target_instance.network_interfaces.each { |iface|
            resp = MU::Cloud::Google.compute(credentials: credentials).list_routes(
              project,
              filter: "network eq #{iface.network}"
            )

            if resp and resp.items
MU.log "ROUTES TO #{target_instance.name}", MU::WARN, details: resp
            end
          }
          false
        end
isGlobal?() click to toggle source

Does this resource type exist as a global (cloud-wide) artifact, or is it localized to a region/zone? @return [Boolean]

# File modules/mu/providers/google/vpc.rb, line 540
def self.isGlobal?
  true
end
new(**args) click to toggle source

Initialize this cloud resource object. Calling super will invoke the initializer defined under {MU::Cloud}, which should set the attribtues listed in {MU::Cloud::PUBLIC_ATTRS} as well as applicable dependency shortcuts, like @vpc, for us. @param args [Hash]: Hash of named arguments passed via Ruby's double-splat

Calls superclass method
# File modules/mu/providers/google/vpc.rb, line 26
def initialize(**args)
  super

  @subnets ||= []
  @subnetcachesemaphore = Mutex.new

  loadSubnets if @cloud_id

  @mu_name ||= @config['scrub_mu_isms'] ? @config['name'] : @deploy.getResourceName(@config['name'])
end
pickVPC(vpc_block, my_config, my_type, configurator) click to toggle source

If the VPC a config block was set to one that's been “split,” try to figure out which of the new VPCs we really want to be in. For use by resource types that don't go in subnets, but do tie to VPCs. @param vpc_block [Hash] @param configurator [MU::Config] @return [Hash]

# File modules/mu/providers/google/vpc.rb, line 736
def self.pickVPC(vpc_block, my_config, my_type, configurator)
  _shortclass, cfg_name, cfg_plural, _classname = MU::Cloud.getResourceNames(my_type)
  return if vpc_block.nil?
  vpc_block['name'] ||= vpc_block['vpc_name']
  return if !vpc_block['name']

  vpcs = configurator.haveLitterMate?(
    nil,
    "vpcs",
    has_multiple: true
  )
  # drop all virtual vpcs that aren't real anymore
  vpcs.reject! { |v| v['virtual_name'] == v['name'] }
  # drop the ones that have nothing to do with us
  vpcs.reject! { |v| v['virtual_name'] != vpc_block['name'] }

  return vpc_block if vpcs.size == 0

  # see if one of this thing's siblings declared a subnet_pref we can
  # use to guess which one we should marry ourselves to
  configurator.kittens.values.each { |siblings|
    siblings.each { |sibling|
      next if !sibling['dependencies']
      sibling['dependencies'].each { |dep|
        if [cfg_name, cfg_plural].include?(dep['type']) and
           dep['name'] == my_config['name']
          vpcs.each { |v|
            if sibling['vpc']['name'] == v['name']
              vpc_block['name'] = v['name']
              return vpc_block
            end
          }
          if sibling['vpc']['subnet_pref']
            vpcs.each { |v|
              gateways = v['route_tables'].map { |rtb|
                rtb['routes'].map { |r| r["gateway"] }
              }.flatten.uniq
              if ["public", "all_public"].include?(sibling['vpc']['subnet_pref']) and
                 gateways.include?("#INTERNET")
                vpc_block['name'] = v['name']
                return vpc_block
              elsif ["private", "all_private"].include?(sibling['vpc']['subnet_pref']) and
                 !gateways.include?("#INTERNET")
                vpc_block['name'] = v['name']
                return vpc_block
              end
            }

          end
        end
      }
    }
  }

  vpc_block
end
quality() click to toggle source

Denote whether this resource implementation is experiment, ready for testing, or ready for production use.

# File modules/mu/providers/google/vpc.rb, line 546
def self.quality
  MU::Cloud::RELEASE
end
schema(_config = nil) click to toggle source

Cloud-specific configuration properties. @param _config [MU::Config]: The calling MU::Config object @return [Array<Array,Hash>]: List of required fields, and json-schema Hash of cloud-specific configuration parameters for this resource

# File modules/mu/providers/google/vpc.rb, line 710
def self.schema(_config = nil)
  toplevel_required = []
  schema = {
    "regions" => {
      "type" => "array",
      "items" => MU::Config.region_primitive
    },
    "project" => {
      "type" => "string",
      "description" => "The project into which to deploy resources. This is shorthand for a +habitat+ key with a +name+ or +id+ set. The config parser will attempt to correctly resolve this."
    },
    "auto_create_subnetworks" => {
      "type" => "boolean",
      "default" => false,
      "description" => "Sets the +auto_create_subnetworks+ flag, which causes Google to generate a set of generic subnets, one per region. This effectively overrides Mu's +create_standard_subnets+ and any explicitly defined +subnets+."
    }
  }
  [toplevel_required, schema]
end
update_route_tables_cache(subnet_key, use_cache: true, region: MU.curRegion) click to toggle source

updates the route table cache (@rtb_cache). @param subnet_key [String]: The subnet/subnets route tables will be extracted from. @param use_cache [Boolean]: If to use the existing cache and add records to cache only if missing, or to also replace exising records in cache. @param region [String]: The cloud provider region of the target subnet.

# File modules/mu/providers/google/vpc.rb, line 518
def self.update_route_tables_cache(subnet_key, use_cache: true, region: MU.curRegion)
end
validateConfig(vpc, configurator) click to toggle source

Cloud-specific pre-processing of {MU::Config::BasketofKittens::vpcs}, bare and unvalidated. @param vpc [Hash]: The resource to process and validate @param configurator [MU::Config]: The overall deployment configurator of which this resource is a member @return [Boolean]: True if validation succeeded, False otherwise

# File modules/mu/providers/google/vpc.rb, line 797
        def self.validateConfig(vpc, configurator)
          ok = true

          vpc['project'] ||= MU::Cloud::Google.defaultProject(vpc['credentials'])

          if vpc["project"] and !vpc["habitat"]
            vpc["habitat"] = MU::Cloud::Google.projectToRef(vpc["project"], config: configurator, credentials: vpc["credentials"])
          end

          # Generate a set of subnets per route, if none are declared
          if !vpc['subnets'] or vpc['subnets'].empty?
            if vpc['regions'].nil? or vpc['regions'].empty?
              vpc['regions'] = MU::Cloud::Google.listRegions(vpc['us_only'])
            end
            blocks = configurator.divideNetwork(vpc['ip_block'], vpc['regions'].size*vpc['route_tables'].size, 29)
            ok = false if blocks.nil?

            vpc["subnets"] = []
            vpc['route_tables'].each { |t|
              is_public = false
              t['routes'].each { |r|
                if !vpc["virtual_name"] and
                   !vpc["create_nat_gateway"] and
                   !vpc['bastion'] and
                   r["gateway"] == "#NAT"
                  r["gateway"] = "#DENY"
                end
                is_public = true if r["gateway"] == "#INTERNET"
              }
              count = 0
              vpc['regions'].each { |r|
                block = blocks.shift
                subnet = {
                  "availability_zone" => r,
                  "route_table" => t["name"],
                  "ip_block" => block.to_s,
                  "name" => "Subnet"+count.to_s+t["name"].capitalize
                }
                if is_public
                  subnet["map_public_ips"] = true
                  subnet["is_public"] = true
                end
                vpc["subnets"] << subnet
                count = count + 1
              }
            }
          end

          vpc['subnets'].each { |s|
            if !s['availability_zone']
              s['availability_zone'] = vpc['region']
              s['availability_zone'] ||= MU::Cloud::Google.myRegion(vpc['credentials'])
            end
          }

          # Google VPCs can't have routes that are anything other than global
          # (they can be tied to individual instances by tags, but w/e). So we
          # decompose our VPCs into littler VPCs, one for each declared route
          # table, so that the routes therein will only apply to the portion of
          # our network we want them to.
          if vpc['route_tables'].size > 1
            blocks = configurator.divideNetwork(vpc['ip_block'], vpc['route_tables'].size*2, 29)
            peernames = []
            vpc['route_tables'].each { |tbl|
              peernames << vpc['name']+"-"+tbl['name']
            }
            vpc['route_tables'].each { |tbl|
              newvpc = {
                "name" => vpc['name']+"-"+tbl['name'],
                "cloud" => "Google",
                "credentials" => vpc['credentials'],
                "virtual_name" => vpc['name'],
                "ip_block" => blocks.shift,
                "route_tables" => [tbl],
                "parent_block" => vpc['ip_block'],
                "subnets" => [],
                "peers" => vpc['peers']
              }
              MU.log "Splitting VPC #{newvpc['name']} off from #{vpc['name']}", MU::NOTICE

              vpc.each_pair { |key, val|
                next if ["name", "route_tables", "subnets", "ip_block"].include?(key)
                newvpc[key] = val
              }
              if vpc["bastion"] and
                 !tbl["routes"].map { |r| r["gateway"] }.include?("#INTERNET")
                newvpc["bastion"] = vpc["bastion"]
                vpc.delete("bastion")
              end
              newvpc['peers'] ||= []
# Add the peer connections we're generating, in addition
              peernames.each { |peer|
                if peer != newvpc['name']
                  newvpc['peers'] << { "vpc" => { "vpc_name" => peer } }
                end
              }
              newvpc['peers'].reject! { |p|
                p.values.first['vpc_name'] == newvpc['name'] or p.values.first['vpc_name'] == vpc['name']
              }

              vpc["subnets"].each { |subnet|
                newvpc["subnets"] << subnet if subnet["route_table"] == tbl["name"]
              }

              ok = false if !configurator.insertKitten(newvpc, "vpcs", true)
            }
            configurator.removeKitten(vpc['name'], "vpcs")
          else
            has_nat = vpc['route_tables'].first["routes"].include?({"gateway"=>"#NAT", "destination_network"=>"0.0.0.0/0"})
            has_deny = vpc['route_tables'].first["routes"].include?({"gateway"=>"#DENY", "destination_network"=>"0.0.0.0/0"})
# XXX we need routes to peered Networks too

            if has_nat or has_deny
              ok = false if !genStandardSubnetACLs(vpc['parent_block'] || vpc['ip_block'], vpc['name'], configurator, vpc["project"], false, credentials: vpc['credentials'])
            else
              ok = false if !genStandardSubnetACLs(vpc['parent_block'] || vpc['ip_block'], vpc['name'], configurator, vpc["project"], credentials: vpc['credentials'])
            end
            if has_nat and !has_deny and !vpc['bastion']
              vpc['route_tables'].first["routes"] << {
                "gateway"=>"#DENY",
                "destination_network"=>"0.0.0.0/0"
              }
            end

            # You know what, let's just guarantee that we'll have a route from
            # this master, always
            # XXX this confuses machines that don't have public IPs
            if !vpc['scrub_mu_isms']
#              vpc['route_tables'].first["routes"] << {
#                'gateway' => "#INTERNET",
#                'destination_network' => MU.mu_public_ip+"/32"
#              }
            end
            vpc['route_tables'].first["routes"].each { |route|
              # No such thing as a NAT gateway in Google... so make an instance
              # that'll do the deed.
              if route['gateway'] == "#NAT"
                # theoretically our upstream validation should have inserted
                # a NAT/bastion host we can use
                nat = if vpc['virtual_name']
                  configurator.haveLitterMate?(vpc['virtual_name']+"-natstion", "servers")
                else
                  configurator.haveLitterMate?(vpc['name']+"-natstion", "servers")
                end

                if !nat
                  MU.log "Google VPC #{vpc['name']} declared a #NAT route, but I don't see an upstream NAT host I can use. Do I even have public subnets?", MU::ERR
                  ok = false
                else
                  route['nat_host_name'] = nat['name']
                  route['priority'] = 100
                  MU::Config.addDependency(vpc, nat['name'], "server", their_phase: "groom", my_phase: "groom")
                  vpc["bastion"] = MU::Config::Ref.get(
                    name: nat['name'],
                    cloud: vpc['cloud'],
                    credentials: vpc['credentials'],
                    type: "servers"
                  )

                end
              end
            }
          end

#          MU.log "GOOGLE VPC", MU::WARN, details: vpc
          ok
        end

Private Class Methods

genStandardSubnetACLs(vpc_cidr, vpc_name, configurator, project, _publicroute = true, credentials: nil) click to toggle source
# File modules/mu/providers/google/vpc.rb, line 1008
        def self.genStandardSubnetACLs(vpc_cidr, vpc_name, configurator, project, _publicroute = true, credentials: nil)
          private_acl = {
            "name" => vpc_name+"-rt",
            "cloud" => "Google",
            "credentials" => credentials,
            "project" => project,
            "vpc" => { "vpc_name" => vpc_name },
            "dependencies" => [ { "type" => "vpc", "name" => vpc_name } ],
            "rules" => [
              { "ingress" => true, "proto" => "all", "hosts" => [vpc_cidr] }
            ]
          }
#          if publicroute
#          XXX distinguish between "I have a NAT" and "I really shouldn't be
#          able to talk to the world"
            private_acl["rules"] << {
              "egress" => true, "proto" => "all", "hosts" => ["0.0.0.0/0"]
            }
#          else
#            private_acl["rules"] << {
#              "egress" => true, "proto" => "all", "hosts" => [vpc_cidr], "weight" => 999
#            }
#            private_acl["rules"] << {
#              "egress" => true, "proto" => "all", "hosts" => ["0.0.0.0/0"], "deny" => true
#            }
#          end
          configurator.insertKitten(private_acl, "firewall_rules", true)
        end
purge_subnets(noop = false, _tagfilters = [{name: "tag:MU-ID", values: [MU.deploy_id]}], regions: MU::Cloud::Google.listRegions, project: nil, credentials: nil) click to toggle source

Remove all subnets associated with the currently loaded deployment. @param noop [Boolean]: If true, will only print what would be done @param _tagfilters [Array<Hash>]: Labels to filter against when search for resources to purge @param regions [Array<String>]: The cloud provider regions to check @return [void]

# File modules/mu/providers/google/vpc.rb, line 1116
def self.purge_subnets(noop = false, _tagfilters = [{name: "tag:MU-ID", values: [MU.deploy_id]}], regions: MU::Cloud::Google.listRegions, project: nil, credentials: nil)
  project ||= MU::Cloud::Google.defaultProject(credentials)
  parent_thread_id = Thread.current.object_id
  regionthreads = []
  regions.each { |r|
    regionthreads << Thread.new {
      MU.dupGlobals(parent_thread_id)
      begin
        MU::Cloud::Google.compute(credentials: credentials).delete(
          "subnetwork",
          project,
          r,
          noop
        )
      rescue MU::Cloud::MuDefunctHabitat
        Thread.exit
      end
    }
  }
  regionthreads.each do |t|
    t.join
  end
end

Public Instance Methods

cloud_desc(use_cache: true) click to toggle source

Describe this VPC from the cloud platform's perspective @return [Google::Apis::Core::Hashable]

# File modules/mu/providers/google/vpc.rb, line 128
def cloud_desc(use_cache: true)
  if @cloud_desc_cache and use_cache
    return @cloud_desc_cache
  end

  resp = MU::Cloud::Google.compute(credentials: @config['credentials']).get_network(@project_id, @cloud_id)

  if @cloud_id.nil? or @cloud_id == "" or resp.nil?
    MU.log "Couldn't describe #{self}, @cloud_id #{@cloud_id.nil? ? "undefined" : "empty" }", MU::ERR
    return nil
  end
  @cloud_desc_cache = resp

  # populate other parts and pieces of ourself
  @url ||= resp.self_link
  routes = MU::Cloud::Google.compute(credentials: @config['credentials']).list_routes(
    @project_id,
    filter: "network = \"#{@url}\""
  ).items
  @routes = routes if routes and routes.size > 0

  @cloud_desc_cache
end
create() click to toggle source

Called automatically by {MU::Deploy#createResources}

# File modules/mu/providers/google/vpc.rb, line 38
def create
  networkobj = MU::Cloud::Google.compute(:Network).new(
    name: MU::Cloud::Google.nameStr(@mu_name),
    description: @deploy.deploy_id,
    auto_create_subnetworks: false
  )
  MU.log "Creating network #{@mu_name} (#{@config['ip_block']}) in project #{@project_id}", details: networkobj

  resp = MU::Cloud::Google.compute(credentials: @config['credentials']).insert_network(@project_id, networkobj)
  @url = resp.self_link
  @cloud_id = resp.name

  if @config['subnets']
    subnetthreads = []
    parent_thread_id = Thread.current.object_id
    @config['subnets'].each { |subnet|
      subnetthreads << Thread.new {
        MU.dupGlobals(parent_thread_id)
        subnet_name = @config['name']+subnet['name']

        subnet_mu_name = @config['scrub_mu_isms'] ? @cloud_id+subnet_name.downcase : MU::Cloud::Google.nameStr(@deploy.getResourceName(subnet_name, max_length: 61))
        MU.log "Creating subnetwork #{subnet_mu_name} (#{subnet['ip_block']}) in project #{@project_id} region #{subnet['availability_zone']}", details: subnet
        subnetobj = MU::Cloud::Google.compute(:Subnetwork).new(
          name: subnet_mu_name,
          description: @deploy.deploy_id,
          ip_cidr_range: subnet['ip_block'],
          network: @url,
          region: subnet['availability_zone']
        )
        MU::Cloud::Google.compute(credentials: @config['credentials']).insert_subnetwork(@project_id, subnet['availability_zone'], subnetobj)

        # make sure the subnet we created exists, before moving on
        subnetdesc = nil
        begin 
          subnetdesc = MU::Cloud::Google.compute(credentials: @config['credentials']).get_subnetwork(@project_id, subnet['availability_zone'], subnet_mu_name)
          if !subnetdesc.nil?
            subnet_cfg = {}
            subnet_cfg["ip_block"] = subnet['ip_block']
            subnet_cfg["name"] = subnet_name
            subnet_cfg['mu_name'] = subnet_mu_name
            subnet_cfg["cloud_id"] = subnetdesc.self_link.gsub(/.*?\/([^\/]+)$/, '\1')
            subnet_cfg['az'] = subnet['availability_zone']
            @subnets << MU::Cloud::Google::VPC::Subnet.new(self, subnet_cfg, precache_description: false)
          end
          sleep 1
        end while subnetdesc.nil?
      }
    }
    subnetthreads.each do |t|
      t.join
    end
  end

  if !@config['route_tables'].nil?
    @config['route_tables'].each { |rtb|
      rtb['routes'].each { |route|
        # GCP does these for us, by default
        next if route['destination_network'] == "0.0.0.0/0" and
                route['gateway'] == "#INTERNET"
        # sibling NAT host routes will get set up our groom phrase
        next if route['gateway'] == "#NAT" and !route['nat_host_name'].nil?
        createRoute(route, network: @url)
      }
    }
  end
end
createRouteForInstance(route, server) click to toggle source

@param route [Hash]: A route description, per the Basket of Kittens schema @param server [MU::Cloud::Google::Server]: Instance to which this route will apply

# File modules/mu/providers/google/vpc.rb, line 967
def createRouteForInstance(route, server)
  createRoute(route, network: @url, tags: [MU::Cloud::Google.nameStr(server.mu_name)])
end
findBastion(nat_name: nil, nat_cloud_id: nil, nat_tag_key: nil, nat_tag_value: nil, nat_ip: nil) click to toggle source

Given some search criteria for a {MU::Cloud::Server}, see if we can locate a NAT host in this VPC. @param nat_name [String]: The name of the resource as defined in its 'name' Basket of Kittens field, typically used in conjunction with deploy_id. @param nat_cloud_id [String]: The cloud provider's identifier for this NAT. @param nat_tag_key [String]: A cloud provider tag to help identify the resource, used in conjunction with tag_value. @param nat_tag_value [String]: A cloud provider tag to help identify the resource, used in conjunction with tag_key. @param nat_ip [String]: An IP address associated with the NAT instance.

# File modules/mu/providers/google/vpc.rb, line 391
def findBastion(nat_name: nil, nat_cloud_id: nil, nat_tag_key: nil, nat_tag_value: nil, nat_ip: nil)
  if nat_name
    svr_obj = @deploy.findLitterMate(name: nat_name, type: "servers")
    return svr_obj if svr_obj
  end

  deploy_id = nil
  nat_name = nat_name.to_s if !nat_name.nil? and nat_name.class.to_s == "MU::Config::Tail"
  nat_ip = nat_ip.to_s if !nat_ip.nil? and nat_ip.class.to_s == "MU::Config::Tail"
  nat_cloud_id = nat_cloud_id.to_s if !nat_cloud_id.nil? and nat_cloud_id.class.to_s == "MU::Config::Tail"
  nat_tag_key = nat_tag_key.to_s if !nat_tag_key.nil? and nat_tag_key.class.to_s == "MU::Config::Tail"
  nat_tag_value = nat_tag_value.to_s if !nat_tag_value.nil? and nat_tag_value.class.to_s == "MU::Config::Tail"
  # If we're searching by name, assume it's part of this here deploy.
  if nat_cloud_id.nil? and !@deploy.nil?
    deploy_id = @deploy.deploy_id
  end
  found = MU::MommaCat.findStray(
    "Google",
    "server",
    name: nat_name,
    cloud_id: nat_cloud_id,
    deploy_id: deploy_id,
    tag_key: nat_tag_key,
    tag_value: nat_tag_value,
    allow_multi: true,
    dummy_ok: true,
    calling_deploy: @deploy
  )

  return nil if found.nil? || found.empty?

  if found.size == 1
    return found.first
  elsif found.size > 1
    found.each { |nat|
      next if !nat.cloud_desc
      # Try some cloud-specific criteria
      nat.cloud_desc.network_interfaces.each { |iface|
        if !nat_ip.nil?
          return nat if iface.network_ip == nat_ip
          if iface.access_configs
            iface.access_configs.each { |public_iface|
              return if public_iface.nat_ip == nat_ip
            }
          end
        end
        if iface.network == @url
          # XXX Strictly speaking we could have different NATs in
          # different subnets, so this can be wrong in corner cases.
          return nat
        end
      }
    }
  end

  return nil
end
findNat(nat_cloud_id: nil, nat_filter_key: nil, nat_filter_value: nil, region: MU.curRegion) click to toggle source

Given some search criteria try locating a NAT Gaateway in this VPC. @param nat_cloud_id [String]: The cloud provider's identifier for this NAT. @param nat_filter_key [String]: A cloud provider filter to help identify the resource, used in conjunction with nat_filter_value. @param nat_filter_value [String]: A cloud provider filter to help identify the resource, used in conjunction with nat_filter_key. @param region [String]: The cloud provider region of the target instance.

# File modules/mu/providers/google/vpc.rb, line 381
def findNat(nat_cloud_id: nil, nat_filter_key: nil, nat_filter_value: nil, region: MU.curRegion)
end
getSubnet(cloud_id: nil, name: nil, tag_key: nil, tag_value: nil, ip_block: nil, region: nil, subnet_mu_name: nil) click to toggle source

Check for a subnet in this VPC matching one or more of the specified criteria, and return it if found.

# File modules/mu/providers/google/vpc.rb, line 451
def getSubnet(cloud_id: nil, name: nil, tag_key: nil, tag_value: nil, ip_block: nil, region: nil, subnet_mu_name: nil)
  if !cloud_id.nil? and cloud_id.match(/^https:\/\//)
    cloud_id.match(/\/regions\/([^\/]+)\/subnetworks\/([^\/]+)$/)
    region = Regexp.last_match[1]
    cloud_id = Regexp.last_match[2]
    cloud_id.gsub!(/.*?\//, "")
  end
  
  if name
    subnet_mu_name ||= @config['scrub_mu_isms'] ? @cloud_id+name.downcase : MU::Cloud::Google.nameStr(@deploy.getResourceName(name, max_length: 61))
  end

  MU.log "getSubnet(cloud_id: #{cloud_id}, name: #{name}, tag_key: #{tag_key}, tag_value: #{tag_value}, ip_block: #{ip_block}, region: #{region}, subnet_mu_name: #{subnet_mu_name})", MU::DEBUG, details: caller[0]
  subnets.each { |subnet|
    next if region and subnet.az != region
    if !cloud_id.nil? and !subnet.cloud_id.nil? and subnet.cloud_id.to_s == cloud_id.to_s
      return subnet
    elsif !name.nil? and !subnet.name.nil? and
          subnet.name.downcase.to_s == name.downcase.to_s
      return subnet
    elsif !subnet_mu_name.nil? and !subnet.name.nil? and
          subnet.name.downcase.to_s == subnet_mu_name.downcase.to_s
      return subnet
    end
  }
  return nil
end
getUnusedAddressBlock(exclude: [], max_bits: 28) click to toggle source

Looks at existing subnets, and attempts to find the next available IP block that's roughly similar to the ones we already have. This checks against secondary IP ranges, as well as each subnet's primary CIDR block. @param exclude [Array<String>]: One or more CIDRs to treat as unavailable, in addition to those allocated to existing subnets @return [String]

# File modules/mu/providers/google/vpc.rb, line 977
        def getUnusedAddressBlock(exclude: [], max_bits: 28)
          used_ranges = exclude.map { |cidr| NetAddr::IPv4Net.parse(cidr) }
          subnets.each { |s|
            used_ranges << NetAddr::IPv4Net.parse(s.cloud_desc.ip_cidr_range)
            if s.cloud_desc.secondary_ip_ranges
              used_ranges.concat(s.cloud_desc.secondary_ip_ranges.map { |r| NetAddr::IPv4Net.parse(r.ip_cidr_range) })
            end
          }
# XXX sort used_ranges
          candidate = used_ranges.first.next_sib

          begin
            if candidate.netmask.prefix_len > max_bits
              candidate = candidate.resize(max_bits)
            end
            try_again = false
            used_ranges.each { |cidr|
              if !cidr.rel(candidate).nil?
                candidate = candidate.next_sib
                try_again = true
                break
              end
            }
            try_again = false if candidate.nil?
          end while try_again

          candidate.to_s
        end
groom() click to toggle source

Called automatically by {MU::Deploy#createResources}

# File modules/mu/providers/google/vpc.rb, line 153
        def groom
          rtb = @config['route_tables'].first # there's only ever one

          rtb['routes'].each { |route|
            # If we had a sibling server being spun up as a NAT, rig up the
            # route that the hosts behind it will need.
            if route['gateway'] == "#NAT" and !route['nat_host_name'].nil?
              createRoute(route, network: @url)
            end
          }

          if !@config['peers'].nil?
            count = 0
            @config['peers'].each { |peer|
              if peer['vpc']['name']
                peer_obj = @deploy.findLitterMate(name: peer['vpc']['name'], type: "vpcs", habitat: peer['vpc']['project'])
              else
                tag_key, tag_value = peer['vpc']['tag'].split(/=/, 2) if !peer['vpc']['tag'].nil?
                if peer['vpc']['deploy_id'].nil? and peer['vpc']['id'].nil? and tag_key.nil?
                  peer['vpc']['deploy_id'] = @deploy.deploy_id
                end

                peer_obj = MU::MommaCat.findStray(
                  "Google",
                  "vpcs",
                  deploy_id: peer['vpc']['deploy_id'],
                  cloud_id: peer['vpc']['id'],
                  name: peer['vpc']['name'],
# XXX project flag tho
                  tag_key: tag_key,
                  tag_value: tag_value,
                  dummy_ok: true
                ).first
              end
if peer_obj.nil?
  MU.log "Failed VPC peer lookup on behalf of #{@cloud_id}", MU::WARN, details: peer
  pr = peer['vpc']['project'] || @project_id
  MU.log "all the VPCs I can see", MU::WARN, details: MU::Cloud::Google.compute(credentials: @config['credentials']).list_networks(pr)

end
              raise MuError, "No result looking for #{@mu_name}'s peer VPCs (#{peer['vpc']})" if peer_obj.nil?

              url = if peer_obj.cloudobj.url
                peer_obj.cloudobj.url
              elsif peer_obj.cloudobj.deploydata
                peer_obj.cloudobj.deploydata['self_link']
              else
                raise MuError, "Can't find the damn URL of my damn peer VPC #{peer['vpc']}"
              end
              cnxn_name = MU::Cloud::Google.nameStr(@mu_name+"-peer-"+count.to_s)
              peerreq = MU::Cloud::Google.compute(:NetworksAddPeeringRequest).new(
                name: cnxn_name,
                auto_create_routes: true,
                peer_network: url
              )

              begin
                MU.log "Peering #{@cloud_id} with #{peer_obj.cloudobj.cloud_id}, connection name is #{cnxn_name}", details: peerreq
                MU::Cloud::Google.compute(credentials: @config['credentials']).add_network_peering(
                  @project_id,
                  @cloud_id,
                  peerreq
                )
              rescue ::Google::Apis::ClientError => e
                if e.message.match(/operation in progress on the local or peer network/)
                  MU.log e.message, MU::DEBUG, details: peerreq
                  sleep 10
                  retry
                end
              end
              count += 1
            }
          end
          loadSubnets(use_cache: false)
        end
loadSubnets(use_cache: true) click to toggle source

Describe subnets associated with this VPC. We'll compose identifying information similar to what MU::Cloud.describe builds for first-class resources. @param use_cache [Boolean]: If available, use saved deployment metadata to describe subnets, instead of querying the cloud API @return [Array<Hash>]: A list of cloud provider identifiers of subnets associated with this VPC.

# File modules/mu/providers/google/vpc.rb, line 286
        def loadSubnets(use_cache: true)
          @subnetcachesemaphore.synchronize {
            return @subnets if use_cache and @subnets and @subnets.size > 0
          }
          network = cloud_desc

          if network.nil?
            MU.log "Unabled to load cloud description in #{self}", MU::ERR
            return nil
          end
          found = []

          if @deploy and @deploy.deployment and
             @deploy.deployment["vpcs"] and
             @deploy.deployment["vpcs"][@config['name']] and
             @deploy.deployment["vpcs"][@config['name']]["subnets"] and
             @deploy.deployment["vpcs"][@config['name']]["subnets"].size > 0
            @deploy.deployment["vpcs"][@config['name']]["subnets"].each { |desc|
              subnet = desc.clone
              subnet['mu_name'] = @config['scrub_mu_isms'] ? @cloud_id+subnet['name'].downcase : MU::Cloud::Google.nameStr(@deploy.getResourceName(subnet['name'], max_length: 61))
              subnet["cloud_id"] ||= desc['self_link'].gsub(/.*?\/([^\/]+)$/, '\1')
              subnet["cloud_id"] ||= subnet['mu_name']
              subnet['az'] ||= desc["region"].gsub(/.*?\/([^\/]+)$/, '\1')
              @subnets << MU::Cloud::Google::VPC::Subnet.new(self, subnet, precache_description: false)
            }
          else
            resp = MU::Cloud::Google.compute(credentials: @config['credentials']).list_subnetwork_usable(
              @project_id,
              filter: "network eq #{network.self_link}"
            )
            resp.items.each { |subnet|
              found << subnet
            }

            @subnetcachesemaphore.synchronize {
              @subnets ||= []
              ext_ids = @subnets.each.collect { |s| s.cloud_id }
              # If we're a plain old Mu resource, load our config and deployment
              # metadata. Like ya do.
              if !@config.nil? and @config.has_key?("subnets")
                @config['subnets'].each { |subnet|
#                  subnet['mu_name'] = @mu_name+"-"+subnet['name'] if !subnet.has_key?("mu_name")
                  subnet_name = @config['name']+subnet['name']
                  subnet['mu_name'] ||= @config['scrub_mu_isms'] ? @cloud_id+subnet_name.downcase : MU::Cloud::Google.nameStr(@deploy.getResourceName(subnet_name, max_length: 61))
                  subnet['region'] = @config['region']
                  found.each { |desc|
                    if desc.ip_cidr_range == subnet["ip_block"]
                      desc.subnetwork.match(/\/projects\/[^\/]+\/regions\/([^\/]+)\/subnetworks\/(.+)$/)
                      subnet['az'] = Regexp.last_match[1]
                      subnet['name'] ||= Regexp.last_match[2]
                      subnet["cloud_id"] = subnet['mu_name']
                      subnet["url"] = desc.subnetwork
                      break
                    end
                  }

                  if !ext_ids.include?(subnet["cloud_id"])
                    @subnets << MU::Cloud::Google::VPC::Subnet.new(self, subnet, precache_description: false)
                  end
                }

              # Of course we might be loading up a dummy subnet object from a
              # foreign or non-Mu-created VPC and subnet. So make something up.
              elsif !found.nil?
                found.each { |desc|
                  subnet = {}
                  desc.subnetwork.match(/\/projects\/[^\/]+\/regions\/([^\/]+)\/subnetworks\/(.+)$/)
                  subnet['az'] = Regexp.last_match[1]
                  subnet['name'] = Regexp.last_match[2]
                  subnet["cloud_id"] = subnet['name']
                  subnet["ip_block"] = desc.ip_cidr_range
                  subnet["url"] = desc.subnetwork
                  subnet['mu_name'] = @mu_name+"-"+subnet['name']
                  if !ext_ids.include?(subnet["cloud_id"])
                    @subnets << MU::Cloud::Google::VPC::Subnet.new(self, subnet, precache_description: false)
                  end
                }
              end
            }
          end


          # The API is filled with lies
          @subnets.reject! { |s|
            !MU::Cloud::Google.listRegions(credentials: @credentials).include?(s.az)
          }

          return @subnets
        end
notify() click to toggle source

Describe this VPC @return [Hash]

# File modules/mu/providers/google/vpc.rb, line 115
def notify
  base = MU.structToHash(cloud_desc, stringify_keys: true)
  base["cloud_id"] = @cloud_id
  base["project_id"] = habitat_id
  base.merge!(@config.to_h)
  if @subnets
    base["subnets"] = @subnets.map { |s| s.notify }
  end
  base
end
subnets() click to toggle source

Return an array of MU::Cloud::Google::VPC::Subnet objects describe the member subnets of this VPC.

@return [Array<MU::Cloud::Google::VPC::Subnet>]

# File modules/mu/providers/google/vpc.rb, line 274
def subnets
  if @subnets.nil? or @subnets.size == 0
    return loadSubnets
  end
  return @subnets
end
toKitten(**_args) click to toggle source

Reverse-map our cloud description into a runnable config hash. We assume that any values we have in +@config+ are placeholders, and calculate our own accordingly based on what's live in the cloud. XXX add flag to return the diff between @config and live cloud

# File modules/mu/providers/google/vpc.rb, line 607
        def toKitten(**_args)
          return nil if cloud_desc.name == "default" # parent project builds these
          bok = {
            "cloud" => "Google",
            "project" => @config['project'],
            "credentials" => @config['credentials']
          }
          MU::Cloud::Google.listRegions.size

          _schema, valid = MU::Config.loadResourceSchema("VPC", cloud: "Google")
          return [nil, nil] if !valid
#          pp schema
#          MU.log "++++++++++++++++++++++++++++++++"

          bok['name'] = cloud_desc.name.dup
          bok['cloud_id'] = cloud_desc.name.dup
          bok['create_standard_subnets'] = false

          if @subnets and @subnets.size > 0
            bok['subnets'] = []
            regions_seen = []
            names_seen = []
            @subnets.reject! { |x| x.cloud_desc.nil? }
            @subnets.map { |x| x.cloud_desc }.each { |s|
              subnet_name = s.name.dup
              names_seen << s.name.dup
              regions_seen << s.region
              bok['subnets'] << {
                "name" => subnet_name,
                "ip_block" => s.ip_cidr_range
              }
            }
            
            # If all of the subnets are named 'default' and there's one per
            # region, we're using GCP-generated subnets instead of explicitly
            # declared ones.
            if names_seen.uniq.size == 1 and names_seen.first == "default" and
               regions_seen.uniq.size == regions_seen.size and
               regions_seen.size >= (MU::Cloud::Google.listRegions.size * 0.8)
              bok.delete("subnets")
              bok['auto_create_subnetworks'] = true
            end
          end

          if cloud_desc.peerings and cloud_desc.peerings.size > 0
            bok['peers'] = []
            cloud_desc.peerings.each { |peer|
              peer.network.match(/projects\/([^\/]+?)\/[^\/]+?\/networks\/([^\/]+)$/)
              vpc_project = Regexp.last_match[1]
              vpc_name = Regexp.last_match[2]
              vpc_id = vpc_name.dup
              # Make sure the peer is something we have permission to look at
              peer_descs = MU::Cloud::Google::VPC.find(cloud_id: vpc_id, project: vpc_project)
              if peer_descs.nil? or peer_descs.empty?
                MU.log "VPC #{@cloud_id} peer #{vpc_id} #{vpc_project} is not accessible, will remove from peer list", MU::WARN
                next
              end
# XXX need to decide which of these parameters to use based on whether the peer is also in the mix of things being harvested, which is above this method's pay grade
              bok['peers'] << { "vpc" => MU::Config::Ref.get(
                id: vpc_id,
                name: vpc_name,
                cloud: "Google",
                habitat: MU::Config::Ref.get(
                  id: vpc_project,
                  cloud: "Google",
                  credentials: @credentials,
                  type: "habitats"
                ),
                credentials: @config['credentials'],
                type: "vpcs"
              ) }
            }
          end

# XXX need to grok VPN tunnels, priorities, and maybe preserve descriptions; make sure we know where next_hop_gateway  and next_hop_ip come from
          if @routes
            routes = []
            @routes.each { |r|
              next if r.next_hop_peering # these are auto-created
              route = {
                "destination_network" => r.dest_range
              }
              if r.next_hop_instance
                route["nat_host_id"] = r.next_hop_instance
              end
            }
            if routes.size > 0
              bok['route_tables'] = [
                {
                  "name" => "default",
                  "routes" => routes
                }
              ]
            end
          end

# XXX validate that we've at least touched every required attribute (maybe upstream?)
          bok
        end
trafficLogging(log_group_name: nil, resource_id: nil, resource_type: "VPC", traffic_type: "ALL") click to toggle source

Configure IP traffic logging on a given VPC/Subnet. Logs are saved in cloudwatch based on the network interface ID of each instance. @param log_group_name [String]: The name of the CloudWatch log group all logs will be saved in. @param resource_id [String]: The cloud provider's identifier of the resource that traffic logging will be enabled on. @param resource_type [String]: What resource type to enable logging on (VPC or Subnet). @param traffic_type [String]: What traffic to log (ALL, ACCEPT or REJECT).

# File modules/mu/providers/google/vpc.rb, line 110
def trafficLogging(log_group_name: nil, resource_id: nil, resource_type: "VPC", traffic_type: "ALL")
end

Private Instance Methods

createRoute(route, network: @url, tags: []) click to toggle source

Helper method for manufacturing routes. Expect to be called from {MU::Cloud::Google::VPC#create} or {MU::Cloud::Google::VPC#groom}. @param route [Hash]: A route description, per the Basket of Kittens schema @param network [String]: Cloud identifier of the VPC to which we're adding this route @param tags [Array<String>]: Instance tags to which this route applies. If empty, applies to entire VPC. @return [Hash]: The modified configuration that was originally passed in.

# File modules/mu/providers/google/vpc.rb, line 1044
        def createRoute(route, network: @url, tags: [])
          routename = MU::Cloud::Google.nameStr(@mu_name+"-route-"+route['destination_network'])
          if !tags.nil? and tags.size > 0
            routename = MU::Cloud::Google.nameStr(routename+"-"+tags.first).slice(0,63)
          end
          route["priority"] ||= 999
          if route['gateway'] == "#NAT"
            if !route['nat_host_name'].nil? or !route['nat_host_id'].nil?
                sleep 5
              nat_instance = findBastion(
                nat_name: route["nat_host_name"],
                nat_cloud_id: route["nat_host_id"]
              )
              if nat_instance.nil? or nat_instance.cloud_desc.nil?
                raise MuError, "Failed to find NAT host for #NAT route in #{@mu_name} (#{route})"
              end

              routeobj = ::Google::Apis::ComputeV1::Route.new(
                name: routename,
                next_hop_instance: nat_instance.cloud_desc.self_link,
                dest_range: route['destination_network'],
                priority: route["priority"],
                description: @deploy.deploy_id,
                tags: tags,
                network: network
              )
            end
# several other cases missing for various types of routers (raw IPs, instance ids, etc) XXX
          elsif route['gateway'] == "#DENY"
            resp = MU::Cloud::Google.compute(credentials: @config['credentials']).list_routes(
              @project_id,
              filter: "network eq #{network}"
            )

            if !resp.nil? and !resp.items.nil?
              resp.items.each { |r|
                next if r.next_hop_gateway.nil? or !r.next_hop_gateway.match(/\/global\/gateways\/default-internet-gateway$/)
                MU.log "Removing standard route #{r.name} per our #DENY entry"
                MU::Cloud::Google.compute(credentials: @config['credentials']).delete_route(@project_id, r.name)
              }
            end
          elsif route['gateway'] == "#INTERNET"
            routeobj = ::Google::Apis::ComputeV1::Route.new(
              name: routename,
              next_hop_gateway: "global/gateways/default-internet-gateway",
              dest_range: route['destination_network'],
              priority: route["priority"],
              description: @deploy.deploy_id,
              tags: tags,
              network: network
            )
          end

          if route['gateway'] != "#DENY" and routeobj
            begin
              MU::Cloud::Google.compute(credentials: @config['credentials']).get_route(@project_id, routename)
            rescue ::Google::Apis::ClientError, MU::MuError => e
              if e.message.match(/notFound/)
                MU.log "Creating route #{routename} in project #{@project_id}", details: routeobj
                MU::Cloud::Google.compute(credentials: @config['credentials']).insert_route(@project_id, routeobj)
              else
                # TODO can't update GCP routes, would have to delete and re-create
              end
            end
          end
        end