class MU::Cloud
Plugins under this namespace serve as interfaces to cloud providers and other provisioning layers.
Plugins under this namespace serve as interfaces to cloud providers and other provisioning layers.
Plugins under this namespace serve as interfaces to cloud providers and other provisioning layers.
Plugins under this namespace serve as interfaces to cloud providers and other provisioning layers.
Plugins under this namespace serve as interfaces to cloud providers and other provisioning layers.
Plugins under this namespace serve as interfaces to cloud providers and other provisioning layers.
Plugins under this namespace serve as interfaces to cloud providers and other provisioning layers.
Plugins under this namespace serve as interfaces to cloud providers and other provisioning layers.
Plugins under this namespace serve as interfaces to cloud providers and other provisioning layers.
Plugins under this namespace serve as interfaces to cloud providers and other provisioning layers.
Constants
- ALPHA
Denotes a resource implementation which is missing significant functionality or is largely untested.
- BASE_IMAGE_BUCKET
The public
AWS
S3 bucket where we expect to find YAML files listing our standard base images for various platforms.- BASE_IMAGE_PATH
The path in the
AWS
S3 bucket where we expect to find YAML files listing our standard base images for various platforms.- BETA
Denotes a resource implementation which supports most or all key API functionality and has seen at least some non-trivial testing.
- PLATFORM_ALIASES
Aliases for platform names, in case we don't have actual images built for them.
- PUBLIC_ATTRS
Public attributes which will be available on all instantiated cloud resource objects
:config
: The fully-resolved {MU::Config} hash describing the object, aka the Basket of Kittens entry:mu_name
: The unique internal name of the object, if one already exists:cloud
: The cloud in which this object is resident:cloud_id
: The cloud provider's official identifier for this object:environment
: The declared environment string for the deployment of which this object is a member:deploy:
The {MU::MommaCat} object representing the deployment of which this object is a member:deploy_id:
The unique string which identifies the deployment of which this object is a member:deploydata:
AHash
containing all metadata reported by resources in this deploy method, via theirnotify
methods:appname:
The declared application name of this deployment:credentials:
The name of the cloud provider credential set frommu.yaml
which is used to manage this object- RELEASE
Denotes a resource implementation which supports all key API functionality and has been substantially tested on real-world applications.
Attributes
Public Class Methods
Raise an exception if the cloud provider specified isn't valid or we don't have any credentials configured for it.
# File modules/mu/cloud/providers.rb, line 57 def self.assertAvailableCloud(cloud) if cloud.nil? or availableClouds.include?(cloud.to_s) raise MuError, "Cloud provider #{cloud} is not available" end end
List of known/supported Cloud
providers for which we have at least one set of credentials configured. @return [Array<String>]
# File modules/mu/cloud/providers.rb, line 41 def self.availableClouds available = [] MU::Cloud.supportedClouds.each { |cloud| begin cloudbase = Object.const_get("MU").const_get("Cloud").const_get(cloud) next if cloudbase.listCredentials.nil? or cloudbase.listCredentials.empty? available << cloud rescue NameError end } available end
Return a list of “container” artifacts, by class, that apply to this resource type in a cloud provider. This is so methods that call find know whether to call find
with identifiers for parent resources. This is similar in purpose to the isGlobal?
resource class method, which tells our search functions whether or not a resource scopes to a region. In almost all cases this is one-entry list consisting of :Habitat
. Notable exceptions include most implementations of Habitat
, which either reside inside a :Folder
or nothing at all; whereas a :Folder
tends to not have any containing parent. Very few resource implementations will need to override this. A nil
entry in this list is interpreted as “this resource can be global.” @return [Array<Symbol,nil>]
# File modules/mu/cloud/wrappers.rb, line 72 def self.canLiveIn if self.shortname == "Folder" [nil, :Folder] elsif self.shortname == "Habitat" [:Folder] else [:Habitat] end end
# File modules/mu/cloud/wrappers.rb, line 41 def self.can_live_in_vpc MU::Cloud.resource_types[shortname.to_sym][:can_live_in_vpc] end
# File modules/mu/cloud/wrappers.rb, line 37 def self.cfg_name MU::Cloud.resource_types[shortname.to_sym][:cfg_name] end
# File modules/mu/cloud/wrappers.rb, line 29 def self.cfg_plural MU::Cloud.resource_types[shortname.to_sym][:cfg_plural] end
Wrapper for the cleanup class method of underlying cloud object implementations.
# File modules/mu/cloud/wrappers.rb, line 130 def self.cleanup(*flags) ok = true params = flags.first clouds = MU::Cloud.supportedClouds if params[:cloud] clouds = [params[:cloud]] params.delete(:cloud) end params[:deploy_id] ||= MU.deploy_id if !params[:deploy_id] or params[:deploy_id].empty? raise MuError, "Can't call cleanup methods without a deploy id" end clouds.each { |cloud| begin cloudclass = MU::Cloud.resourceClass(cloud, shortname) if cloudclass.isGlobal? params.delete(:region) end raise MuCloudResourceNotImplemented if !cloudclass.respond_to?(:cleanup) or cloudclass.method(:cleanup).owner.to_s != "#<Class:#{cloudclass}>" MU.log "Invoking #{cloudclass}.cleanup from #{shortname}", MU::DEBUG, details: flags cloudclass.cleanup(params) rescue MuCloudResourceNotImplemented MU.log "No #{cloud} implementation of #{shortname}.cleanup, skipping", MU::DEBUG, details: flags rescue StandardError => e in_msg = cloud if params and params[:region] in_msg += " "+params[:region] end if params and params[:flags] and params[:flags]["project"] and !params[:flags]["project"].empty? in_msg += " project "+params[:flags]["project"] end MU.log "Skipping #{shortname} cleanup method in #{in_msg} due to #{e.class.name}: #{e.message}", MU::WARN, details: e.backtrace ok = false end } MU::MommaCat.unlockAll ok end
Raise an exception if the cloud provider specified isn't valid
# File modules/mu/cloud/providers.rb, line 31 def self.cloudClass(cloud) if cloud.nil? or !supportedClouds.include?(cloud.to_s) raise MuError, "Cloud provider #{cloud} is not supported" end Object.const_get("MU").const_get("Cloud").const_get(cloud.to_s) end
# File modules/mu/cloud/wrappers.rb, line 49 def self.deps_wait_on_my_creation MU::Cloud.resource_types[shortname.to_sym][:deps_wait_on_my_creation] end
Fetch our baseline userdata argument (read: “script that runs on first boot”) for a given platform. XXX both the eval() and the blind File.read() based on the platform variable are dangerous without cleaning. Clean them. @param platform [String]: The target OS. @param template_variables [Hash]: A list of variable substitutions to pass as globals to the ERB parser when loading the userdata script. @param custom_append [String]: Arbitrary extra code to append to our default userdata behavior. @return [String]
# File modules/mu/cloud.rb, line 519 def self.fetchUserdata(platform: "linux", template_variables: {}, custom_append: nil, cloud: "AWS", scrub_mu_isms: false, credentials: nil) return nil if platform.nil? or platform.empty? userdata_mutex.synchronize { script = "" if !scrub_mu_isms if template_variables.nil? or !template_variables.is_a?(Hash) raise MuError, "My second argument should be a hash of variables to pass into ERB templates" end template_variables["credentials"] ||= credentials $mu = OpenStruct.new(template_variables) userdata_dir = File.expand_path(MU.myRoot+"/modules/mu/providers/#{cloud.downcase}/userdata") platform = if %w{win2k12r2 win2k12 win2k8 win2k8r2 win2k16 windows win2k19}.include?(platform) "windows" else "linux" end erbfile = "#{userdata_dir}/#{platform}.erb" if !File.exist?(erbfile) MU.log "No such userdata template '#{erbfile}'", MU::WARN, details: caller return "" end userdata = File.read(erbfile) begin erb = ERB.new(userdata) script = erb.result rescue NameError => e raise MuError, "Error parsing userdata script #{erbfile} as an ERB template: #{e.inspect}" end MU.log "Parsed #{erbfile} as ERB", MU::DEBUG, details: script end if !custom_append.nil? if custom_append['path'].nil? raise MuError, "Got a custom userdata script argument, but no ['path'] component" end erbfile = File.read(custom_append['path']) MU.log "Loaded userdata script from #{custom_append['path']}" if custom_append['use_erb'] begin erb = ERB.new(erbfile, 1) if custom_append['skip_std'] script = +erb.result else script = script+"\n"+erb.result end rescue NameError => e raise MuError, "Error parsing userdata script #{erbfile} as an ERB template: #{e.inspect}" end MU.log "Parsed #{custom_append['path']} as ERB", MU::DEBUG, details: script else if custom_append['skip_std'] script = erbfile else script = script+"\n"+erbfile end MU.log "Parsed #{custom_append['path']} as flat file", MU::DEBUG, details: script end end return script } end
# File modules/mu/cloud/wrappers.rb, line 82 def self.find(*flags) allfound = {} MU::Cloud.availableClouds.each { |cloud| begin args = flags.first next if args[:cloud] and args[:cloud] != cloud # skip this cloud if we have a region argument that makes no # sense there cloudbase = MU::Cloud.cloudClass(cloud) next if cloudbase.listCredentials.nil? or cloudbase.listCredentials.empty? or cloudbase.credConfig(args[:credentials]).nil? if args[:region] and cloudbase.respond_to?(:listRegions) if !cloudbase.listRegions(credentials: args[:credentials]) MU.log "Failed to get region list for credentials #{args[:credentials]} in cloud #{cloud}", MU::ERR, details: caller else next if !cloudbase.listRegions(credentials: args[:credentials]).include?(args[:region]) end end begin cloudclass = MU::Cloud.resourceClass(cloud, shortname) rescue MU::MuError next end credsets = if args[:credentials] [args[:credentials]] else cloudbase.listCredentials end credsets.each { |creds| args[:credentials] = creds found = cloudclass.find(args) if !found.nil? if found.is_a?(Hash) allfound.merge!(found) else raise MuError, "#{cloudclass}.find returned a non-Hash result" end end } rescue MuCloudResourceNotImplemented end } allfound end
Shorthand lookup for resource type names. Given any of the shorthand class name, configuration name (singular or plural), or full class name, return all four as a set. @param type [String]: A string that looks like our short or full class name or singular or plural configuration names. @param assert [Boolean]: Raise an exception if the type isn't valid @return [Array]: Class name (Symbol), singular config name (String), plural config name (String), full class name (Object
)
# File modules/mu/cloud.rb, line 482 def self.getResourceNames(type, assert = true) if !type if assert raise MuError, "nil resource type requested in getResourceNames" else return [nil, nil, nil, nil, {}] end end @@resource_types.each_pair { |name, cloudclass| if name == type.to_sym or cloudclass[:cfg_name] == type or cloudclass[:cfg_plural] == type or MU::Cloud.const_get(name) == type type = name return [type.to_sym, cloudclass[:cfg_name], cloudclass[:cfg_plural], MU::Cloud.const_get(name), cloudclass] end } if assert raise MuError, "Invalid resource type #{type} requested in getResourceNames" end [nil, nil, nil, nil, {}] end
Locate a base image for a {MU::Cloud::Server} resource. First we check Mu's public bucket, which should list the latest and greatest. If we can't fetch that, then we fall back to a YAML file that's bundled as part of Mu, but which will typically be less up-to-date. @param cloud [String]: The cloud provider for which to return an image list @param platform [String]: The supported platform for which to return an image or images. If not specified, we'll return our entire library for the appropriate cloud provider. @param region [String]: The region for which the returned image or images should be supported, for cloud providers which require it (such as AWS
). @param fail_hard [Boolean]: Raise an exception on most errors, such as an inability to reach our public listing, lack of matching images, etc. @return [Hash,String,nil]
# File modules/mu/cloud/machine_images.rb, line 77 def self.getStockImage(cloud = MU::Config.defaultCloud, platform: nil, region: nil, fail_hard: false, quiet: false) if !MU::Cloud.supportedClouds.include?(cloud) MU.log "'#{cloud}' is not a supported cloud provider! Available providers:", MU::ERR, details: MU::Cloud.supportedClouds raise MuError, "'#{cloud}' is not a supported cloud provider!" end urls = ["http://"+BASE_IMAGE_BUCKET+".s3-website-us-east-1.amazonaws.com"+BASE_IMAGE_PATH] if $MU_CFG and $MU_CFG['custom_images_url'] urls << $MU_CFG['custom_images_url'] end images = nil urls.each { |base_url| @@image_fetch_semaphore.synchronize { if @@image_fetch_cache[cloud] and (Time.now - @@image_fetch_cache[cloud]['time']) < 30 images = @@image_fetch_cache[cloud]['contents'].dup else begin Timeout.timeout(2) do response = URI.open("#{base_url}/#{cloud}.yaml").read images ||= {} images.deep_merge!(YAML.load(response)) break end rescue StandardError => e if fail_hard raise MuError, "Failed to fetch stock images from #{base_url}/#{cloud}.yaml (#{e.message})" else MU.log "Failed to fetch stock images from #{base_url}/#{cloud}.yaml (#{e.message})", MU::WARN if !quiet end end end } } @@image_fetch_semaphore.synchronize { @@image_fetch_cache[cloud] = { 'contents' => images.dup, 'time' => Time.now } } backwards_compat = { "AWS" => "amazon_images", "Google" => "google_images", } # Load from inside our repository, if we didn't get images elsewise if images.nil? [backwards_compat[cloud], cloud].each { |file| next if file.nil? if File.exist?("#{MU.myRoot}/modules/mu/defaults/#{file}.yaml") images = YAML.load(File.read("#{MU.myRoot}/modules/mu/defaults/#{file}.yaml")) break end } end # Now overlay local overrides, both of the systemwide (/opt/mu/etc) and # per-user (~/.mu/etc) variety. [backwards_compat[cloud], cloud].each { |file| next if file.nil? if File.exist?("#{MU.etcDir}/#{file}.yaml") images ||= {} images.deep_merge!(YAML.load(File.read("#{MU.etcDir}/#{file}.yaml"))) end if Process.uid != 0 basepath = Etc.getpwuid(Process.uid).dir+"/.mu/etc" if File.exist?("#{basepath}/#{file}.yaml") images ||= {} images.deep_merge!(YAML.load(File.read("#{basepath}/#{file}.yaml"))) end end } if images.nil? if fail_hard raise MuError, "Failed to find any base images for #{cloud}" else MU.log "Failed to find any base images for #{cloud}", MU::WARN if !quiet return nil end end PLATFORM_ALIASES.each_pair { |a, t| if images[t] and !images[a] images[a] = images[t] end } if platform if !images[platform] if fail_hard raise MuError, "No base image for platform #{platform} in cloud #{cloud}" else MU.log "No base image for platform #{platform} in cloud #{cloud}", MU::WARN if !quiet return nil end end images = images[platform] if region # We won't fuss about the region argument if this isn't a cloud that # has regions, just quietly don't bother. if images.is_a?(Hash) if images[region] images = images[region] else if fail_hard raise MuError, "No base image for platform #{platform} in cloud #{cloud} region #{region} found" else MU.log "No base image for platform #{platform} in cloud #{cloud} region #{region} found", MU::WARN if !quiet return nil end end end end else if region images.values.each { |regions| # Filter to match our requested region, but for all the platforms, # since we didn't specify one. if regions.is_a?(Hash) regions.delete_if { |r| r != region } end } end end images end
Net::SSH exceptions seem to have their own behavior vis a vis threads, and our regular call stack gets circumvented when they're thrown. Cheat here to catch them gracefully.
# File modules/mu/cloud/ssh_sessions.rb, line 28 def self.handleNetSSHExceptions Thread.handle_interrupt(Net::SSH::Exception => :never) { begin Thread.handle_interrupt(Net::SSH::Exception => :immediate) { MU.log "(Probably harmless) Caught a Net::SSH Exception in #{Thread.current.inspect}", MU::DEBUG, details: Thread.current.backtrace } ensure # raise NetSSHFail, "Net::SSH had a nutty" end } end
# File modules/mu/cloud/wrappers.rb, line 33 def self.has_multiples MU::Cloud.resource_types[shortname.to_sym][:has_multiples] end
Rifle our image lists from {MU::Cloud.getStockImage} and return a list of valid platform
names. @return [Array<String>]
# File modules/mu/cloud/machine_images.rb, line 47 def self.listPlatforms return @@platform_cache if @@platform_cache and !@@platform_cache.empty? @@platform_cache = MU::Cloud.supportedClouds.map { |cloud| begin resourceClass(cloud, :Server) rescue MU::Cloud::MuCloudResourceNotImplemented, MU::MuError next end images = MU::Cloud.getStockImage(cloud, quiet: true) if images images.keys else nil end }.flatten.uniq @@platform_cache.delete(nil) @@platform_cache.sort @@platform_cache end
Given a resource type, validate that it's legit and return its base class from the {MU::Cloud} module @param type [String] @return [MU::Cloud]
# File modules/mu/cloud.rb, line 586 def self.loadBaseType(type) raise MuError, "Argument to MU::Cloud.loadBaseType cannot be nil" if type.nil? shortclass, cfg_name, _cfg_plural, _classname = MU::Cloud.getResourceNames(type) if !shortclass raise MuCloudResourceNotImplemented, "#{type} does not appear to be a valid resource type" end Object.const_get("MU").const_get("Cloud").const_get(shortclass) end
@param mommacat [MU::MommaCat]: The deployment containing this cloud resource @param mu_name [String]: Optional- specify the full Mu resource name of an existing resource to load, instead of creating a new one @param cloud_id [String]: Optional- specify the cloud provider's identifier for an existing resource to load, instead of creating a new one @param kitten_cfg [Hash]: The parse configuration for this object from {MU::Config}
# File modules/mu/cloud/resource_base.rb, line 78 def initialize(**args) raise MuError, "Cannot invoke Cloud objects without a configuration" if args[:kitten_cfg].nil? # We are a parent wrapper object. Initialize our child object and # housekeeping bits accordingly. if self.class.name =~ /^MU::Cloud::([^:]+)$/ @live = true @delayed_save = args[:delayed_save] @method_semaphore = Mutex.new @method_locks = {} if args[:mommacat] MU.log "Initializing an instance of #{self.class.name} in #{args[:mommacat].deploy_id} #{mu_name}", MU::DEBUG, details: args[:kitten_cfg] elsif args[:mu_name].nil? raise MuError, "Can't instantiate a MU::Cloud object with a live deploy or giving us a mu_name" else MU.log "Initializing a detached #{self.class.name} named #{args[:mu_name]}", MU::DEBUG, details: args[:kitten_cfg] end my_cloud = args[:kitten_cfg]['cloud'].to_s || MU::Config.defaultCloud if (my_cloud.nil? or my_cloud.empty?) and args[:mommacat] my_cloud = args[:mommacat].original_config['cloud'] end if my_cloud.nil? or !MU::Cloud.supportedClouds.include?(my_cloud) raise MuError, "Can't instantiate a MU::Cloud object without a valid cloud (saw '#{my_cloud}')" end @cloudclass = MU::Cloud.resourceClass(my_cloud, self.class.shortname) @cloud_desc_cache ||= args[:from_cloud_desc] if args[:from_cloud_desc] @cloudparentclass = MU::Cloud.cloudClass(my_cloud) @cloudobj = @cloudclass.new( mommacat: args[:mommacat], kitten_cfg: args[:kitten_cfg], cloud_id: args[:cloud_id], mu_name: args[:mu_name] ) raise MuError, "Unknown error instantiating #{self}" if @cloudobj.nil? # These should actually call the method live instead of caching a static value PUBLIC_ATTRS.each { |a| begin instance_variable_set(("@"+a.to_s).to_sym, @cloudobj.send(a)) rescue NoMethodError => e MU.log "#{@cloudclass.name} failed to implement method '#{a}'", MU::ERR, details: e.message raise e end } @deploy ||= args[:mommacat] @deploy_id ||= @deploy.deploy_id if @deploy # Register with the containing deployment if !@deploy.nil? and !@cloudobj.mu_name.nil? and !@cloudobj.mu_name.empty? and !args[:delay_descriptor_load] describe # XXX is this actually safe here? @deploy.addKitten(self.class.cfg_name, @config['name'], self) elsif !@deploy.nil? and @cloudobj.mu_name.nil? MU.log "#{self} in #{@deploy.deploy_id} didn't generate a mu_name after being loaded/initialized, dependencies on this resource will probably be confused!", MU::ERR, details: [caller, args.keys] end # We are actually a child object invoking this via super() from its # own initialize(), so initialize all the attributes and instance # variables we know to be universal. else class << self # Declare attributes that everyone should have PUBLIC_ATTRS.each { |a| attr_reader a } end # XXX this butchers ::Id and ::Ref objects that might be used by dependencies() to good effect, but we also can't expect our implementations to cope with knowing when a .to_s has to be appended to things at random @config = MU::Config.manxify(args[:kitten_cfg]) || MU::Config.manxify(args[:config]) if !@config MU.log "Missing config arguments in setInstanceVariables, can't initialize a cloud object without it", MU::ERR, details: args.keys raise MuError, "Missing config arguments in setInstanceVariables" end @deploy = args[:mommacat] || args[:deploy] @cloud_desc_cache ||= args[:from_cloud_desc] if args[:from_cloud_desc] @credentials = args[:credentials] @credentials ||= @config['credentials'] @cloud = @config['cloud'] if !@cloud if self.class.name =~ /^MU::Cloud::([^:]+)(?:::.+|$)/ cloudclass_name = Regexp.last_match[1] if MU::Cloud.supportedClouds.include?(cloudclass_name) @cloud = cloudclass_name end end end if !@cloud raise MuError, "Failed to determine what cloud #{self} should be in!" end @environment = @config['environment'] if @deploy @deploy_id = @deploy.deploy_id @appname = @deploy.appname end @cloudclass = MU::Cloud.resourceClass(@cloud, self.class.shortname) @cloudparentclass = MU::Cloud.cloudClass(@cloud) # A pre-existing object, you say? if args[:cloud_id] # TODO implement ::Id for every cloud... and they should know how to get from # cloud_desc to a fully-resolved ::Id object, not just the short string @cloud_id = args[:cloud_id] describe(cloud_id: @cloud_id) @habitat_id = habitat_id # effectively, cache this # If we can build us an ::Id object for @cloud_id instead of a # string, do so. begin idclass = @cloudparentclass.const_get(:Id) long_id = if @deploydata and @deploydata[idclass.idattr.to_s] @deploydata[idclass.idattr.to_s] elsif self.respond_to?(idclass.idattr) self.send(idclass.idattr) end @cloud_id = idclass.new(long_id) if !long_id.nil? and !long_id.empty? # 1 see if we have the value on the object directly or in deploy data # 2 set an attr_reader with the value # 3 rewrite our @cloud_id attribute with a ::Id object rescue NameError, MU::Cloud::MuCloudResourceNotImplemented end end # Use pre-existing mu_name (we're probably loading an extant deploy) # if available if args[:mu_name] @mu_name = args[:mu_name].dup # If scrub_mu_isms is set, our mu_name is always just the bare name # field of the resource. elsif @config['scrub_mu_isms'] @mu_name = @config['name'].dup # XXX feck it insert an inheritable method right here? Set a default? How should resource implementations determine whether they're instantiating a new object? end @tags = {} if !@config['scrub_mu_isms'] @tags = @deploy ? @deploy.listStandardTags : MU::MommaCat.listStandardTags end if @config['tags'] @config['tags'].each { |tag| @tags[tag['key']] = tag['value'] } end MU::MommaCat.listOptionalTags.each_pair { |k, v| @tags[k] ||= v if v } if @cloudparentclass.respond_to?(:resourceInitHook) @cloudparentclass.resourceInitHook(self, @deploy) end # Add cloud-specific instance methods for our resource objects to # inherit. if @cloudparentclass.const_defined?(:AdditionalResourceMethods) self.extend @cloudparentclass.const_get(:AdditionalResourceMethods) end if ["Server", "ServerPool"].include?(self.class.shortname) and @deploy @mu_name ||= @deploy.getResourceName(@config['name'], need_unique_string: @config.has_key?("basis")) if self.class.shortname == "Server" @groomer = MU::Groomer.new(self) end @groomclass = MU::Groomer.loadGroomer(@config["groomer"]) if windows? or @config['active_directory'] and !@mu_windows_name if !@deploydata.nil? and !@deploydata['mu_windows_name'].nil? @mu_windows_name = @deploydata['mu_windows_name'] else # Use the same random differentiator as the "real" name if we're # from a ServerPool. Helpful for admin sanity. unq = @mu_name.sub(/^.*?-(...)$/, '\1') if @config['basis'] and !unq.nil? and !unq.empty? @mu_windows_name = @deploy.getResourceName(@config['name'], max_length: 15, need_unique_string: true, use_unique_string: unq, reuse_unique_string: true) else @mu_windows_name = @deploy.getResourceName(@config['name'], max_length: 15, need_unique_string: true) end end end class << self attr_reader :groomer attr_reader :groomerclass attr_accessor :mu_windows_name # XXX might be ok as reader now end end @tags["Name"] ||= @mu_name if @mu_name end end
Defaults any resources that don't declare their release-readiness to ALPHA
. That'll learn 'em.
# File modules/mu/cloud/wrappers.rb, line 55 def self.quality MU::Cloud::ALPHA end
Given a cloud layer and resource type, return the class which implements it. @param cloud [String]: The Cloud
layer @param type [String]: The resource type. Can be the full class name, symbolic name, or Basket of Kittens configuration shorthand for the resource type. @return [Class]: The cloud-specific class implementing this resource
# File modules/mu/cloud.rb, line 600 def self.resourceClass(cloud, type) raise MuError, "cloud argument to MU::Cloud.resourceClass cannot be nil" if cloud.nil? shortclass, cfg_name, _cfg_plural, _classname = MU::Cloud.getResourceNames(type) if @cloud_class_cache.has_key?(cloud) and @cloud_class_cache[cloud].has_key?(type) if @cloud_class_cache[cloud][type].nil? raise MuError, "The '#{type}' resource is not supported in cloud #{cloud} (tried MU::#{cloud}::#{type})", caller end return @cloud_class_cache[cloud][type] end if cfg_name.nil? raise MuError, "Can't find a cloud resource type named '#{type}'" end if !File.size?(MU.myRoot+"/modules/mu/providers/#{cloud.downcase}.rb") raise MuError, "Requested to use unsupported provisioning layer #{cloud}" end begin require "mu/providers/#{cloud.downcase}/#{cfg_name}" rescue LoadError => e raise MuCloudResourceNotImplemented, "MU::Cloud::#{cloud} does not currently implement #{shortclass}, or implementation does not load correctly (#{e.message})" end @cloud_class_cache[cloud] = {} if !@cloud_class_cache.has_key?(cloud) begin cloudclass = const_get("MU").const_get("Cloud").const_get(cloud) myclass = Object.const_get("MU").const_get("Cloud").const_get(cloud).const_get(shortclass) @@resource_types[shortclass.to_sym][:class].each { |class_method| if !myclass.respond_to?(class_method) or myclass.method(class_method).owner.to_s != "#<Class:#{myclass}>" raise MuError, "MU::Cloud::#{cloud}::#{shortclass} has not implemented required class method #{class_method}" end } @@resource_types[shortclass.to_sym][:instance].each { |instance_method| if !myclass.public_instance_methods.include?(instance_method) raise MuCloudResourceNotImplemented, "MU::Cloud::#{cloud}::#{shortclass} has not implemented required instance method #{instance_method}" end } cloudclass.required_instance_methods.each { |instance_method| if !myclass.public_instance_methods.include?(instance_method) MU.log "MU::Cloud::#{cloud}::#{shortclass} has not implemented required instance method #{instance_method}, will declare as attr_accessor", MU::DEBUG end } @cloud_class_cache[cloud][type] = myclass return myclass rescue NameError => e @cloud_class_cache[cloud][type] = nil raise MuCloudResourceNotImplemented, "The '#{type}' resource is not supported in cloud #{cloud} (tried MU::Cloud::#{cloud}::#{shortclass})", e.backtrace end end
A list of supported cloud resource types as Mu classes
# File modules/mu/cloud.rb, line 474 def self.resource_types; @@resource_types end
# File modules/mu/cloud/wrappers.rb, line 25 def self.shortname name.sub(/.*?::([^:]+)$/, '\1') end
List of known/supported Cloud
providers @return [Array<String>]
# File modules/mu/cloud/providers.rb, line 26 def self.supportedClouds @@supportedCloudList end
@return [Mutex]
# File modules/mu/cloud.rb, line 507 def self.userdata_mutex @userdata_mutex ||= Mutex.new end
# File modules/mu/cloud/wrappers.rb, line 45 def self.waits_on_parent_completion MU::Cloud.resource_types[shortname.to_sym][:waits_on_parent_completion] end
Public Instance Methods
If applicable, allow this resource's NAT host blanket access via rules in its associated admin
firewall rule set.
# File modules/mu/cloud/resource_base.rb, line 825 def allowBastionAccess return nil if !@nat or !@nat.is_a?(MU::Cloud::Server) myFirewallRules.each { |acl| if acl.config["admin"] acl.addRule(@nat.listIPs, proto: "tcp") acl.addRule(@nat.listIPs, proto: "udp") acl.addRule(@nat.listIPs, proto: "icmp") end } end
# File modules/mu/cloud/resource_base.rb, line 277 def cloud if @cloud @cloud elsif @config and @config['cloud'] @config['cloud'] elsif self.class.name =~ /^MU::Cloud::([^:]+)::.+/ cloudclass_name = Regexp.last_match[1] if MU::Cloud.supportedClouds.include?(cloudclass_name) cloudclass_name else nil end else nil end end
# File modules/mu/cloud/resource_base.rb, line 386 def cloud_desc(use_cache: true) describe if !@cloudobj.nil? if @cloudobj.class.instance_methods(false).include?(:cloud_desc) @cloud_desc_cache ||= @cloudobj.cloud_desc end end if !@config.nil? and !@cloud_id.nil? and (!use_cache or @cloud_desc_cache.nil?) # The find() method should be returning a Hash with the cloud_id # as a key and a cloud platform descriptor as the value. begin args = { :region => @config['region'], :cloud => @config['cloud'], :cloud_id => @cloud_id, :credentials => @credentials, :project => habitat_id, # XXX this belongs in our required_instance_methods hack :flags => @config } @cloudparentclass.required_instance_methods.each { |m| # if respond_to?(m) # args[m] = method(m).call # else args[m] = instance_variable_get(("@"+m.to_s).to_sym) # end } matches = self.class.find(args) if !matches.nil? and matches.is_a?(Hash) # XXX or if the hash is keyed with an ::Id element, oh boy # puts matches[@cloud_id][:self_link] # puts matches[@cloud_id][:url] # if matches[@cloud_id][:self_link] # @url ||= matches[@cloud_id][:self_link] # elsif matches[@cloud_id][:url] # @url ||= matches[@cloud_id][:url] # elsif matches[@cloud_id][:arn] # @arn ||= matches[@cloud_id][:arn] # end if matches[@cloud_id] @cloud_desc_cache = matches[@cloud_id] else matches.each_pair { |k, v| # flatten out ::Id objects just in case if @cloud_id.to_s == k.to_s @cloud_desc_cache = v break end } end end if !@cloud_desc_cache MU.log "cloud_desc via #{self.class.name}.find() failed to locate a live object.\nWas called by #{caller(1..1)}", MU::WARN, details: args end rescue StandardError => e MU.log "Got #{e.inspect} trying to find cloud handle for #{self.class.shortname} #{@mu_name} (#{@cloud_id})", MU::WARN raise e end end return @cloud_desc_cache end
Merge the passed hash into the existing configuration hash of this cloud object. Currently this is only used by the {MU::Adoption} module. I don't love exposing this to the whole internal API, but I'm probably overthinking that. @param newcfg [Hash]
# File modules/mu/cloud/resource_base.rb, line 382 def config!(newcfg) @config.merge!(newcfg) end
Fetch MU::Cloud
objects for each of this object's dependencies, and return in an easily-navigable Hash
. This can include things listed in @config, implicitly-defined dependencies such as add_firewall_rules or vpc stanzas, and may refer to objects internal to this deployment or external. Will populate the instance variables @dependencies (general dependencies, which can only be sibling resources in this deployment), as well as for certain config stanzas which can refer to external resources (@vpc, @loadbalancers, @add_firewall_rules)
# File modules/mu/cloud/resource_base.rb, line 500 def dependencies(use_cache: false, debug: false) @dependencies ||= {} @loadbalancers ||= [] @firewall_rules ||= [] if @config.nil? return [@dependencies, @vpc, @loadbalancers] end if use_cache and @dependencies.size > 0 return [@dependencies, @vpc, @loadbalancers] end @config['dependencies'] = [] if @config['dependencies'].nil? loglevel = debug ? MU::NOTICE : MU::DEBUG # First, general dependencies. These should all be fellow members of # the current deployment. @config['dependencies'].each { |dep| @dependencies[dep['type']] ||= {} next if @dependencies[dep['type']].has_key?(dep['name']) handle = @deploy.findLitterMate(type: dep['type'], name: dep['name']) if !@deploy.nil? if !handle.nil? MU.log "Loaded dependency for #{self}: #{dep['name']} => #{handle}", loglevel @dependencies[dep['type']][dep['name']] = handle else # XXX yell under circumstances where we should expect to have # our stuff available already? end } # Special dependencies: my containing VPC if self.class.can_live_in_vpc and !@config['vpc'].nil? @config['vpc']["id"] ||= @config['vpc']["vpc_id"] # old deploys @config['vpc']["name"] ||= @config['vpc']["vpc_name"] # old deploys # If something hash-ified a MU::Config::Ref here, fix it if !@config['vpc']["id"].nil? and @config['vpc']["id"].is_a?(Hash) @config['vpc']["id"] = MU::Config::Ref.new(@config['vpc']["id"]) end if !@config['vpc']["id"].nil? if @config['vpc']["id"].is_a?(MU::Config::Ref) and !@config['vpc']["id"].kitten.nil? @vpc = @config['vpc']["id"].kitten(@deploy) else if @config['vpc']['habitat'] @config['vpc']['habitat'] = MU::Config::Ref.get(@config['vpc']['habitat']) end vpc_ref = MU::Config::Ref.get(@config['vpc']) @vpc = vpc_ref.kitten(@deploy) end elsif !@config['vpc']["name"].nil? and @deploy MU.log "Attempting findLitterMate on VPC for #{self}", loglevel, details: @config['vpc'] sib_by_name = @deploy.findLitterMate(name: @config['vpc']['name'], type: "vpcs", return_all: true, habitat: @config['vpc']['project'], debug: debug) if sib_by_name.is_a?(Hash) if sib_by_name.size == 1 @vpc = sib_by_name.values.first MU.log "Single VPC match for #{self}", loglevel, details: @vpc.to_s else # XXX ok but this is the wrong place for this really the config parser needs to sort this out somehow # we got multiple matches, try to pick one by preferred subnet # behavior MU.log "Sorting a bunch of VPC matches for #{self}", loglevel, details: sib_by_name.map { |s| s.to_s }.join(", ") sib_by_name.values.each { |sibling| all_private = sibling.subnets.map { |s| s.private? }.all?(true) all_public = sibling.subnets.map { |s| s.private? }.all?(false) names = sibling.subnets.map { |s| s.name } ids = sibling.subnets.map { |s| s.cloud_id } if all_private and ["private", "all_private"].include?(@config['vpc']['subnet_pref']) @vpc = sibling break elsif all_public and ["public", "all_public"].include?(@config['vpc']['subnet_pref']) @vpc = sibling break elsif @config['vpc']['subnet_name'] and names.include?(@config['vpc']['subnet_name']) #puts "CHOOSING #{@vpc.to_s} 'cause it has #{@config['vpc']['subnet_name']}" @vpc = sibling break elsif @config['vpc']['subnet_id'] and ids.include?(@config['vpc']['subnet_id']) @vpc = sibling break end } if !@vpc sibling = sib_by_name.values.sample MU.log "Got multiple matching VPCs for #{self.class.cfg_name} #{@mu_name}, so I'm arbitrarily choosing #{sibling.mu_name}", MU::WARN, details: @config['vpc'] @vpc = sibling end end else @vpc = sib_by_name MU.log "Found exact VPC match for #{self}", loglevel, details: sib_by_name.to_s end else MU.log "No shortcuts available to fetch VPC for #{self}", loglevel, details: @config['vpc'] end if !@vpc and !@config['vpc']["name"].nil? and @dependencies.has_key?("vpc") and @dependencies["vpc"].has_key?(@config['vpc']["name"]) MU.log "Grabbing VPC I see in @dependencies['vpc']['#{@config['vpc']["name"]}'] for #{self}", loglevel, details: @config['vpc'] @vpc = @dependencies["vpc"][@config['vpc']["name"]] elsif !@vpc tag_key, tag_value = @config['vpc']['tag'].split(/=/, 2) if !@config['vpc']['tag'].nil? if !@config['vpc'].has_key?("id") and !@config['vpc'].has_key?("deploy_id") and !@deploy.nil? @config['vpc']["deploy_id"] = @deploy.deploy_id end MU.log "Doing findStray for VPC for #{self}", loglevel, details: @config['vpc'] vpcs = MU::MommaCat.findStray( @config['cloud'], "vpc", deploy_id: @config['vpc']["deploy_id"], cloud_id: @config['vpc']["id"], name: @config['vpc']["name"], tag_key: tag_key, tag_value: tag_value, habitats: [@project_id], region: @config['vpc']["region"], calling_deploy: @deploy, credentials: @credentials, dummy_ok: true, debug: debug ) @vpc = vpcs.first if !vpcs.nil? and vpcs.size > 0 end if @vpc and @vpc.config and @vpc.config['bastion'] and @vpc.config['bastion'].to_h['name'] != @config['name'] refhash = @vpc.config['bastion'].to_h refhash['deploy_id'] ||= @vpc.deploy.deploy_id natref = MU::Config::Ref.get(refhash) if natref and natref.kitten(@vpc.deploy) @nat = natref.kitten(@vpc.deploy) end end if @nat.nil? and !@vpc.nil? and ( @config['vpc'].has_key?("nat_host_id") or @config['vpc'].has_key?("nat_host_tag") or @config['vpc'].has_key?("nat_host_ip") or @config['vpc'].has_key?("nat_host_name") ) nat_tag_key, nat_tag_value = @config['vpc']['nat_host_tag'].split(/=/, 2) if !@config['vpc']['nat_host_tag'].nil? @nat = @vpc.findBastion( nat_name: @config['vpc']['nat_host_name'], nat_cloud_id: @config['vpc']['nat_host_id'], nat_tag_key: nat_tag_key, nat_tag_value: nat_tag_value, nat_ip: @config['vpc']['nat_host_ip'] ) if @nat.nil? if !@vpc.cloud_desc.nil? @nat = @vpc.findNat( nat_cloud_id: @config['vpc']['nat_host_id'], nat_filter_key: "vpc-id", region: @config['vpc']["region"], nat_filter_value: @vpc.cloud_id, credentials: @config['credentials'] ) else @nat = @vpc.findNat( nat_cloud_id: @config['vpc']['nat_host_id'], region: @config['vpc']["region"], credentials: @config['credentials'] ) end end end if @vpc.nil? and @config['vpc'] feck = MU::Config::Ref.get(@config['vpc']) feck.kitten(@deploy, debug: true) pp feck raise MuError.new "#{self.class.cfg_name} #{@config['name']} failed to locate its VPC", details: @config['vpc'] end elsif self.class.cfg_name == "vpc" @vpc = self end # Google accounts usually have a useful default VPC we can use if @vpc.nil? and @project_id and @cloud == "Google" and self.class.can_live_in_vpc MU.log "Seeing about default VPC for #{self}", MU::NOTICE vpcs = MU::MommaCat.findStray( "Google", "vpc", cloud_id: "default", habitats: [@project_id], credentials: @credentials, dummy_ok: true, debug: debug ) @vpc = vpcs.first if !vpcs.nil? and vpcs.size > 0 end # Special dependencies: LoadBalancers I've asked to attach to an # instance. if @config.has_key?("loadbalancers") @loadbalancers = [] if !@loadbalancers @config['loadbalancers'].each { |lb| MU.log "Loading LoadBalancer for #{self}", MU::DEBUG, details: lb if @dependencies.has_key?("loadbalancer") and @dependencies["loadbalancer"].has_key?(lb['concurrent_load_balancer']) @loadbalancers << @dependencies["loadbalancer"][lb['concurrent_load_balancer']] else if !lb.has_key?("existing_load_balancer") and !lb.has_key?("deploy_id") and !@deploy.nil? lb["deploy_id"] = @deploy.deploy_id end lbs = MU::MommaCat.findStray( @config['cloud'], "loadbalancer", deploy_id: lb["deploy_id"], cloud_id: lb['existing_load_balancer'], name: lb['concurrent_load_balancer'], region: @config["region"], calling_deploy: @deploy, dummy_ok: true ) @loadbalancers << lbs.first if !lbs.nil? and lbs.size > 0 end } end # Munge in external resources referenced by the existing_deploys # keyword if @config["existing_deploys"] && !@config["existing_deploys"].empty? @config["existing_deploys"].each { |ext_deploy| if ext_deploy["cloud_id"] found = MU::MommaCat.findStray( @config['cloud'], ext_deploy["cloud_type"], cloud_id: ext_deploy["cloud_id"], region: @config['region'], dummy_ok: false ).first MU.log "Couldn't find existing resource #{ext_deploy["cloud_id"]}, #{ext_deploy["cloud_type"]}", MU::ERR if found.nil? @deploy.notify(ext_deploy["cloud_type"], found.config["name"], found.deploydata, mu_name: found.mu_name, triggering_node: @mu_name) elsif ext_deploy["mu_name"] && ext_deploy["deploy_id"] MU.log "#{self}: Importing metadata for #{ext_deploy["cloud_type"]} #{ext_deploy["mu_name"]} from #{ext_deploy["deploy_id"]}" found = MU::MommaCat.findStray( @config['cloud'], ext_deploy["cloud_type"], deploy_id: ext_deploy["deploy_id"], mu_name: ext_deploy["mu_name"], region: @config['region'], dummy_ok: false ).first if found.nil? MU.log "Couldn't find existing resource #{ext_deploy["mu_name"]}/#{ext_deploy["deploy_id"]}, #{ext_deploy["cloud_type"]}", MU::ERR else @deploy.notify(ext_deploy["cloud_type"], found.config["name"], found.deploydata, mu_name: ext_deploy["mu_name"], triggering_node: @mu_name) end else MU.log "Trying to find existing deploy, but either the cloud_id is not valid or no mu_name and deploy_id where provided", MU::ERR end } end if @config['dns_records'] && !@config['dns_records'].empty? @config['dns_records'].each { |dnsrec| if dnsrec.has_key?("name") if dnsrec['name'].start_with?(@deploy.deploy_id.downcase) && !dnsrec['name'].start_with?(@mu_name.downcase) MU.log "DNS records for #{@mu_name} seem to be wrong, deleting from current config", MU::WARN, details: dnsrec dnsrec.delete('name') dnsrec.delete('target') end end } end return [@dependencies, @vpc, @loadbalancers] end
Retrieve all of the known metadata for this resource. @param cloud_id [String]: The cloud platform's identifier for the resource we're describing. Makes lookups more efficient. @return [Array<Hash>]: mu_name, config, deploydata
# File modules/mu/cloud/resource_base.rb, line 453 def describe(cloud_id: nil) if cloud_id.nil? and !@cloudobj.nil? @cloud_id ||= @cloudobj.cloud_id end res_type = self.class.cfg_plural res_name = @config['name'] if !@config.nil? @credentials ||= @config['credentials'] if !@config.nil? deploydata = nil if !@deploy.nil? and @deploy.is_a?(MU::MommaCat) and !@deploy.deployment.nil? and !@deploy.deployment[res_type].nil? and !@deploy.deployment[res_type][res_name].nil? deploydata = @deploy.deployment[res_type][res_name] else # XXX This should only happen on a brand new resource, but we should # probably complain under other circumstances, if we can # differentiate them. end if self.class.has_multiples and !@mu_name.nil? and deploydata.is_a?(Hash) and deploydata.has_key?(@mu_name) @deploydata = deploydata[@mu_name] elsif deploydata.is_a?(Hash) @deploydata = deploydata end if @cloud_id.nil? and @deploydata.is_a?(Hash) if @mu_name.nil? and @deploydata.has_key?('#MU_NAME') @mu_name = @deploydata['#MU_NAME'] end if @deploydata.has_key?('cloud_id') @cloud_id ||= @deploydata['cloud_id'] end end return [@mu_name, @config, @deploydata] end
Remove all metadata and cloud resources associated with this object
# File modules/mu/cloud/resource_base.rb, line 296 def destroy if self.class.cfg_name == "server" begin ip = canonicalIP MU::Master.removeIPFromSSHKnownHosts(ip) if ip if @deploy and @deploy.deployment and @deploy.deployment['servers'] and @config['name'] me = @deploy.deployment['servers'][@config['name']][@mu_name] if me ["private_ip_address", "public_ip_address"].each { |field| if me[field] MU::Master.removeIPFromSSHKnownHosts(me[field]) end } if me["private_ip_list"] me["private_ip_list"].each { |private_ip| MU::Master.removeIPFromSSHKnownHosts(private_ip) } end end end rescue MU::MuError => e MU.log e.message, MU::WARN end end if !@cloudobj.nil? and !@cloudobj.groomer.nil? @cloudobj.groomer.cleanup elsif !@groomer.nil? @groomer.cleanup end if !@deploy.nil? if !@cloudobj.nil? and !@config.nil? and !@cloudobj.mu_name.nil? @deploy.notify(self.class.cfg_plural, @config['name'], nil, mu_name: @cloudobj.mu_name, remove: true, triggering_node: @cloudobj, delayed_save: @delayed_save) elsif !@mu_name.nil? @deploy.notify(self.class.cfg_plural, @config['name'], nil, mu_name: @mu_name, remove: true, triggering_node: self, delayed_save: @delayed_save) end @deploy.removeKitten(self) end # Make sure that if notify gets called again it won't go returning a # bunch of now-bogus metadata. @destroyed = true if !@cloudobj.nil? def @cloudobj.notify {} end else def notify {} end end end
@param max_retries [Integer]: Number of connection attempts to make before giving up @param retry_interval [Integer]: Number of seconds to wait between connection attempts @return [Net::SSH::Connection::Session]
# File modules/mu/cloud/ssh_sessions.rb, line 137 def getSSHSession(max_retries = 12, retry_interval = 30) ssh_keydir = Etc.getpwnam(@deploy.mu_user).dir+"/.ssh" nat_ssh_key, nat_ssh_user, nat_ssh_host, canonical_ip, ssh_user, _ssh_key_name = getSSHConfig session = nil retries = 0 # XXX WHY is this a thing Thread.handle_interrupt(Errno::ECONNREFUSED => :never) { } begin MU::Cloud.handleNetSSHExceptions if !nat_ssh_host.nil? proxy_cmd = "ssh -q -o StrictHostKeyChecking=no -W %h:%p #{nat_ssh_user}@#{nat_ssh_host}" MU.log "Attempting SSH to #{canonical_ip} (#{@mu_name}) as #{ssh_user} with key #{@deploy.ssh_key_name} using proxy '#{proxy_cmd}'" if retries == 0 proxy = Net::SSH::Proxy::Command.new(proxy_cmd) session = Net::SSH.start( canonical_ip, ssh_user, :config => false, :keys_only => true, :keys => [ssh_keydir+"/"+nat_ssh_key, ssh_keydir+"/"+@deploy.ssh_key_name], :verify_host_key => false, # :verbose => :info, :host_key => "ssh-rsa", :port => 22, :auth_methods => ['publickey'], :proxy => proxy ) else MU.log "Attempting SSH to #{canonical_ip} (#{@mu_name}) as #{ssh_user} with key #{ssh_keydir}/#{@deploy.ssh_key_name}" if retries == 0 session = Net::SSH.start( canonical_ip, ssh_user, :config => false, :keys_only => true, :keys => [ssh_keydir+"/"+@deploy.ssh_key_name], :verify_host_key => false, # :verbose => :info, :host_key => "ssh-rsa", :port => 22, :auth_methods => ['publickey'] ) end retries = 0 rescue Net::SSH::HostKeyMismatch => e MU.log("Remembering new key: #{e.fingerprint}") e.remember_host! session.close retry # rescue SystemCallError, Timeout::Error, Errno::ECONNRESET, Errno::EHOSTUNREACH, Net::SSH::Proxy::ConnectError, SocketError, Net::SSH::Disconnect, Net::SSH::AuthenticationFailed, IOError, Net::SSH::ConnectionTimeout, Net::SSH::Proxy::ConnectError, MU::Cloud::NetSSHFail => e rescue SystemExit, Timeout::Error, Net::SSH::AuthenticationFailed, Net::SSH::Disconnect, Net::SSH::ConnectionTimeout, Net::SSH::Proxy::ConnectError, Net::SSH::Exception, Errno::ECONNRESET, Errno::EHOSTUNREACH, Errno::ECONNREFUSED, Errno::EPIPE, SocketError, IOError => e if !active? raise MuError, "Server #{@mu_name} disappeared while I was attempting to log into it" end begin session.close if !session.nil? rescue Net::SSH::Disconnect, IOError => e if windows? MU.log "Windows has probably closed the ssh session before we could. Waiting before trying again", MU::NOTICE else MU.log "ssh session was closed unexpectedly, waiting before trying again", MU::NOTICE end sleep 10 end if retries < max_retries retries = retries + 1 msg = "ssh #{ssh_user}@#{@mu_name}: #{e.message}, waiting #{retry_interval}s (attempt #{retries}/#{max_retries})" if retries == 1 or (retries/max_retries <= 0.5 and (retries % 3) == 0) MU.log msg, MU::NOTICE if !MU::Cloud.resourceClass(@cloud, "VPC").haveRouteToInstance?(cloud_desc, credentials: @credentials) and canonical_ip.match(/(^127\.)|(^192\.168\.)|(^10\.)|(^172\.1[6-9]\.)|(^172\.2[0-9]\.)|(^172\.3[0-1]\.)|(^::1$)|(^[fF][cCdD])/) and !nat_ssh_host MU.log "Node #{@mu_name} at #{canonical_ip} looks like it's in a private address space, and I don't appear to have a direct route to it. It may not be possible to connect with this routing!", MU::WARN end elsif retries/max_retries > 0.5 MU.log msg, MU::WARN, details: e.inspect end sleep retry_interval retry else raise MuError, "#{@mu_name}: #{e.inspect} trying to connect with SSH, max_retries exceeded", e.backtrace end end return session end
Get a privileged Powershell session on the server in question, using SSL-encrypted WinRM with certificate authentication. @param max_retries [Integer]: @param retry_interval [Integer]: @param timeout [Integer]: @param winrm_retries [Integer]: @param reboot_on_problems [Boolean]:
# File modules/mu/cloud/winrm_sessions.rb, line 175 def getWinRMSession(max_retries = 40, retry_interval = 60, timeout: 30, winrm_retries: 2, reboot_on_problems: false) _nat_ssh_key, _nat_ssh_user, _nat_ssh_host, canonical_ip, _ssh_user, _ssh_key_name = getSSHConfig @mu_name ||= @config['mu_name'] shell = nil opts = nil # and now, a thing I really don't want to do MU::Master.addInstanceToEtcHosts(canonical_ip, @mu_name) # catch exceptions that circumvent our regular call stack Thread.abort_on_exception = false Thread.handle_interrupt(WinRM::WinRMWSManFault => :never) { begin Thread.handle_interrupt(WinRM::WinRMWSManFault => :immediate) { MU.log "(Probably harmless) Caught a WinRM::WinRMWSManFault in #{Thread.current.inspect}", MU::DEBUG, details: Thread.current.backtrace } ensure # Reraise something useful end } retries = 0 rebootable_fails = 0 begin loglevel = retries > 4 ? MU::NOTICE : MU::DEBUG MU.log "Calling WinRM on #{@mu_name}", loglevel, details: opts opts = { retry_limit: winrm_retries, no_ssl_peer_verification: true, # XXX this should not be necessary; we get 'hostname "foo" does not match the server certificate' even when it clearly does match ca_trust_path: "#{MU.mySSLDir}/Mu_CA.pem", transport: :ssl, operation_timeout: timeout, } if retries % 2 == 0 # NTLM password over https opts[:endpoint] = 'https://'+canonical_ip+':5986/wsman' opts[:user] = @config['windows_admin_username'] opts[:password] = getWindowsAdminPassword else # certificate auth over https opts[:endpoint] = 'https://'+@mu_name+':5986/wsman' opts[:client_cert] = "#{MU.mySSLDir}/#{@mu_name}-winrm.crt" opts[:client_key] = "#{MU.mySSLDir}/#{@mu_name}-winrm.key" end conn = WinRM::Connection.new(opts) conn.logger.level = :debug if retries > 2 MU.log "WinRM connection to #{@mu_name} created", MU::DEBUG, details: conn shell = conn.shell(:powershell) shell.run('ipconfig') # verify that we can do something rescue Errno::EHOSTUNREACH, Errno::ECONNREFUSED, HTTPClient::ConnectTimeoutError, OpenSSL::SSL::SSLError, SocketError, WinRM::WinRMError, Timeout::Error => e retries, rebootable_fails = handleWindowsFail(e, retries, rebootable_fails, max_retries: max_retries, reboot_on_problems: reboot_on_problems, retry_interval: retry_interval) retry ensure MU::Master.removeInstanceFromEtcHosts(@mu_name) end shell end
Return the cloud object's idea of where it lives (project, account, etc) in the form of an identifier. If not applicable for this object, we expect to return nil
. @return [String,nil]
# File modules/mu/cloud/resource_base.rb, line 352 def habitat(nolookup: true) return nil if ["folder", "habitat"].include?(self.class.cfg_name) if @cloudobj @cloudparentclass.habitat(@cloudobj, nolookup: nolookup, deploy: @deploy) else @cloudparentclass.habitat(self, nolookup: nolookup, deploy: @deploy) end end
# File modules/mu/cloud/resource_base.rb, line 361 def habitat_id(nolookup: false) @habitat_id ||= habitat(nolookup: nolookup) @habitat_id end
Gracefully message and attempt to accommodate the common transient errors peculiar to Windows nodes @param e [Exception]: The exception that we're handling @param retries [Integer]: The current number of retries, which we'll increment and pass back to the caller @param rebootable_fails [Integer]: The current number of reboot-worthy failures, which we'll increment and pass back to the caller @param max_retries [Integer]: Maximum number of retries to attempt; we'll raise an exception if this is exceeded @param reboot_on_problems [Boolean]: Whether we should try to reboot a “stuck” machine @param retry_interval [Integer]: How many seconds to wait before returning for another attempt
# File modules/mu/cloud/winrm_sessions.rb, line 32 def handleWindowsFail(e, retries, rebootable_fails, max_retries: 30, reboot_on_problems: false, retry_interval: 45) msg = "WinRM connection to https://"+@mu_name+":5986/wsman: #{e.message}, waiting #{retry_interval}s (attempt #{retries}/#{max_retries})" if e.class.name == "WinRM::WinRMAuthorizationError" or e.message.match(/execution expired/) and reboot_on_problems if rebootable_fails > 0 and (rebootable_fails % 7) == 0 MU.log "#{@mu_name} still misbehaving, forcing Stop and Start from API", MU::WARN reboot(true) # vicious API stop/start sleep retry_interval*3 rebootable_fails = 0 else if rebootable_fails == 5 MU.log "#{@mu_name} misbehaving, attempting to reboot from API", MU::WARN reboot # graceful API restart sleep retry_interval*2 end rebootable_fails = rebootable_fails + 1 end end if retries < max_retries if retries == 1 or (retries/max_retries <= 0.5 and (retries % 3) == 0 and retries != 0) MU.log msg, MU::NOTICE elsif retries/max_retries > 0.5 MU.log msg, MU::WARN, details: e.inspect end sleep retry_interval retries = retries + 1 else raise MuError, "#{@mu_name}: #{e.inspect} trying to connect with WinRM, max_retries exceeded", e.backtrace end return [retries, rebootable_fails] end
Basic setup tasks performed on a new node during its first initial ssh connection. Most of this is terrible Windows glue. @param ssh [Net::SSH::Connection::Session]: The active SSH session to the new node.
# File modules/mu/cloud/ssh_sessions.rb, line 46 def initialSSHTasks(ssh) win_env_fix = %q{echo 'export PATH="$PATH:/cygdrive/c/opscode/chef/embedded/bin"' > "$HOME/chef-client"; echo 'prev_dir="`pwd`"; for __dir in /proc/registry/HKEY_LOCAL_MACHINE/SYSTEM/CurrentControlSet/Control/Session\ Manager/Environment;do cd "$__dir"; for __var in `ls * | grep -v TEMP | grep -v TMP`;do __var=`echo $__var | tr "[a-z]" "[A-Z]"`; test -z "${!__var}" && export $__var="`cat $__var`" >/dev/null 2>&1; done; done; cd "$prev_dir"; /cygdrive/c/opscode/chef/bin/chef-client.bat $@' >> "$HOME/chef-client"; chmod 700 "$HOME/chef-client"; ( grep "^alias chef-client=" "$HOME/.bashrc" || echo 'alias chef-client="$HOME/chef-client"' >> "$HOME/.bashrc" ) ; ( grep "^alias mu-groom=" "$HOME/.bashrc" || echo 'alias mu-groom="powershell -File \"c:/Program Files/Amazon/Ec2ConfigService/Scripts/UserScript.ps1\""' >> "$HOME/.bashrc" )} win_installer_check = %q{ls /proc/registry/HKEY_LOCAL_MACHINE/SOFTWARE/Microsoft/Windows/CurrentVersion/Installer/} lnx_installer_check = %q{ps auxww | awk '{print $11}' | egrep '(/usr/bin/yum|apt-get|dpkg)'} lnx_updates_check = %q{( test -f /.mu-installer-ran-updates || ! test -d /var/lib/cloud/instance ) || echo "userdata still running"} win_set_pw = nil if windows? and !@config['use_cloud_provider_windows_password'] # This covers both the case where we have a windows password passed from a vault and where we need to use a a random Windows Admin password generated by MU::Cloud::Server.generateWindowsPassword pw = @groomer.getSecret( vault: @config['mu_name'], item: "windows_credentials", field: "password" ) win_check_for_pw = %Q{powershell -Command '& {Add-Type -AssemblyName System.DirectoryServices.AccountManagement; $Creds = (New-Object System.Management.Automation.PSCredential("#{@config["windows_admin_username"]}", (ConvertTo-SecureString "#{pw}" -AsPlainText -Force)));$DS = New-Object System.DirectoryServices.AccountManagement.PrincipalContext([System.DirectoryServices.AccountManagement.ContextType]::Machine); $DS.ValidateCredentials($Creds.GetNetworkCredential().UserName, $Creds.GetNetworkCredential().password); echo $Result}'} win_set_pw = %Q{powershell -Command "& {(([adsi]('WinNT://./#{@config["windows_admin_username"]}, user')).psbase.invoke('SetPassword', '#{pw}'))}"} end # There shouldn't be a use case where a domain joined computer goes through initialSSHTasks. Removing Active Directory specific computer rename. set_hostname = true hostname = nil if !@config['active_directory'].nil? if @config['active_directory']['node_type'] == "domain_controller" && @config['active_directory']['domain_controller_hostname'] hostname = @config['active_directory']['domain_controller_hostname'] @mu_windows_name = hostname set_hostname = true else # Do we have an AD specific hostname? hostname = @mu_windows_name set_hostname = true end else hostname = @mu_windows_name end win_check_for_hostname = %Q{powershell -Command '& {hostname}'} win_set_hostname = %Q{powershell -Command "& {Rename-Computer -NewName '#{hostname}' -Force -PassThru -Restart; Restart-Computer -Force }"} begin # Set our admin password first, if we need to if windows? and !win_set_pw.nil? and !win_check_for_pw.nil? output = ssh.exec!(win_check_for_pw) raise MU::Cloud::BootstrapTempFail, "Got nil output from ssh session, waiting and retrying" if output.nil? if !output.match(/True/) MU.log "Setting Windows password for user #{@config['windows_admin_username']}", details: ssh.exec!(win_set_pw) end end if windows? output = ssh.exec!(win_env_fix) output += ssh.exec!(win_installer_check) raise MU::Cloud::BootstrapTempFail, "Got nil output from ssh session, waiting and retrying" if output.nil? if output.match(/InProgress/) raise MU::Cloud::BootstrapTempFail, "Windows Installer service is still doing something, need to wait" end if set_hostname and !@hostname_set and @mu_windows_name output = ssh.exec!(win_check_for_hostname) raise MU::Cloud::BootstrapTempFail, "Got nil output from ssh session, waiting and retrying" if output.nil? if !output.match(/#{@mu_windows_name}/) MU.log "Setting Windows hostname to #{@mu_windows_name}", details: ssh.exec!(win_set_hostname) @hostname_set = true # Reboot from the API too, in case Windows is flailing if !@cloudobj.nil? @cloudobj.reboot else reboot end raise MU::Cloud::BootstrapTempFail, "Set hostname in Windows, waiting for reboot" end end else output = ssh.exec!(lnx_installer_check) if !output.nil? and !output.empty? raise MU::Cloud::BootstrapTempFail, "Linux package manager is still doing something, need to wait (#{output})" end if !@config['skipinitialupdates'] and !@config['scrub_mu_isms'] and !@config['userdata_script'] output = ssh.exec!(lnx_updates_check) if !output.nil? and output.match(/userdata still running/) raise MU::Cloud::BootstrapTempFail, "Waiting for initial userdata system updates to complete" end end end rescue RuntimeError, IOError => e raise MU::Cloud::BootstrapTempFail, "Got #{e.inspect} performing initial SSH connect tasks, will try again" end end
Basic setup tasks performed on a new node during its first WinRM connection. Most of this is terrible Windows glue. @param shell [WinRM::Shells::Powershell]: An active Powershell session to the new node.
# File modules/mu/cloud/winrm_sessions.rb, line 98 def initialWinRMTasks(shell) retries = 0 rebootable_fails = 0 begin if !@config['use_cloud_provider_windows_password'] pw = @groomer.getSecret( vault: @config['mu_name'], item: "windows_credentials", field: "password" ) win_check_for_pw = %Q{Add-Type -AssemblyName System.DirectoryServices.AccountManagement; $Creds = (New-Object System.Management.Automation.PSCredential("#{@config["windows_admin_username"]}", (ConvertTo-SecureString "#{pw}" -AsPlainText -Force)));$DS = New-Object System.DirectoryServices.AccountManagement.PrincipalContext([System.DirectoryServices.AccountManagement.ContextType]::Machine); $DS.ValidateCredentials($Creds.GetNetworkCredential().UserName, $Creds.GetNetworkCredential().password); echo $Result} resp = shell.run(win_check_for_pw) if resp.stdout.chomp != "True" win_set_pw = %Q{(([adsi]('WinNT://./#{@config["windows_admin_username"]}, user')).psbase.invoke('SetPassword', '#{pw}'))} resp = shell.run(win_set_pw) puts resp.stdout MU.log "Resetting Windows host password", MU::NOTICE, details: resp.stdout end end # Install Cygwin here, because for some reason it breaks inside Chef # XXX would love to not do this here pkgs = ["bash", "mintty", "vim", "curl", "openssl", "wget", "lynx", "openssh"] admin_home = "c:/bin/cygwin/home/#{@config["windows_admin_username"]}" install_cygwin = %Q{ If (!(Test-Path "c:/bin/cygwin/Cygwin.bat")){ $WebClient = New-Object System.Net.WebClient $WebClient.DownloadFile("http://cygwin.com/setup-x86_64.exe","$env:Temp/setup-x86_64.exe") Start-Process -wait -FilePath $env:Temp/setup-x86_64.exe -ArgumentList "-q -n -l $env:Temp/cygwin -R c:/bin/cygwin -s http://mirror.cs.vt.edu/pub/cygwin/cygwin/ -P #{pkgs.join(',')}" } if(!(Test-Path #{admin_home})){ New-Item -type directory -path #{admin_home} } if(!(Test-Path #{admin_home}/.ssh)){ New-Item -type directory -path #{admin_home}/.ssh } if(!(Test-Path #{admin_home}/.ssh/authorized_keys)){ New-Item #{admin_home}/.ssh/authorized_keys -type file -force -value "#{@deploy.ssh_public_key}" } } resp = shell.run(install_cygwin) if resp.exitcode != 0 MU.log "Failed at installing Cygwin", MU::ERR, details: resp end hostname = nil if !@config['active_directory'].nil? if @config['active_directory']['node_type'] == "domain_controller" && @config['active_directory']['domain_controller_hostname'] hostname = @config['active_directory']['domain_controller_hostname'] @mu_windows_name = hostname else # Do we have an AD specific hostname? hostname = @mu_windows_name end else hostname = @mu_windows_name end resp = shell.run(%Q{hostname}) if resp.stdout.chomp != hostname resp = shell.run(%Q{Rename-Computer -NewName '#{hostname}' -Force -PassThru -Restart; Restart-Computer -Force}) MU.log "Renaming Windows host to #{hostname}; this will trigger a reboot", MU::NOTICE, details: resp.stdout reboot(true) sleep 30 end rescue WinRM::WinRMError, HTTPClient::ConnectTimeoutError => e retries, rebootable_fails = handleWindowsFail(e, retries, rebootable_fails, max_retries: 10, reboot_on_problems: true, retry_interval: 30) retry end end
Set our deploy
and deploy_id
attributes, optionally doing so even if they have already been set.
@param mommacat [MU::MommaCat]: The deploy to which we're being told we belong @param force [Boolean]: Set even if we already have a deploy object @return [String]: Our new deploy_id
# File modules/mu/cloud/resource_base.rb, line 51 def intoDeploy(mommacat, force: false) if force or (!@deploy) MU.log "Inserting #{self} [#{self.object_id}] into #{mommacat.deploy_id} as a #{@config['name']}", MU::DEBUG @deploy = mommacat @deploy.addKitten(@cloudclass.cfg_plural, @config['name'], self) @deploy_id = @deploy.deploy_id @cloudobj.intoDeploy(mommacat, force: force) if @cloudobj end @deploy_id end
We're fundamentally a wrapper class, so go ahead and reroute requests that are meant for our wrapped object.
# File modules/mu/cloud/resource_base.rb, line 368 def method_missing(method_sym, *arguments) if @cloudobj MU.log "INVOKING #{method_sym} FROM PARENT CLOUD OBJECT #{self}", MU::DEBUG, details: arguments @cloudobj.method(method_sym).call(*arguments) else raise NoMethodError, "No such instance method #{method_sym} available on #{self.class.name}" end end
@return [Array<MU::Cloud::FirewallRule>]
# File modules/mu/cloud/resource_base.rb, line 811 def myFirewallRules dependencies rules = [] if @dependencies.has_key?("firewall_rule") rules = @dependencies['firewall_rule'].values end # XXX what other ways are these specified? rules end
Using the automatically-defined +@vpc+ from {dependencies} in conjunction with our config, return our configured subnets. @return [Array<MU::Cloud::VPC::Subnet>]
# File modules/mu/cloud/resource_base.rb, line 780 def mySubnets dependencies if !@vpc or !@config["vpc"] return nil end if @config["vpc"]["subnet_id"] or @config["vpc"]["subnet_name"] @config["vpc"]["subnets"] ||= [] subnet_block = {} subnet_block["subnet_id"] = @config["vpc"]["subnet_id"] if @config["vpc"]["subnet_id"] subnet_block["subnet_name"] = @config["vpc"]["subnet_name"] if @config["vpc"]["subnet_name"] @config["vpc"]["subnets"] << subnet_block @config["vpc"]["subnets"].uniq! end if (!@config["vpc"]["subnets"] or @config["vpc"]["subnets"].empty?) and !@config["vpc"]["subnet_id"] return @vpc.subnets end subnets = [] @config["vpc"]["subnets"].each { |subnet| subnet_obj = @vpc.getSubnet(cloud_id: subnet["subnet_id"].to_s, name: subnet["subnet_name"].to_s) raise MuError.new "Couldn't find a live subnet for #{self} matching #{subnet} in #{@vpc}", details: @vpc.subnets.map { |s| s.name }.join(",") if subnet_obj.nil? subnets << subnet_obj } subnets end
# File modules/mu/cloud/resource_base.rb, line 342 def notify {} end
A hook that is always called just before each instance method is invoked, so that we can ensure that repetitive setup tasks (like resolving :resource_group
for Azure
resources) have always been done.
# File modules/mu/cloud/resource_base.rb, line 841 def resourceInitHook @cloud ||= cloud if @cloudparentclass.respond_to?(:resourceInitHook) @cloudparentclass.resourceInitHook(@cloudobj, @deploy) end end
Print something palatable when we're called in a string context.
# File modules/mu/cloud/resource_base.rb, line 31 def to_s fullname = "#{self.class.shortname}" if !@cloudobj.nil? and !@cloudobj.mu_name.nil? @mu_name ||= @cloudobj.mu_name end if !@mu_name.nil? and !@mu_name.empty? fullname = fullname + " '#{@mu_name}'" end if !@cloud_id.nil? fullname = fullname + " (#{@cloud_id})" end return fullname end
Return the virtual_name
config field, if it is set. @param name [String]: If set, will only return a value if virtual_name
matches this string @return [String,nil]
# File modules/mu/cloud/resource_base.rb, line 66 def virtual_name(name = nil) if @config and @config['virtual_name'] and (!name or name == @config['virtual_name']) return @config['virtual_name'] end nil end
# File modules/mu/cloud/server.rb, line 24 def windows? return true if %w{win2k16 win2k12r2 win2k12 win2k8 win2k8r2 win2k19 windows}.include?(@config['platform']) begin return true if cloud_desc.respond_to?(:platform) and cloud_desc.platform == "Windows" # XXX ^ that's AWS-speak, doesn't cover GCP or anything else; maybe we should require cloud layers to implement this so we can just call @cloudobj.windows? rescue MU::MuError return false end false end
# File modules/mu/cloud/winrm_sessions.rb, line 63 def windowsRebootPending?(shell = nil) if shell.nil? shell = getWinRMSession(1, 30) end # if (Get-Item "HKLM:/SOFTWARE/Microsoft/Windows/CurrentVersion/WindowsUpdate/Auto Update/RebootRequired" -EA Ignore) { exit 1 } cmd = %Q{ if (Get-ChildItem "HKLM:/Software/Microsoft/Windows/CurrentVersion/Component Based Servicing/RebootPending" -EA Ignore) { echo "Component Based Servicing/RebootPending is true" exit 1 } if (Get-ItemProperty "HKLM:/SYSTEM/CurrentControlSet/Control/Session Manager" -Name PendingFileRenameOperations -EA Ignore) { echo "Control/Session Manager/PendingFileRenameOperations is true" exit 1 } try { $util = [wmiclass]"\\\\.\\root\\ccm\\clientsdk:CCM_ClientUtilities" $status = $util.DetermineIfRebootPending() if(($status -ne $null) -and $status.RebootPending){ echo "WMI says RebootPending is true" exit 1 } } catch { exit 0 } exit 0 } resp = shell.run(cmd) returnval = resp.exitcode == 0 ? false : true shell.close returnval end