class Jets::Gems::Check
Attributes
Public Class Methods
# File lib/jets/gems/check.rb, line 9 def initialize(options={}) @options = options @missing_gems = [] # keeps track of gems that are not found in any of the serverlessgems source end
Public Instance Methods
# File lib/jets/gems/check.rb, line 79 def agree Agree.new end
Use pre-compiled gem because the gem could have development header shared object file dependencies. The shared dependencies are packaged up as part of the pre-compiled gem so it is available in the Lambda execution environment.
Example paths: Macosx:
opt/ruby/gems/2.5.0/extensions/x86_64-darwin-16/2.5.0-static/nokogiri-1.8.1 opt/ruby/gems/2.5.0/extensions/x86_64-darwin-16/2.5.0-static/byebug-9.1.0
Official AWS Lambda Linux AMI:
opt/ruby/gems/2.5.0/extensions/x86_64-linux/2.5.0-static/nokogiri-1.8.1
Circleci Ubuntu based Linux:
opt/ruby/gems/2.5.0/extensions/x86_64-linux/2.5.0/pg-0.21.0
# File lib/jets/gems/check.rb, line 180 def compiled_gem_paths Dir.glob("#{Jets.build_root}/stage/opt/ruby/gems/*/extensions/**/**/*.{so,bundle}") end
Context, observations, and history:
Two ways to check if gem is compiled.
1. compiled_gem_paths - look for .so and .bundle extension files in the folder itself. 2. gemspec - uses the gemspec metadata.
Observations:
* The gemspec approach generally finds more compiled gems than the compiled_gem_paths approach. * So when using the compiled_gem_paths some compiled are missed and not properly detected like http-parser. * However, some gemspec found compiled gems like json are weird and they don't work when they get replaced.
History:
* Started with compiled_gem_paths approach * Tried to gemspec approach, but ran into json-2.1.0 gem issues. bundler removes? http://bit.ly/39T8uln * Went to selective checking approach with `cli: true` option. This helped gather more data. * jets deploy - compiled_gem_paths * jets gems:check - gemspec_compiled_gems * Going back to compiled_gem_paths with: * Using the `weird_gem?` check to filter out gems removed by bundler. Note: Only happens with specific versions of json. * Keeping compiled_gem_paths for Jets Afterburner mode. Default to gemspec_compiled_gems otherwise
# File lib/jets/gems/check.rb, line 109 def compiled_gems # @use_gemspec option finds compile gems with Gem::Specification # The normal build process does not use this and checks the file system. # So @use_gemspec is only used for this command: # # jets gems:check # # This is because it seems like some gems like json are remove and screws things up. # We'll filter out for the json gem as a hacky workaround, unsure if there are more # gems though that exhibit this behavior. if @options[:use_gemspec] == false # Afterburner mode compiled_gems = compiled_gem_paths.map { |p| gem_name_from_path(p) }.uniq # Double check that the gems are also in the gemspec list since that # one is scoped to Bundler and will only included gems used in the project. # This handles the possiblity of stale gems leftover from previous builds # in the cache. # TODO: figure out if we need # compiled_gems.select { |g| gemspec_compiled_gems.include?(g) } else # default when use_gemspec not set # # jets deploy # jets gems:check # gems = gemspec_compiled_gems gems += other_compiled_gems gems += registered_compiled_gems gems.uniq end end
Input: opt/ruby/gems/2.5.0/extensions/x86_64-darwin-16/2.5.0-static/byebug-9.1.0 Output: byebug-9.1.0
# File lib/jets/gems/check.rb, line 186 def gem_name_from_path(path) regexp = %r{opt/ruby/gems/\d+\.\d+\.\d+/extensions/.*?/.*?/(.*?)/} path.match(regexp)[1] # gem_name end
# File lib/jets/gems/check.rb, line 19 def gems_source ENV['SG_API'] || Jets.config.gems.source end
So can also check for compiled gems with Gem::Specification But then also includes the json gem, which then bundler removes? We'll figure out the the json gems. gist.github.com/tongueroo/16f4aa5ac5393424103347b0e529495e
This is a faster way to check but am unsure if there are more gems than just json that exhibit this behavior. So only using this technique for this commmand:
jets gems:check
Thanks: gist.github.com/aelesbao/1414b169a79162b1d795 and
https://stackoverflow.com/questions/5165950/how-do-i-get-a-list-of-gems-that-are-installed-that-have-native-extensions
# File lib/jets/gems/check.rb, line 203 def gemspec_compiled_gems specs = Gem::Specification.each.select { |spec| spec.extensions.any? } specs.reject! { |spec| weird_gem?(spec.name) } specs.map(&:full_name) end
# File lib/jets/gems/check.rb, line 44 def missing? !@missing_gems.empty? end
# File lib/jets/gems/check.rb, line 48 def missing_message template = <<-EOL Your project requires compiled gems that are not currently available. Unavailable pre-compiled gems: <% missing_gems.each do |gem| %> * <%= gem -%> <% end %> Your current serverlessgems source: #{gems_source} Jets is unable to build a deployment package that will work on AWS Lambda without the required pre-compiled gems. To remedy this, you can: * Use another gem that does not require compilation. * Create your own custom layer with the gem: http://rubyonjets.com/docs/extras/custom-lambda-layers/ <% if agree.yes? -%> * No need to report this to us, as we've already been notified. <% elsif agree.no? -%> * You have choosen not to report data to serverlessgems so we will not be notified about these missing gems. * You can edit ~/.jets/agree to change this. * Reporting gems generally allows Serverless Gems to build the missing gems within a few minutes. * You can try redeploying again after a few minutes. * Non-reported gems may take days or even longer to be built. <% end -%> Compiled gems usually take some time to figure out how to build as they each depend on different libraries and packages. More info: http://rubyonjets.com/docs/serverlessgems/ EOL erb = ERB.new(template, nil, '-') # trim mode https://stackoverflow.com/questions/4632879/erb-template-removing-the-trailing-line erb.result(binding) end
note 2.7.0/gems vs 2.7.0/extensions
/tmp/jets/demo/stage/opt/ruby/gems/2.7.0/gems/nokogiri-1.11.1-x86_64-darwin/ /tmp/jets/demo/stage/opt/ruby/gems/2.7.0/extensions/x86_64-linux/2.7.0/
Also the platform is appended to the gem folder now
/tmp/jets/demo/stage/opt/ruby/gems/2.7.0/gems/nokogiri-1.11.1-x86_64-darwin /tmp/jets/demo/stage/opt/ruby/gems/2.7.0/gems/nokogiri-1.11.1-x86_64-linux
# File lib/jets/gems/check.rb, line 151 def other_compiled_gems paths = Dir.glob("#{Jets.build_root}/stage/opt/ruby/gems/#{Jets::Gems.ruby_folder}/gems/*{-darwin,-linux}") paths.map { |p| File.basename(p).sub(/-x\d+.*-(darwin|linux)/,'') } end
# File lib/jets/gems/check.rb, line 156 def registered_compiled_gems registered = Jets::Gems::Registered.new registered_gems = registered.all # no version numbers in this list paths = Dir.glob("#{Jets.build_root}/stage/opt/ruby/gems/#{Jets::Gems.ruby_folder}/gems/*") project_gems = paths.map { |p| File.basename(p).sub(/-x\d+.*-(darwin|linux)/,'') } project_gems.select do |name| name_only = name.sub(/-\d+\.\d+\.\d+.*/,'') registered_gems.include?(name_only) end end
Checks whether the gem is found at the serverlessgems source.
# File lib/jets/gems/check.rb, line 24 def run(exit_early: false) puts "Checking projects gems for binary serverlessgems..." compiled_gems.each do |gem_name| puts "Checking #{gem_name}..." if @options[:verbose] exist = Jets::Gems::Exist.new @missing_gems << gem_name unless exist.check(gem_name) end if exit_early && !@missing_gems.empty? # Exits early if not all the linux gems are available. # Better to error now than deploy a broken package to AWS Lambda. # Provide users with message about using their own serverlessgems source. puts missing_message Report.new(@options).report(@missing_gems) if agree.yes? exit 1 end compiled_gems end
# File lib/jets/gems/check.rb, line 14 def run! puts "Gems source: #{gems_source}" if @options[:verbose] run(exit_early: true) end
Filter out the weird special case gems that bundler deletes? Probably to fix some bug.
$ bundle show json The gem json has been deleted. It was installed at: /home/ec2-user/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/json-2.1.0
# File lib/jets/gems/check.rb, line 216 def weird_gem?(name) command = "bundle show #{name} 2>&1" output = `#{command}` output.include?("has been deleted") end