class PEROBS::DynamoDB

This class implements an Amazon DynamoDB storage engine for PEROBS.

Constants

INTERNAL_ITEMS

Attributes

item_counter[R]

Public Class Methods

delete_db(table_name) click to toggle source
# File lib/perobs/DynamoDB.rb, line 99
def DynamoDB::delete_db(table_name)
  dynamodb = Aws::DynamoDB::Client.new
  dynamodb.delete_table(:table_name => table_name)
  dynamodb.wait_until(:table_not_exists, table_name: table_name)
end
new(db_name, options = {}) click to toggle source

Create a new DynamoDB object. @param db_name [String] name of the DB directory @param options [Hash] options to customize the behavior. Currently only

the following options are supported:
:serializer    : Can be :json and :yaml
:aws_id        : AWS credentials ID
:aws_key       : AWS credentials key
:aws_region    : AWS region to host the data
Calls superclass method PEROBS::DataBase::new
# File lib/perobs/DynamoDB.rb, line 51
def initialize(db_name, options = {})
  # :marshal serialization results in a binary format that cannot easily
  # be stored in DynamoDB. We fall back to :yaml.
  if options[:serializer] == :marshal
    options[:serializer] = :yaml
  end

  super(options)

  if options.include?(:aws_id) && options.include?(:aws_key)
    Aws.config[:credentials] = Aws::Credentials.new(options[:aws_id],
                                                    options[:aws_key])
  end
  if options.include?(:aws_region)
    Aws.config[:region] = options[:aws_region]
  end

  @dynamodb = Aws::DynamoDB::Client.new
  @table_name = db_name
  @config = nil
  # The number of items currently stored in the DB.
  @item_counter = nil
  if create_table(@table_name)
    @config = { 'serializer' => @serializer }
    put_hash('config', @config)
    @item_counter = 0
    dynamo_put_item('item_counter', @item_counter.to_s)
  else
    @config = get_hash('config')
    if @config['serializer'] != @serializer
      raise ArgumentError, "DynamoDB #{@table_name} was created with " +
        "serializer #{@config['serializer']} but was now opened with " +
        "serializer #{@serializer}."
    end
    @item_counter = dynamo_get_item('item_counter').to_i
  end

  # Read the existing DB config.
end

Public Instance Methods

check(id, repair) click to toggle source

Check if the stored object is syntactically correct. @param id [Integer] Object ID @param repair [TrueClass/FalseClass] True if an repair attempt should be

made.

@return [TrueClass/FalseClass] True if the object is OK, otherwise

false.
# File lib/perobs/DynamoDB.rb, line 210
def check(id, repair)
  begin
    get_object(id)
  rescue => e
    PEROBS.log.error "Cannot read object with ID #{id}: #{e.message}"
    return false
  end

  true
end
check_db(repair = false) click to toggle source

Basic consistency check. @param repair [TrueClass/FalseClass] True if found errors should be

repaired.
# File lib/perobs/DynamoDB.rb, line 190
def check_db(repair = false)
  unless (item_counter = dynamo_get_item('item_counter')) &&
         item_counter == @item_counter
    PEROBS.log.error "@item_counter variable (#{@item_counter}) and " +
      "item_counter table entry (#{item_counter}) don't match"
  end
  item_counter = 0
  each_item { item_counter += 1 }
  unless item_counter == @item_counter
    PEROBS.log.error "Table contains #{item_counter} items but " +
      "@item_counter is #{@item_counter}"
  end
end
clear_marks() click to toggle source

This method must be called to initiate the marking process.

# File lib/perobs/DynamoDB.rb, line 152
def clear_marks
  each_item do |id|
    dynamo_mark_item(id, false)
  end
end
delete_database() click to toggle source

Delete the entire database. The database is no longer usable after this method was called.

# File lib/perobs/DynamoDB.rb, line 93
def delete_database
  dynamodb = Aws::DynamoDB::Client.new
  dynamodb.delete_table(:table_name => @table_name)
  dynamodb.wait_until(:table_not_exists, table_name: @table_name)
end
delete_unmarked_objects() click to toggle source

Permanently delete all objects that have not been marked. Those are orphaned and are no longer referenced by any actively used object. @return [Integer] Count of the deleted objects.

# File lib/perobs/DynamoDB.rb, line 161
def delete_unmarked_objects
  deleted_objects_count = 0
  each_item do |id|
    unless dynamo_is_marked?(id)
      dynamo_delete_item(id)
      deleted_objects_count += 1
      @item_counter -= 1
    end
  end
  dynamo_put_item('item_counter', @item_counter.to_s)

  deleted_objects_count
end
get_hash(name) click to toggle source

Load the Hash with the given name. @param name [String] Name of the hash. @return [Hash] A Hash that maps String objects to strings or numbers.

# File lib/perobs/DynamoDB.rb, line 122
def get_hash(name)
  if (item = dynamo_get_item(name))
    JSON.parse(item)
  else
    ::Hash.new
  end
end
get_object(id) click to toggle source

Load the given object from the filesystem. @param id [Integer] object ID @return [Hash] Object as defined by PEROBS::ObjectBase or nil if ID does

not exist
# File lib/perobs/DynamoDB.rb, line 147
def get_object(id)
  (item = dynamo_get_item(id.to_s)) ? deserialize(item) : nil
end
include?(id) click to toggle source

Return true if the object with given ID exists @param id [Integer]

# File lib/perobs/DynamoDB.rb, line 107
def include?(id)
  !dynamo_get_item(id.to_s).nil?
end
is_marked?(id) click to toggle source

Check if the object is marked. @param id [Integer] ID of the object to check

# File lib/perobs/DynamoDB.rb, line 183
def is_marked?(id)
  dynamo_is_marked?(id.to_s)
end
mark(id) click to toggle source

Mark an object. @param id [Integer] ID of the object to mark

# File lib/perobs/DynamoDB.rb, line 177
def mark(id)
  dynamo_mark_item(id.to_s, true)
end
put_hash(name, hash) click to toggle source

Store a simple Hash as a JSON encoded file into the DB directory. @param name [String] Name of the hash. Will be used as file name. @param hash [Hash] A Hash that maps String objects to strings or numbers.

# File lib/perobs/DynamoDB.rb, line 115
def put_hash(name, hash)
  dynamo_put_item(name, hash.to_json)
end
put_object(obj, id) click to toggle source

Store the given object into the cluster files. @param obj [Hash] Object as defined by PEROBS::ObjectBase

# File lib/perobs/DynamoDB.rb, line 132
def put_object(obj, id)
  id_str = id.to_s
  unless dynamo_get_item(id_str)
    # The is no object with this ID yet. Increase the item counter.
    @item_counter += 1
    dynamo_put_item('item_counter', @item_counter.to_s)
  end

  dynamo_put_item(id.to_s, serialize(obj))
end

Private Instance Methods

create_table(table_name) click to toggle source
# File lib/perobs/DynamoDB.rb, line 223
def create_table(table_name)
  begin
    @dynamodb.describe_table(:table_name => table_name)
    # The table exists already. No need to create it.
    return false
  rescue Aws::DynamoDB::Errors::ResourceNotFoundException
    @dynamodb.create_table(
      :table_name => table_name,
      :attribute_definitions => [
        {
          :attribute_name => :Id,
          :attribute_type => :S
        }
      ],
      :key_schema => [
        {
          :attribute_name => :Id,
          :key_type => :HASH
        }
      ],
      :provisioned_throughput => {
        :read_capacity_units => 1,
        :write_capacity_units => 1,
      }
    )

    @dynamodb.wait_until(:table_exists, table_name: table_name)
    # The table was successfully created.
    return true
  end
end
dynamo_delete_item(id) click to toggle source
# File lib/perobs/DynamoDB.rb, line 266
def dynamo_delete_item(id)
  @dynamodb.delete_item(:table_name => @table_name,
                        :key => { :Id => id })
end
dynamo_get_item(id) click to toggle source
# File lib/perobs/DynamoDB.rb, line 255
def dynamo_get_item(id)
  resp = @dynamodb.get_item(:table_name => @table_name,
                            :key => { :Id => id })
  resp[:item] ? resp[:item]['Value'] : nil
end
dynamo_is_marked?(id) click to toggle source
# File lib/perobs/DynamoDB.rb, line 279
def dynamo_is_marked?(id)
  resp = @dynamodb.get_item(:table_name => @table_name,
                           :key => { :Id => id })
  resp[:item] && resp[:item]['Mark']
end
dynamo_mark_item(id, set_mark = true) click to toggle source
# File lib/perobs/DynamoDB.rb, line 271
def dynamo_mark_item(id, set_mark = true)
  @dynamodb.update_item(:table_name => @table_name,
                        :key => { :Id => id },
                        :attribute_updates => {
                          :Mark => { :value => set_mark,
                                     :action => "PUT" }})
end
dynamo_put_item(id, value) click to toggle source
# File lib/perobs/DynamoDB.rb, line 261
def dynamo_put_item(id, value)
  @dynamodb.put_item(:table_name => @table_name,
                     :item => { :Id => id, :Value => value })
end
each_item() { |item| ... } click to toggle source
# File lib/perobs/DynamoDB.rb, line 285
def each_item
  start_key = nil
  loop do
    resp = @dynamodb.scan(:table_name => @table_name,
                          :exclusive_start_key => start_key)
    break if resp.count <= 0

    resp.items.each do |item|
      # Skip all internal items
      next if INTERNAL_ITEMS.include?(item['Id'])

      yield(item['Id'])
    end

    break unless resp.last_evaluated_key
    start_key = resp.last_evaluated_key['AttributeName']
  end
end