class PEROBS::FlatFileDB

The FlatFileDB is a storage backend that uses a single flat file to store the value blobs.

Constants

VERSION

This version number increases whenever the on-disk format changes in a way that requires conversion actions after an update.

Attributes

max_blob_size[R]

Public Class Methods

delete_db(db_name) click to toggle source
# File lib/perobs/FlatFileDB.rb, line 94
def FlatFileDB::delete_db(db_name)
  close
  FileUtils.rm_rf(db_name)
end
new(db_name, options = {}) click to toggle source

Create a new FlatFileDB object. @param db_name [String] name of the DB directory @param options [Hash] options to customize the behavior. Currently only

the following options are supported:
:serializer    : Can be :marshal, :json, :yaml
:progressmeter : Reference to a ProgressMeter object
:log           : IO that should be used for logging
:log_level     : Minimum Logger level to log
Calls superclass method PEROBS::DataBase::new
# File lib/perobs/FlatFileDB.rb, line 57
def initialize(db_name, options = {})
  super(options)

  @db_dir = db_name
  # Create the database directory if it doesn't exist yet.
  ensure_dir_exists(@db_dir)
  PEROBS.log.level = options[:log_level] if options[:log_level]
  PEROBS.log.open(options[:log] || File.join(@db_dir, 'log'))
  check_version_and_upgrade

  # Read the existing DB config.
  @config = get_hash('config')
  check_option('serializer')

  put_hash('config', @config)
end

Public Instance Methods

check(id, repair) click to toggle source

Check if the stored object is syntactically correct. @param id [Integer] Object ID @param repair [TrueClass/FalseClass] True if an repair attempt should be

made.

@return [TrueClass/FalseClass] True if the object is OK, otherwise

false.
# File lib/perobs/FlatFileDB.rb, line 200
def check(id, repair)
  begin
    return get_object(id) != nil
  rescue PEROBS::FatalError => e
    PEROBS.log.error "Cannot read object with ID #{id}: #{e.message}"
    if repair
      begin
        PEROBS.log.error "Discarding broken object with ID #{id}"
        @flat_file.delete_obj_by_id(id)
      rescue PEROBS::FatalError
      end
    end
  end

  return false
end
check_db(repair = false) click to toggle source

Basic consistency check. @param repair [TrueClass/FalseClass] True if found errors should be

repaired.

@return number of errors found

# File lib/perobs/FlatFileDB.rb, line 186
def check_db(repair = false)
  if repair
    @flat_file.repair
  else
    @flat_file.check
  end
end
clear_marks() click to toggle source

This method must be called to initiate the marking process.

# File lib/perobs/FlatFileDB.rb, line 157
def clear_marks
  @flat_file.clear_all_marks
end
close() click to toggle source

Close the FlatFileDB.

# File lib/perobs/FlatFileDB.rb, line 82
def close
  @flat_file.close
  @flat_file = nil
  PEROBS.log.info "FlatFile '#{@db_dir}' closed"
end
delete_database() click to toggle source

Delete the entire database. The database is no longer usable after this method was called.

# File lib/perobs/FlatFileDB.rb, line 90
def delete_database
  FileUtils.rm_rf(@db_dir)
end
delete_unmarked_objects(&block) click to toggle source

Permanently delete all objects that have not been marked. Those are orphaned and are no longer referenced by any actively used object. @return [Integer] Number of the removed objects from the DB.

# File lib/perobs/FlatFileDB.rb, line 164
def delete_unmarked_objects(&block)
  @flat_file.delete_unmarked_objects(&block)
end
get_hash(name) click to toggle source

Load the Hash with the given name. @param name [String] Name of the hash. @return [Hash] A Hash that maps String objects to strings or numbers.

# File lib/perobs/FlatFileDB.rb, line 121
def get_hash(name)
  file_name = File.join(@db_dir, name + '.json')
  return ::Hash.new unless File.exist?(file_name)

  begin
    json = File.read(file_name)
  rescue => e
    PEROBS.log.fatal "Cannot read hash file '#{file_name}': #{e.message}"
  end
  JSON.parse(json, :create_additions => true)
end
get_object(id) click to toggle source

Load the given object from the filesystem. @param id [Integer] object ID @return [Hash] Object as defined by PEROBS::ObjectBase or nil if ID does

not exist
# File lib/perobs/FlatFileDB.rb, line 143
def get_object(id)
  if (raw_obj = @flat_file.read_obj_by_id(id))
    return deserialize(raw_obj)
  else
    nil
  end
end
include?(id) click to toggle source

Return true if the object with given ID exists @param id [Integer]

# File lib/perobs/FlatFileDB.rb, line 101
def include?(id)
  !@flat_file.find_obj_addr_by_id(id).nil?
end
is_marked?(id, ignore_errors = false) click to toggle source

Check if the object is marked. @param id [Integer] ID of the object to check @param ignore_errors [Boolean] If set to true no errors will be raised

for non-existing objects.
# File lib/perobs/FlatFileDB.rb, line 178
def is_marked?(id, ignore_errors = false)
  @flat_file.is_marked_by_id?(id)
end
item_counter() click to toggle source

@return [Integer] Number of objects stored in the DB.

# File lib/perobs/FlatFileDB.rb, line 152
def item_counter
  @flat_file.item_counter
end
mark(id) click to toggle source

Mark an object. @param id [Integer] ID of the object to mark

# File lib/perobs/FlatFileDB.rb, line 170
def mark(id)
  @flat_file.mark_obj_by_id(id)
end
open() click to toggle source

Open the FlatFileDB for transactions.

# File lib/perobs/FlatFileDB.rb, line 75
def open
  @flat_file = FlatFile.new(@db_dir, @progressmeter)
  @flat_file.open
  PEROBS.log.info "FlatFile '#{@db_dir}' opened"
end
put_hash(name, hash) click to toggle source

Store a simple Hash as a JSON encoded file into the DB directory. @param name [String] Name of the hash. Will be used as file name. @param hash [Hash] A Hash that maps String objects to strings or numbers.

# File lib/perobs/FlatFileDB.rb, line 109
def put_hash(name, hash)
  file_name = File.join(@db_dir, name + '.json')
  begin
    RobustFile.write(file_name, hash.to_json)
  rescue IOError => e
    PEROBS.log.fatal "Cannot write hash file '#{file_name}': #{e.message}"
  end
end
put_object(obj, id) click to toggle source

Store the given object into the cluster files. @param obj [Hash] Object as defined by PEROBS::ObjectBase

# File lib/perobs/FlatFileDB.rb, line 135
def put_object(obj, id)
  @flat_file.write_obj_by_id(id, serialize(obj))
end
put_raw_object(raw, id) click to toggle source

Store the given serialized object into the cluster files. This method is for internal use only! @param raw [String] Serialized Object as defined by PEROBS::ObjectBase @param id [Integer] Object ID

# File lib/perobs/FlatFileDB.rb, line 221
def put_raw_object(raw, id)
  @flat_file.write_obj_by_id(id, raw)
end

Private Instance Methods

check_version_and_upgrade() click to toggle source
# File lib/perobs/FlatFileDB.rb, line 227
def check_version_and_upgrade
  version_file = File.join(@db_dir, 'version')
  version = 1

  if File.exist?(version_file)
    begin
      version = File.read(version_file).to_i
    rescue => e
      PEROBS.log.fatal "Cannot read version number file " +
                       "'#{version_file}': " + e.message
    end
  else
    # The DB is brand new.
    version = VERSION
    write_version_file(version_file)
  end

  if version > VERSION
    PEROBS.log.fatal "Cannot downgrade the FlatFile database from " +
                     "version #{version} to version #{VERSION}"
  end
  if version < 3
    PEROBS.log.fatal "The upgrade of this version of the PEROBS database " +
      "is not supported by this version of PEROBS. Please try an earlier " +
      "version of PEROBS to upgrade the database before using this version."
  end

  # Version upgrades must be done one version number at a time. If the
  # existing DB is multiple versions older than what the current PEROBS
  # version expects than multiple upgrade runs will be needed.
  while version < VERSION
    if version == 3
      PEROBS.log.warn "Updating FlatFileDB #{@db_dir} from version 3 to " +
        "version 4 ..."
      # Version 4 adds checksums for blob file headers. We have to convert
      # the blob file to include the checksums.
      FlatFile.insert_header_checksums(@db_dir)
      open
      @flat_file.regenerate_index_and_spaces
      close
    end

    # After a successful upgrade change the version number in the DB as
    # well.
    write_version_file(version_file)
    PEROBS.log.warn "Update of FlatFileDB '#{@db_dir}' from version " +
      "#{version} to version #{version + 1} completed"

    # Update version variable to new version.
    version += 1
  end
end
write_version_file(version_file) click to toggle source
# File lib/perobs/FlatFileDB.rb, line 280
def write_version_file(version_file)

  begin
    RobustFile.write(version_file, VERSION)
  rescue IOError => e
    PEROBS.log.fatal "Cannot write version number file " +
                     "'#{version_file}': " + e.message
  end
end