class RocketJob::DirmonEntry

Public Class Methods

add_whitelist_path(path) click to toggle source

Add a path to the whitelist Raises: Errno::ENOENT: No such file or directory

# File lib/rocket_job/dirmon_entry.rb, line 137
def self.add_whitelist_path(path)
  # Confirms that path exists
  path = IOStreams.path(path).realpath.to_s
  whitelist_paths << path
  whitelist_paths.uniq!
  path
end
counts_by_state() click to toggle source

Returns [Hash<String:Integer>] of the number of dirmon entries in each state. Note: If there are no workers in that particular state then the hash will not have a value for it.

Example dirmon entries in every state:

RocketJob::DirmonEntry.counts_by_state
# => {
       :pending => 1,
       :enabled => 37,
       :failed => 1,
       :disabled => 3
     }

Example no dirmon entries:

RocketJob::Job.counts_by_state
# => {}
# File lib/rocket_job/dirmon_entry.rb, line 170
def self.counts_by_state
  counts = {}
  collection.aggregate([{"$group" => {_id: "$state", count: {"$sum" => 1}}}]).each do |result|
    counts[result["_id"].to_sym] = result["count"]
  end
  counts
end
delete_whitelist_path(path) click to toggle source

Deletes a path from the whitelist paths Raises: Errno::ENOENT: No such file or directory

# File lib/rocket_job/dirmon_entry.rb, line 147
def self.delete_whitelist_path(path)
  # Confirms that path exists
  path = IOStreams.path(path).realpath.to_s
  whitelist_paths.delete(path)
  whitelist_paths.uniq!
  path
end
get_whitelist_paths() click to toggle source

Security Settings

A whitelist of paths from which to process files. This prevents accidental or malicious ‘pattern`s from processing files from anywhere in the system that the user under which Dirmon is running can access.

All resolved ‘pattern`s must start with one of the whitelisted path, otherwise they will be rejected

Note:

  • If no whitelist paths have been added, then a whitelist check is not performed

  • Relative paths can be used, but are not considered safe since they can be manipulated

  • These paths should be assigned in an initializer and not editable via the Web UI to ensure that they are not tampered with

Default: [] ==> Do not enforce whitelists

Returns [Array<String>] a copy of the whitelisted paths

# File lib/rocket_job/dirmon_entry.rb, line 131
def self.get_whitelist_paths
  whitelist_paths.dup
end

Public Instance Methods

each() { |path| ... } click to toggle source

Yields [IOStreams::Path] for each file found that matches the current pattern.

# File lib/rocket_job/dirmon_entry.rb, line 179
def each
  SemanticLogger.named_tagged(dirmon_entry: id.to_s) do
    # Case insensitive filename matching
    IOStreams.each_child(pattern) do |path|
      path = path.realpath
      # Skip archive directories
      next if path.to_s.include?(archive_directory || self.class.default_archive_directory)

      # Security check?
      if whitelist_paths.size.positive? && whitelist_paths.none? { |whitepath| path.to_s.start_with?(whitepath) }
        logger.warn "Skipping file: #{path} since it is not in any of the whitelisted paths: #{whitelist_paths.join(', ')}"
        next
      end

      # File must be writable so it can be removed after processing
      if path.respond_to?(:writable?) && !path.writable?
        logger.warn "Skipping file: #{file_name} since it is not writable by the current user. Must be able to delete/move the file after queueing the job"
        next
      end
      yield(path)
    end
  end
end
job_class() click to toggle source

Returns the Job to be created.

# File lib/rocket_job/dirmon_entry.rb, line 219
def job_class
  return if job_class_name.nil?

  job_class_name.constantize
rescue NameError
  nil
end
later(iopath) click to toggle source

Archives the file, then kicks off a file upload job to upload the archived file.

# File lib/rocket_job/dirmon_entry.rb, line 228
def later(iopath)
  job_id       = BSON::ObjectId.new
  archive_path = archive_iopath(iopath).join("#{job_id}_#{iopath.basename}")
  iopath.move_to(archive_path)

  job = RocketJob::Jobs::UploadFileJob.create!(
    job_class_name:     job_class_name,
    properties:         properties,
    description:        "#{name}: #{iopath.basename}",
    upload_file_name:   archive_path,
    original_file_name: iopath.to_s,
    job_id:             job_id
  )

  logger.info(
    message: "Created RocketJob::Jobs::UploadFileJob",
    payload: {
      dirmon_entry_name:  name,
      upload_file_name:   archive_path,
      original_file_name: iopath.to_s,
      job_class_name:     job_class_name,
      job_id:             job_id.to_s,
      upload_job_id:      job.id.to_s
    }
  )
  job
end
set_exception(worker_name, exc_or_message) click to toggle source

Set exception information for this DirmonEntry and fail it

# File lib/rocket_job/dirmon_entry.rb, line 204
def set_exception(worker_name, exc_or_message)
  if exc_or_message.is_a?(Exception)
    self.exception        = JobException.from_exception(exc_or_message)
    exception.worker_name = worker_name
  else
    build_exception(
      class_name:  "RocketJob::DirmonEntryException",
      message:     exc_or_message,
      backtrace:   [],
      worker_name: worker_name
    )
  end
end

Private Instance Methods

archive_iopath(iopath) click to toggle source

Returns [Pathname] to the archive directory, and creates it if it does not exist.

If ‘archive_directory` is a relative path, it is appended to the `file_pathname`. If `archive_directory` is an absolute path, it is returned as-is.

# File lib/rocket_job/dirmon_entry.rb, line 271
def archive_iopath(iopath)
  path = IOStreams.path(archive_directory)
  path.relative? ? iopath.directory.join(archive_directory) : path
end
job_has_properties() click to toggle source

Does the job have all the supplied properties

# File lib/rocket_job/dirmon_entry.rb, line 285
def job_has_properties
  klass = job_class
  return unless klass

  properties.each_pair do |k, _v|
    next if klass.public_method_defined?("#{k}=".to_sym)

    if %i[output_categories input_categories].include?(k)
      category_class = k == :input_categories ? RocketJob::Category::Input : RocketJob::Category::Output
      properties[k].each do |category|
        category.each_pair do |key, _value|
          next if category_class.public_method_defined?("#{key}=".to_sym)

          errors.add(
            :properties,
            "Unknown Property in #{k}: Attempted to set a value for #{key}.#{k} which is not allowed on the job #{job_class_name}"
          )
        end
      end
      next
    end

    errors.add(
      :properties,
      "Unknown Property: Attempted to set a value for #{k.inspect} which is not allowed on the job #{job_class_name}"
    )
  end
end
job_is_a_rocket_job() click to toggle source

Validates job_class is a Rocket Job

# File lib/rocket_job/dirmon_entry.rb, line 277
def job_is_a_rocket_job
  klass = job_class
  return if job_class_name.nil? || klass&.ancestors&.include?(RocketJob::Job)

  errors.add(:job_class_name, "Job #{job_class_name} must be defined and inherit from RocketJob::Job")
end
strip_whitespace() click to toggle source

strip whitespaces from all variables that reference paths or patterns

# File lib/rocket_job/dirmon_entry.rb, line 259
def strip_whitespace
  self.pattern           = pattern.strip unless pattern.nil?
  self.archive_directory = archive_directory.strip unless archive_directory.nil?
end