class RocketJob::DirmonEntry
Public Class Methods
Add a path to the whitelist Raises: Errno::ENOENT: No such file or directory
# File lib/rocket_job/dirmon_entry.rb, line 137 def self.add_whitelist_path(path) # Confirms that path exists path = IOStreams.path(path).realpath.to_s whitelist_paths << path whitelist_paths.uniq! path end
Returns [Hash<String:Integer>] of the number of dirmon entries in each state. Note: If there are no workers in that particular state then the hash will not have a value for it.
Example dirmon entries in every state:
RocketJob::DirmonEntry.counts_by_state # => { :pending => 1, :enabled => 37, :failed => 1, :disabled => 3 }
Example no dirmon entries:
RocketJob::Job.counts_by_state # => {}
# File lib/rocket_job/dirmon_entry.rb, line 170 def self.counts_by_state counts = {} collection.aggregate([{"$group" => {_id: "$state", count: {"$sum" => 1}}}]).each do |result| counts[result["_id"].to_sym] = result["count"] end counts end
Deletes a path from the whitelist paths Raises: Errno::ENOENT: No such file or directory
# File lib/rocket_job/dirmon_entry.rb, line 147 def self.delete_whitelist_path(path) # Confirms that path exists path = IOStreams.path(path).realpath.to_s whitelist_paths.delete(path) whitelist_paths.uniq! path end
Security Settings
A whitelist of paths from which to process files. This prevents accidental or malicious ‘pattern`s from processing files from anywhere in the system that the user under which Dirmon is running can access.
All resolved ‘pattern`s must start with one of the whitelisted path, otherwise they will be rejected
Note:
-
If no whitelist paths have been added, then a whitelist check is not performed
-
Relative paths can be used, but are not considered safe since they can be manipulated
-
These paths should be assigned in an initializer and not editable via the Web UI to ensure that they are not tampered with
Default: [] ==> Do not enforce whitelists
Returns [Array<String>] a copy of the whitelisted paths
# File lib/rocket_job/dirmon_entry.rb, line 131 def self.get_whitelist_paths whitelist_paths.dup end
Public Instance Methods
Yields [IOStreams::Path] for each file found that matches the current pattern.
# File lib/rocket_job/dirmon_entry.rb, line 179 def each SemanticLogger.named_tagged(dirmon_entry: id.to_s) do # Case insensitive filename matching IOStreams.each_child(pattern) do |path| path = path.realpath # Skip archive directories next if path.to_s.include?(archive_directory || self.class.default_archive_directory) # Security check? if whitelist_paths.size.positive? && whitelist_paths.none? { |whitepath| path.to_s.start_with?(whitepath) } logger.warn "Skipping file: #{path} since it is not in any of the whitelisted paths: #{whitelist_paths.join(', ')}" next end # File must be writable so it can be removed after processing if path.respond_to?(:writable?) && !path.writable? logger.warn "Skipping file: #{file_name} since it is not writable by the current user. Must be able to delete/move the file after queueing the job" next end yield(path) end end end
Returns the Job
to be created.
# File lib/rocket_job/dirmon_entry.rb, line 219 def job_class return if job_class_name.nil? job_class_name.constantize rescue NameError nil end
Archives the file, then kicks off a file upload job to upload the archived file.
# File lib/rocket_job/dirmon_entry.rb, line 228 def later(iopath) job_id = BSON::ObjectId.new archive_path = archive_iopath(iopath).join("#{job_id}_#{iopath.basename}") iopath.move_to(archive_path) job = RocketJob::Jobs::UploadFileJob.create!( job_class_name: job_class_name, properties: properties, description: "#{name}: #{iopath.basename}", upload_file_name: archive_path, original_file_name: iopath.to_s, job_id: job_id ) logger.info( message: "Created RocketJob::Jobs::UploadFileJob", payload: { dirmon_entry_name: name, upload_file_name: archive_path, original_file_name: iopath.to_s, job_class_name: job_class_name, job_id: job_id.to_s, upload_job_id: job.id.to_s } ) job end
Set exception information for this DirmonEntry
and fail it
# File lib/rocket_job/dirmon_entry.rb, line 204 def set_exception(worker_name, exc_or_message) if exc_or_message.is_a?(Exception) self.exception = JobException.from_exception(exc_or_message) exception.worker_name = worker_name else build_exception( class_name: "RocketJob::DirmonEntryException", message: exc_or_message, backtrace: [], worker_name: worker_name ) end end
Private Instance Methods
Returns [Pathname] to the archive directory, and creates it if it does not exist.
If ‘archive_directory` is a relative path, it is appended to the `file_pathname`. If `archive_directory` is an absolute path, it is returned as-is.
# File lib/rocket_job/dirmon_entry.rb, line 271 def archive_iopath(iopath) path = IOStreams.path(archive_directory) path.relative? ? iopath.directory.join(archive_directory) : path end
Does the job have all the supplied properties
# File lib/rocket_job/dirmon_entry.rb, line 285 def job_has_properties klass = job_class return unless klass properties.each_pair do |k, _v| next if klass.public_method_defined?("#{k}=".to_sym) if %i[output_categories input_categories].include?(k) category_class = k == :input_categories ? RocketJob::Category::Input : RocketJob::Category::Output properties[k].each do |category| category.each_pair do |key, _value| next if category_class.public_method_defined?("#{key}=".to_sym) errors.add( :properties, "Unknown Property in #{k}: Attempted to set a value for #{key}.#{k} which is not allowed on the job #{job_class_name}" ) end end next end errors.add( :properties, "Unknown Property: Attempted to set a value for #{k.inspect} which is not allowed on the job #{job_class_name}" ) end end
Validates job_class
is a Rocket Job
# File lib/rocket_job/dirmon_entry.rb, line 277 def job_is_a_rocket_job klass = job_class return if job_class_name.nil? || klass&.ancestors&.include?(RocketJob::Job) errors.add(:job_class_name, "Job #{job_class_name} must be defined and inherit from RocketJob::Job") end
strip whitespaces from all variables that reference paths or patterns
# File lib/rocket_job/dirmon_entry.rb, line 259 def strip_whitespace self.pattern = pattern.strip unless pattern.nil? self.archive_directory = archive_directory.strip unless archive_directory.nil? end