class Arachni::Support::Database::Queue

Flat-file Queue implementation

Behaves pretty much like a Ruby Queue however it transparently serializes and saves its entries to the file-system under the OS’s temp directory after a specified {#max_buffer_size} (for in-memory entries) has been exceeded.

It’s pretty useful when you want to reduce memory footprint without having to refactor any code since it behaves just like a Ruby Queue implementation.

@author Tasos “Zapotek” Laskos <tasos.laskos@arachni-scanner.com>

Constants

DEFAULT_MAX_BUFFER_SIZE

Default {#max_buffer_size}.

Attributes

buffer[R]

@return [Array<Object>]

Objects stored in the memory buffer.
disk[R]

@return [Array<String>]

Paths to files stored to disk.
max_buffer_size[RW]

@return [Integer]

How many entries to keep in memory before starting to off-load to disk.

Public Class Methods

new( *args ) click to toggle source

@see Arachni::Database::Base#initialize

Calls superclass method Arachni::Support::Database::Base::new
# File lib/arachni/support/database/queue.rb, line 41
def initialize( *args )
    super( *args )
    @disk    = []
    @buffer  = []
    @waiting = []
    @mutex   = Mutex.new
end

Public Instance Methods

<<( obj ) click to toggle source

@param [Object] obj

Object to add to the queue.
# File lib/arachni/support/database/queue.rb, line 59
def <<( obj )
    synchronize do
        if @buffer.size < max_buffer_size
            @buffer << obj
        else
            @disk << dump( obj )
        end

        begin
            t = @waiting.shift
            t.wakeup if t
        rescue ThreadError
            retry
        end
    end
end
Also aliased as: push, enq
buffer_size() click to toggle source
# File lib/arachni/support/database/queue.rb, line 108
def buffer_size
    @buffer.size
end
clear() click to toggle source

Removes all objects from the queue.

# File lib/arachni/support/database/queue.rb, line 125
def clear
    synchronize do
        @buffer.clear

        while !@disk.empty?
            path = @disk.pop
            next if !path
            delete_file path
        end
    end
end
deq( non_block = false )
Alias for: pop
disk_size() click to toggle source
# File lib/arachni/support/database/queue.rb, line 112
def disk_size
    @disk.size
end
empty?() click to toggle source

@return [Bool]

`true` if the queue if empty, `false` otherwise.
# File lib/arachni/support/database/queue.rb, line 118
def empty?
    synchronize do
        internal_empty?
    end
end
enq( obj )
Alias for: <<
free_buffer_size() click to toggle source
# File lib/arachni/support/database/queue.rb, line 104
def free_buffer_size
    max_buffer_size - buffer_size
end
length()
Alias for: size
num_waiting() click to toggle source
# File lib/arachni/support/database/queue.rb, line 137
def num_waiting
    @waiting.size
end
pop( non_block = false ) click to toggle source

@return [Object]

Removes an object from the queue and returns it.
# File lib/arachni/support/database/queue.rb, line 80
def pop( non_block = false )
    synchronize do
        loop do
            if internal_empty?
                raise ThreadError, 'queue empty' if non_block
                @waiting.push Thread.current
                @mutex.sleep
            else
                return @buffer.shift if !@buffer.empty?
                return load_and_delete_file( @disk.shift )
            end
        end
    end
end
Also aliased as: deq, shift
push( obj )
Alias for: <<
shift( non_block = false )
Alias for: pop
size() click to toggle source

@return [Integer]

Size of the queue, the number of objects it currently holds.
# File lib/arachni/support/database/queue.rb, line 99
def size
    buffer_size + disk_size
end
Also aliased as: length

Private Instance Methods

internal_empty?() click to toggle source
# File lib/arachni/support/database/queue.rb, line 143
def internal_empty?
    @buffer.empty? && @disk.empty?
end
synchronize( &block ) click to toggle source
# File lib/arachni/support/database/queue.rb, line 147
def synchronize( &block )
    @mutex.synchronize( &block )
end