class UkParliament::QueueManager

Class to create/manage a queue for a set of items that will be scraped.

Constants

QUEUE_ERROR

Unique identifier for the error queue.

QUEUE_MAIN

Unique identifier for the main work queue.

Attributes

active_queue[R]

Instance data accessor(s).

error_queue[R]

Instance data accessor(s).

main_queue[R]

Instance data accessor(s).

Public Class Methods

new(name = 'commons') click to toggle source

Set up queue states.

# File lib/uk_parliament/queue_manager.rb, line 17
def initialize(name = 'commons')
  config = configuration
  main_queue_file_name = File.join(config[:queue_file_path], "#{name}.queue")
  error_queue_file_name = File.join(config[:queue_file_path], "#{name}.error.queue")

  @main_queue = FileQueue.new(main_queue_file_name)
  @error_queue = FileQueue.new(error_queue_file_name)

  reset_main_queue
  set_active_queue
end

Public Instance Methods

enqueue(members) click to toggle source

Set up the queue, either with provided items, or from the error queue.

# File lib/uk_parliament/queue_manager.rb, line 53
def enqueue(members)
  if @active_queue == QUEUE_ERROR
    populate_from_error_queue
  else
    populate(members, 'id')
  end

  log.info("Populated queue with #{@main_queue.length} items...")
end
error_queue_size() click to toggle source

Return the current size of the error queue.

This is a bit of a work around FileQueue. github.com/pezra/filequeue/pull/4

# File lib/uk_parliament/queue_manager.rb, line 42
def error_queue_size
  size = 0

  if File.exists?(@error_queue.file_name)
    size = @error_queue.length
  end

  size
end
scrape_errors?() click to toggle source

Identify if there were errors from the last scrape.

# File lib/uk_parliament/queue_manager.rb, line 30
def scrape_errors?
  if @active_queue == QUEUE_ERROR
    true
  else
    false
  end
end

Private Instance Methods

populate(items, key) click to toggle source

Populate the main queue.

# File lib/uk_parliament/queue_manager.rb, line 84
def populate(items, key)
  items.each { |member|
    @main_queue.push(member[key].to_s)
  }
end
populate_from_error_queue() click to toggle source

Populate the main queue with items from the error queue.

# File lib/uk_parliament/queue_manager.rb, line 91
def populate_from_error_queue
  log.info('Populating queue from error queue...')

  until @error_queue.empty?
    # Could prevent potentially a lot of disk IO by just overwriting the
    # file directly and clear() the error queue...
    id = @error_queue.pop
    @main_queue.push(id)
  end
end
reset_main_queue() click to toggle source

Empty the main queue for a house.

# File lib/uk_parliament/queue_manager.rb, line 66
def reset_main_queue
  if File.exists?(@main_queue.file_name)
    @main_queue.clear
  end
end
set_active_queue() click to toggle source

Identify the currently active queue, main or error.

# File lib/uk_parliament/queue_manager.rb, line 73
def set_active_queue
  @active_queue = QUEUE_MAIN

  if File.exists?(@error_queue.file_name)
    unless @error_queue.empty?
      @active_queue = QUEUE_ERROR
    end
  end
end