class UkParliament::QueueManager
Class to create/manage a queue for a set of items that will be scraped.
Constants
- QUEUE_ERROR
Unique identifier for the error queue.
- QUEUE_MAIN
Unique identifier for the main work queue.
Attributes
active_queue[R]
Instance data accessor(s).
error_queue[R]
Instance data accessor(s).
main_queue[R]
Instance data accessor(s).
Public Class Methods
new(name = 'commons')
click to toggle source
Set up queue states.
# File lib/uk_parliament/queue_manager.rb, line 17 def initialize(name = 'commons') config = configuration main_queue_file_name = File.join(config[:queue_file_path], "#{name}.queue") error_queue_file_name = File.join(config[:queue_file_path], "#{name}.error.queue") @main_queue = FileQueue.new(main_queue_file_name) @error_queue = FileQueue.new(error_queue_file_name) reset_main_queue set_active_queue end
Public Instance Methods
enqueue(members)
click to toggle source
Set up the queue, either with provided items, or from the error queue.
# File lib/uk_parliament/queue_manager.rb, line 53 def enqueue(members) if @active_queue == QUEUE_ERROR populate_from_error_queue else populate(members, 'id') end log.info("Populated queue with #{@main_queue.length} items...") end
error_queue_size()
click to toggle source
Return the current size of the error queue.
This is a bit of a work around FileQueue. github.com/pezra/filequeue/pull/4
# File lib/uk_parliament/queue_manager.rb, line 42 def error_queue_size size = 0 if File.exists?(@error_queue.file_name) size = @error_queue.length end size end
scrape_errors?()
click to toggle source
Identify if there were errors from the last scrape.
# File lib/uk_parliament/queue_manager.rb, line 30 def scrape_errors? if @active_queue == QUEUE_ERROR true else false end end
Private Instance Methods
populate(items, key)
click to toggle source
Populate the main queue.
# File lib/uk_parliament/queue_manager.rb, line 84 def populate(items, key) items.each { |member| @main_queue.push(member[key].to_s) } end
populate_from_error_queue()
click to toggle source
Populate the main queue with items from the error queue.
# File lib/uk_parliament/queue_manager.rb, line 91 def populate_from_error_queue log.info('Populating queue from error queue...') until @error_queue.empty? # Could prevent potentially a lot of disk IO by just overwriting the # file directly and clear() the error queue... id = @error_queue.pop @main_queue.push(id) end end
reset_main_queue()
click to toggle source
Empty the main queue for a house.
# File lib/uk_parliament/queue_manager.rb, line 66 def reset_main_queue if File.exists?(@main_queue.file_name) @main_queue.clear end end
set_active_queue()
click to toggle source
Identify the currently active queue, main or error.
# File lib/uk_parliament/queue_manager.rb, line 73 def set_active_queue @active_queue = QUEUE_MAIN if File.exists?(@error_queue.file_name) unless @error_queue.empty? @active_queue = QUEUE_ERROR end end end