class Scruber::QueueAdapters::AbstractAdapter
Abstract Queue
Adapter
@abstract @author Ivan Goncharov
Public Class Methods
# File lib/scruber/queue_adapters/abstract_adapter.rb, line 204 def initialize(options={}) @options = options end
Public Instance Methods
Add page to queue @param url [String] URL of page @param options [Hash] Other options, see {Scruber::QueueAdapters::AbstractAdapter::Page}
@return [void]
# File lib/scruber/queue_adapters/abstract_adapter.rb, line 214 def add(url, options={}) raise NotImplementedError end
Count of downloaded pages Using to show downloading progress.
@return [Integer] count of downloaded pages
# File lib/scruber/queue_adapters/abstract_adapter.rb, line 267 def downloaded_count raise NotImplementedError end
Fetch downloaded and not processed pages for feching @param count [Integer] count of pages to fetch
@return [Scruber::QueueAdapters::AbstractAdapter::Page|Array<Scruber::QueueAdapters::AbstractAdapter::Page>] page of count = nil, or array of pages of count > 0
# File lib/scruber/queue_adapters/abstract_adapter.rb, line 249 def fetch_downloaded(count=nil) raise NotImplementedError end
Fetch error page @param count [Integer] count of pages to fetch
@return [Scruber::QueueAdapters::AbstractAdapter::Page|Array<Scruber::QueueAdapters::AbstractAdapter::Page>] page of count = nil, or array of pages of count > 0
# File lib/scruber/queue_adapters/abstract_adapter.rb, line 258 def fetch_error(count=nil) raise NotImplementedError end
Fetch pending page for fetching @param count [Integer] count of pages to fetch
@return [Scruber::QueueAdapters::AbstractAdapter::Page|Array<Scruber::QueueAdapters::AbstractAdapter::Page>] page of count = nil, or array of pages of count > 0
# File lib/scruber/queue_adapters/abstract_adapter.rb, line 240 def fetch_pending(count=nil) raise NotImplementedError end
Search page by id @param id [Object] id of page
@return [Page] page object
# File lib/scruber/queue_adapters/abstract_adapter.rb, line 223 def find(id) raise NotImplementedError end
Used by Core
. It checks for pages that are not downloaded or not parsed yet.
@return [Boolean] true if queue still has work for scraper
# File lib/scruber/queue_adapters/abstract_adapter.rb, line 286 def has_work? raise NotImplementedError end
Check if queue was initialized. Using for `seed` method. If queue was initialized, then no need to run seed block.
@return [Boolean] true if queue already was initialized
# File lib/scruber/queue_adapters/abstract_adapter.rb, line 277 def initialized? raise NotImplementedError end
Size of queue
@return [Integer] count of pages in queue
# File lib/scruber/queue_adapters/abstract_adapter.rb, line 231 def size raise NotImplementedError end