class Krikri::AsyncUriGetter

Helper class for fetching multiple URLs concurrently.

@example to fetch 5 URLs in 5 threads

urls = ['http://example.com/one',
        'http://example.com/two',
        'http://example.com/three',
        'http://example.com/four',
        'http://example.com/five']
       .map { |url| URI.parse(url) }

getter = Krikri::AsyncUriGetter.new

requests = urls.map do |url|
  getter.add_request(uri: url, opts: { follow_redirects: true })
end

At this point, 5 threads are launched to fetch the list of URLs. We can wait for them all to finish if we want to make sure we don't continue until all threads have terminated: `requests.map(&:join)`

Or simply access the responses and have our current thread block until they're available:

requests.each do |request|
  request.with_response do |response|
    if response.status == 200
      puts "Response body: #{response.body}"
    else
      puts "Got return status: #{response.status}"
    end
  end
end

Constants

MAX_REDIRECTS
Request

Public Class Methods

new(opts: {}) click to toggle source

Create a new asynchronous URL fetcher.

@param opts [Hash] a hash of the supported options, which are: @option opts [Boolean] :follow_redirects Whether to follow HTTP 3xx

redirects.

@option opts [Integer] :max_redirects Number of redirects to follow before

giving up. (default: 10)

@option opts [Boolean] :inline_exceptions If true, pass exceptions as a

5xx response with the exception string in the body. (default: false)
# File lib/krikri/async_uri_getter.rb, line 56
def initialize(opts: {})
  @default_opts = { max_redirects: MAX_REDIRECTS }.merge(opts)
end

Public Instance Methods

add_request(uri: nil, headers: {}, opts: {}) click to toggle source

Run a request (in a new thread) and return a promise-like object for the response.

@param uri [URI] the URI to be fetched @param headers [Hash<String, String>] HTTP headers to include with the

request

@param opts [Hash] options to override the ones provided when

AsyncUriGetter was initialized. All supported options from `#initialize`
are available here as well.
# File lib/krikri/async_uri_getter.rb, line 70
def add_request(uri: nil, headers: {}, opts: {})
  fail ArgumentError, "uri must be a URI; got: #{uri}" unless uri.is_a?(URI)
  Request.new(uri, headers, @default_opts.merge(opts))
end