class ProxyFetcher::Configuration

ProxyFetcher configuration. Stores all the options for dealing with HTTP requests, adapters, custom classes.

Constants

DEFAULT_ADAPTER

HTML parser adapter name.

Default is Nokogiri, but can be changed in ProxyFetcher.config.

DEFAULT_USER_AGENT

User-Agent string that will be used by the ProxyFetcher HTTP client (to send requests via proxy) and to fetch proxy lists from the sources.

Default is Google Chrome 60, but can be changed in ProxyFetcher.config.

Attributes

adapter[R]

@!attribute [r] adapter

@return [#to_s] HTML parser adapter
client_timeout[RW]

@!attribute client_timeout

@return [Integer]
  HTTP request timeout (connect / open) for [ProxyFetcher::Client]
http_client[R]

@!attribute [r] http_client

@return [Object] HTTP client class
logger[RW]

@!attribute [r] logger

@return [Logger] Logger object
pool_size[RW]

@!attribute pool_size

@return [Integer] proxy validator pool size (max number of threads)
provider[R]

@!attribute [r] providers

@return [Array<String, Symbol>] proxy providers list to be used
provider_proxies_load_timeout[RW]

@!attribute provider_proxies_load_timeout

@return [Integer]
  HTTP request timeout (connect / open) for loading
  of proxies list by provider
providers[R]

@!attribute [r] providers

@return [Array<String, Symbol>] proxy providers list to be used
proxy_validation_timeout[RW]

@!attribute proxy_validation_timeout

@return [Integer]
  HTTP request timeout (connect / open) for proxy
  validation with [ProxyFetcher::ProxyValidator]
proxy_validator[R]

@!attribute [r] proxy_validator

@return [Object] proxy validator class
timeout[RW]

@!attribute client_timeout

@return [Integer]
  HTTP request timeout (connect / open) for [ProxyFetcher::Client]
timeout=[RW]

@!attribute client_timeout

@return [Integer]
  HTTP request timeout (connect / open) for [ProxyFetcher::Client]
user_agent[RW]

@!attribute user_agent

@return [String] User-Agent string

Public Class Methods

new() click to toggle source

Initialize ProxyFetcher configuration with default options.

@return [ProxyFetcher::Configuration]

ProxyFetcher gem configuration object
# File lib/proxy_fetcher/configuration.rb, line 111
def initialize
  reset!
end
providers_registry() click to toggle source

Registry for handling proxy providers.

@return [ProxyFetcher::ProvidersRegistry]

providers registry
# File lib/proxy_fetcher/configuration.rb, line 79
def providers_registry
  @providers_registry ||= ProvidersRegistry.new
end
register_provider(name, klass) click to toggle source

Register new proxy provider. Requires provider name and class that will process proxy list.

@param name [String, Symbol]

name of the provider

@param klass [Class]

Class that will fetch and process proxy list
# File lib/proxy_fetcher/configuration.rb, line 92
def register_provider(name, klass)
  providers_registry.register(name, klass)
end
registered_providers() click to toggle source

Returns registered providers names.

@return [Array<String, Symbol>]

registered providers names
# File lib/proxy_fetcher/configuration.rb, line 101
def registered_providers
  providers_registry.providers.keys
end

Public Instance Methods

adapter=(value) click to toggle source
# File lib/proxy_fetcher/configuration.rb, line 130
def adapter=(value)
  remove_instance_variable(:@adapter_class) if defined?(@adapter_class)
  @adapter = value
end
adapter_class() click to toggle source
# File lib/proxy_fetcher/configuration.rb, line 135
def adapter_class
  self.class.instance_variable_get(:@__adapter_lock__).synchronize do
    return @adapter_class if defined?(@adapter_class)

    @adapter_class = ProxyFetcher::Document::Adapters.lookup(adapter)
    @adapter_class.setup!
    @adapter_class
  end
end
http_client=(klass) click to toggle source

Setups HTTP client class that will be used to fetch proxy lists. Validates class for the required methods to be defined.

@param klass [Class]

HTTP client class
# File lib/proxy_fetcher/configuration.rb, line 163
def http_client=(klass)
  @http_client = setup_custom_class(klass, required_methods: :fetch)
end
provider=(value)
Alias for: providers=
providers=(value) click to toggle source

Setups collection of providers that will be used to fetch proxies.

@param value [String, Symbol, Array<String, Symbol>]

provider names
# File lib/proxy_fetcher/configuration.rb, line 150
def providers=(value)
  @providers = Array(value)
end
Also aliased as: provider=
proxy_validator=(klass) click to toggle source

Setups class that will be used to validate proxy lists. Validates class for the required methods to be defined.

@param klass [Class]

Proxy validator class
# File lib/proxy_fetcher/configuration.rb, line 173
def proxy_validator=(klass)
  @proxy_validator = setup_custom_class(klass, required_methods: :connectable?)
end
reset!() click to toggle source

Sets default configuration options

# File lib/proxy_fetcher/configuration.rb, line 116
def reset!
  @logger = Logger.new($stdout)
  @user_agent = DEFAULT_USER_AGENT
  @pool_size = 10
  @client_timeout = 3
  @provider_proxies_load_timeout = 30
  @proxy_validation_timeout = 3

  @http_client = HTTPClient
  @proxy_validator = ProxyValidator

  self.providers = self.class.registered_providers
end

Private Instance Methods

setup_custom_class(klass, required_methods: []) click to toggle source

Checks if custom class has some required class methods

# File lib/proxy_fetcher/configuration.rb, line 180
def setup_custom_class(klass, required_methods: [])
  unless klass.respond_to?(*required_methods)
    raise ProxyFetcher::Exceptions::WrongCustomClass.new(klass, required_methods)
  end

  klass
end