class Roundhouse::Scheduled::Poller

The Poller checks Redis every N seconds for jobs in the retry or scheduled set have passed their timestamp and should be enqueued. If so, it just pops the job back onto its original queue so the workers can pick it up like any other job.

Constants

INITIAL_WAIT

Public Class Methods

new() click to toggle source
# File lib/roundhouse/scheduled.rb, line 46
def initialize
  @enq = (Roundhouse.options[:scheduled_enq] || Roundhouse::Scheduled::Enq).new
end

Public Instance Methods

poll(first_time=false) click to toggle source
# File lib/roundhouse/scheduled.rb, line 50
def poll(first_time=false)
  watchdog('scheduling poller thread died!') do
    initial_wait if first_time

    begin
      @enq.enqueue_jobs
    rescue => ex
      # Most likely a problem with redis networking.
      # Punt and try again at the next interval
      logger.error ex.message
      logger.error ex.backtrace.first
    end

    after(random_poll_interval) { poll }
  end
end

Private Instance Methods

initial_wait() click to toggle source
# File lib/roundhouse/scheduled.rb, line 100
def initial_wait
  begin
    # Have all processes sleep between 5-15 seconds.  10 seconds
    # to give time for the heartbeat to register (if the poll interval is going to be calculated by the number
    # of workers), and 5 random seconds to ensure they don't all hit Redis at the same time.
    sleep(INITIAL_WAIT) unless Roundhouse.options[:poll_interval_average]
    sleep(5 * rand)
  rescue Celluloid::TaskTerminated
    # Hit Ctrl-C when Roundhouse is finished booting and we have a chance
    # to get here.
  end
end
poll_interval_average() click to toggle source

We do our best to tune the poll interval to the size of the active Roundhouse cluster. If you have 30 processes and poll every 15 seconds, that means one Roundhouse is checking Redis every 0.5 seconds - way too often for most people and really bad if the retry or scheduled sets are large.

Instead try to avoid polling more than once every 15 seconds. If you have 30 Roundhouse processes, we’ll poll every 30 * 15 or 450 seconds. To keep things statistically random, we’ll sleep a random amount between 225 and 675 seconds for each poll or 450 seconds on average. Otherwise restarting all your Roundhouse processes at the same time will lead to them all polling at the same time: the thundering herd problem.

We only do this if poll_interval is unset (the default).

# File lib/roundhouse/scheduled.rb, line 87
def poll_interval_average
  Roundhouse.options[:poll_interval_average] ||= scaled_poll_interval
end
random_poll_interval() click to toggle source

Calculates a random interval that is ±50% the desired average.

# File lib/roundhouse/scheduled.rb, line 70
def random_poll_interval
  poll_interval_average * rand + poll_interval_average.to_f / 2
end
scaled_poll_interval() click to toggle source

Calculates an average poll interval based on the number of known Roundhouse processes. This minimizes a single point of failure by dispersing check-ins but without taxing Redis if you run many Roundhouse processes.

# File lib/roundhouse/scheduled.rb, line 94
def scaled_poll_interval
  pcount = Roundhouse::ProcessSet.new.size
  pcount = 1 if pcount == 0
  pcount * Roundhouse.options[:average_scheduled_poll_interval]
end