class Breakers::Service

A service defines a backend system that your application relies upon. This class allows you to configure the outage detection for a service as well as to define which requests belong to it.

Constants

DEFAULT_OPTS

Public Class Methods

new(opts) click to toggle source

Create a new service

@param [Hash] opts the options to create the service with @option opts [String] :name The name of the service for reporting and logging purposes @option opts [Proc] :request_matcher A proc taking a Faraday::Env as an argument that returns true if the service handles that request @option opts [Integer] :seconds_before_retry The number of seconds to wait after an outage begins before testing with a new request @option opts [Integer] :error_threshold The percentage of errors over the last two minutes that indicates an outage @option opts [Integer] :data_retention_seconds The number of seconds to retain success and error data in Redis @option opts [Proc] :exception_handler A proc taking an exception and returns true if it represents an error on the service

# File lib/breakers/service.rb, line 21
def initialize(opts)
  @configuration = DEFAULT_OPTS.merge(opts)
end

Public Instance Methods

add_error() click to toggle source

Indicate that an error has occurred and potentially create an outage

# File lib/breakers/service.rb, line 55
def add_error
  increment_key(key: errors_key)
  maybe_create_outage
end
add_success() click to toggle source

Indicate that a successful response has occurred

# File lib/breakers/service.rb, line 61
def add_success
  increment_key(key: successes_key)
end
begin_forced_outage!() click to toggle source

Force an outage to begin on the service. Forced outages are not periodically retested.

# File lib/breakers/service.rb, line 66
def begin_forced_outage!
  Outage.create(service: self, forced: true)
end
end_forced_outage!() click to toggle source

End a forced outage on the service.

# File lib/breakers/service.rb, line 71
def end_forced_outage!
  latest = Outage.find_latest(service: self)
  if latest.forced?
    latest.end!
  end
end
errors_in_range(start_time:, end_time:, sample_minutes: 60) click to toggle source

Return data about the failed request counts in the time range

@param start_time [Time] the beginning of the range @param end_time [Time] the end of the range @param sample_minutes [Integer] the rate at which to sample the data @return [Array] a list of hashes in the form: { count: Integer, time: Unix Timestamp }

# File lib/breakers/service.rb, line 112
def errors_in_range(start_time:, end_time:, sample_minutes: 60)
  values_in_range(start_time: start_time, end_time: end_time, type: :errors, sample_minutes: sample_minutes)
end
exception_represents_server_error?(exception) click to toggle source

Returns true if a given exception represents an error with the service

@return [Boolean] is it an error?

# File lib/breakers/service.rb, line 50
def exception_represents_server_error?(exception)
  @configuration[:exception_handler]&.call(exception)
end
handles_request?(request_env:) click to toggle source

Given a Faraday::Env, return true if this service handles the request, via its matcher

@param request_env [Faraday::Env] the request environment @return [Boolean] should the service handle the request

# File lib/breakers/service.rb, line 36
def handles_request?(request_env:)
  @configuration[:request_matcher].call(request_env)
end
latest_outage() click to toggle source

Return the most recent outage on the service

# File lib/breakers/service.rb, line 79
def latest_outage
  Outage.find_latest(service: self)
end
name() click to toggle source

Get the name of the service

@return [String] the name

# File lib/breakers/service.rb, line 28
def name
  @configuration[:name]
end
outages_in_range(start_time:, end_time:) click to toggle source

Return a list of all outages in the given time range

@param start_time [Time] the beginning of the range @param end_time [Time] the end of the range @return [Array] a list of outages that began in the range

# File lib/breakers/service.rb, line 88
def outages_in_range(start_time:, end_time:)
  Outage.in_range(
    service: self,
    start_time: start_time,
    end_time: end_time
  )
end
seconds_before_retry() click to toggle source

Get the seconds before retry parameter

@return [Integer] the value

# File lib/breakers/service.rb, line 43
def seconds_before_retry
  @configuration[:seconds_before_retry]
end
successes_in_range(start_time:, end_time:, sample_minutes: 60) click to toggle source

Return data about the successful request counts in the time range

@param start_time [Time] the beginning of the range @param end_time [Time] the end of the range @param sample_minutes [Integer] the rate at which to sample the data @return [Array] a list of hashes in the form: { count: Integer, time: Unix Timestamp }

# File lib/breakers/service.rb, line 102
def successes_in_range(start_time:, end_time:, sample_minutes: 60)
  values_in_range(start_time: start_time, end_time: end_time, type: :successes, sample_minutes: sample_minutes)
end

Protected Instance Methods

align_time_on_minute(time: nil) click to toggle source

Take the current or given time and round it down to the nearest minute

# File lib/breakers/service.rb, line 153
def align_time_on_minute(time: nil)
  time = (time || Time.now.utc).to_i
  time - (time % 60)
end
errors_key(time: nil) click to toggle source
# File lib/breakers/service.rb, line 118
def errors_key(time: nil)
  "#{Breakers.redis_prefix}#{name}-errors-#{align_time_on_minute(time: time).to_i}"
end
increment_key(key:) click to toggle source
# File lib/breakers/service.rb, line 145
def increment_key(key:)
  Breakers.client.redis_connection.multi do
    Breakers.client.redis_connection.incr(key)
    Breakers.client.redis_connection.expire(key, @configuration[:data_retention_seconds])
  end
end
maybe_create_outage() click to toggle source
# File lib/breakers/service.rb, line 158
def maybe_create_outage
  data = Breakers.client.redis_connection.multi do
    Breakers.client.redis_connection.get(errors_key(time: Time.now.utc))
    Breakers.client.redis_connection.get(errors_key(time: Time.now.utc - 60))
    Breakers.client.redis_connection.get(successes_key(time: Time.now.utc))
    Breakers.client.redis_connection.get(successes_key(time: Time.now.utc - 60))
  end
  failure_count = data[0].to_i + data[1].to_i
  success_count = data[2].to_i + data[3].to_i

  if failure_count > 0 && success_count == 0
    Outage.create(service: self)
  else
    failure_rate = failure_count / (failure_count + success_count).to_f
    if failure_rate >= @configuration[:error_threshold] / 100.0
      Outage.create(service: self)
    end
  end
end
successes_key(time: nil) click to toggle source
# File lib/breakers/service.rb, line 122
def successes_key(time: nil)
  "#{Breakers.redis_prefix}#{name}-successes-#{align_time_on_minute(time: time).to_i}"
end
values_in_range(start_time:, end_time:, type:, sample_minutes:) click to toggle source
# File lib/breakers/service.rb, line 126
def values_in_range(start_time:, end_time:, type:, sample_minutes:)
  start_time = align_time_on_minute(time: start_time)
  end_time = align_time_on_minute(time: end_time)
  keys = []
  times = []
  while start_time <= end_time
    times << start_time
    if type == :errors
      keys << errors_key(time: start_time)
    elsif type == :successes
      keys << successes_key(time: start_time)
    end
    start_time += sample_minutes * 60
  end
  Breakers.client.redis_connection.mget(keys).each_with_index.map do |value, idx|
    { count: value.to_i, time: times[idx] }
  end
end