class Faulty::Circuit

Runs code protected by a circuit breaker

www.martinfowler.com/bliki/CircuitBreaker.html

A circuit is intended to protect against repeated calls to a failing external dependency. For example, a vendor API may be failing continuously. In that case, we trip the circuit breaker and stop calling that API for a specified cool-down period.

Once the cool-down passes, we try the API again, and if it succeeds, we reset the circuit.

Why isn't there a timeout option?


Timeout is inherently unsafe, and should not be used blindly. See [Why Ruby's timeout is Dangerous](jvns.ca/blog/2015/11/27/why-rubys-timeout-is-dangerous-and-thread-dot-raise-is-terrifying).

You should prefer a network timeout like `open_timeout` and `read_timeout`, or write your own code to periodically check how long it has been running. If you're sure you want ruby's generic Timeout, you can apply it yourself inside the circuit run block.

Constants

CACHE_REFRESH_SUFFIX
Options

Options for {Circuit}

@!attribute [r] cache_expires_in

@return [Integer, nil] The number of seconds to keep
  cached results. A value of nil will keep the cache indefinitely.
  Default `86400`.

@!attribute [r] cache_refreshes_after

@return [Integer, nil] The number of seconds after which we attempt
  to refresh the cache even if it's not expired. If the circuit fails,
  we continue serving the value from cache until `cache_expires_in`.
  A value of `nil` disables cache refreshing.
  Default `900`.

@!attribute [r] cache_refresh_jitter

@return [Integer] The maximum number of seconds to randomly add or
  subtract from `cache_refreshes_after` when determining whether to
  refresh the cache.  A non-zero value helps reduce a "thundering herd"
  cache refresh in most scenarios. Set to `0` to disable jitter.
  Default `0.2 * cache_refreshes_after`.

@!attribute [r] cool_down

@return [Integer] The number of seconds the circuit will
  stay open after it is tripped. Default 300.

@!attribute [r] error_module

@return [Module] Used by patches to set the namespace module for
  the faulty errors that will be raised. Default `Faulty`

@!attribute [r] evaluation_window

@return [Integer] The number of seconds of history that
  will be evaluated to determine the failure rate for a circuit.
  Default `60`.

@!attribute [r] rate_threshold

@return [Float] The minimum failure rate required to trip
  the circuit. For example, `0.5` requires at least a 50% failure rate to
  trip. Default `0.5`.

@!attribute [r] sample_threshold

@return [Integer] The minimum number of runs required before
  a circuit can trip. A value of 1 means that the circuit will trip
  immediately when a failure occurs. Default `3`.

@!attribute [r] errors

@return [Error, Array<Error>] An array of errors that are considered circuit
  failures. Default `[StandardError]`.

@!attribute [r] exclude

@return [Error, Array<Error>] An array of errors that will not be
  captured by Faulty. These errors will not be considered circuit
  failures. Default `[]`.

@!attribute [r] cache

@return [Cache::Interface] The cache backend. Default
`Cache::Null.new`. Unlike {Faulty#initialize}, this is not wrapped in
{Cache::AutoWire} by default.

@!attribute [r] notifier

@return [Events::Notifier] A Faulty notifier. Default `Events::Notifier.new`

@!attribute [r] storage

@return [Storage::Interface] The storage backend. Default
`Storage::Memory.new`. Unlike {Faulty#initialize}, this is not wrapped
 in {Storage::AutoWire} by default.

Attributes

name[R]
options[R]

Public Class Methods

new(name, **options, &block) click to toggle source

@param name [String] The name of the circuit @param options [Hash] Attributes for {Options} @yield [Options] For setting options in a block

# File lib/faulty/circuit.rb, line 149
def initialize(name, **options, &block)
  raise ArgumentError, 'name must be a String' unless name.is_a?(String)

  @name = name
  @options = Options.new(options, &block)
end

Public Instance Methods

history() click to toggle source

Get the history of runs of this circuit

The history is an array of tuples where the first value is the run time, and the second value is a boolean which is true if the run was successful.

@return [Array<Array>>] An array of tuples of [run_time, is_success]

# File lib/faulty/circuit.rb, line 281
def history
  storage.history(self)
end
lock_closed!() click to toggle source

Force the circuit to stay closed until unlocked

@return [self]

# File lib/faulty/circuit.rb, line 240
def lock_closed!
  storage.lock(self, :closed)
  self
end
lock_open!() click to toggle source

Force the circuit to stay open until unlocked

@return [self]

# File lib/faulty/circuit.rb, line 232
def lock_open!
  storage.lock(self, :open)
  self
end
reset!() click to toggle source

Reset this circuit to its initial state

This removes the current state, all history, and locks

@return [self]

# File lib/faulty/circuit.rb, line 258
def reset!
  storage.reset(self)
  self
end
run(cache: nil, &block) click to toggle source

Run a block protected by this circuit

If the circuit is closed, the block will run. Any exceptions raised inside the block will be checked against the error and exclude options to determine whether that error should be captured. If the error is captured, this run will be recorded as a failure.

If the circuit exceeds the failure conditions, this circuit will be tripped and marked as open. Any future calls to run will not execute the block, but instead wait for the cool down period. Once the cool down period passes, the circuit transitions to half-open, and the block will be allowed to run.

If the circuit fails again while half-open, the circuit will be closed for a second cool down period. However, if the circuit completes successfully, the circuit will be closed and reset to its initial state.

@param cache [String, nil] A cache key, or nil if caching is not desired @yield The block to protect with this circuit @raise If the block raises an error not in the error list, or if the error

is excluded.

@raise {OpenCircuitError} if the circuit is open @raise {CircuitTrippedError} if this run causes the circuit to trip. It's

possible for concurrent runs to simultaneously trip the circuit if the
storage engine is not concurrency-safe.

@raise {CircuitFailureError} if this run fails, but doesn't cause the

circuit to trip

@return The return value of the block

# File lib/faulty/circuit.rb, line 218
def run(cache: nil, &block)
  cached_value = cache_read(cache)
  # return cached unless cached.nil?
  return cached_value if !cached_value.nil? && !cache_should_refresh?(cache)

  current_status = status
  return run_skipped(cached_value) unless current_status.can_run?

  run_exec(current_status, cached_value, cache, &block)
end
status() click to toggle source

Get the current status of the circuit

This method is not safe for concurrent operations, so it's unsafe to check this method and make runtime decisions based on that. However, it's useful for getting a non-synchronized snapshot of a circuit.

@return [Status]

# File lib/faulty/circuit.rb, line 270
def status
  storage.status(self)
end
try_run(**options, &block) click to toggle source

Run the circuit as with {#run}, but return a {Result}

This is syntax sugar for running a circuit and rescuing an error

@example

result = Faulty.circuit(:api).try_run do
  api.get
end

response = if result.ok?
  result.get
else
  { error: result.error.message }
end

@example

# The Result object has a fetch method that can return a default value
# if an error occurs
result = Faulty.circuit(:api).try_run do
  api.get
end.fetch({})

@param (see run) @yield (see run) @raise If the block raises an error not in the error list, or if the error

is excluded.

@return [Result<Object, Error>] A result where the ok value is the return

value of the block, or the error value is an error captured by the
circuit.
# File lib/faulty/circuit.rb, line 185
def try_run(**options, &block)
  Result.new(ok: run(**options, &block))
rescue FaultyError => e
  Result.new(error: e)
end
unlock!() click to toggle source

Remove any open or closed locks

@return [self]

# File lib/faulty/circuit.rb, line 248
def unlock!
  storage.unlock(self)
  self
end

Private Instance Methods

cache_read(key) click to toggle source

Read from the cache if it is configured

@param key The key to read from the cache @return The cached value, or nil if not present

# File lib/faulty/circuit.rb, line 375
def cache_read(key)
  return unless key

  result = options.cache.read(key.to_s)
  event = result.nil? ? :circuit_cache_miss : :circuit_cache_hit
  options.notifier.notify(event, circuit: self, key: key)
  result
end
cache_refresh_key(key) click to toggle source

Get the corresponding cache refresh key for a given cache key

We use this to force a cache entry to refresh before it has expired

@return [String] The cache refresh key

# File lib/faulty/circuit.rb, line 421
def cache_refresh_key(key)
  "#{key}#{CACHE_REFRESH_SUFFIX}"
end
cache_should_refresh?(key) click to toggle source

Check whether the cache should be refreshed

Should be called only if cache is present

@return [Boolean] true if the cache should be refreshed

# File lib/faulty/circuit.rb, line 404
def cache_should_refresh?(key)
  time = options.cache.read(cache_refresh_key(key.to_s)).to_i
  time + (rand * 2 - 1) * options.cache_refresh_jitter < Faulty.current_time
end
cache_write(key, value) click to toggle source

Write to the cache if it is configured

@param key The key to read from the cache @return [void]

# File lib/faulty/circuit.rb, line 388
def cache_write(key, value)
  return unless key

  options.notifier.notify(:circuit_cache_write, circuit: self, key: key)
  options.cache.write(key.to_s, value, expires_in: options.cache_expires_in)

  unless options.cache_refreshes_after.nil?
    options.cache.write(cache_refresh_key(key.to_s), next_refresh_time, expires_in: options.cache_expires_in)
  end
end
close!() click to toggle source

@return [Boolean] True if the circuit transitioned from half-open to closed

# File lib/faulty/circuit.rb, line 365
def close!
  closed = storage.close(self)
  options.notifier.notify(:circuit_closed, circuit: self) if closed
  closed
end
failure!(status, error) click to toggle source

@return [Boolean] True if the circuit transitioned to open

# File lib/faulty/circuit.rb, line 330
def failure!(status, error)
  entries = storage.entry(self, Faulty.current_time, false)
  status = Status.from_entries(entries, **status.to_h)
  options.notifier.notify(:circuit_failure, circuit: self, status: status, error: error)

  opened = if status.half_open?
    reopen!(error, status.opened_at)
  elsif status.fails_threshold?
    open!(error)
  else
    false
  end

  opened
end
next_refresh_time() click to toggle source

Get the next time to refresh the cache when writing to it

@return [Integer] The timestamp to refresh at

# File lib/faulty/circuit.rb, line 412
def next_refresh_time
  (Faulty.current_time + options.cache_refreshes_after).floor
end
open!(error) click to toggle source

@return [Boolean] True if the circuit transitioned from closed to open

# File lib/faulty/circuit.rb, line 351
def open!(error)
  opened = storage.open(self, Faulty.current_time)
  options.notifier.notify(:circuit_opened, circuit: self, error: error) if opened
  opened
end
rand() click to toggle source

Get a random number from 0.0 to 1.0 for use with cache jitter

@return [Float] A random number from 0.0 to 1.0

# File lib/faulty/circuit.rb, line 428
def rand
  SecureRandom.random_number
end
reopen!(error, previous_opened_at) click to toggle source

@return [Boolean] True if the circuit was reopened

# File lib/faulty/circuit.rb, line 358
def reopen!(error, previous_opened_at)
  reopened = storage.reopen(self, Faulty.current_time, previous_opened_at)
  options.notifier.notify(:circuit_reopened, circuit: self, error: error) if reopened
  reopened
end
run_exec(status, cached_value, cache_key) { || ... } click to toggle source

Excecute a run

@param cached_value The cached value if one is available @param cache_key [String, nil] The cache key if one is given @return The run result

# File lib/faulty/circuit.rb, line 303
def run_exec(status, cached_value, cache_key)
  result = yield
  success!(status)
  cache_write(cache_key, result)
  result
rescue *options.errors => e
  raise if options.exclude.any? { |ex| e.is_a?(ex) }

  if cached_value.nil?
    raise options.error_module::CircuitTrippedError.new(nil, self) if failure!(status, e)

    raise options.error_module::CircuitFailureError.new(nil, self)
  else
    cached_value
  end
end
run_skipped(cached_value) click to toggle source

Process a skipped run

@param cached_value The cached value if one is available @return The result from cache if available

# File lib/faulty/circuit.rb, line 291
def run_skipped(cached_value)
  skipped!
  raise options.error_module::OpenCircuitError.new(nil, self) if cached_value.nil?

  cached_value
end
skipped!() click to toggle source
# File lib/faulty/circuit.rb, line 346
def skipped!
  options.notifier.notify(:circuit_skipped, circuit: self)
end
storage() click to toggle source

Alias to the storage engine from options

@return [Storage::Interface]

# File lib/faulty/circuit.rb, line 435
def storage
  return Faulty::Storage::Null.new if Faulty.disabled?

  options.storage
end
success!(status) click to toggle source

@return [Boolean] True if the circuit transitioned to closed

# File lib/faulty/circuit.rb, line 321
def success!(status)
  storage.entry(self, Faulty.current_time, true)
  closed = close! if status.half_open?

  options.notifier.notify(:circuit_success, circuit: self)
  closed
end