class Faulty::Circuit
Runs code protected by a circuit breaker
www.martinfowler.com/bliki/CircuitBreaker.html
A circuit is intended to protect against repeated calls to a failing external dependency. For example, a vendor API may be failing continuously. In that case, we trip the circuit breaker and stop calling that API for a specified cool-down period.
Once the cool-down passes, we try the API again, and if it succeeds, we reset the circuit.
Why isn't there a timeout option?
Timeout is inherently unsafe, and should not be used blindly. See [Why Ruby's timeout is Dangerous](jvns.ca/blog/2015/11/27/why-rubys-timeout-is-dangerous-and-thread-dot-raise-is-terrifying).
You should prefer a network timeout like `open_timeout` and `read_timeout`, or write your own code to periodically check how long it has been running. If you're sure you want ruby's generic Timeout, you can apply it yourself inside the circuit run block.
Constants
- CACHE_REFRESH_SUFFIX
- Options
Options
for {Circuit}@!attribute [r] cache_expires_in
@return [Integer, nil] The number of seconds to keep cached results. A value of nil will keep the cache indefinitely. Default `86400`.
@!attribute [r] cache_refreshes_after
@return [Integer, nil] The number of seconds after which we attempt to refresh the cache even if it's not expired. If the circuit fails, we continue serving the value from cache until `cache_expires_in`. A value of `nil` disables cache refreshing. Default `900`.
@!attribute [r] cache_refresh_jitter
@return [Integer] The maximum number of seconds to randomly add or subtract from `cache_refreshes_after` when determining whether to refresh the cache. A non-zero value helps reduce a "thundering herd" cache refresh in most scenarios. Set to `0` to disable jitter. Default `0.2 * cache_refreshes_after`.
@!attribute [r] cool_down
@return [Integer] The number of seconds the circuit will stay open after it is tripped. Default 300.
@!attribute [r] error_module
@return [Module] Used by patches to set the namespace module for the faulty errors that will be raised. Default `Faulty`
@!attribute [r] evaluation_window
@return [Integer] The number of seconds of history that will be evaluated to determine the failure rate for a circuit. Default `60`.
@!attribute [r] rate_threshold
@return [Float] The minimum failure rate required to trip the circuit. For example, `0.5` requires at least a 50% failure rate to trip. Default `0.5`.
@!attribute [r] sample_threshold
@return [Integer] The minimum number of runs required before a circuit can trip. A value of 1 means that the circuit will trip immediately when a failure occurs. Default `3`.
@!attribute [r] errors
@return [Error, Array<Error>] An array of errors that are considered circuit failures. Default `[StandardError]`.
@!attribute [r] exclude
@return [Error, Array<Error>] An array of errors that will not be captured by Faulty. These errors will not be considered circuit failures. Default `[]`.
@!attribute [r] cache
@return [Cache::Interface] The cache backend. Default `Cache::Null.new`. Unlike {Faulty#initialize}, this is not wrapped in {Cache::AutoWire} by default.
@!attribute [r] notifier
@return [Events::Notifier] A Faulty notifier. Default `Events::Notifier.new`
@!attribute [r] storage
@return [Storage::Interface] The storage backend. Default `Storage::Memory.new`. Unlike {Faulty#initialize}, this is not wrapped in {Storage::AutoWire} by default.
Attributes
Public Class Methods
@param name [String] The name of the circuit @param options [Hash] Attributes for {Options} @yield [Options] For setting options in a block
# File lib/faulty/circuit.rb, line 149 def initialize(name, **options, &block) raise ArgumentError, 'name must be a String' unless name.is_a?(String) @name = name @options = Options.new(options, &block) end
Public Instance Methods
Get the history of runs of this circuit
The history is an array of tuples where the first value is the run time, and the second value is a boolean which is true if the run was successful.
@return [Array<Array>>] An array of tuples of [run_time, is_success]
# File lib/faulty/circuit.rb, line 281 def history storage.history(self) end
Force the circuit to stay closed until unlocked
@return [self]
# File lib/faulty/circuit.rb, line 240 def lock_closed! storage.lock(self, :closed) self end
Force the circuit to stay open until unlocked
@return [self]
# File lib/faulty/circuit.rb, line 232 def lock_open! storage.lock(self, :open) self end
Reset this circuit to its initial state
This removes the current state, all history, and locks
@return [self]
# File lib/faulty/circuit.rb, line 258 def reset! storage.reset(self) self end
Run a block protected by this circuit
If the circuit is closed, the block will run. Any exceptions raised inside the block will be checked against the error and exclude options to determine whether that error should be captured. If the error is captured, this run will be recorded as a failure.
If the circuit exceeds the failure conditions, this circuit will be tripped and marked as open. Any future calls to run will not execute the block, but instead wait for the cool down period. Once the cool down period passes, the circuit transitions to half-open, and the block will be allowed to run.
If the circuit fails again while half-open, the circuit will be closed for a second cool down period. However, if the circuit completes successfully, the circuit will be closed and reset to its initial state.
@param cache [String, nil] A cache key, or nil if caching is not desired @yield The block to protect with this circuit @raise If the block raises an error not in the error list, or if the error
is excluded.
@raise {OpenCircuitError} if the circuit is open @raise {CircuitTrippedError} if this run causes the circuit to trip. It's
possible for concurrent runs to simultaneously trip the circuit if the storage engine is not concurrency-safe.
@raise {CircuitFailureError} if this run fails, but doesn't cause the
circuit to trip
@return The return value of the block
# File lib/faulty/circuit.rb, line 218 def run(cache: nil, &block) cached_value = cache_read(cache) # return cached unless cached.nil? return cached_value if !cached_value.nil? && !cache_should_refresh?(cache) current_status = status return run_skipped(cached_value) unless current_status.can_run? run_exec(current_status, cached_value, cache, &block) end
Get the current status of the circuit
This method is not safe for concurrent operations, so it's unsafe to check this method and make runtime decisions based on that. However, it's useful for getting a non-synchronized snapshot of a circuit.
@return [Status]
# File lib/faulty/circuit.rb, line 270 def status storage.status(self) end
Run the circuit as with {#run}, but return a {Result}
This is syntax sugar for running a circuit and rescuing an error
@example
result = Faulty.circuit(:api).try_run do api.get end response = if result.ok? result.get else { error: result.error.message } end
@example
# The Result object has a fetch method that can return a default value # if an error occurs result = Faulty.circuit(:api).try_run do api.get end.fetch({})
@param (see run
) @yield (see run
) @raise If the block raises an error not in the error list, or if the error
is excluded.
@return [Result<Object, Error>] A result where the ok value is the return
value of the block, or the error value is an error captured by the circuit.
# File lib/faulty/circuit.rb, line 185 def try_run(**options, &block) Result.new(ok: run(**options, &block)) rescue FaultyError => e Result.new(error: e) end
Remove any open or closed locks
@return [self]
# File lib/faulty/circuit.rb, line 248 def unlock! storage.unlock(self) self end
Private Instance Methods
Read from the cache if it is configured
@param key The key to read from the cache @return The cached value, or nil if not present
# File lib/faulty/circuit.rb, line 375 def cache_read(key) return unless key result = options.cache.read(key.to_s) event = result.nil? ? :circuit_cache_miss : :circuit_cache_hit options.notifier.notify(event, circuit: self, key: key) result end
Get the corresponding cache refresh key for a given cache key
We use this to force a cache entry to refresh before it has expired
@return [String] The cache refresh key
# File lib/faulty/circuit.rb, line 421 def cache_refresh_key(key) "#{key}#{CACHE_REFRESH_SUFFIX}" end
Check whether the cache should be refreshed
Should be called only if cache is present
@return [Boolean] true if the cache should be refreshed
# File lib/faulty/circuit.rb, line 404 def cache_should_refresh?(key) time = options.cache.read(cache_refresh_key(key.to_s)).to_i time + (rand * 2 - 1) * options.cache_refresh_jitter < Faulty.current_time end
Write to the cache if it is configured
@param key The key to read from the cache @return [void]
# File lib/faulty/circuit.rb, line 388 def cache_write(key, value) return unless key options.notifier.notify(:circuit_cache_write, circuit: self, key: key) options.cache.write(key.to_s, value, expires_in: options.cache_expires_in) unless options.cache_refreshes_after.nil? options.cache.write(cache_refresh_key(key.to_s), next_refresh_time, expires_in: options.cache_expires_in) end end
@return [Boolean] True if the circuit transitioned from half-open to closed
# File lib/faulty/circuit.rb, line 365 def close! closed = storage.close(self) options.notifier.notify(:circuit_closed, circuit: self) if closed closed end
@return [Boolean] True if the circuit transitioned to open
# File lib/faulty/circuit.rb, line 330 def failure!(status, error) entries = storage.entry(self, Faulty.current_time, false) status = Status.from_entries(entries, **status.to_h) options.notifier.notify(:circuit_failure, circuit: self, status: status, error: error) opened = if status.half_open? reopen!(error, status.opened_at) elsif status.fails_threshold? open!(error) else false end opened end
Get the next time to refresh the cache when writing to it
@return [Integer] The timestamp to refresh at
# File lib/faulty/circuit.rb, line 412 def next_refresh_time (Faulty.current_time + options.cache_refreshes_after).floor end
@return [Boolean] True if the circuit transitioned from closed to open
# File lib/faulty/circuit.rb, line 351 def open!(error) opened = storage.open(self, Faulty.current_time) options.notifier.notify(:circuit_opened, circuit: self, error: error) if opened opened end
Get a random number from 0.0 to 1.0 for use with cache jitter
@return [Float] A random number from 0.0 to 1.0
# File lib/faulty/circuit.rb, line 428 def rand SecureRandom.random_number end
@return [Boolean] True if the circuit was reopened
# File lib/faulty/circuit.rb, line 358 def reopen!(error, previous_opened_at) reopened = storage.reopen(self, Faulty.current_time, previous_opened_at) options.notifier.notify(:circuit_reopened, circuit: self, error: error) if reopened reopened end
Excecute a run
@param cached_value The cached value if one is available @param cache_key [String, nil] The cache key if one is given @return The run result
# File lib/faulty/circuit.rb, line 303 def run_exec(status, cached_value, cache_key) result = yield success!(status) cache_write(cache_key, result) result rescue *options.errors => e raise if options.exclude.any? { |ex| e.is_a?(ex) } if cached_value.nil? raise options.error_module::CircuitTrippedError.new(nil, self) if failure!(status, e) raise options.error_module::CircuitFailureError.new(nil, self) else cached_value end end
Process a skipped run
@param cached_value The cached value if one is available @return The result from cache if available
# File lib/faulty/circuit.rb, line 291 def run_skipped(cached_value) skipped! raise options.error_module::OpenCircuitError.new(nil, self) if cached_value.nil? cached_value end
# File lib/faulty/circuit.rb, line 346 def skipped! options.notifier.notify(:circuit_skipped, circuit: self) end
Alias to the storage engine from options
@return [Storage::Interface]
# File lib/faulty/circuit.rb, line 435 def storage return Faulty::Storage::Null.new if Faulty.disabled? options.storage end
@return [Boolean] True if the circuit transitioned to closed
# File lib/faulty/circuit.rb, line 321 def success!(status) storage.entry(self, Faulty.current_time, true) closed = close! if status.half_open? options.notifier.notify(:circuit_success, circuit: self) closed end