module NdrDevSupport::IntegrationTesting::FlakeyTests

Grudging handling of flakey integration tests. Allows tests to be declared with `flakey_test`. Our CI reporter gathers information on flakey failures.

Public Instance Methods

flakes() click to toggle source
# File lib/ndr_dev_support/integration_testing/flakey_tests.rb, line 26
def flakes
  @flakes ||= []
end
flakey_test(description, attempts: 3, &block) click to toggle source
# File lib/ndr_dev_support/integration_testing/flakey_tests.rb, line 13
def flakey_test(description, attempts: 3, &block)
  test(description, &block).tap do |test_name|
    self.attempts_per_test = attempts_per_test.merge(test_name.to_s => attempts)
  end
end
run() click to toggle source
Calls superclass method
# File lib/ndr_dev_support/integration_testing/flakey_tests.rb, line 30
def run
  attempts_remaining = attempts_per_test[name]
  return super unless attempts_remaining

  previous_failure = failures.last
  failed_attempts = []
  result = nil

  loop do
    break if attempts_remaining < 1

    result = super

    # No failure was added; we passed!
    break if failures.last == previous_failure

    # Ran out of attempts:
    break if (attempts_remaining -= 1) < 1

    # Loop round and have another go:
    failed_attempts << failures.pop
  end

  # Attempts were only flakey if we eventually passed:
  flakes.concat(failed_attempts) if failures.last == previous_failure

  result
end
test_repeatedly(description, times: 100, &block) click to toggle source
# File lib/ndr_dev_support/integration_testing/flakey_tests.rb, line 19
def test_repeatedly(description, times: 100, &block)
  (1..times).map do |n|
    test("#{description} - #{n}/#{times}", &block)
  end
end