CircuitBreaker

CircuitBreaker protects external resources against overload under failure

Operates in three states:

• Closed (initial state / normal operation): calls are let through normally. Call failures and successes update the call statistics, eg failure count. When the statistics satisfy some criteria, the circuit breaker is 'tripped' and set to the Open state. Note that after this switch, in-flight calls are not canceled. Their success or failure does not affect the circuit breaker anymore though.

• Open: all calls fail fast with a CircuitBreakerOpen error. After the reset timeout, the states changes to HalfOpen

• HalfOpen: the first call is let through. Meanwhile all other calls fail with a CircuitBreakerOpen error. If the first call succeeds, the state changes to Closed again (normal operation). If it fails, the state changes back to Open. The reset timeout is governed by a reset policy, which is typically an exponential backoff.

Two tripping strategies are implemented: 1) Failure counting. When the number of successive failures exceeds a threshold, the circuit breaker is tripped.

Note that the maximum number of failures before tripping the circuit breaker is not absolute under concurrent execution. I.e. if you make 20 calls to a failing system in parallel via a circuit breaker with max 10 failures, the calls will be running concurrently. The circuit breaker will trip after 10 calls, but the remaining 10 that are in-flight will continue to run and fail as well.

TODO what to do if you want this kind of behavior, or should we make it an option?

2. Failure rate. When the fraction of failed calls in some sample period exceeds a threshold (between 0 and 1), the circuit breaker is tripped. The decision to trip the Circuit Breaker is made after every call (including successful ones!)