Elixir Circuit Breakers
A circuit breaker is a state machine that stops calling a failing service after too many failures in a row. While open it blocks calls, and a trial step later checks whether the service has recovered.
Three failures flip the breaker open
The threshold is three failures. Two failures leave the breaker closed, and the third failure flips it open.
defmodule Breaker do
def fail(%{state: :closed, failures: failures}) when failures + 1 >= 3, do: %{state: :open, failures: 3}
def fail(%{state: :closed, failures: failures}), do: %{state: :closed, failures: failures + 1}
def fail(%{state: :half_open}), do: %{state: :open, failures: 3}
def fail(%{state: :open} = state), do: state
def succeed(%{state: :half_open}), do: %{state: :closed, failures: 0}
def succeed(%{state: :closed}), do: %{state: :closed, failures: 0}
def succeed(%{state: :open} = state), do: state
def trial(%{state: :open}), do: %{state: :half_open, failures: 3}
def trial(state), do: state
def call(%{state: :open} = state), do: {:blocked, state}
def call(state), do: {:allowed, state}
end
state = %{state: :closed, failures: 0}
state = Breaker.fail(state)
state = Breaker.fail(state)
IO.puts("after 2 failures: state #{state.state}")
IO.puts("after 2 failures: failures #{state.failures}")
state = Breaker.fail(state)
IO.puts("after 3 failures: state #{state.state}")
IO.puts("after 3 failures: failures #{state.failures}")An open breaker blocks calls
An open breaker blocks a call without trying the service. A closed breaker allows a call through.
defmodule Breaker do
def fail(%{state: :closed, failures: failures}) when failures + 1 >= 3, do: %{state: :open, failures: 3}
def fail(%{state: :closed, failures: failures}), do: %{state: :closed, failures: failures + 1}
def fail(%{state: :half_open}), do: %{state: :open, failures: 3}
def fail(%{state: :open} = state), do: state
def succeed(%{state: :half_open}), do: %{state: :closed, failures: 0}
def succeed(%{state: :closed}), do: %{state: :closed, failures: 0}
def succeed(%{state: :open} = state), do: state
def trial(%{state: :open}), do: %{state: :half_open, failures: 3}
def trial(state), do: state
def call(%{state: :open} = state), do: {:blocked, state}
def call(state), do: {:allowed, state}
end
open_state = %{state: :open, failures: 3}
{open_result, _open} = Breaker.call(open_state)
IO.puts("open call: #{open_result}")
closed_state = %{state: :closed, failures: 0}
{closed_result, _closed} = Breaker.call(closed_state)
IO.puts("closed call: #{closed_result}")A trial success closes the breaker
A trial step moves an open breaker to half-open. A success in half-open closes the breaker and resets the failure count to 0.
defmodule Breaker do
def fail(%{state: :closed, failures: failures}) when failures + 1 >= 3, do: %{state: :open, failures: 3}
def fail(%{state: :closed, failures: failures}), do: %{state: :closed, failures: failures + 1}
def fail(%{state: :half_open}), do: %{state: :open, failures: 3}
def fail(%{state: :open} = state), do: state
def succeed(%{state: :half_open}), do: %{state: :closed, failures: 0}
def succeed(%{state: :closed}), do: %{state: :closed, failures: 0}
def succeed(%{state: :open} = state), do: state
def trial(%{state: :open}), do: %{state: :half_open, failures: 3}
def trial(state), do: state
def call(%{state: :open} = state), do: {:blocked, state}
def call(state), do: {:allowed, state}
end
state = %{state: :open, failures: 3}
state = Breaker.trial(state)
IO.puts("after trial: state #{state.state}")
IO.puts("after trial: failures #{state.failures}")
state = Breaker.succeed(state)
IO.puts("after success: state #{state.state}")
IO.puts("after success: failures #{state.failures}")Example
defmodule Breaker do
def fail(%{state: :closed, failures: failures}) when failures + 1 >= 3, do: %{state: :open, failures: 3}
def fail(%{state: :closed, failures: failures}), do: %{state: :closed, failures: failures + 1}
def fail(%{state: :half_open}), do: %{state: :open, failures: 3}
def fail(%{state: :open} = state), do: state
def succeed(%{state: :half_open}), do: %{state: :closed, failures: 0}
def succeed(%{state: :closed}), do: %{state: :closed, failures: 0}
def succeed(%{state: :open} = state), do: state
def trial(%{state: :open}), do: %{state: :half_open, failures: 3}
def trial(state), do: state
def call(%{state: :open} = state), do: {:blocked, state}
def call(state), do: {:allowed, state}
end
state = %{state: :closed, failures: 0}
IO.puts("start: state #{state.state}")
state = Breaker.fail(state)
state = Breaker.fail(state)
state = Breaker.fail(state)
IO.puts("after 3 failures: state #{state.state}")
{call_result, _state} = Breaker.call(state)
IO.puts("call while open: #{call_result}")
state = Breaker.trial(state)
IO.puts("after trial: state #{state.state}")
state = Breaker.succeed(state)
IO.puts("after trial success: state #{state.state}")
IO.puts("after trial success: failures #{state.failures}")Key points
- Closed: the normal state, where calls pass through.
- Open: the state after too many failures, where calls are blocked without trying the service.
- Half-open: the trial state, where one call is allowed through to test recovery.
- Failure count: the number of failures in a row, reset by a success.
- Threshold: the failure count that flips the breaker from closed to open.
In this example, a breaker starts closed and fails three times, which flips it open. A call while open is blocked, so the service is not tried. A trial step moves the breaker to half-open, and a success there closes it again with the failure count reset to 0.
Note
the breaker tracks counts and states as plain values, so its moves can be checked without waiting.
Try it Yourself
Edit the code, then press Run.
Running is simulated
Exercise
A breaker is closed with a threshold of three failures. It fails twice and then succeeds. What state is it in?