🎓 A1Learner
🔍
0 of 46 lessons completed

Elixir Let It Crash

Let it crash is a strategy where a failing worker is allowed to stop and be restarted clean instead of being defended against every possible failure. The supervisor brings the worker back with fresh state, so the rest of the system keeps running.

A restarted worker starts clean

Crash logs are silenced first so only the counts show. The worker adds 3 to its count, then a crash message stops it and the supervisor starts it again from 0.

Logger.configure(level: :none)
defmodule Waiter do
  def until_named(name) do
    if Process.whereis(name) == nil do
      Process.sleep(10)
      until_named(name)
    else
      :ok
    end
  end
  def until_dead(pid) do
    if Process.alive?(pid) do
      Process.sleep(10)
      until_dead(pid)
    else
      :ok
    end
  end
end
defmodule Flaky do
  use GenServer
  def start_link(arg), do: GenServer.start_link(__MODULE__, arg, name: __MODULE__)
  def init(count), do: {:ok, count}
  def handle_call(:get, _from, count), do: {:reply, count, count}
  def handle_call({:add, n}, _from, count), do: {:reply, count + n, count + n}
  def handle_cast(:boom, _state), do: raise "boom"
end
{:ok, sup} = Supervisor.start_link([%{id: Flaky, start: {Flaky, :start_link, [0]}}], strategy: :one_for_one)
GenServer.call(Flaky, {:add, 3})
IO.puts("before: #{GenServer.call(Flaky, :get)}")
old = Process.whereis(Flaky)
GenServer.cast(Flaky, :boom)
Waiter.until_dead(old)
Waiter.until_named(Flaky)
IO.puts("after: #{GenServer.call(Flaky, :get)}")
Supervisor.stop(sup)

A crash in one worker leaves its sibling alone

Two workers share one supervisor. Only the first worker crashes, so only its count resets.

Logger.configure(level: :none)
defmodule Waiter do
  def until_named(name) do
    if Process.whereis(name) == nil do
      Process.sleep(10)
      until_named(name)
    else
      :ok
    end
  end
  def until_dead(pid) do
    if Process.alive?(pid) do
      Process.sleep(10)
      until_dead(pid)
    else
      :ok
    end
  end
end
defmodule Counter do
  use GenServer
  def start_link(name, arg), do: GenServer.start_link(__MODULE__, arg, name: name)
  def init(count), do: {:ok, count}
  def handle_call(:get, _from, count), do: {:reply, count, count}
  def handle_call({:add, n}, _from, count), do: {:reply, count + n, count + n}
  def handle_cast(:boom, _state), do: raise "boom"
end
children = [
  %{id: :a, start: {Counter, :start_link, [:a, 0]}},
  %{id: :b, start: {Counter, :start_link, [:b, 0]}}
]
{:ok, sup} = Supervisor.start_link(children, strategy: :one_for_one)
GenServer.call(:a, {:add, 5})
GenServer.call(:b, {:add, 7})
old = Process.whereis(:a)
GenServer.cast(:a, :boom)
Waiter.until_dead(old)
Waiter.until_named(:a)
IO.puts("crashed worker: #{GenServer.call(:a, :get)}")
IO.puts("sibling worker: #{GenServer.call(:b, :get)}")
Supervisor.stop(sup)

Without a supervisor, a stopped worker stays stopped

A parent that traps exits can watch a linked worker. When the worker exits, the parent learns the reason and finds the worker gone.

Process.flag(:trap_exit, true)
pid = spawn_link(fn -> exit(:boom) end)
receive do
  {:EXIT, _pid, reason} -> IO.puts("reason: #{reason}")
end
IO.puts("alive: #{Process.alive?(pid)}")

Example

Logger.configure(level: :none)
defmodule Waiter do
  def until_named(name) do
    if Process.whereis(name) == nil do
      Process.sleep(10)
      until_named(name)
    else
      :ok
    end
  end
  def until_dead(pid) do
    if Process.alive?(pid) do
      Process.sleep(10)
      until_dead(pid)
    else
      :ok
    end
  end
end
defmodule Flaky do
  use GenServer
  def start_link(arg), do: GenServer.start_link(__MODULE__, arg, name: __MODULE__)
  def init(count), do: {:ok, count}
  def handle_call(:get, _from, count), do: {:reply, count, count}
  def handle_call({:add, n}, _from, count), do: {:reply, count + n, count + n}
  def handle_cast(:boom, _state), do: raise "boom"
end
{:ok, sup} = Supervisor.start_link([%{id: Flaky, start: {Flaky, :start_link, [0]}}], strategy: :one_for_one)
GenServer.call(Flaky, {:add, 4})
IO.puts("count before crash: #{GenServer.call(Flaky, :get)}")
old = Process.whereis(Flaky)
GenServer.cast(Flaky, :boom)
Waiter.until_dead(old)
Waiter.until_named(Flaky)
IO.puts("count after restart: #{GenServer.call(Flaky, :get)}")
GenServer.call(Flaky, {:add, 2})
IO.puts("count after adding again: #{GenServer.call(Flaky, :get)}")
Supervisor.stop(sup)

Key points

  • Worker: a process that does one job and holds its own state.
  • Supervisor: a process that watches workers and restarts a worker that stops.
  • Restart: starting a failed worker again from its initial state.
  • State reset: a restarted worker starts from its initial value, not from the value it had when it stopped.
  • Isolation: one worker stopping does not stop its sibling workers.

In this example, a counter worker starts under a supervisor at 0 and adds 4. A crash message stops the worker, the supervisor restarts it, and the count reads 0 again. Adding 2 after the restart gives 2, which shows the restarted worker works normally with fresh state.

Note

a supervisor restarts a worker even when the failure was not predicted in advance.

Try it Yourself

Edit the code, then press Run.

Running is simulated

Exercise

A supervised worker holds a count of 9 and then crashes. What count does the restarted worker hold?