🎓 A1Learner
🔍
0 of 46 lessons completed

Elixir Bulkhead Isolation

A bulkhead is a wall between pools of work that keeps one failing service from using another service's slots. Each pool has its own limit, so trouble stays inside one pool.

A pool accepts calls up to its own limit

The payments pool has a limit of 2. Two calls fill it, and the used count is the proof.

defmodule Bulkhead do
  def new(), do: %{payments: %{limit: 2, used: 0}, search: %{limit: 2, used: 0}}
  def acquire(pools, name) do
    pool = pools[name]
    if pool.used < pool.limit do
      {Map.put(pools, name, %{pool | used: pool.used + 1}), :ok}
    else
      {pools, :full}
    end
  end
  def used(pools, name), do: pools[name].used
end
pools = Bulkhead.new()
{pools, first} = Bulkhead.acquire(pools, :payments)
{pools, second} = Bulkhead.acquire(pools, :payments)
IO.inspect([first, second])
IO.puts(Bulkhead.used(pools, :payments))

A full pool leaves the other pool open

Filling the payments pool does not touch the search pool. A search call still succeeds while payments is full.

defmodule Bulkhead do
  def new(), do: %{payments: %{limit: 2, used: 0}, search: %{limit: 2, used: 0}}
  def acquire(pools, name) do
    pool = pools[name]
    if pool.used < pool.limit do
      {Map.put(pools, name, %{pool | used: pool.used + 1}), :ok}
    else
      {pools, :full}
    end
  end
  def used(pools, name), do: pools[name].used
end
pools = Bulkhead.new()
{pools, _a} = Bulkhead.acquire(pools, :payments)
{pools, _b} = Bulkhead.acquire(pools, :payments)
{pools, pay_result} = Bulkhead.acquire(pools, :payments)
{pools, search_result} = Bulkhead.acquire(pools, :search)
IO.inspect([pay_result, search_result])
IO.puts(Bulkhead.used(pools, :search))

Releasing a slot opens the pool again

Releasing one payments slot drops its used count to 1. The next payments call then succeeds.

defmodule Bulkhead do
  def new(), do: %{payments: %{limit: 2, used: 0}, search: %{limit: 2, used: 0}}
  def acquire(pools, name) do
    pool = pools[name]
    if pool.used < pool.limit do
      {Map.put(pools, name, %{pool | used: pool.used + 1}), :ok}
    else
      {pools, :full}
    end
  end
  def release(pools, name) do
    pool = pools[name]
    Map.put(pools, name, %{pool | used: pool.used - 1})
  end
  def used(pools, name), do: pools[name].used
end
pools = Bulkhead.new()
{pools, _a} = Bulkhead.acquire(pools, :payments)
{pools, _b} = Bulkhead.acquire(pools, :payments)
pools = Bulkhead.release(pools, :payments)
IO.puts(Bulkhead.used(pools, :payments))
{pools, result} = Bulkhead.acquire(pools, :payments)
IO.inspect(result)
IO.puts(Bulkhead.used(pools, :payments))

Example

defmodule Bulkhead do
  def new(), do: %{payments: %{limit: 2, used: 0}, search: %{limit: 2, used: 0}}
  def acquire(pools, name) do
    pool = pools[name]
    if pool.used < pool.limit do
      {Map.put(pools, name, %{pool | used: pool.used + 1}), :ok}
    else
      {pools, :full}
    end
  end
  def release(pools, name) do
    pool = pools[name]
    Map.put(pools, name, %{pool | used: pool.used - 1})
  end
  def used(pools, name), do: pools[name].used
end
pools = Bulkhead.new()
{pools, first} = Bulkhead.acquire(pools, :payments)
{pools, second} = Bulkhead.acquire(pools, :payments)
{pools, third} = Bulkhead.acquire(pools, :payments)
{pools, fourth} = Bulkhead.acquire(pools, :search)
pools = Bulkhead.release(pools, :payments)
{pools, fifth} = Bulkhead.acquire(pools, :payments)
IO.inspect([first, second, third, fourth, fifth])
IO.puts(Bulkhead.used(pools, :payments))
IO.puts(Bulkhead.used(pools, :search))

Key points

  • Pool: a separate set of slots reserved for one service.
  • Limit: the largest number of slots one pool can use at once.
  • Acquire: a call takes a slot from its own pool when one is free.
  • Full: a call is refused when its own pool has no free slot.
  • Isolation: a full pool blocks only its own calls and leaves other pools open.
  • Release: a finished call returns its slot so the next call can use it.

In this example, the payments pool fills after two calls, refuses a third, and the search pool still accepts a call. Releasing one payments slot then lets a later payments call succeed, while the final counts show the two pools separately.

Try it Yourself

Edit the code, then press Run.

Running is simulated

Exercise

The payments pool has a limit of 2 and is full, while the search pool has a free slot. What happens to a new search call?