Design a Load Balancer

The ask

Requests arrive at one public address.

Many app servers sit behind that address.

The system picks one server for each request.

The work is spread so no server drowns while others sit idle.

A server that stops answering is taken out of use.

Estimate the scale

Traffic reaches 1,000,000,000 requests a day.

1,000,000,000 requests / 86,400 seconds = 11,574 requests per second.

One app server handles 500 requests per second.

11,574 requests per second / 500 per server rounds up to 24 servers.

The design runs 30 servers for headroom.

30 servers x 500 requests per second = 15,000 requests per second of capacity.

API and data

A request arrives with its address and its content.

The balancer forwards the request to the chosen server.

The answer travels back through the balancer to the sender.

The balancer keeps a list of its app servers.

For each server it keeps a state of up or down.

It also keeps a count of open requests per server.

The data is small and changes fast. It lives in memory.

High-level design

The balancer is the front door for all traffic.

It holds the set of servers that can take work.

For each request it picks one server from the set.

It forwards the request and waits for the answer.

It hands the answer back to the sender.

A server that stops answering leaves the set at once.

Choosing a server

The simplest rule takes turns. Each server gets the next request in order.

Taking turns works when all requests cost about the same.

A second rule picks the server with the fewest open requests.

That rule handles slow requests better.

A hash of the user name can send the same user to the same server.

That helps when a server holds state for the user.

A server near its limit is skipped under every rule.

Deep dive: health checks

A health check is a small request sent on a schedule to ask a server if it can take work.

Health checks run every 5 seconds per server.

60 seconds / 5 seconds = 12 checks a minute for each server.

30 servers x 12 checks a minute = 360 checks a minute.

360 checks / 60 seconds = 6 checks per second.

A server that fails checks in a row leaves the set.

A server returns only after it passes checks again.

Checks are cheap and run outside the request path.

When servers fail

A server can fail in the middle of a request.

The balancer then has no answer to hand back.

A request that is safe to repeat is sent to another server.

A request that is not safe to repeat returns an error instead.

The remaining servers share the load of the failed one.

The headroom in the server count absorbs that extra load.

One balancer is a risk

A single balancer is a single point of failure.

The design therefore runs balancers in pairs.

One balancer of the pair takes the traffic.

The other stands ready with the same server list.

The public address moves to the ready one when the first fails.

Evaluation and trade-offs

A simple balancer reads only addresses and ports.

It is fast and cheap per request.

A smarter balancer reads the request itself.

It can route by content but pays more time per request.

Fast checks catch failures sooner but add steady check traffic.

The design pays that small cost for a set that is rarely wrong.

Scale at a glance