Design a Pub-Sub System

The ask

Publishers announce events, such as an order placed or a payment settled. Subscribers register interest in kinds of events. Publishers never learn who subscribes.

Every subscriber of a topic receives every event on that topic. Delivery is at least once. A slow subscriber must not slow the publisher or the other subscribers.

Subscribers come in two kinds. Internal services read events from a stored stream. External systems receive events at their own HTTP endpoints.

Events stay stored for a short replay window. A subscriber that was down can catch up. The window is days, not months.

Estimate the scale

The system holds 10,000 topics. Publishers send 1,000,000 events a day across all of them.

1,000,000 events / 86,400 seconds = 12 events per second arriving.

One published event reaches 1,000 subscribers on average.

1,000,000 events x 1,000 subscribers = 1,000,000,000 deliveries a day.

1,000,000,000 deliveries / 86,400 seconds = 11,574 deliveries per second leaving.

Fan-out is the whole problem. Ingest is light at 12 events per second. Delivery is heavy at 11,574 per second.

API and data

Publish takes a topic name and an event body. The call returns after the event is stored. Publishing never waits for deliveries.

Subscribe takes a topic name and a delivery target. The target is either an internal stream reader or an external HTTP endpoint. Unsubscribe removes that target from the topic.

A topic record holds the topic name, its owner and its retention window. A subscription record holds one topic and one target. One published event therefore reads many subscription records.

10,000 topics x 1,000 subscribers = 10,000,000 subscription records. The subscription store is small next to the delivery load. It is read on every publish, so it sits in memory near the publishers.

Each event gets a sequence number within its topic. The number grows by one for each event on that topic. Subscribers use it to spot gaps and duplicates.

High-level design

A publish lands in the topic's stored stream first. Storage comes before delivery. A crash after storage loses no event.

A fan-out worker reads the stored event and its subscription list. The worker creates one delivery job per subscriber. Delivery jobs wait in per-subscriber queues.

Per-subscriber queues isolate failures. One dead endpoint fills only its own queue. Deliveries to healthy endpoints keep flowing.

Internal subscribers read from the stored stream at their own pace. External endpoints receive pushed deliveries from delivery workers. Both paths start from the same stored event.

Topics and subscriptions

A topic is a named channel for one kind of event. Names carry the kind, such as orders-placed. Publishers and subscribers agree on names out of band.

Topics are cheap to create and heavily reused. One topic gathers many publishers of the same event kind. Subscribers follow the kind, not the publisher.

A subscription can carry a simple filter on event fields. Filtering happens at fan-out, before a delivery job exists. Unwanted events never reach the endpoint.

Subscription changes apply to new events only. An event already fanning out keeps its original list. That rule keeps delivery behaviour predictable during changes.

Fan-out and delivery

Fan-out turns one stored event into many delivery jobs. The jobs are independent. Each job tracks its own attempts and its own outcome.

A delivery job ends in one of three states. It is delivered, it is still retrying, or it is parked after its final failure. Parked jobs stay visible to the subscriber.

Delivery workers pull jobs from many per-subscriber queues in fair rotation. Fair rotation stops one busy topic from starving quiet ones. Capacity follows delivery rate, not publish rate.

Duplicates come from retries after uncertain outcomes. An endpoint may receive an event and fail to confirm it. The retry then delivers the same event again, and endpoints handle events idempotently.

Deep dive: webhook delivery

A webhook is an HTTP call the system makes to a subscriber's own endpoint when an event arrives. The endpoint answers with a success status or a failure status. No answer at all also counts as failure.

A failed webhook is retried up to 5 times. The gaps between retries grow, from seconds to minutes to hours. Growing gaps give a broken endpoint time to recover.

Each delivery carries a signature over its body. The signature uses a secret shared by the system and that endpoint. The endpoint recomputes the signature and rejects deliveries that fail the check.

Each delivery also carries the event's sequence number and the time it was sent. The time lets endpoints reject replayed old deliveries. The sequence number lets them spot missing events.

Ordering is a designed seam, not a promise. Retries run beside newer deliveries, so events can arrive out of order. An endpoint that needs order buffers by sequence number and asks for a replay of any gap.

An endpoint that fails for a long stretch is paused automatically. Its events keep collecting in its own queue. The subscriber restarts delivery from that queue after fixing the endpoint.

When a subscriber falls behind

Falling behind shows up as a growing per-subscriber queue. The stored stream still holds the events. Only the delivery side is late.

Old events leave the stream when the replay window closes. A subscriber still behind at that point loses the oldest events. The system warns the subscriber before the window closes on its backlog.

Publishers are never blocked by a slow subscriber. Backpressure stops at the per-subscriber queue. That isolation is the reason the queues exist per subscriber.

Evaluation and trade-offs

Pub-sub loosens coupling further than a shared queue. Publishers know only topic names. Subscribers come and go without touching publishers.

The cost is fan-out. One event becomes about a thousand deliveries at the worked scale. Storage is cheap here, and delivery capacity is the real budget.

Ordering holds inside one topic's stored stream. Pushed delivery weakens it at the endpoint. Systems that need strict order read the stream instead of taking webhooks.

At-least-once delivery plus signatures plus sequence numbers is the honest external contract. Endpoints verify origin, detect gaps and absorb duplicates. Exactly-once delivery across companies is not promised.

Fan-out in numbers