Design a Distributed Message Queue

The ask

The system carries messages between services inside one company. A producer hands a message over and moves on. A consumer picks the message up later.

Delivery is at least once. A message may arrive twice, and consumers must cope with that. Losing a message is not acceptable.

Order matters inside one stream of related messages. An order payment should not be read before the order itself. Order across unrelated streams does not matter.

Messages stay stored for days, not seconds. A consumer that falls behind can catch up. A new consumer can replay old messages.

Estimate the scale

The worked queue carries 1,000,000,000 messages a day.

1,000,000,000 messages / 86,400 seconds = 11,574 messages per second on average.

One message is 1 KB.

1,000,000,000 messages x 1 KB = 1 TB of new messages a day.

Retention is 7 days. One copy of the stored messages therefore takes 7 TB.

The load is steady compared with public traffic. Internal producers grow with the company's own services. Capacity planning follows the busiest internal hour, not a viral spike.

API and data

Send takes a queue name, a message body and an optional ordering key. The ordering key groups related messages, such as all messages for one order. The call returns once the message is stored safely.

Receive takes a queue name and returns a small batch of messages. The consumer then acknowledges each message it handled. Acknowledge means the consumer confirms the message is done.

A message record holds the queue name, the partition number and a sequence number inside that partition. It also holds the body and the arrival time. The sequence number fixes the reading order.

Dead-letter handling is part of the interface. A consumer can send a message it cannot process to a dead-letter queue. Operators inspect that queue later.

High-level design

Producers send to a front door that checks the queue name and the ordering key. The front door routes the message to one partition of the queue. Brokers are the servers that store partitions and serve reads.

Each partition is an append-only log on its broker. New messages land at the end. Consumers read from a saved position in the log.

Every partition is copied to three brokers. One broker leads the partition and takes the writes. The copies let reading continue when a broker fails.

Three copies lift stored data from 7 TB to 21 TB. That cost buys survival of a broker failure with no message loss.

A small coordination store tracks which broker leads each partition. It also tracks each group's reading position. Brokers ask it after any failure.

Partitions and ordering

A partition is one ordered slice of a queue. The worked queue has 100 partitions. Messages spread across partitions, and each partition keeps its own order.

100 partitions share the average rate. Each partition takes about 116 messages per second.

Each partition stores about 10 GB of new messages a day. One day of the worked queue splits into small, movable pieces.

The ordering key chooses the partition by hashing. All messages with the same key land in one partition. Their order is the order they arrived.

A queue with no ordering key spreads messages evenly. Even spread gives the best balance. It gives up order between messages.

Consumer groups and delivery

A consumer group is a named set of consumers that share the messages of one queue. Each message goes to one member of the group. Two groups on the same queue each receive every message.

Groups are how the same messages feed different work. The billing group and the reporting group read the same queue independently. Each keeps its own position.

Inside a group, one partition is read by one member at a time. Adding members helps only up to the partition count. Extra members wait as spares.

A member acknowledges a message after handling it. The group's position then moves past that message. No acknowledgement means the message is delivered again.

Lag is the distance between a group's position and the newest message. Growing lag shows a group falling behind. Lag is the queue's clearest warning sign.

Deep dive: the poison message

A poison message is a message that makes its consumer fail every time. A malformed body is the usual cause. Redelivery alone never fixes it.

The broker counts delivery attempts for each message. After five failed attempts, the message moves to the dead-letter queue. Five attempts is the stated rule of the design.

Moving the message aside unblocks its partition. Later messages keep flowing in order. The skipped message waits for a person to inspect it.

The dead-letter queue keeps the original body and the failure count. A fixed consumer can replay the message from there. Nothing is silently dropped.

Duplicates still happen on ordinary failures. A consumer may handle a message and crash before acknowledging it. Consumers therefore make handling idempotent, so a second delivery changes nothing.

Retention and replay

Stored messages stay for 7 days whether they were read or not. Reading never deletes a message. Deletion follows the retention clock alone.

A group can move its position backwards inside the retention window. That move is a replay. Replays rebuild derived data after a bug.

A brand-new group starts at the oldest stored message or at the newest one. The choice is made when the group is created. Old data and live data are both common needs.

Retention is also the storage bill. Longer retention means more brokers. Seven days covers a long weekend plus repair time for most internal failures.

Evaluation and trade-offs

The queue decouples producers from consumers. Producers stay fast when consumers slow down. The price is storage and a delivery delay.

Ordering is strong inside a partition and absent across partitions. More partitions mean more parallel reading and weaker overall order. The partition count is the main tuning knob.

At-least-once delivery is simpler than exactly-once promises. The cost lands on consumers, which must handle duplicates safely. That trade suits internal traffic well.

A hot ordering key can overload one partition. The fix is a finer key or a separate queue for the heavy stream. Detection starts with per-partition lag and stored bytes.

The worked queue in numbers