Design a Chat System

Problem statement

The prompt is short: design a chat system. Behind those few words sits a familiar product: people hold private conversations with one other person, hold shared conversations in groups, and expect each short text message to reach the other side within a moment of sending. The sender also expects proof of how far a message travelled, and everyone expects to see which of their contacts are online right now.

The scale class is a large consumer service rather than a small team tool. The service carries 50,000,000 messages a day, and 10,000,000 users can be online at the same time, each holding a connection that must stay ready for the next message. That combination, a high message rate plus a very large number of mostly idle connections, is what makes chat a distinct design problem rather than a small messaging feature.

Two properties shape everything that follows. A message is tiny, at 100 bytes of stored data, yet messages are never thrown away casually, because the stored copy is both the delivery guarantee for an offline receiver and the history the conversation shows later. And delivery is a conversation with the receiver phone, not a single handoff: the service must know whether a message was accepted, whether it reached the phone, and whether the person actually saw it.

In scope

• One-to-one chat: two people exchange messages in a private conversation, shown in the order they were sent
• Group chat: several people exchange messages in one shared conversation, and every member receives every message
• Delivery proof: the sender can see whether a message was accepted by the server, reached the receiver phone, or was seen by the receiver
• Presence: the service shows whether a contact is online now, as a guide that may lag slightly behind reality

Out of scope

• Voice and video calls of any kind
• Photos, files and other attachments; the scope is text messages, with calls and photos left out
• Search across old messages, and message editing, replies and reactions
• Moderation, account administration and business tooling

Everything outside the lists stays out even where it sounds close. Presence covers contacts in conversation, not a general activity feed, and delivery proof covers the states named in the requirements, not typing indicators or group read counts. Keeping the boundary explicit protects the estimates: every figure that follows prices text delivery, ordering, storage and presence, and nothing else.

Requirements

The requirements split into two kinds, and keeping the kinds separate matters because they fail in different ways. Functional requirements say what the service must do: send a message to one person, send a message to a group, and show whether a contact is online. Non-functional requirements say how well the service must do it: answers must come within a moment even at peak times, and the service must keep working when one server fails.

A small record of the ask shows the split in counted form. The recorded decisions describe behaviour the user can see on one side, and qualities the service must hold while delivering that behaviour on the other. The count is trivial, and the discipline behind it is not: a design review can test every functional item against a flow, and every non-functional item against a number or a failure story.

requirements = [
    ("send a message to one person", "functional"),
    ("send a message to a group", "functional"),
    ("show whether a contact is online", "functional"),
    ("answer within a moment, even at peak times", "non-functional"),
    ("keep working when one server fails", "non-functional"),
]
counts = {"functional": 0, "non-functional": 0}
for text, kind in requirements:
    counts[kind] += 1
for kind in ["functional", "non-functional"]:
    print(kind, "decisions:", counts[kind])

That split drives the order of the whole design. Functional items come first, because a connection style or a storage shape chosen before the behaviours are named will quietly decide those behaviours by accident. Non-functional items then act as the bars each candidate design must clear, and a candidate that cannot clear the peak bar or the single-failure bar is rejected however elegant it looks.

Functional requirements

• Send a message to one person: a private conversation between two people, and the message must appear in that conversation in sending order, because a shuffled conversation is unreadable even when every word arrived
• Send a message to a group: one sent message reaches every member of the group, so sending stays one action for the sender while delivery becomes one copy per member
• Prove delivery: the sender sees how far each message travelled, accepted by the server, delivered to the receiver phone, or read by the receiver, because silence about the fate of a message reads as loss
• Show presence: the service shows whether a contact is online now, and the sender can use that signal to judge whether a reply should come quickly or the message will wait in storage

Non-functional requirements

• Answer within a moment, even at peak times: a sent message reaches an online receiver almost at once, and the design is sized for about 1,700 messages per second at peak rather than for the average of 579 messages per second
• Keep working when one server fails: the failure of a single machine loses no stored message and drops no conversation permanently, because individual servers are expected to fail as a matter of routine
• Deliver in order within each conversation: messages are displayed in the order they were sent in that conversation, even when the network delivers them out of order
• Wait for offline receivers: a message sent to an offline receiver is stored and delivered when that receiver returns, because people close the app far more often than they leave a conversation
• Presence may be briefly wrong: an online marker can lag reality by a short margin, and the product treats presence as a guide rather than a promise, because exact presence would cost far more traffic than it is worth

Scale estimates

The estimates start from four stated inputs, and every figure that follows is derived from those inputs in the open. The inputs are 50,000,000 messages a day, 100 bytes of stored data per message, 10,000,000 users online at the same time, and 100,000 open connections held by one connection server. A day holds 86,400 seconds, and that constant converts every daily count into the per-second rates the fleet is actually sized by.

One more input runs quietly underneath the others: the heartbeat. A heartbeat is a small still-here message a phone sends at a fixed interval while the app is open, and the stated convention of the design is one heartbeat every 30 seconds from each online user. Heartbeats carry no message content at all, yet their traffic is priced with the rest, because at full online scale the still-here stream rivals everything else the service moves.

The scale of the problem

The first conversion turns the daily message count into the rate the service must sustain. Dividing by the seconds in a day gives the average, and the peak is then taken as three times that average, rounded, because busy periods in a consumer service bunch messages into the same waking hours. One short calculation produces both figures, and the gap between them is the first real design fact.

messages_per_day = 50000000
seconds_per_day = 86400
average = round(messages_per_day / seconds_per_day)
peak = int(round(messages_per_day / seconds_per_day * 3, -2))
print("messages per day:", messages_per_day)
print("average per second:", average)
print("peak per second:", peak)

The average of 579 messages per second is the planning floor, not the design target. A fleet sized to the average would fall behind every busy period and catch up only when the rush passed, so messages would arrive late exactly when people are most actively talking. Sizing for about 1,700 messages per second at peak costs more idle capacity at quiet times, and that idle capacity is the price of answers that stay fast when the service is busiest.

Storage grows from the same daily count by a different multiplication. Each message leaves 100 bytes of stored data behind, so the daily count times the stored size gives the new storage a day, and multiplying the daily figure across a year gives the growth the storage fleet must absorb. No cleanup decision is assumed anywhere in that arithmetic, which is what makes the result a commitment rather than a hope.

messages_per_day = 50000000
bytes_per_message = 100
bytes_per_day = messages_per_day * bytes_per_message
gb_per_day = bytes_per_day // 1000000000
tb_per_year = round(bytes_per_day * 365 / 1000000000000, 1)
print("bytes per message:", bytes_per_message)
print("GB per day:", gb_per_day)
print("TB per year:", tb_per_year)

The answer is 5 GB of new stored messages a day and about 1.8 TB a year, or 1,825 GB a year before rounding. Those are small numbers next to photo or video storage, and that smallness is itself a design fact: keeping full message history is affordable at text sizes. The storage problem in chat is therefore the number of rows and the speed of loading one conversation, not the raw volume of bytes, and the retention question returns in the trade-offs as a product choice rather than a capacity rescue.

One combined calculation reprices the headline figures together: the message rate, the daily storage growth and the size of the connection fleet, all from the same starting inputs. The fleet figure surprises most first-time designers, because it comes from people merely being online, not from anyone sending anything. Holding a connection open for an idle user still occupies a place on a connection server, and idle users are the overwhelming majority at any given moment.

messages_per_day = 50000000
bytes_per_message = 100
online_users = 10000000
connections_per_server = 100000
average = round(messages_per_day / 86400)
gb_per_day = messages_per_day * bytes_per_message // 1000000000
servers = online_users // connections_per_server
print("average messages per second:", average)
print("GB of new messages per day:", gb_per_day)
print("users online at once:", online_users)
print("connection servers needed:", servers)

Read together, the estimates name the two loads the architecture must carry at once. The message load is the visible one: 579 messages per second on average and about 1,700 at peak, each message needing acceptance, storage, ordering and delivery. The connection load is the quiet one: 10,000,000 open connections spread across 100 connection servers, most of them idle at any given moment, and every one of them a delivery path that must work the moment a message arrives.

API design

A small set of operations carries the whole product surface, and the held-open connection does most of the work. A WebSocket is a connection held open between a phone and a server so that either side can send at any moment, and sending and receiving ride that connection rather than separate request endpoints. Operations that change durable state, fetching history and reporting a delivery state among them, are the ones shaped as explicit requests with explicit answers.

Core operations

The split between immediate and queued work decides where a sender waits. A queue is a holding line where work waits briefly until a worker, a background process that takes jobs from the line, picks it up and carries it out. Acceptance is immediate and synchronous: the sender phone waits only until the message is stored and numbered, because that stored copy is the guarantee everything else relies on, while delivery to the receiver, group copying and state updates travel through the queue, so a slow receiver or a large group never holds the request of the sender open.

Retries are safe because acceptance is tied to storage. If a phone loses its connection after sending but before hearing acceptance, it can send again on reconnecting, and the service can recognise a repeat of a message it already stored rather than delivering the same words twice as two separate messages. The sequence number returned at acceptance gives the sender a fixed handle for the message, and every later state report refers to that same stored row.

Data model

The model is small, because a chat message is a small thing. The durable heart is one stored row per message, holding the text, the conversation it belongs to, its sequence number and its delivery state. Around that row sit the conversation record, the group membership record, the presence record and the connection record, and each of them exists because a named flow reads it constantly.

Core entities

Delivery states

The three delivery states are the vocabulary of delivery proof, and each state is a claim about a different handoff. Sent is a claim about the server: it accepted the message and stored it, so the sender can stop worrying that the words vanished. Delivered is a claim about the receiver phone, and read is a claim about the person, and the gaps between the three states are exactly where senders look when a conversation stalls.

The partition choice is the conversation. Partitioning means splitting stored data into separate homes by a key, and messages are stored per conversation, in the limiting form one table per conversation, because every read in the product loads the messages of one conversation in sequence order and never scans across conversations. The conversation id is the key every access already carries, so a load touches only the rows it will actually display.

The alternative shape, one shared table holding the messages of every conversation, is examined as a real option in the storage deep dive. An index is a sorted lookup structure that lets a store jump straight to matching rows instead of reading every row, and an index can cut the cost of finding one conversation inside a shared table. The index narrows the search, but its rows still live in one large table that every conversation shares, and that sharing is the cost the per-conversation shape avoids.

High-level architecture

Phones connect to connection servers, whose whole job is holding open connections. One connection server holds many idle connections cheaply and pushes down a connection the moment a message for that phone arrives, because push, the server starting delivery instead of waiting for the phone to ask, is only possible while a path stands open. Behind the connection servers sits the chat service, which accepts messages, numbers them, stores them and hands delivery work to a queue, while presence is kept from heartbeats in a fast store and message storage holds the durable rows everything else can be rebuilt from.

+----------------+  WebSocket   +----------------------+
|  Sender phone  | -----------> |  Connection server   |
+----------------+              |  (sender side)       |
                                +----------------------+
                                           | message
                                           v
                                +----------------------+
                                |     Chat service     |
                                | accept, number,      |
                                | store                |
                                +----------------------+
                                   |              |
                     stored row    v              v  delivery job
                          +----------------+   +---------+
                          | Message        |   |  Queue  |
                          | storage        |   +---------+
                          +----------------+        | job
                                                      v
                                                 +---------+
                                                 | Worker  |
                                                 +---------+
                                                      | push
                                                      v
+----------------+  WebSocket   +----------------------+
| Receiver phone | <----------- |  Connection server   |
+----------------+              |  (receiver side)     |
                                +----------------------+
  Heartbeats from both phones feed the presence store,
  which keeps only the latest heartbeat per user.

The send flow

1. The sender phone sends the message text and the conversation id up its open WebSocket connection
2. The connection server on the sender side passes the message to the chat service
3. The chat service assigns the next sequence number for that conversation and stores the row with state sent
4. The chat service places a delivery job on the queue and confirms acceptance to the sender
5. A worker picks up the job and routes the message to the connection server holding the connection of the receiver
6. That connection server pushes the message down the open connection of the receiver, or the stored row waits if the receiver is offline
7. The receiver phone reports delivered, and reports read when the conversation is opened, updating the state on the stored row

The path in counted steps is short enough to list in full. One message passes through each stage in turn, from the sender phone to the receiver phone, and the stages in the printout are the same components the diagram draws. The queue and the worker sit in the middle of the path, which is why a message can be safely stored and accepted before anyone has tried to deliver it.

stages = ["sender phone", "sender connection server", "chat service", "queue", "worker", "receiver connection server", "receiver phone"]
for i, stage in enumerate(stages, 1):
    print(i, stage)

Two flows run beside the send flow without touching its path. Presence flows upward from every open app as heartbeats and is read whenever a contact list is drawn, and history flows downward from message storage whenever a conversation is opened or a phone reconnects after time offline. Neither flow blocks sending: a presence store failure makes online markers stale, and a slow history read makes old messages load late, while new messages keep travelling.

Every functional requirement lands on a named component. One-to-one and group sending land on the chat service and the queue, delivery proof lands on the delivery state held on the stored row, and ordering lands on the sequence number assigned by the chat service. Presence lands on the heartbeat-fed presence store, and no requirement depends on a component the diagram does not show, just as no component exists without a requirement behind it.

Deep dive: holding the connection open

The hard part of chat is not moving a message; it is being ready to move one. With 10,000,000 users online at the same time, the service must know, for every one of those users, how the next message will reach their phone within a moment of being sent. A design where the server can only answer when asked forces the phone to ask constantly, while a design where the server can push must pay for a held-open connection for every online user, idle or not.

Three styles compete for that readiness. Polling means the phone asks again and again whether anything new has arrived, on a fixed rhythm, whether or not anything has arrived. Long polling means the server holds one request open until a message arrives or time runs out, then answers, and the phone immediately asks again, while a WebSocket connection carries messages in both directions for as long as it stays open, which suits chat well.

Three ways to keep a phone reachable

A short scripted run delivers the same small set of messages by all three styles and counts what each style spends. The messages arrive at fixed steps, and the styles differ only in how the phone learns about them. The pattern in the counts is what the run proves, and the pattern only sharpens as the run gets longer or the conversation gets quieter.

arrival_steps = [3, 6, 9]
steps = list(range(1, 11))
polling_requests = 0
polling_delivered = 0
for step in steps:
    polling_requests += 1
    if step in arrival_steps:
        polling_delivered += 1
long_polling_requests = 0
long_polling_delivered = 0
for step in arrival_steps:
    long_polling_requests += 1
    long_polling_delivered += 1
websocket_connections = 1
websocket_delivered = len(arrival_steps)
print("polling:", polling_requests, "requests,", polling_delivered, "messages")
print("long polling:", long_polling_requests, "requests,", long_polling_delivered, "messages")
print("WebSocket:", websocket_connections, "connection,", websocket_delivered, "messages")

Polling spends a request on every step, including every empty step, and most of its requests learn nothing. Long polling spends a request per message, because each held request ends when a message lands, and the WebSocket style spends one connection for the whole run. The difference grows with time and with idleness: the longer a conversation sits quiet, the more polling pays for nothing, while the held connection costs the same whether it carries nothing or everything.

The cost of the held connection is capacity, and it prices out directly. Each connection server holds 100,000 open connections, and dividing the online population by that capacity gives the fleet the design must run before a single message moves. The arithmetic is one division, and its result is the largest fleet in the whole design.

online_users = 10000000
connections_per_server = 100000
servers = online_users // connections_per_server
print("users online at once:", online_users)
print("connections per server:", connections_per_server)
print("connection servers needed:", servers)

The fleet comes to 100 connection servers for 10,000,000 online users. Those servers do little computing per connection, because holding a connection open is mostly memory and patience, so they are a different kind of machine from the chat service behind them. Separating the connection fleet from the chat service lets each side scale on its own axis: connections grow with online users, and message work grows with messages sent.

One held connection carries both directions, which is the property chat actually needs. A phone sends up the same connection it receives on, so a reply never needs a second path to be opened or found. The run below opens a single connection and carries messages both ways across it, counting connections opened and messages carried as it goes.

sent = ["hi", "how are you"]
received = ["hello", "fine, thanks"]
connections_opened = 1
carried = 0
for text in sent:
    carried += 1
    print("phone to server:", text)
for text in received:
    carried += 1
    print("server to phone:", text)
print("connections opened:", connections_opened)
print("messages carried:", carried)

Placement on the fleet is simple filling: each connection server takes users until it reaches capacity, and the next user goes to the next server. The closing model fills a small fleet that way and then pushes a message to a named user through the connection that user already holds. Real fleets rebalance as servers fail and users drift between them, but the fill rule shows the shape clearly: a directory from user to connection server is all a push needs.

users = ["Ana", "Bo", "Cy", "Dee", "Eli", "Fay", "Gil", "Hal", "Ivy", "Jay"]
servers = ["server A", "server B", "server C"]
capacity = 4
placed = {name: [] for name in servers}
for i, user in enumerate(users):
    placed[servers[i // capacity]].append(user)
for name in servers:
    print(name, "holds", len(placed[name]), "connections")
print("total connections:", sum(len(placed[name]) for name in servers))
print("push to Jay:", "one new message")

The verdict is WebSocket connections, held open by a dedicated fleet of connection servers. Push removes the repeated asking that makes polling wasteful and the request churn that long polling keeps, and one bidirectional connection serves sending and receiving with a single path to maintain. The cost is stated plainly in the fleet estimate: 100 connection servers exist purely to hold connections open, and that fleet must be run, monitored and failed over even though it stores nothing durable.

Failure behaviour follows from what the fleet holds. When a connection server fails, its connections drop and the phones reconnect to surviving servers, and any messages that arrived during the gap are waiting in storage, because delivery proof and offline waiting never depended on the connection. A reconnecting phone fetches what it missed from the stored rows, so a dropped connection costs a pause and a catch-up, never a lost message.

Deep dive: keeping messages in order

Delivery paths do not run at one speed. A message can wait briefly in a queue while a later message takes a faster path, so arrival order at the receiver phone is not guaranteed to match sending order. Shown in arrival order, a conversation can answer a question before the question appears, and the damage is to meaning rather than tidiness, because people read chat as a record of who said what in response to whom.

The fix is a sequence number: a number given to each message in one conversation, rising by one with each new message, assigned by the chat service at acceptance. Message ordering then becomes a display rule rather than a transport hope, because messages are shown in sequence number order even when they arrive out of order. Because the number is assigned once, at the moment of storage, every later copy and every later read can sort by the same key.

A shuffled arrival shows the rule working. Three messages arrive in the wrong order in the model, each carrying the sequence number it was sent with, and sorting by that number restores the conversation exactly as it was written. Nothing about the text is consulted, and nothing about arrival time is trusted, which is precisely why the rule survives messy networks.

arrivals = [(2, "yes, five works"), (3, "see you then"), (1, "are you free today")]
print("arrival order:", [seq for seq, text in arrivals])
displayed = sorted(arrivals)
for seq, text in displayed:
    print(seq, text)

Ordering is fixed at display time, and that placement is deliberate. The transport is allowed to be messy, because parallel paths, retries and queue delays can shuffle arrivals freely while the receiver rebuilds the true order from the numbers. A late message still lands in its proper place when it finally arrives, slotting into the conversation above the replies it triggered rather than appearing at the bottom as if newly said.

Sequence numbers are per conversation, never global. Two conversations running side by side each keep their own run of numbers, and each sorts its own messages by its own numbers before display. A global counter would force unrelated conversations to share one ordering they have no use for, and it would make a single counter a bottleneck for the whole service.

arrivals = [("chat A", 2, "yes, it worked"), ("chat B", 1, "the photos are up"), ("chat A", 1, "did it work"), ("chat B", 2, "thanks")]
by_chat = {}
for chat, seq, text in arrivals:
    by_chat.setdefault(chat, []).append((seq, text))
for chat in sorted(by_chat):
    for seq, text in sorted(by_chat[chat]):
        print(chat, seq, text)

A longer run shows the full cycle end to end. Messages are numbered as they are sent, delivered in a shuffled scripted order, and displayed in sequence order at the receiver, and the printout ends with the next sequence number waiting for the conversation. The counter is small state with an outsized job, because it is the single source of truth for the order of a conversation.

sent = ["are you free today", "yes, five works", "see you then", "bring the keys"]
numbered = [(i + 1, text) for i, text in enumerate(sent)]
arrival_order = [numbered[2], numbered[0], numbered[3], numbered[1]]
print("arrival order:", [seq for seq, text in arrival_order])
for seq, text in sorted(arrival_order):
    print("display", seq, text)
print("next sequence number:", len(sent) + 1)

The verdict is per-conversation sequence numbers, assigned by the chat service when a message is accepted and stored. The gain is a true reading order that survives shuffled delivery, retries and offline catch-up, because every path sorts by the same numbers before display. The cost is one counter per conversation to maintain correctly: numbers must rise without two messages sharing a number, or the display rule loses the order it exists to protect.

Edge cases cluster around absence. A missing number at display time usually means a message still in transit rather than a message lost, and the stored rows are the arbiter, because a reconnecting phone fetches history in sequence order and fills any gap from storage. A duplicated arrival is harmless for the same reason: the number identifies the message, so a second copy of a numbered message is recognised as the same row rather than displayed as a repeat.

Deep dive: storing messages and proving delivery

Storage in a chat system serves two masters at once. Stored rows are the waiting room for offline delivery, because a message for a phone that is not connected has nowhere else to be, and they are also the history every conversation reopens from, day after day. The shape chosen for that storage decides how much work a single conversation load performs, and conversation loads happen every time anyone opens the app.

Two shapes for message storage

A small model prices the difference in rows touched. Loading one conversation from its own table touches only the rows of that conversation, while loading the same conversation by scanning one big shared table touches every row in that table. The model is tiny, and the contrast between the two counts is the whole argument in miniature.

tables = {
    "chat A": ["a1", "a2", "a3", "a4"],
    "chat B": ["b1", "b2", "b3"],
    "chat C": ["c1", "c2", "c3", "c4", "c5"],
}
own_rows = len(tables["chat A"])
big_rows = sum(len(rows) for rows in tables.values())
print("rows loaded:", own_rows)
print("rows touched from its own table:", own_rows)
print("rows touched scanning one big table:", big_rows)

The gap widens with scale rather than with conversation size. The history of one conversation grows with how much its members talk, while the shared table grows with how much everyone talks, so the scan cost of the shared shape is set by strangers. An index on the shared table can cut a scan down sharply, and that rescue is real, but the indexed rows still live in one large table, with one table's contention and one table's operational fate shared by every conversation.

Delivery proof is stored, never merely remembered. The state lives on the stored row itself, and each step of the journey updates the row: sent when the server has accepted and stored the message, delivered when the message reaches the receiver phone, and read when the receiver opens the conversation and sees the message. The printout reads the row back after each step, which is what the screen of the sender ultimately does.

row = {"text": "hello", "state": "sent"}
print("stored state:", row["state"])
row["state"] = "delivered"
print("stored state:", row["state"])
row["state"] = "read"
print("stored state:", row["state"])

Keeping the state on the row is what makes proof survive everything else in the design. A sender who closes the app and returns still sees the state the row carries, and a receiver phone that reports read long after delivery updates the one place the sender will ever look. Proof kept on a screen or in memory would vanish at the first restart, while proof kept on the row lasts exactly as long as the message does.

Offline delivery is storage doing its simplest job: waiting. A stored message waits in sequence order while the receiver is offline, and when the receiver comes back online the waiting rows are delivered in that same order. No separate outbox structure is needed, because the stored history and the waiting delivery set are the same rows read in two different moments.

stored = [(1, "are you free today"), (2, "yes, five works"), (3, "see you then")]
receiver_online = False
print("waiting while offline:", len(stored))
receiver_online = True
delivered = 0
if receiver_online:
    for seq, text in sorted(stored):
        delivered += 1
        print("delivered", seq, text)
print("delivered after return:", delivered)

The same mechanism covers short absences and long ones without distinction. A phone in a tunnel and a phone switched off for a long stretch both return to stored rows waiting in order, and the only difference is how many rows greet them on return. Delivery states make the catch-up honest: rows delivered on return move to delivered together, and the proof shown to the sender catches up with reality in one pass.

The closing model combines the pieces at toy scale. Each conversation keeps its own table of stored rows, a new message joins one table with the state sent, and the state then moves to delivered as the row travels. Loading that table returns its rows in sequence order with their states, which is the whole storage design in miniature: per-conversation homes, states on rows, and order from numbers.

tables = {
    "chat A": [{"seq": 1, "text": "did it work", "state": "read"}, {"seq": 2, "text": "yes, it worked", "state": "read"}],
    "chat B": [{"seq": 1, "text": "the photos are up", "state": "delivered"}],
}
new_row = {"seq": 2, "text": "thanks", "state": "sent"}
tables["chat B"].append(new_row)
new_row["state"] = "delivered"
loaded = tables["chat B"]
print("rows loaded for chat B:", len(loaded))
for row in sorted(loaded, key=lambda r: r["seq"]):
    print(row["seq"], row["text"], "-", row["state"])
print("total stored rows:", sum(len(rows) for rows in tables.values()))

The verdict is one table per conversation, with the delivery state held on each stored row. Loads stay proportional to the conversation being read, ordering comes from the sequence numbers the rows already carry, and proof is exactly as durable as the message itself. The given-up side is operational: very many small tables must be created as conversations start, carried as they grow and archived when they go quiet, and that management burden is the standing cost of the fast load.

Failure behaviour is local by construction. Damage to the table of one conversation endangers the history of that conversation and no other, which is the failure-isolation argument for the shape as much as the speed argument. The shared shape inverts that bargain: one large table is simpler to protect as a whole, and its failure is the failure of everyone at once, which is why the verdict accepts the management burden.

Deep dive: presence and group chats

Presence is the live record the service keeps of which users are online now, and it is the one part of the design allowed to be wrong. A message delivered to the wrong place is a defect, while an online marker that lags reality by a short margin harms nobody, because no one acts irreversibly on it. That tolerance is a resource: presence can be built cheaply from signals the phones already send, as long as the design stays honest that the signal can be stale.

The signal is the heartbeat, and the server-side record is last seen: the time of the latest heartbeat from a phone, kept by the server. A user reads as online while the gap since the last heartbeat stays within the interval, and the design uses one heartbeat every 30 seconds while the app is open. When heartbeats stop because the app closed or the phone died, the gap grows past the interval and the user reads as offline, with no sign-off message needed from the phone.

Scripted heartbeats arrive at the fixed interval while the app stays open, and the printout checks the gap since the last heartbeat at scripted moments. While the gap stays within the interval the user shows online, and the first check after heartbeats stop shows the built-in cost: presence notices an absence only once the interval has fully elapsed. The delay is not a bug in the mechanism; the delay is the mechanism, traded openly for a signal that costs almost nothing.

heartbeats = [0, 30, 60]
checks = [60, 89, 91]
for now in checks:
    last = max(t for t in heartbeats if t <= now)
    gap = now - last
    status = "online" if gap <= 30 else "offline"
    print("at", now, "last heartbeat", last, "gap", gap, status)

That delay is the definition of stale presence: presence built from a last heartbeat that is too old to trust, treated as offline exactly because trusting it for longer would mislead for longer. Between the last heartbeat and the end of the interval there is a window where a departed user still shows online, and the product accepts that window as the price of the cheap signal. Presence is therefore treated as a guide rather than a promise everywhere it appears, from contact lists to delivery decisions.

A contact list is the same record read in bulk. Each contact carries a last heartbeat time, and splitting the list at one scripted moment into online and offline is one comparison per contact against the same interval. The printout performs that split on a small scripted list, and the contacts with the freshest heartbeats land on the online side.

last_seen = {"Ana": 95, "Bo": 40, "Cy": 100, "Dee": 10}
now = 100
online = sorted(name for name, t in last_seen.items() if now - t <= 30)
offline = sorted(name for name, t in last_seen.items() if now - t > 30)
print("online:", online)
print("offline:", offline)

Reading by last heartbeat also gives the natural ordering for a contact list, because the most recently seen contacts are the ones most likely to answer. The record is tiny, one timestamp per user, and it is rewritten constantly, which is why it lives in a fast store apart from message storage. Losing the record entirely would cost nothing durable: fresh heartbeats refill it within an interval, and only the online markers are wrong in the meantime.

Three ways to know who is online

The verdict for presence is the heartbeat with expiry, at one heartbeat every 30 seconds. Sign-off designs fail exactly where phones fail, silently and without a closing message, while the heartbeat treats silence itself as the signal, so every failure mode of a phone produces the correct offline result in the end. The costs are a staleness window of up to one interval and a constant heartbeat load across the online population, priced openly in the bottlenecks discussion at about 333,333 heartbeats per second.

Group chat reuses every piece already built, with one multiplication. Group delivery means one sent message is copied to each member of the group: the sender sends once, the chat service stores the message once in the conversation of the group, and delivery then makes one copy per other member. Ordering, storage and delivery proof all work unchanged, because the conversation of a group has its own sequence numbers and its own stored rows, exactly like a private conversation.

A small group shows the multiplication plainly. One member sends a single message to the group, and a copy waits in the inbox of each other member, so the copies made equal the membership apart from the sender. Nothing else about the message changes as the group grows; only the copy count moves, which is why large groups are a delivery-capacity question rather than a new design question.

members = ["Ana", "Bo", "Cy", "Dee"]
sender = "Ana"
inboxes = {name: [] for name in members if name != sender}
message = "practice moves to Saturday"
for name in inboxes:
    inboxes[name].append(message)
total = 0
for name in sorted(inboxes):
    total += len(inboxes[name])
    print(name, "has", len(inboxes[name]), "copy")
print("total copies:", total)

Presence decides how each of those copies travels. In the closing model, one group message meets a mixed group: members whose last heartbeat is fresh receive their copy at once over their open connections, and the member whose heartbeat is old has the copy wait in storage until that member returns. The two mechanisms, heartbeat presence and stored waiting, meet in a single delivery decision taken per member.

members = ["Ana", "Bo", "Cy", "Dee"]
sender = "Ana"
last_seen = {"Bo": 98, "Cy": 95, "Dee": 20}
now = 100
instant = 0
waiting = 0
for name in members:
    if name == sender:
        continue
    if now - last_seen[name] <= 30:
        instant += 1
        print(name, "- online, copy delivered at once")
    else:
        waiting += 1
        print(name, "- offline, copy waits in storage")
print("delivered at once:", instant)
print("waiting:", waiting)

That meeting point is why presence, for all its looseness, earns its place in the delivery path. Sending at once to online members and storing for offline members is the same stored-row mechanism running at two speeds, and presence is merely the switch that chooses the speed for each member. A wrong switch costs little: a member marked online who has just left collects the message from storage on return, and a member marked offline who is actually present receives the waiting copy on the next fetch.

Bottlenecks and failure modes

The first component to saturate is the connection fleet, and it saturates by standing still. Connections grow with online users rather than with messages, so success at attracting users consumes connection capacity even on days when nobody sends very much. The fleet answer is mechanical, more connection servers each holding 100,000 connections, and the operational signal is the count of online users per server climbing toward the per-server capacity.

Heartbeat traffic is the second standing load, and it is larger than the message load it serves. With 10,000,000 users online and one heartbeat every 30 seconds from each phone, the service receives about 333,333 heartbeats per second, set against an average message rate of 579 messages per second. The heartbeat stream is therefore the highest-rate stream in the design, and it stays cheap only because each heartbeat is tiny, touches one small presence record, and needs no storage, ordering or delivery work.

• A connection server fails: its connections drop and the phones reconnect to surviving servers, while messages that arrived in the gap wait in storage and are delivered on return, so nothing is lost
• The chat service is overloaded at peak: acceptance slows before it fails, and sizing for about 1,700 messages per second at peak is the standing protection against that failure
• The queue backs up: delivery lags while acceptance and storage continue, so senders see the state sent for longer and receivers see messages late rather than never
• The table of one conversation is damaged: the history of that conversation is at risk and no other conversation is affected, which is the isolation the storage shape was chosen for
• The presence store is lost: online markers empty and refill from fresh heartbeats within an interval, while messages, ordering and delivery proof stay untouched
• A receiver stays offline for a long stretch: stored rows wait in sequence order and deliver in order on return, and the store grows by exactly the messages of that conversation in the meantime

What must never degrade is narrower than what can degrade. Messages already accepted must never be lost, order within a conversation must never scramble, and delivery proof must never claim a state the stored row does not hold. Freshness can degrade, presence can degrade and delivery speed can degrade, and the design spends its protection budget in that order: storage first, ordering second, proof third, and speed last.

Trade-offs

No choice in this design is free, and each cost named here buys a named gain. The summary table collects the decisions the deep dives actually contested, with the chosen side and the given-up side stated side by side. Three small runs then reprice the standing costs, connection style, heartbeat interval and history retention, in counted form.

The decisions and their costs

The connection cost over a longer run sharpens the first row of the table. The same three arrivals are carried over more empty steps than the earlier run, and the asking styles spend more requests with every added step while the held connection spends nothing extra. Time, not traffic, is what polling pays for, and quiet conversations are where the bill grows fastest.

arrival_steps = [4, 9, 15]
steps = list(range(1, 22))
polling_requests = 0
polling_delivered = 0
for step in steps:
    polling_requests += 1
    if step in arrival_steps:
        polling_delivered += 1
long_polling_requests = len(arrival_steps)
print("polling:", polling_requests, "requests,", polling_delivered, "messages")
print("long polling:", long_polling_requests, "requests,", polling_delivered, "messages")
print("WebSocket:", 1, "connection,", polling_delivered, "messages")

Read against the first connection run, the longer run changes nothing about the ranking and everything about the margin. Long polling still spends one request per arrival and nothing between arrivals, which keeps it respectable for quiet services. At chat scale, with conversations that sit idle for hours between bursts, the single held connection remains the only style whose cost does not grow with waiting.

The heartbeat interval is a dial with traffic on one side and freshness on the other. A small scripted fleet sends heartbeats at three different intervals, and the printout shows the traffic each interval creates together with the longest time presence can be stale at that setting. The chosen setting sits in the middle of the dial deliberately, and the runs show exactly what moving the dial in either direction would buy and what it would cost.

phones = 600
for interval in [10, 30, 60]:
    per_minute = phones * (60 // interval)
    print("interval", interval, "- heartbeats per minute:", per_minute, "- stale up to", interval, "seconds")

Shortening the interval buys fresher presence with more traffic, and lengthening it buys quiet with staler markers, and neither direction changes the mechanism at all. At full online scale the middle setting produces about 333,333 heartbeats per second, which the design absorbs because heartbeats are tiny and touch nothing durable. A product that prized freshness over cost would slide the dial down and pay the traffic willingly; the dial is what changes, never the design.

Keeping history is the third standing cost, and a scripted run shows its shape without any large numbers. The same small number of stored messages arrives each day of the run; keeping everything grows the store steadily, while keeping only the last week holds the store flat. The shape is the decision: growth without a limit, or a hard line that deletes the past of old conversations.

messages_per_day = 5
days = 30
keep_all = messages_per_day * days
keep_last_week = messages_per_day * 7
print("stored after the run, keep everything:", keep_all)
print("stored after the run, keep the last week:", keep_last_week)

At real scale the growth is 5 GB a day and about 1.8 TB a year, which the estimates already showed is affordable for text. The verdict is to keep the messages, because history is half the product: reopening an old conversation and finding it whole is what stored rows are for. Deletion remains available as a policy for quiet conversations, but it is a product decision about memory rather than a capacity rescue the design needs.

The closing run reprices the whole design in one pass, from message rate to storage growth to the connection fleet and the heartbeat traffic. Every figure in the run matches the estimates the design was built on, which is the test a wrap-up should pass, because the trade-offs changed what the design costs rather than what it carries. A design whose final pricing disagreed with its opening estimates would be two designs wearing one name, not one.

messages_per_day = 50000000
online_users = 10000000
average = round(messages_per_day / 86400)
peak = int(round(messages_per_day / 86400 * 3, -2))
gb_per_day = messages_per_day * 100 // 1000000000
servers = online_users // 100000
heartbeats_per_second = round(online_users / 30)
print("average messages per second:", average)
print("peak messages per second:", peak)
print("GB of new messages per day:", gb_per_day)
print("connection servers needed:", servers)
print("heartbeats per second:", heartbeats_per_second)

Taken together, the design spends standing capacity, a connection fleet, constant heartbeat traffic and ever-growing storage, to buy three properties users feel immediately. Messages arrive within a moment, conversations read in true order, and proof shows how far every message travelled, and those properties are bought with capacity that stands ready even when nobody is sending. A much smaller service should make the opposite exchange on several of these rows, keeping the asking styles and the shared table that this scale rejected, because the right answer follows the shape of the load.

Common questions

How does a chat system deliver a message in real time?

Delivery rides a connection that is already open. The phone of the receiver holds a WebSocket connection to a connection server, so when the chat service has accepted and stored a message, a worker hands the message to the connection server holding that connection and the server pushes it down at once. No asking rhythm sets the pace, which is why delivery stays fast at the peak rate of about 1,700 messages per second as well as at the average of 579 messages per second.

How are chat messages kept in order?

Each message receives a sequence number inside its own conversation when the chat service accepts it, rising by one with each new message in that conversation. Phones display messages in sequence number order rather than arrival order, so a message that travelled slowly still takes its proper place in the conversation on arrival. Ordering is per conversation because conversations are what people read, and unrelated conversations share no order worth paying for.

What do sent, delivered and read mean in a chat system?

The three words are delivery states held on the stored row of a message. Sent means the server has accepted the message and stored it, delivered means the message has reached the receiver phone, and read means the receiver has opened the conversation and seen the message. Each state is a claim about a different handoff, and the proof shown to the sender is exactly as durable as the stored row carrying it.

How does a chat system know who is online?

Presence runs on heartbeats rather than on trust. While the app is open, each phone sends a small still-here heartbeat every 30 seconds, and the server keeps the time of the latest heartbeat for each user, so a user whose last heartbeat falls within the interval shows as online. When the app closes the heartbeats simply stop, the record ages past the interval, and the user shows as offline, with the honest caveat that a marker can be stale by up to one interval.

The estimate at a glance