Design a Notification System

Problem statement

The prompt is short: design a notification system. A notification system is the service that tells a user something happened, such as a new like on their photo, a comment on their post or a change to their password. Two plain words carry the whole problem. A trigger is the event that causes a notification, and a channel is one of the routes a notification can travel: push notification, email or text message.

The scale is what turns a simple sounding service into a design problem. The system sends 200,000,000 notifications a day, which is 2,315 per second on average, and that average hides sharp peaks when something big happens. Every one of those sends is a small promise made to a person, and a system that breaks its promises at that volume, by spamming, duplicating or silently dropping messages, loses the trust that makes notifications worth receiving at all.

Not every event deserves a notification, and deciding which ones do is part of the clarification. A like, a comment and a password change each change something a user cares about, so each one triggers a notification. A profile view and an app open change nothing the user needs to hear about, so both stay silent. The first small model draws exactly that line, event by event.

events = [
    ("like", True),
    ("comment", True),
    ("profile view", False),
    ("password change", True),
    ("app open", False),
]

sent = 0
for name, triggers in events:
    if triggers:
        sent += 1
        print(name, "triggers a notification")
    else:
        print(name, "triggers nothing")
print(sent, "of", len(events), "events trigger a notification")

The model triggers a notification for the like, the comment and the password change, and stays silent for the profile view and the app open. That split is a product decision recorded in the design, not a technical accident. Every later component exists to honour decisions of exactly that kind, at full scale and without exceptions leaking through.

One word in the prompt needs an honest definition before anything is built: delivered. A notification passes through stages on its way to the user, and the system can confirm the early stages but not the final appearance on the device. Delivered therefore means the message was handed to the provider, the outside service that carries push notifications, email or text messages onward. A good clarification names what the system will not promise, not only what it will, and the promise stops at the handover.

stages = ["queued", "handed to the provider", "shown on the device"]
confirmed = ["queued", "handed to the provider"]

for stage in stages:
    if stage in confirmed:
        print(stage, "- the system can confirm this")
    else:
        print(stage, "- the system cannot confirm this")
print("the honest promise stops at:", confirmed[-1])

The stages model shows the boundary plainly. Queued and handed to the provider are stages the system itself controls and can confirm from its own records. Shown on the device sits beyond the provider, on hardware and networks the system cannot observe, so claiming it would be a promise made on someone else's behalf. The design reports the stages it can prove and stays silent about the one it cannot.

In scope

• Accepting trigger events, such as likes, comments and password changes, from the rest of the product
• Choosing a channel for each notification from the recipient's preferences, across push notification, email and text message
• Sending reliably: retrying failed sends, blocking duplicates and parking hopeless sends where a human can see them
• Respecting provider limits, message priority and quiet hours on every send
• Recording every notification and reporting its status honestly, up to handover to the provider

Out of scope

• The product features that create the events; the system receives triggers, it does not decide what a like is
• The providers' own networks and devices; delivery after handover belongs to the provider
• Any guarantee that a notification appears on a device or is read by its owner
• Writing marketing campaigns; a campaign arrives as ordinary trigger events like any other
• Direct messages and chat between users, which are a separate design with a different shape

Requirements

The requirements turn the clarified prompt into a contract. Each functional requirement names a behaviour the system must show, together with the consequence that makes the behaviour matter. Each non-functional requirement names a bar the behaviour must clear, and those bars, honesty about delivery, freedom from duplicates and respect for the user's night, are what make the design interesting rather than routine.

Functional requirements

• Accept a trigger: an event such as a like, a comment or a password change creates a notification, and an event that is not a trigger, such as a profile view, creates nothing at all
• Choose by preference: preferences are the channels a user has switched on or off for each kind of event, and the send uses a channel the user has actually switched on
• Honour the opt-out: an opt-out is a user choice that stops a kind of notification completely, and a user who has opted out of every channel receives nothing, however important the sender thinks the news is
• Send on three channels: push notification, email and text message are all supported, because different news suits different routes and users keep different channels open
• Keep the record: every notification is stored as a record holding the event id, the user, the channel and the text, so any send can be traced, retried or checked afterwards
• Report status honestly: the recorded status of a notification stops at handed to the provider, and never claims the message was shown on the device

Non-functional requirements

• Sustain the volume: 200,000,000 notifications a day, an average of 2,315 per second, with peaks of about 23,000 per second absorbed rather than refused
• Notify exactly once: one event produces one notification per user, even when sends are retried and queues deliver the same event more than once
• Lose nothing silently: a failed send is retried, and a send that keeps failing is parked in the dead-letter queue where a human can inspect it
• Stay inside provider limits: sends to each provider are capped at the rate that provider accepts, so the provider never cuts the system off for flooding
• Respect quiet hours: non-urgent notifications are held from 22:00 until 07:00, while urgent messages such as a password reset still arrive at night
• Treat preferences as law: a send to a user who opted out is a correctness failure, not an accident, so the preference check runs before any send can happen

Preferences deserve a closer statement, because they decide more sends than any other part of the design. A preference is stored per user and per kind of event, so the same person can welcome push notifications for likes while refusing email about them entirely. The channels are tried in a fixed order and the first switched-on channel wins, which keeps the choice predictable and easy to explain to a user who asks why a message arrived the way it did.

preferences = {
    "Ana": {"push": True, "email": True, "text message": False},
    "Ben": {"push": False, "email": True, "text message": False},
    "Cy": {"push": False, "email": False, "text message": False},
}
order = ["push", "email", "text message"]

for user in ["Ana", "Ben", "Cy"]:
    chosen = "none - opted out of every channel"
    for channel in order:
        if preferences[user][channel]:
            chosen = channel
            break
    print("like event for", user, "->", chosen)

One like event reaches three users in the model, and the preferences produce three different outcomes. The first user takes push, the second falls back to email because push is switched off, and the third receives nothing at all after opting out of every channel. Silence is a legitimate outcome of the routing step, and designing for it deliberately is what separates a notification system from a broadcast system.

Scale estimates

The estimates rest on three stated assumptions, and every figure that follows is derived from them in the open. The first assumption is the volume: 200,000,000 notifications a day. The second is the channel split: 70% of notifications travel as push, 20% as email and 10% as text message. The third is the peak: a big event, such as a major announcement or a widely followed account acting at once, pushes traffic to 10 times the average rate.

The scale of the problem

The average rate comes first, because it sizes the steady fleet. One day holds 86,400 seconds, so 200,000,000 ÷ 86,400 = 2,315 sends per second on average, sustained around the clock. That rate is not large for a fleet of sender machines, and a design that stopped at the average would look comfortably simple. The average is not the design problem, though, because notifications arrive in bursts tied to events in the world, not spread evenly across the day.

The peak is the rate that shapes the architecture. At 10 times the average, the exact figure is 200,000,000 × 10 ÷ 86,400 = 23,148 sends per second, stated as about 23,000 per second. A system built only for the average would face that burst with a tenth of the capacity it needs, and the missing capacity would show up as queues growing faster than they drain. The queue exists precisely so that a short burst becomes a delay, spread over the following minutes, instead of a refusal.

The channel split decides where the volume lands, because each channel has its own workers and its own provider. Push carries 70% of 200,000,000, which is 140,000,000 notifications a day. Email carries 20%, which is 40,000,000 a day, and text message carries 10%, which is 20,000,000 a day. A small batch shows the same proportions in miniature, close enough to count by eye.

batch = ["push"] * 7 + ["email"] * 2 + ["text message"] * 1

counts = {"push": 0, "email": 0, "text message": 0}
for channel in batch:
    counts[channel] += 1

for channel in ["push", "email", "text message"]:
    print(channel, counts[channel], "of", len(batch))

The batch of ten splits into push, email and text message in the same proportions as the full day. The point of the miniature is the shape, not the size: push always dominates, email is a substantial second, and text message is a small, expensive tail. Capacity planning follows the same shape, channel by channel, rather than treating the total as one undifferentiated stream.

Dividing the peak by the same split shows each channel's worst minute. Push faces 16,100 sends per second at peak, email faces 4,600 per second and text message faces 2,300 per second. Those three numbers, not the single headline peak, are what each group of channel workers and each provider limit has to survive.

peak = 23000
split = [("push", 70), ("email", 20), ("text message", 10)]

total = 0
for channel, percent in split:
    count = peak * percent // 100
    total += count
    print(channel, count, "per second at peak")
print("total", total, "per second")

The peak model confirms the arithmetic in code, and the three channel figures add back to the total peak rate. Each channel can now be designed against its own number: its own worker fleet, its own throttling and its own provider relationship. A weakness in one channel, an email provider slowing down, stays inside that channel instead of dragging the others with it.

API design

The interface is small, because most of the system is internal machinery. Product services submit events and read back status, users manage their own preferences, and operators inspect the parked failures. Everything a caller can do is one of those three kinds of conversation.

Core operations

The important split is between what finishes inside the request and what leaves through the queue. Submitting an event finishes inside the request, because the caller needs to know the event was accepted, and acceptance only requires the event to be stored safely in the queue. Sending finishes later, in the background, because a send may wait behind a burst, pause for quiet hours or retry against a struggling provider, and none of that waiting should hold the caller's request open.

The event id is the thread that ties the interface together. It is chosen when the event is created, printed on the notification record, checked against the sent-log before sending and quoted in every outcome report. Because the same id names the event at every stage, a duplicate submission, a redelivery and a retried send are all recognisable as the same event rather than three pieces of news.

Data model

The stored shapes are few and deliberately plain. A notification is a small record of facts, not a rich object, because the system writes and reads these records at the full send rate. The entities below cover everything the interface touches: the record itself, the preferences that route it, the sent-log that protects it and the two holding places, the held store and the dead-letter queue, where notifications wait.

Core entities

The notification record is the centre of the model, and its fields are exactly the facts a channel worker needs to send. The record holds the event id, the user, the channel and the text, so a worker reads one small record and has everything required, with no joins and no lookups into the product's own data. The text is stored ready to send rather than rebuilt at send time, because a send retried an hour later must carry the wording that was true when the event happened.

record = {
    "event id": "like-41",
    "user": "Ana",
    "channel": "push",
    "text": "Ben liked your photo",
}

for field in ["event id", "user", "channel", "text"]:
    print(field, ":", record[field])

The record model prints the four fields in order, and those four fields are the whole contract between the routing step and the sending step. Anything the worker needs that is not on the record does not get sent, which keeps the record honest as the single description of a notification. Status changes are written back onto the same record, so the record a caller reads through the interface is the record the workers acted on.

Two key choices carry the design. The sent-log is keyed by event id alone, because the dedupe question is always the same question: has this event been sent, to anyone, on any channel. The preferences are keyed by user and kind of event together, because the preference check never asks about a user in general, only about this user and this kind of news. Both stores sit on the send path and are read before every send, so both are kept small, flat and fast, and the sent-log in particular must live in a durable store rather than in a worker's memory, since a worker that restarts and forgets would resend everything it had already delivered.

High-level architecture

An event enters from the product and lands on a pub-sub topic, the named stream where a new event lands for workers to pick up. From the topic the event moves into a queue, where events wait so that a burst of events does not crush the senders. A routing step reads the recipient's preferences, builds the notification record and places the notification in front of the worker for the chosen channel, and each channel worker sends on one channel only, through that channel's provider.

┌──────────┐   event    ┌─────────┐   ┌────────┐
│ Product  │ ─────────▶ │ Pub-sub │ ─▶│ Queue  │
│ services │            │ topic   │   └────────┘
└──────────┘            └─────────┘        │
                                           ▼
                              ┌────────────────────┐
                              │ Preference check + │
                              │ notification record│
                              └────────────────────┘
                                 │       │       │
                    ┌────────────┘       │       └────────────┐
                    ▼                    ▼                    ▼
             ┌───────────┐       ┌───────────┐       ┌───────────┐
             │ Push      │       │ Email     │       │ Text      │
             │ worker    │       │ worker    │       │ worker    │
             └───────────┘       └───────────┘       └───────────┘
                    │                    │                    │
                    ▼                    ▼                    ▼
                providers        providers        providers ──▶ user
  ┌──────────────┐        ┌──────────────────┐
  │ Sent-log     │ guards │ Dead-letter queue│ parks failures
  └──────────────┘        └──────────────────┘

The send flow

1. A trigger event happens in the product, and the product service submits it with its event id
2. The event lands on the notifications topic and moves into the queue, where it waits its turn
3. A routing worker takes the event and runs the preference check for the recipient and the kind of event
4. If every channel is switched off, the event ends here as a deliberate silence, with the outcome recorded
5. Otherwise the worker builds the notification record for the first switched-on channel and hands it to that channel's queue, with its priority attached
6. The channel worker checks the sent-log, sends through the provider, and records the event id in the sent-log at the moment of sending
7. A failed send returns for another attempt, and a send that fails on every attempt moves to the dead-letter queue

The queue is the shock absorber of the design. Events land in the queue and a worker takes them in arrival order, so the sending side works at the rate it can sustain while the arriving side bursts freely. When the queue drains, it holds nothing and the system is caught up, and its depth at any moment is the clearest single measure of whether the system is keeping up.

queue = []
queue.append("event 1: a like")
queue.append("event 2: a comment")
queue.append("event 3: a password change")

while queue:
    event = queue.pop(0)
    print("worker takes", event)
print("queue holds", len(queue), "events")

The queue model shows the discipline in miniature: three events go in, the worker takes them in the order they arrived, and the queue ends empty. Arrival order is the default fairness rule, and it is only overridden deliberately, by priority, when capacity runs short. Nothing in the model is lost or reordered, which is exactly the behaviour the real queue is bought for.

One worker per channel keeps each channel's problems inside that channel. Each notification names its channel, and the matching worker sends it, so the push fleet, the email fleet and the text message fleet scale, throttle and fail independently. A separate worker per channel also matches the provider structure, since each channel answers to a different provider with a different limit and a different failure style.

notifications = [
    ("n1", "push"),
    ("n2", "email"),
    ("n3", "push"),
    ("n4", "text message"),
    ("n5", "push"),
]
workers = {"push": 0, "email": 0, "text message": 0}

for note_id, channel in notifications:
    workers[channel] += 1
    print(note_id, "goes to the", channel, "worker")

for channel in ["push", "email", "text message"]:
    print(channel, "worker sent", workers[channel])

Five notifications in the model spread across the three workers, with push taking the largest share just as it does at full scale. The counts at the end are the capacity story of the whole design in miniature: staff each channel for its own share, and no channel's workers sit idle while another channel drowns. The split of work is decided once, by the preference check, and never revisited downstream.

The preference check sits before the channel workers, and that placement is a correctness decision rather than an optimisation. A check that ran after sending could not unsend a message, so the only position that honours an opt-out is before any worker is involved. The complete path, from event to channel, runs end to end in the closing model of the design.

event = {"id": "like-41", "kind": "like", "text": "Ben liked your photo"}
preferences = {
    "Ana": {"push": True, "email": True},
    "Ben": {"push": False, "email": True},
}

print("event", event["id"], "lands on the notifications topic")
for user in ["Ana", "Ben"]:
    prefs = preferences[user]
    if prefs["push"]:
        channel = "push"
    elif prefs["email"]:
        channel = "email"
    else:
        channel = "none"
    print("preferences checked for", user, "->", channel, "worker sends:", event["text"])

One like event lands on the topic in the model, and the preference check routes it differently for two users whose preferences differ. The first user's switched-on push sends the event to the push worker, while the second user's switched-off push falls back to the email worker instead. Same event, same text, two channels, and the difference is entirely the recipients' own choices, applied before either send began.

Deep dive: retries, duplicates and dead letters

Sending fails in ordinary ways, constantly. A provider times out, a network drops a connection, a worker restarts mid-send, and at 200,000,000 notifications a day even rare failures arrive by the thousand. A retry is the basic answer: a failed send is tried again, up to a stated number of attempts, and this design allows three attempts for one send before it gives up. The difficulty is that the cure creates its own disease, because a retry is another chance to send a message that may already have been sent.

Three ways to handle a failed send

The happy path of a retry is short. The provider fails the first attempts and accepts a later one, and the notification arrives without any human noticing that anything went wrong. Most real failures are of exactly that kind, brief and self-clearing, which is why a modest attempt limit rescues the great majority of troubled sends.

failures_before_success = 2
attempt = 0
sent = False

while not sent and attempt < 3:
    attempt += 1
    if attempt <= failures_before_success:
        print("attempt", attempt, "failed")
    else:
        sent = True
        print("attempt", attempt, "sent the notification")
print("total attempts:", attempt)

Two failures precede the success in the model, and the third attempt sends the notification. The count at the end is the whole story: the send needed persistence, not intervention. A design that gave up after a single attempt would have lost a notification over a problem that cleared itself moments later.

Some sends never succeed, and they need a destination that is not the bin. A dead-letter queue is the queue where a send that keeps failing is parked for a human to inspect. Parking keeps the failure visible, with its event id and its history attached, so an operator can find a bad address, a malformed text or a provider rejecting one user, and decide what to do with open eyes.

dead_letter_queue = []
attempt = 0

while attempt < 3:
    attempt += 1
    print("attempt", attempt, "failed")

dead_letter_queue.append("event e-9")
print("after", attempt, "attempts, event e-9 is parked in the dead-letter queue")
print("dead-letter queue holds:", dead_letter_queue)

Every attempt fails in the model, and after the last attempt the event is parked rather than dropped. The parked event waits in the dead-letter queue, counted and inspectable, instead of disappearing from a log line. Retries fix short failures, and the dead-letter queue keeps long failures visible instead of silent, and between those two behaviours no failure mode is left unowned.

The duplicate is the failure that retries make possible. A duplicate is the same event arriving more than once, through a retry or a repeated delivery, and the sharpest case is the success that goes unreported: the provider sent the message, the confirmation was lost, and the system, believing the send failed, sends it again. Dedupe is the check that stops the same event from notifying twice, and the sent-log, the record of event ids already sent, is where the check looks.

sent_log = {}

def deliver(event_id):
    if event_id in sent_log:
        print("event", event_id, "already sent - duplicate blocked")
        return
    sent_log[event_id] = True
    print("event", event_id, "sent")

deliver("e-1")
deliver("e-1")
deliver("e-2")
print("notifications sent:", len(sent_log))

The same event is delivered twice in the model, and the sent-log blocks the second delivery cold. The first delivery sends and records the event id, the repeat meets the record and stops, and a different event passes through untouched. The user is notified once, which is the requirement, and the mechanism that guarantees it is a single lookup against a small record.

The verdict is the third option: retries checked against a durable sent-log, with a dead-letter queue behind them. Retrying without dedupe trades lost notifications for duplicated ones, and never retrying trades duplicates for silent loss, and both trades spend the user's trust. The sent-log breaks the exchange by making repetition harmless, and its cost is stated plainly: one lookup before every send, on the hottest path in the system, forever.

The pattern underneath is idempotency, the property that repeating an action has the same effect as doing it once. Dedupe builds on idempotency: because the sent-log records the effect of the first send, a second attempt at the same event converges on the same outcome instead of doubling it. The guard only works if the record is durable and the check and the record travel together, with the event id entering the sent-log at the moment of sending rather than afterwards. The closing model runs the whole policy at once: one event that succeeds after a retry, one hopeless event, and one repeat.

sent_log = {}
dead_letter_queue = []
plan = {"e-1": 1, "e-2": 99}

def process(event_id):
    if event_id in sent_log:
        print("event", event_id, "duplicate - blocked by the sent-log")
        return
    attempt = 0
    while attempt < 3:
        attempt += 1
        if attempt > plan[event_id]:
            sent_log[event_id] = True
            print("event", event_id, "sent on attempt", attempt)
            return
        print("event", event_id, "attempt", attempt, "failed")
    dead_letter_queue.append(event_id)
    print("event", event_id, "parked in the dead-letter queue after", attempt, "attempts")

process("e-1")
process("e-2")
process("e-1")
print("sent:", len(sent_log), "| dead-letter:", len(dead_letter_queue))

The final line of the model shows one sent and one parked, with no duplicate, and that one line is the reliability contract in miniature. The retried event arrived, the hopeless event is visible to a human, and the repeated event changed nothing. A policy that can show all three outcomes in a single small run is a policy whose behaviour an operator can reason about at full scale.

Deep dive: peaks, provider limits and quiet hours

Scale in a notification system means the design still works when a big event multiplies the normal traffic. The peak rate of about 23,000 notifications per second arrives with push facing 16,100 of them, and the system's own capacity is not the only ceiling in the room. Each outside provider accepts only a stated number of sends per second, a provider limit, and a provider that is flooded past its limit protects itself by cutting the sender off, which converts a busy minute into a silent hour.

Three ways to face a peak

Rate limiting is the tool that holds sends to a provider at its limit, so the provider never cuts the system off for flooding. The sends above the limit are not refused; they wait in the queue, and the queue drains at the accepted rate until the burst is cleared. That single change of shape, from burst to backlog, is what makes a peak survivable, because a backlog is a delay the design can price while a cut-off is a failure it cannot.

When sending capacity is scarce, order matters as much as rate. Priority is an ordering where an urgent message, like a password reset, jumps ahead of a routine one, like a marketing offer. A user locked out of an account experiences a slow password reset as a broken product, while a marketing offer that arrives an hour late is experienced as nothing at all, so the queue spends its scarce early slots where delay hurts most.

messages = [
    ("marketing offer", 3),
    ("new like", 2),
    ("password reset", 1),
    ("security alert", 1),
]

ordered = sorted(messages, key=lambda m: m[1])
for name, priority in ordered:
    print("priority", priority, "-", name)

Messages wait with a priority number in the model, and the lowest number sends first. The password reset and the security alert lead, the new like follows, and the marketing offer goes last, overtaken by everything more urgent that arrived behind it. Sorting a small queue is trivial, and sorting is all the mechanism is; the design work lies in deciding honestly which kinds of news deserve which rank.

The second limit on sending is not technical at all: the user's night. Quiet hours are the night hours when non-urgent notifications are held until morning, and this design holds non-urgent messages from 22:00 until 07:00, a window of 9 hours. A phone that buzzes at midnight with a routine like teaches its owner to switch notifications off entirely, which is the worst outcome a notification system can produce.

hour = 23
messages = [
    ("password reset", True),
    ("new like", False),
    ("weekly digest", False),
]

held = []
for name, urgent in messages:
    if urgent or 7 <= hour < 22:
        print(name, "sends now")
    else:
        held.append(name)
        print(name, "is held for the morning")
print("held:", len(held))

The clock sits at a night hour in the model, and the split is immediate: the urgent password reset sends now, while the new like and the weekly digest are held for the morning. Urgent messages ignore quiet hours by design, so a password reset still arrives at night, because some news is worth waking for and the user is the one who decided which kinds qualify. The held messages are delayed, never deleted, and they leave in the morning with their value intact.

The verdict is the throttled design: rate limiting to each provider's limit, priority ordering inside the queue, and quiet hours holding the routine at night. Sending everything at once is rejected because it spends the provider relationship and the user's patience in the same gesture. Shedding routine load is rejected because a notification system that silently drops news has broken its one promise. The admitted cost is lateness: routine messages wait behind bursts and sleep through the night, and that lateness is the price of every send that does arrive being welcome.

Priority and quiet hours compose, and the composition is where the policy earns its keep. A message held by the night does not lose its rank, and a message released in the morning does not jump ahead of urgent traffic that arrived at dawn. The closing model runs a night batch through both rules together, sorting by priority first and then applying the quiet hours hold to everything non-urgent.

now_hour = 23
messages = [
    ("weekly digest", 3, False),
    ("password reset", 1, True),
    ("marketing offer", 3, False),
    ("new like", 2, False),
]

ordered = sorted(messages, key=lambda m: m[1])
held = []
print("sending now, in order:")
for name, priority, urgent in ordered:
    if urgent:
        print(" ", name)
    else:
        held.append((name, priority))
print("held until morning, in order:")
for name, priority in held:
    print(" ", name)

Only the password reset sends in the night model, and the rest wait for the morning in priority order, the new like ahead of the digest and the marketing offer. Two simple rules, applied in a fixed order, produce a behaviour a user would describe as considerate. Consideration, at this scale, is not a tone of voice; it is a sorting rule and a clock check, applied the same way on the busiest night of the year as on the quietest.

Bottlenecks and failure modes

The first component to saturate is the sending edge: the channel workers pressing against the provider limits. Arrivals average 2,315 per second and peak at about 23,000 per second, while each provider accepts only its stated rate, so during a big event the queues grow at exactly the difference between those two rates. The symptom is honest and visible, queue depth climbing and routine notifications arriving late, and operators watch the depth because it announces saturation long before anything actually breaks.

The slow bottleneck is attention rather than machinery. Every parked send in the dead-letter queue waits for a human to read it, and a provider problem that parks sends faster than operators clear them turns a technical failure into a staffing failure. The queue's size is therefore an operational budget as much as a technical one, and a design that parks freely without alerting on the queue's growth has only moved its silent failure to a new address.

• A channel provider fails: sends on that channel retry up to the attempt limit, persistent failures park in the dead-letter queue, and the other channels keep sending, because each channel's workers and provider are independent
• A channel worker fleet falls behind: the queue for that channel grows, priority ordering decides what the reduced capacity sends first, and nothing is dropped to catch up
• The preference store is unavailable: sends pause rather than proceed blind, because sending without the check risks notifying a user who opted out, the one outcome the design treats as unforgivable
• The sent-log is unavailable: sending pauses for the same reason, since a send that cannot check the log cannot promise it is not a duplicate
• The topic or queue is unavailable: product services cannot submit events, submissions fail loudly at the interface instead of being accepted and lost, and recovery starts from a queue that holds nothing it cannot account for
• Quiet hours end during a backlog: held messages release into the morning queue behind fresh urgent traffic, in priority order, so the morning begins with a surge the throttling already knows how to pace

The pattern across those failures is a deliberate ordering of what may degrade. Timeliness degrades first and most freely: routine notifications arrive late under bursts, behind retries and after the night, and the design treats that lateness as a fair price. Completeness degrades last and only into the open: a send that cannot be delivered is parked where a human will see it, never silently dropped. Two outcomes never degrade at all, a notification reaching a user who opted out and a user being notified twice about the same event, because both spend trust that no later reliability can buy back.

Trade-offs

Every mechanism in the design was bought with something, and the honest place to finish is the bill. More attempts rescue more sends, and each retry is another chance to send a duplicate. Holding the night quiet keeps users subscribed, and every held message arrives later than it was created. The models below price three of those exchanges directly, in the smallest runs that show them.

The first exchange is attempts. One event fails twice before it would succeed, so the number of attempts allowed decides whether that notification exists at all. A single permitted attempt is cheaper and simpler in every way except the one that matters, which is that the message never arrives.

def try_send(failures, max_attempts):
    attempt = 0
    while attempt < max_attempts:
        attempt += 1
        if attempt > failures:
            return attempt
    return 0

result = try_send(2, 1)
if result == 0:
    print("one attempt only: not sent - the send gives up")
else:
    print("one attempt only: sent on attempt", result)
print("three attempts allowed: sent on attempt", try_send(2, 3))

The model shows a single attempt giving up where three attempts deliver. The extra attempts cost time, provider calls and the duplicate risk that the sent-log exists to neutralise. What they buy is measured in notifications that exist, and a notification system is ultimately judged on exactly that count.

The second exchange is the sent-log itself. The check adds one lookup before every send, on every one of the day's notifications, in exchange for a guarantee about a case that might look rare. The model prices the guarantee by running the same event through three deliveries in a row, with and without the check.

deliveries = ["e-1", "e-1", "e-1"]

without_log = 0
for event_id in deliveries:
    without_log += 1

sent_log = {}
with_log = 0
for event_id in deliveries:
    if event_id not in sent_log:
        sent_log[event_id] = True
        with_log += 1

print("without a sent-log:", without_log, "notifications for one event")
print("with a sent-log:", with_log, "notification for one event")

Without the log, three deliveries become three notifications for one event, and with the log they become one. The lookup is the cheapest component in the design and the guarantee it buys is absolute rather than statistical, which is a rare bargain. Where the bargain turns bad is operational: the log grows with every event ever sent, and keeping its lookups fast is a permanent cost of the guarantee.

The third exchange runs across the channels. Each channel trades speed against cost per message, and the split of traffic across them is partly a financial decision wearing a technical costume. Text message is fast and reaches a locked phone, and it carries the highest cost per message by a wide margin.

channels = [
    ("push", "fast", "lowest"),
    ("email", "slower", "low"),
    ("text message", "fast", "highest"),
]

for name, speed, cost in channels:
    print(name, "- speed:", speed, "| cost per message:", cost)

Channels compared

The comparison explains the shape of the channel split. Push is fast and cheapest, so it carries the great bulk of traffic for users who allow it. Email is slower and cheap, a good home for longer and less urgent news. A text message costs the most per message, so the design saves it for messages that matter, where its speed and insistence earn the price, rather than spending it on routine news that push would carry for almost nothing.

The main trade-offs

The decisions and their costs

The finished policy is easiest to judge running whole, with retries, dedupe and quiet hours acting on the same small night batch. One event sends at once, one hopeless event is parked for a human, one repeat is blocked by the sent-log, and one routine event waits for the morning. Each outcome was produced by a different rule, and no rule had to be suspended for the others to work.

sent_log = {}
held = []
dead_letter_queue = []
hour = 23
plan = {"e-1": 0, "e-2": 99}
events = [
    ("e-1", True),
    ("e-2", True),
    ("e-1", True),
    ("e-3", False),
]

for event_id, urgent in events:
    if event_id in sent_log:
        print("event", event_id, "duplicate - blocked")
        continue
    if not urgent and (hour >= 22 or hour < 7):
        held.append(event_id)
        print("event", event_id, "held by quiet hours")
        continue
    attempt = 0
    sent = False
    while attempt < 3:
        attempt += 1
        if attempt > plan[event_id]:
            sent = True
            break
    if sent:
        sent_log[event_id] = True
        print("event", event_id, "sent on attempt", attempt)
    else:
        dead_letter_queue.append(event_id)
        print("event", event_id, "parked in the dead-letter queue")

print("sent:", len(sent_log), "| held:", len(held), "| dead-letter:", len(dead_letter_queue))

The closing line of the model counts one sent, one held and one parked, and the whole design is visible in those three outcomes. Taken together, the choices clarify first, queue events, check preferences, retry with dedupe, and respect limits and quiet hours. Every trade-off in that list was chosen for the user's trust, which a notification system spends fast when it spams, duplicates or cries wolf, and earns slowly, one welcome message at a time.

Common questions

How does a notification system stop duplicates?

Every event carries an event id, and the delivery worker checks that id against the sent-log, the record of ids already sent, before it sends. A retried send or a redelivered event whose id is already recorded is blocked, so persistence and safety stop pulling in opposite directions. Sends can be retried until they land, because landed sends are never repeated, and the guarantee rests on the log being durable rather than held in a worker's memory.

What happens when a notification send keeps failing?

The send is retried up to three attempts, because most provider failures are brief and clear themselves. A send that fails on every attempt is parked in the dead-letter queue, a holding place where a human can inspect it, find the bad address or the rejecting provider, and replay or abandon it deliberately. Nothing is retried forever, and nothing is deleted without a record, so a hopeless send stays visible instead of becoming a silent loss.

Why are routine notifications held at night?

Quiet hours hold non-urgent messages from 22:00 until 07:00, because a routine buzz at midnight teaches the user to switch notifications off entirely, which ends the channel for every future message including the urgent ones. Held messages are delayed rather than deleted, and they release in the morning in priority order. Urgent messages ignore the hold completely, so a password reset or a security alert still arrives the moment it is created, at any hour.

What does delivered mean for a notification?

Delivered honestly means the message was handed to the provider, and the promise stops there. The system can confirm that a notification was queued and that the provider accepted it, from its own records. It cannot confirm that the message was shown on the device, because the device, its network and its settings sit beyond the provider, outside anything the system can observe, so a design that claimed more would be promising on someone else's behalf.

Related designs: the rate limiter design faces the same throttling choice in its pure form, and the news feed and chat system designs both lean on queues carrying events to workers in the background. The social app design brings the threads together, since its likes, comments and follows are exactly the triggers a notification system of this shape exists to deliver.

The estimate at a glance