🎓 A1Learner
🔍
0 of 42 lessons completed

Notifications: Retries, Duplicates and Dead Letters

Reliability in a notification system means a failed send is handled openly instead of vanishing. This lesson takes the deep dive step, into retries, duplicates and dead letters.

A send that succeeds on a retry

The provider fails the first attempts and accepts a later one. The printout counts the attempts the send needed.

failures_before_success = 2
attempt = 0
sent = False

while not sent and attempt < 3:
    attempt += 1
    if attempt <= failures_before_success:
        print("attempt", attempt, "failed")
    else:
        sent = True
        print("attempt", attempt, "sent the notification")
print("total attempts:", attempt)

A send that never succeeds

Some sends fail on every attempt. After the last attempt the event is parked in the dead-letter queue instead of being dropped.

dead_letter_queue = []
attempt = 0

while attempt < 3:
    attempt += 1
    print("attempt", attempt, "failed")

dead_letter_queue.append("event e-9")
print("after", attempt, "attempts, event e-9 is parked in the dead-letter queue")
print("dead-letter queue holds:", dead_letter_queue)

The same event delivered twice

A queue can deliver the same event more than once. The sent-log blocks the second delivery, so the user is notified once.

sent_log = {}

def deliver(event_id):
    if event_id in sent_log:
        print("event", event_id, "already sent - duplicate blocked")
        return
    sent_log[event_id] = True
    print("event", event_id, "sent")

deliver("e-1")
deliver("e-1")
deliver("e-2")
print("notifications sent:", len(sent_log))

Example

sent_log = {}
dead_letter_queue = []
plan = {"e-1": 1, "e-2": 99}

def process(event_id):
    if event_id in sent_log:
        print("event", event_id, "duplicate - blocked by the sent-log")
        return
    attempt = 0
    while attempt < 3:
        attempt += 1
        if attempt > plan[event_id]:
            sent_log[event_id] = True
            print("event", event_id, "sent on attempt", attempt)
            return
        print("event", event_id, "attempt", attempt, "failed")
    dead_letter_queue.append(event_id)
    print("event", event_id, "parked in the dead-letter queue after", attempt, "attempts")

process("e-1")
process("e-2")
process("e-1")
print("sent:", len(sent_log), "| dead-letter:", len(dead_letter_queue))

Key points

  • Retry: a failed send is tried again, up to a stated number of attempts.
  • Attempts: this design allows three attempts for one send before it gives up.
  • Dead-letter queue: the queue where a send that keeps failing is parked for a human to inspect.
  • Duplicate: the same event arriving more than once, through a retry or a repeated delivery.
  • Dedupe: the check that stops the same event from notifying twice.
  • Sent-log: the record of event ids already sent, checked before every send.
  • Idempotency: dedupe builds on idempotency from the Beginner course, where repeating an action has the same effect as doing it once.

In this example, one event succeeds after a retry, one event fails every attempt and lands in the dead-letter queue, and a repeat of the first event is blocked. The final line shows one sent and one parked, with no duplicate.

Note

retries fix short failures, and the dead-letter queue keeps long failures visible instead of silent.

Try it Yourself

Edit the code, then press Run.

Running is simulated

Exercise

A send has failed on every allowed attempt. What happens next in this design?