🎓 A1Learner
🔍
0 of 47 lessons completed

Designing for Failure

Designing for failure means planning how a system responds when a part stops or answers slowly. The plan keeps the rest of the system working instead of letting one stuck part hold everything up.

A timeout frees a stuck caller

The part being called never answers and the caller waits tick by tick. The timeout frees the caller at tick 3.

timeout = 3
for tick in range(1, timeout + 1):
    if tick < timeout:
        print("tick", tick, "waiting")
    else:
        print("tick", tick, "timeout: caller moves on")

Retries wait longer each time

The waits between tries are 1, then 2, then 4 ticks. The total wait is 7 ticks.

waits = [1, 2, 4]
for attempt, wait in enumerate(waits, 1):
    print("try", attempt, "failed; waiting ticks:", wait)
print("total waiting ticks:", sum(waits))

A page that survives a missing part

The page is built from 3 parts and prints in full when all parts work. It still prints, with 2 lines, when the recommendations part is down.

def build_page(recommendations_up):
    lines = ["title", "photo"]
    if recommendations_up:
        lines.append("recommendations")
    return lines

complete = build_page(True)
shorter = build_page(False)
print("complete page lines:", len(complete))
print("shorter page lines:", len(shorter))
print("shorter page:", shorter)

Example

timeout = 3
print("timeout fires at tick:", timeout)

waits = [1, 2, 4]
print("retry waits:", waits)
print("total waiting ticks:", sum(waits))

def build_page(recommendations_up):
    lines = ["title", "photo"]
    if recommendations_up:
        lines.append("recommendations")
    return lines

print("complete page lines:", len(build_page(True)))
print("shorter page lines:", len(build_page(False)))

Key points

  • Timeout: A limit on how long a caller waits for an answer before giving up.
  • Caller: The program that asks another part for an answer and waits for it.
  • Retry: Sending a request again after it fails or times out.
  • Growing wait: Each retry waits longer than the one before, so a struggling part gets breathing room.
  • Graceful degradation: Keeping a shorter, still useful answer when one part of it is down.

In this example, the timeout frees the caller at tick 3 and the retries wait 1, 2 and 4 ticks, which is 7 ticks in total. The page still prints with 2 lines instead of 3 when one part is down, so each tool limits the damage in a different way.

Note

Timeouts free the caller, retries give a request another chance, and graceful degradation protects the parts that still work.

Try it Yourself

Edit the code, then press Run.

Running is simulated

Exercise

A page still shows its title and photo while its recommendations part is down. Which tool is at work?