Hinted Handoff and Read Repair

Define the problem

Consistent hashing assigns each value to a small set of replicas. In the quorum design, a value has N = 3 copies, a write counts at W = 2 copies, and a read asks R = 2 copies.

Machines go down at awkward moments. A write can arrive while one of its three replicas is restarting, and a read can meet a copy that missed the last few writes.

Refusing every write that cannot reach its full set would tank availability. Accepting writes and reads anyway leaves gaps, and the gaps need cheap repairs.

Hinted handoff

The replica that takes a request and organises the others is called the coordinator here. It sends the write to the value's replicas as usual.

When one target replica is down, the coordinator asks a healthy replica outside the set to hold the write instead. That stand-in stores the write with a hint, a note naming the replica the write truly belongs to.

The held write still counts toward the write quorum. The write succeeds now, and its durability rests on a promise to deliver later.

When the down replica returns, the stand-in hands the write over and deletes its hinted copy. The set is whole again, and nobody outside the repair ever noticed the gap.

The cost of holding hints

Hints pile up while a replica is down. The pile is easy to estimate with stated demo figures.

A replica is down for 1 hour, and writes aimed at it arrive at 1,000 per second.

1,000 writes per second × 3,600 seconds = 3,600,000 writes waiting as hints.

At 1 KB per write, the stand-ins hold 3,600,000 × 1 KB = 3,600,000,000 bytes, about 3.6 GB of hints.

Hints are held for a limited time, and the stand-in can fail too. A hint is a favour, not a guarantee, so a long outage can still lose the writes nobody else copied.

Read repair

A read asks R = 2 copies of the value. The coordinator compares the versions that come back, using their vector clocks or timestamps.

When the copies disagree, the reader returns the newest version to the client. It then writes that version back to the stale copies, quietly fixing them for the next reader.

The repair load follows the disagreement rate. At 1,000,000 reads a day with 1 read in 100 finding a stale copy, that is 1,000,000 ÷ 100 = 10,000 repairs a day.

Read repair only touches values that get read. Hot values heal almost at once, and cold values never heal this way at all.

Where the pair breaks

Both repairs are best-effort by design. A hint can expire before its owner returns, and a stale copy can sit unread for months.

Read repair adds write work to the read path. On a very hot value with constant disagreement, the repairs themselves become load worth watching.

Neither repair proves anything about the whole dataset. They close the gaps they happen to meet, and stay silent about the rest.

That silence is why anti-entropy exists behind them. The two cheap repairs cover the common gaps, and the background comparison covers everything else, slowly.

Trade-offs and wrap-up

The strict alternative is to refuse a write until its real replicas are back. That keeps every promise and fails requests that hinted handoff would have accepted.

This is the PACELC trade in daily clothes. When a replica is down, the store chooses availability now and repairs the certainty later.

The pattern to remember is layered repair. Fix at write time what can be held, fix at read time what can be compared, and sweep the rest in the background.

Three repairs, three gaps