Design a Unique ID Generator

The ask

Many services need an ID for every new record they create. IDs must never repeat, even when many machines create records at the same moment. A repeat would merge two different records in every index.

IDs sort in time order. A later ID is a larger number than an earlier one. Sorted IDs keep database inserts at the end of the index.

One ID is a single 64-bit number. It fits ordinary database columns and travels cheaply. Gaps in the sequence are acceptable, and missing numbers harm nothing.

The service is the sequencer for the whole platform. Its one job is handing out ordered unique IDs. Everything else, from records to files, builds on those numbers.

Estimate the scale

Demand is 100,000,000 IDs a day across all calling services.

100,000,000 IDs / 86,400 seconds = 1,157 IDs per second on average.

The rate is modest. The hard part is the guarantee, not the volume. Even a short outage stops every caller from creating records.

Availability therefore matters more than raw speed. The design spreads generation across many machines so no single failure stops the supply.

API and data

Next ID takes the caller's service name and returns one ID. A batch call returns a run of IDs for callers that create records in bulk. Both calls are small and carry no record data.

The service stores almost nothing per ID. It stores the register of machine numbers and which service holds each one. Issued IDs are never recorded centrally.

Callers store the ID inside their own record. The ID itself carries its creation millisecond. A reader can recover the creation time from the number alone.

No call ever asks for a specific ID. The service cannot reissue, reserve or take back a number. Those absences keep the interface tiny and the guarantee simple.

High-level design

Generation happens on numbered generating machines inside the service. Each machine builds IDs from its own clock and its own machine number. No machine coordinates with another while building an ID.

A caller reaches any healthy generating machine. The answer comes back in one round trip inside the data center, about 0.5 milliseconds of travel. Generation itself is local arithmetic.

A small register service hands out machine numbers. A generating machine claims its number once, at startup. Two machines never share a number.

Losing one generating machine loses no IDs. Callers fail over to the next machine. Its different machine number keeps its IDs distinct from the lost machine's.

The snowflake layout

A snowflake ID is an identifier built time first, so plain numeric order is also time order. Each ID packs three parts into 64 bits. The parts are time, machine and sequence.

The sign takes 1 bit and stays zero. Time takes 41 bits, machine takes 10 bits and sequence takes 12 bits. Together the four parts fill all 64 bits.

The time part counts milliseconds since a start moment the platform chooses. Forty-one bits of milliseconds last about 69 years from that moment. The platform picks a recent start moment, so the range covers its whole life.

2 to the power 10 = 1,024, so the machine part names 1,024 generating machines. 2 to the power 12 = 4,096, so the sequence part numbers 4,096 IDs inside one millisecond.

The fleet ceiling combines both parts. 1,024 machines x 4,096 IDs = 4,194,304 IDs inside one millisecond, far above the demand of 1,157 per second.

One machine, one millisecond

A generating machine remembers the last millisecond it used. A new request in the same millisecond raises the sequence by one. A request in a newer millisecond restarts the sequence at zero.

One machine alone could number 4,096,000 IDs per second. That is 4,096 per millisecond across 1,000 milliseconds. Demand uses a small fraction of one machine's ceiling.

Many machines exist for availability and for distance, not for rate. Callers in each data center use nearby machines. A regional failure still leaves other regions generating.

When the sequence runs out inside a millisecond, the machine waits for the next millisecond. The wait is under a millisecond. Callers see a tiny delay, never a wrong ID.

Deep dive: the clock moves backwards

Machine clocks get corrected. A correction can move a clock backwards by a few milliseconds. A generator trusting the moved clock could reissue time it already used.

Each machine compares every reading with the last millisecond it issued. A reading behind that mark is refused. The machine pauses generation and raises a warning to its operators.

The pause lasts until the clock passes the last issued millisecond again. For a small correction that is milliseconds of waiting. Callers fail over to other machines during the pause.

A large backwards jump keeps the machine out of service. An operator inspects it before it generates again. Safety wins over one machine's availability, every time.

The rule is simple because the failure is unforgiving. A duplicated ID corrupts data quietly and permanently. A brief pause corrupts nothing.

The sequencer as a service

Standing alone, the service answers one question only. What is the next number. That narrow job is what sequencer means here, a service whose output is an ordered stream of unique numbers.

Callers that need ordered events borrow the same numbers. An event stamped with the next ID sorts correctly everywhere. No caller builds its own ordering machinery.

Batch calls keep eager callers cheap. A caller can hold a small run of unused IDs and refill when it runs low. A crashed caller wastes its leftover numbers, and gaps stay harmless.

Evaluation and trade-offs

A single database counter would be simpler. It would also put every record creation in the platform behind one row on one machine. The snowflake layout removes that bottleneck and that single point of failure.

Random identifiers avoid clocks and machine numbers entirely. They give up time order, so index inserts scatter. They are also longer than 64 bits in common use.

The design depends on disciplined clocks and a guarded machine register. Both are operational duties, not code paths. The register is small, replicated and rarely changed.

Time order is coarse inside one millisecond across machines. Two machines can issue IDs in either order within the same millisecond. Callers needing a strict total order must add their own agreement step.

Inside one 64-bit ID