Design a Rate Limiter

Problem statement

The prompt is to design a rate limiter: a component that caps how many requests one user may make in a set time, and refuses the rest. A rate limit is a rule of the form “a given identity may make at most a stated number of requests inside a stated stretch of time”, and the limiter is the machinery that checks every request against that rule before the request does any real work. The worked limit is 100 requests per minute per user, and every mechanism, estimate and verdict that follows is built around that figure.

Three questions have to be settled before any counting rule is chosen, because each one changes the shape of the answer. Who is limited: the signed-in user, the network address of an anonymous caller, or the access key an app uses to call the service. Where the check sits: in one place that every request passes, or spread across the machines that serve requests. What happens on rejection: the refused request stops at the check with a clear refusal, and never reaches the app servers behind it.

The check sits in the API gateway. An API gateway is the single entry point that receives every request before it reaches the app servers, so a limiter inside the gateway sees the whole traffic of the service in one place. A refused request costs almost nothing, because it dies at the front door, while an allowed request passes on to the app servers exactly as it would without a limiter. Two users at the limit together account for 200 requests in one minute, and each of those requests is checked on arrival, so the gateway placement is also what makes per user counting possible at all.

In scope

• Checking every incoming request against a limit of 100 requests per minute before the request reaches the app servers
• Counting per user for signed-in traffic, per network address for anonymous traffic, and per access key for app-to-app traffic
• Allowing a request that is inside the limit and refusing a request that is over the limit, with a clear refusal the caller can act on
• Keeping one exact count per identity even when many gateway machines receive that identity's requests
• Choosing the counting rule, the failure behaviour of the limiter itself, and the fairness guarantee between users

Out of scope

• Billing, paid tiers and quota plans sold to customers; the design enforces one stated limit, it does not price it
• Blocking abusive traffic by content, such as attack detection or bot classification; the limiter counts requests, it does not judge them
• Limits inside a single app server process; every count that matters is shared across the gateway fleet
• Long-term analytics over request history; counters expire and are gone, by design

The scope of a limit is the identity its count follows, and the choice of scope decides whose requests share one budget. A per user limit follows the signed-in user, so one busy user cannot spend another user's share. A per address limit follows the network address when no user is signed in, and a per key limit follows an access key that an app uses to call the service. Each scope keeps its own separate count, even when the same human stands behind more than one of them.

arrivals = [("user", "ana"), ("user", "ana"), ("address", "203.0.113.10"), ("key", "key-7"), ("user", "ana")]
counts = {}

for scope, name in arrivals:
    label = scope + ":" + name
    counts[label] = counts.get(label, 0) + 1
    print(label, "count is now", counts[label])

Three scopes appear in the arrivals and each builds an independent tally. The user scope reaches a count of 3 for the returning user, while the address and the key each sit at a count of 1, untouched by the user's activity. Separate counts are the whole point of scoping: a limit only protects the resource it names if the budget it guards cannot be spent by a different identity arriving under a different scope.

Requirements

The requirements are the contract the counting rule, the counter store and the failure policy are all judged against. Each functional requirement names a behaviour the limiter must show, and each one carries the consequence that makes it a requirement rather than a preference. Each non-functional requirement names a bar the behaviour must clear, and two of those bars pull against each other throughout the design: the count must be exact, and the check must be fast enough to sit in front of every request the service receives.

Functional requirements

• Check every request: each request that enters the gateway is checked against the limit of its identity, so no path into the service bypasses the count
• Count per identity: each user, address or key has a separate count, so one busy identity reaches only its own limit and leaves every other identity untouched
• Allow inside the limit: a request whose count is at or under 100 requests per minute passes to the app servers, and the caller sees a normal response
• Refuse over the limit: a request whose count passes the limit is refused at the gateway with status 429 and a Retry-After hint, and it never reaches the app servers
• Share the count across gateways: whichever gateway machine receives a request, it updates the same count for that identity, so spreading traffic across machines cannot multiply the limit
• Expire old counts: a count that belongs to a finished stretch of time clears itself away, so yesterday's traffic never taxes today's budget

Non-functional requirements

• A fast check: the check sits on the path of every request, so it must complete in a small fraction of the time the request itself takes, or the limiter becomes the bottleneck it exists to prevent
• Exact counts under concurrency: two gateway machines updating one count at the same moment must both be counted, because a count that loses updates quietly raises the real limit above the stated one
• Fairness: a per user limit keeps one heavy user from crowding out every other user, so the busiest identity is the only one that feels the limit
• A stated failure policy: when the limiter itself is down, the design says in advance whether requests pass unchecked or are blocked, and the choice is deliberate rather than accidental
• Cheap refusals: a refused request stops at the gateway and consumes no app server work, because a flood of over-limit requests is exactly the load the design must survive

Fairness is the requirement that decides the default scope. If one shared count covered every user together, the first busy user would spend the whole budget and every later arrival would be refused through no fault of its own. Separate counts invert that outcome: the heavy user meets the limit alone, and a light user arriving at the same moment passes in full. Per user counting is therefore the default scope, with per address counting covering callers who are not signed in and per key counting covering apps that call the service.

limit = 3
counts = {}
arrivals = ["ana", "ana", "ana", "ana", "ben", "ben"]

for user in arrivals:
    counts[user] = counts.get(user, 0) + 1
    if counts[user] <= limit:
        print(user, "request allowed, count", counts[user])
    else:
        print(user, "request refused, count", counts[user])

One user's tally climbs past the small demo limit and that user's later request is refused, while the second user's tally starts fresh and both of that user's requests are allowed. Nothing about the first user's refusal leaks into the second user's count, because the gateway keeps one count per user. That separation, scaled up to the worked limit, is what fairness means in practice: the limit is a wall around each identity, not a ceiling over the whole service.

Scale estimates

The estimates start from one stated assumption and derive everything else in the open. The stated assumption is the worked limit itself: 100 requests per minute per user. A rate limiter has no single global traffic figure in the prompt, because its load is the traffic of whatever service it guards, so the decisive arithmetic is per identity: what one identity may do in a minute, in an hour and in a day, what two identities may do together, and what a counting rule can accidentally let through at a boundary.

The scale of the problem

The per identity chains run as follows. One minute at the limit is 100 requests, so one hour at the limit is 100 × 60 = 6,000 requests, and one full day at the limit is 100 × 60 × 24 = 144,000 requests from a single identity. Two users at the limit together account for 100 × 2 = 200 requests in one minute, which is the smallest demonstration that limits add per identity rather than dividing a shared pool. A token bucket refilling at the limit rate earns 100 × 30 ÷ 60 = 50 tokens in 30 seconds, the figure that sets how fast a drained budget recovers.

The boundary figure is the one that shapes the counting rule. A fixed window counts inside a period with a set start and end, so 100 requests can land at the end of one minute and 100 more at the start of the next, and both halves obey their own window while 200 requests pass inside a few seconds. That is twice the stated limit arriving as one burst, and any design that picks a fixed window has accepted that burst whether it says so or not. The refusal side has its own fixed figure: a request over the limit gets status 429, the response that means Too Many Requests, and the response may carry a Retry-After hint in seconds.

API design

A rate limiter exposes very little surface, because most of its work happens inside the request path rather than behind endpoints a client calls. The client-facing surface is the ordinary response behaviour of the guarded service: allowed requests are answered normally, and refused requests receive a refusal that explains itself. The internal surface is the check itself, the single call the gateway makes for every arrival, and its shape decides whether the count stays exact when many gateway machines call it at once.

Core operations

The check is synchronous, and nothing about it can be queued. The gateway needs the decision before it forwards or refuses the request, so the answer has to come back inside the same moment as the arrival itself. That is the opposite of the slow spreading work in other designs, where a queue can absorb delay: a queued rate limit check would either hold every request hostage to the queue or let requests through unchecked and count them afterwards, which is no longer a limit. The price of a synchronous check is that the counter store sits on the hot path of the whole service, and its speed and its failure behaviour become properties of every request.

The refusal is part of the interface, not an error afterthought. Status 429 tells the caller plainly that the limit, not the service, stopped the request, and the Retry-After hint tells the caller how many seconds to wait before trying again, so a polite client backs off instead of hammering the gateway with retries that each cost a check. A refusal without the hint invites immediate retries; a refusal without a distinct status looks like a broken service. Both pieces are cheap to send and expensive to omit.

limit = 3
count = 0

for i in range(1, 5):
    count += 1
    if count <= limit:
        print("request", i, "status 200")
    else:
        print("request", i, "status 429, retry after 30 seconds")

Requests under the limit pass with a normal status 200 response, and the first request over the limit receives status 429 with a hint to retry after 30 seconds in the demo. The demo's figures are small, but the contract is the production contract: one status for allowed, one status for refused, and a wait hint attached to every refusal. Callers can build back-off behaviour against that contract without knowing anything about buckets, windows or counters behind it.

Data model

The limiter stores almost nothing, and what it stores is shaped entirely by the counting rule in force. Every record is keyed by identity: a counter key names one user, one address or one access key, so users never share a count and a lookup never scans. The key is the partition key of the design — the value that decides which record a request touches — because every operation in the system is of the form “update the record for this one identity, then decide”. No record ever needs to be found by anything other than its identity.

Core entities

The three shapes trade storage for accuracy in a fixed order. A window counter holds a single number and an expiry, which is the least state a limiter can keep. Token bucket state holds two numbers, a balance and a refill time, and buys burst tolerance for that small extra cost. A sliding window log holds one time per recent request — as many as 100 stored times for one identity at the worked limit — and buys exactness over every stretch of time. Whichever shape is chosen, the record lives in one shared counter store: a single fast store that holds the counts for every gateway machine, so all gateways work from the same count for an identity.

TTL is what keeps the model self-cleaning. TTL means the time after which a stored count is dropped, so old counts clear themselves away without any sweeper process hunting for them. A fixed window counter carries an expiry at the end of its window, and bucket state for an identity that has gone quiet can expire once a full bucket would have refilled anyway, because a fresh record and a refilled record behave identically. Expiry is therefore not housekeeping bolted onto the model; it is part of the counting rule, stated in the record itself.

store = {}

def hit(user, now):
    if user not in store or now >= store[user][1]:
        store[user] = [0, now + 60]
    store[user][0] += 1
    return store[user][0]

print("ana at second 10, count", hit("ana", 10))
print("ana at second 20, count", hit("ana", 20))
print("ben at second 20, count", hit("ben", 20))
print("ana at second 80, count", hit("ana", 80))

Each count lives under a key for one user and carries an expiry time, and the arrivals show all three properties at once. The first user's count climbs inside the window, the second user starts an independent count under a different key, and an arrival after the expiry begins a fresh count for the first user instead of reviving the old one. Keying gives separation, the expiry gives renewal, and together they mean the store holds only the present window's truth about each identity.

High-level architecture

Every request enters through the API gateway, and the rate limiter lives inside that gateway, in front of the app servers, so every request is checked in one place. The gateway asks the shared counter store to check and count the arrival, and the store answers allowed or refused. An allowed request is forwarded to the app servers, which never learn that a limiter exists, and a refused request is answered by the gateway itself with status 429 and a Retry-After hint. Many gateway machines can run side by side, because none of them keeps a private count: the counts live in the one shared store below them.

┌──────────┐   request    ┌───────────────┐  allowed  ┌─────────────┐
│  Client  │ ──────────▶ │ API gateway   │ ───────▶ │ App servers  │
└──────────┘              │ + rate limiter│           └─────────────┘
     ▲                    └───────────────┘
     │ 429 + Retry-After        │ check: read + atomic increase
     └────────────────────────▼
                          ┌───────────────┐
                          │ Shared counter│
                          │ store         │
                          └───────────────┘

The request flow

1. The client sends a request, and the API gateway receives it before any app server sees it
2. The gateway builds the identity key for the arrival: the signed-in user, or the network address, or the access key
3. The gateway asks the shared counter store to increase that key's count as one atomic step and report the new count
4. If the new count is at or under the limit of 100 requests per minute, the gateway forwards the request to the app servers
5. If the new count passes the limit, the gateway refuses the request itself with status 429 and a Retry-After hint in seconds
6. Counts from finished windows expire by TTL, so the store only ever holds current state

The placement is what makes the guarantees possible. Because the gateway stands in front, a refused request stops there and the app server only ever sees allowed requests, so over-limit load costs the guarded service nothing. Because the counter store is shared, the count an identity faces is the same whichever gateway machine receives the request, so a caller cannot multiply a limit by spreading requests across machines. Because the store increase is atomic — performed as one single step, with no gap between reading the count and writing it back — two gateways arriving together cannot lose each other's updates.

limit = 3
counts = {}
app_log = []
arrivals = ["ana", "ana", "ana", "ana", "ben"]

for user in arrivals:
    counts[user] = counts.get(user, 0) + 1
    if counts[user] <= limit:
        app_log.append(user)
        print(user, "passed the gateway")
    else:
        print(user, "stopped at the gateway")

print("app server saw", len(app_log), "requests")

The gateway checks each arrival before anything is forwarded, and the refused arrival stops at the gateway while the allowed ones pass. The app server's own tally at the end is the proof of placement: it saw only the allowed requests, and the refusal cost it no work at all. Placement, not the counting rule, is what turns a limit from a measurement into a guard.

limit = 3
counts = {}
app_counts = {}
arrivals = ["ana", "ben", "ana", "cy", "ana", "ben", "ana", "cy"]

for user in arrivals:
    counts[user] = counts.get(user, 0) + 1
    if counts[user] <= limit:
        app_counts[user] = app_counts.get(user, 0) + 1
        print(user, "request passed")
    else:
        print(user, "request refused")

for user in ["ana", "ben", "cy"]:
    print(user, "reached the app server", app_counts.get(user, 0), "times")

Three users arrive at one gateway in the closing model of the ask, each with a separate count, and only the busiest user is refused. The final tallies show exactly what each user sent through, which is the clarified ask running end to end: name who is limited, check in the gateway, count per user, and refuse the excess where it arrives. Every functional requirement lands on a component in the diagram: checking on the gateway, counting on the shared store, allowing on the path to the app servers, and refusing on the gateway's own response path.

Deep dive: the token bucket

A token bucket is a store of tokens that fills at a steady rate and pays one token for each request. The bucket holds up to a stated capacity, tokens arrive at the refill rate, and a request that finds a token spends it and passes, while a request that finds the bucket empty is rejected until a new token arrives. For the worked limit, the refill rate is 100 tokens per minute and a full bucket holds 100 tokens, so the long-run pace is the limit itself while the stored balance is what a burst spends.

The bucket's behaviour has two halves that pull in opposite directions, and both are deliberate. A bucket holding 100 tokens lets a burst of 100 requests pass at once, which suits real callers whose traffic arrives in clumps rather than on a metronome. Once the burst has drained the bucket, the pace falls to the refill rate and no faster, so the average over any long stretch never exceeds the limit. The bucket therefore enforces a rate, not a rhythm: it caps how much arrives over time, without punishing a caller for arriving unevenly inside that allowance.

tokens = 5

for i in range(1, 8):
    if tokens > 0:
        tokens -= 1
        print("request", i, "accepted, tokens left", tokens)
    else:
        print("request", i, "rejected, tokens left", tokens)

The bucket starts full in the model, with a small capacity of 5 tokens standing in for the production capacity, and each request spends one token. The first arrivals drain the balance token by token until the bucket stands empty, and the later requests are rejected because there is nothing left to spend. Nothing is borrowed and nothing goes negative: an empty bucket is a hard stop, and only the passage of time can soften it.

tokens = 0
last_time = 0
arrivals = [(0, "request 1"), (1, "request 2"), (2, "request 3"), (5, "request 4")]

for time, name in arrivals:
    tokens = min(5, tokens + (time - last_time))
    last_time = time
    if tokens > 0:
        tokens -= 1
        print(name, "at second", time, "accepted, tokens left", tokens)
    else:
        print(name, "at second", time, "rejected, tokens left", tokens)

Refill is computed from elapsed time rather than dripped in by a timer. The model scripts time as data and earns one token per second up to its cap, and the balance grows between arrivals exactly by the seconds that passed. A request that arrives before any refill is rejected, later requests spend tokens as fast as they are earned, and a longer gap leaves a small surplus waiting. The same arithmetic at the worked rate earns 50 tokens in 30 seconds, which is why a drained production bucket is half full again half a minute later.

Three counting rules compared

tokens = 5
last = 0
arrivals = [0, 0, 0, 0, 0, 0, 0, 1, 1, 2, 5]

for i, t in enumerate(arrivals, 1):
    tokens = min(5, tokens + (t - last))
    last = t
    if tokens > 0:
        tokens -= 1
        print("request", i, "at second", t, "accepted")
    else:
        print("request", i, "at second", t, "rejected")

A scripted burst hits the bucket first and empties it, and then further arrivals land close together while the refill is still thin. The burst passes in full, the immediate followers are rejected, and the survivors afterwards pass one at a time at the refill pace. The shape is the signature of the rule: generous at the start of a burst, strict immediately after it, and exactly fair over any longer stretch, because every accepted request was paid for by a token that time had genuinely supplied.

capacity = 5
tokens = 5
last = 0
arrivals = [0, 0, 0, 0, 0, 0, 0, 1, 2, 3, 4, 10]
accepted = 0
rejected = 0

for t in arrivals:
    tokens = min(capacity, tokens + (t - last))
    last = t
    if tokens > 0:
        tokens -= 1
        accepted += 1
        print("second", t, "accepted, tokens left", tokens)
    else:
        rejected += 1
        print("second", t, "rejected, tokens left", tokens)

print("accepted", accepted)
print("rejected", rejected)

The closing model runs the whole cycle in one pass. A burst spends the full bucket, the next requests are rejected until refills arrive, later requests pass at the refill pace, and a long quiet gap lets the balance rebuild towards capacity. The accepted and rejected totals at the end summarise the bargain: most arrivals pass, the excess is refused promptly, and no stretch of the run lets the sustained pace exceed what the refill rate allows.

The verdict is the token bucket as the default counting rule. Real traffic is bursty, and a rule that forgives a burst while capping the sustained rate matches how callers actually behave, which keeps refusals for the callers who are genuinely over pace. The cost is spoken plainly: a full bucket releases its whole capacity at once, so a burst of 100 requests can legitimately arrive in a single moment at the worked limit, and a design that cannot tolerate that moment must choose a window rule instead and pay its own costs.

Deep dive: fixed and sliding windows

A fixed window is a counting period with a set start and end, such as one minute, and a count that resets at each new period. A sliding window is a counting period that moves with time, covering the most recent stretch up to now. The worked limit under either rule is 100 requests inside one window, and the two rules differ in exactly one place: what happens at the edges of the period, which turns out to be where the limit is won or lost.

limit = 5
window = 0
count = 0
arrivals = [(10, 1), (20, 2), (30, 3), (40, 4), (50, 5), (55, 6), (70, 7)]

for time, n in arrivals:
    current = time // 60
    if current != window:
        window = current
        count = 0
    count += 1
    print("request", n, "at second", time, "accepted" if count <= limit else "rejected")

Requests are counted inside the current window only, and the model shows the reset doing its work. Arrivals fill the count towards the small demo limit, one arrival passes the limit and is rejected, and the last arrival falls in a new window, so its count starts again from the bottom. The reset is the rule's simplicity and its weakness in a single mechanism: the past is forgotten completely at the boundary, whether it ended a second ago or a minute ago.

limit = 5
counts = {}
arrivals = [56, 57, 58, 59, 59, 60, 60, 61, 62, 63]
passed = 0

for t in arrivals:
    w = t // 60
    counts[w] = counts.get(w, 0) + 1
    if counts[w] <= limit:
        passed += 1
        print("second", t, "accepted")
    else:
        print("second", t, "rejected")

print("passed in total", passed)
print("first at second", arrivals[0], "last at second", arrivals[-1])

The boundary flaw appears when a burst straddles the reset. A burst lands just before a boundary and another just after it, and each half stays within its own window, so every request passes inside a short span. At the worked limit the same shape passes 200 requests in a few seconds: 100 requests at the end of one minute plus 100 at the start of the next, each half legal in its own window. The rule was obeyed at every instant and the limit was still doubled, which is the sharpest failure a counting rule can have.

limit = 5
log = []
arrivals = [56, 57, 58, 59, 59, 60, 60, 61, 62, 63]

for t in arrivals:
    log = [x for x in log if x > t - 60]
    if len(log) < limit:
        log.append(t)
        print("second", t, "accepted, in window", len(log))
    else:
        print("second", t, "rejected, in window", len(log))

A sliding log closes the hole by remembering when each recent request arrived. The store keeps the time of each recent request, and the count covers only times inside the moving period, so late arrivals still see the earlier burst inside the period and are rejected. The same arrivals that the fixed window passed in full are held to the limit here, because no arrival gets to pretend the burst a few seconds earlier belongs to a forgotten past. Exactness is bought with memory: one stored time per recent request, held for as long as the window can still see it.

arrivals = [56, 57, 58, 59, 59, 60, 60, 61, 62, 63]
limit = 5
counts = {}
fixed_passed = 0

for t in arrivals:
    w = t // 60
    counts[w] = counts.get(w, 0) + 1
    if counts[w] <= limit:
        fixed_passed += 1

log = []
sliding_passed = 0

for t in arrivals:
    log = [x for x in log if x > t - 60]
    if len(log) < limit:
        log.append(t)
        sliding_passed += 1

print("fixed window passed", fixed_passed)
print("sliding window passed", sliding_passed)
print("limit per window", limit)

The closing model runs the same scripted burst against both rules in turn. The fixed form passes the whole burst across its boundary, and the sliding form holds the burst to the limit, with the two totals printed side by side. Nothing else differs between the runs — same arrivals, same limit — so the gap between the totals is purely the price of the reset. A rule choice that looks like a detail of bookkeeping is, measured this way, a choice about whether the stated limit is the real limit.

Fixed window against sliding window

The verdict is the sliding window wherever the limit must hold over every stretch of time, and the fixed window only where a boundary burst of twice the limit is genuinely harmless. Most limits exist to protect a downstream resource that feels the burst, not the average, so the default leans to the sliding form despite its storage cost. Weighed against the token bucket, the sliding window is the stricter and more literal reading of “100 requests per minute”: the bucket caps the sustained rate and forgives bursts up to capacity, while the sliding window caps every minute-long stretch an arrival can see. The choice between them is a choice about which promise the limit is making.

Deep dive: where the counters live

One gateway machine could keep its counts in its own memory, and the design would work until the second gateway machine started. The moment traffic spreads across a fleet, a count kept privately on one machine understates the truth by everything the other machines have seen, and an identity can multiply its limit simply by being routed around the fleet. The counters therefore live in one shared counter store: a single fast store that holds the counts for every gateway machine, building on the cache idea of a small, fast, shared place for hot values.

Sharing the store is not enough on its own, because the obvious way to update a shared count is broken. The obvious way is to read the count, add one, and write the result back, and it fails whenever two machines interleave: both read the same value, both add one, and both write back the same increased value. One of the two increases is lost, the count drifts below the truth, and the real limit creeps above the stated limit by exactly the number of lost updates.

store = {"ana": 100}
read_by_a = store["ana"]
read_by_b = store["ana"]
store["ana"] = read_by_a + 1
store["ana"] = read_by_b + 1

print("two requests arrived")
print("final count", store["ana"])

Two gateway machines read the same count of 100 before either writes back, and each writes back its own increase. Two requests arrived, and the store ends at 101, not 102, because the second write overwrote the first instead of building on it. The numbers are the whole argument: a lost update is not a rounding error or a rare curiosity, it is a guaranteed outcome whenever two updates race in the read-then-write style, and busy identities race constantly.

store = {"ana": 100}

def atomic_add(key):
    store[key] = store[key] + 1
    return store[key]

print("machine A adds, count", atomic_add("ana"))
print("machine B adds, count", atomic_add("ana"))
print("final count", store["ana"])

An atomic counter removes the gap the race lives in. An atomic counter is a counter whose increase the store performs as one single step, so no other update can slip in between the read and the write. Both machines add to the same key in the model, and the final count keeps both increases, ending at 102 as it should. The gateways no longer coordinate with each other at all; the store serialises the increases, and exactness becomes a property of the store rather than a hope about timing.

Three ways to keep the count

store = {}
limit = 4

def atomic_add(key):
    store[key] = store.get(key, 0) + 1
    return store[key]

arrivals = [("gateway 1", "ana"), ("gateway 2", "ana"), ("gateway 3", "ben"), ("gateway 1", "ana"), ("gateway 2", "ben"), ("gateway 3", "ana"), ("gateway 2", "ana")]

for gateway, user in arrivals:
    count = atomic_add(user)
    print(gateway, user, "count", count, "accepted" if count <= limit else "rejected")

print("final counts", store)

Three gateway machines share one store in the closing model, and every increase happens as a single atomic step on the user's key. The counts stay exact across machines even though arrivals interleave between gateways, and the busiest user is rejected only after passing the shared limit. The model is small, but nothing about it changes at fleet scale: the store, not the routing, is what makes many gateways behave as one limiter.

The verdict is one shared counter store with atomic increases, keyed per identity, with TTL expiry on every record. The shared store makes the count a single truth, the atomic increase keeps that truth exact under concurrency, the per identity key keeps users from sharing budgets, and the TTL keeps finished windows from accumulating forever. The admitted cost is dependence: the store is now on the path of every request, so the design owes an answer for the day the store itself is slow or down, and that answer is the failure policy covered next.

Bottlenecks and failure modes

The first component to saturate is the shared counter store, because it is the one component every request touches. Each arrival costs the store one atomic increase and one read of the result, and those operations are tiny, but they are also uncacheable: a count is only worth reading at the moment it is written, so no copy of it can be served from anywhere else. As the guarded service grows, gateway machines can be added freely, yet every added gateway only presses harder on the same store, which is why the store is chosen for raw speed on small operations above every other virtue.

The limiter itself can fail, and the design must say what requests do while it is down. Failing open lets requests pass unchecked while the limiter is down, so the service stays reachable without its guard. Failing closed blocks requests while the limiter is down, so the guard holds and the service takes the outage. Neither answer is a bug or a virtue in itself; each one trades a different risk, and the trade has to be made deliberately, per service, before the outage arrives.

arrivals = ["ana", "ben", "ana", "cy", "ben"]

for mode in ["fail open", "fail closed"]:
    passed = 0
    for user in arrivals:
        if mode == "fail open":
            passed += 1
    print(mode, "passed", passed, "of", len(arrivals))

The same arrivals face each policy in turn while the limiter is down, and the gap between the two choices is total. Failing open passes every arrival, and failing closed passes none of them, with no middle setting available in the moment itself. The policy is therefore a property of the design, fixed in advance, because an outage is the worst possible time to improvise it.

The cost of failing open can be priced exactly at the worked limit. An outage that lasts 2 minutes can let a single user send 200 requests past a failed limiter, because 100 requests per minute times 2 minutes is 200 requests that no count ever sees. For most services that price is acceptable for a short outage: the traffic is real user traffic, merely uncounted, and the alternative is refusing every user for the sake of a guard. For a service whose limit protects something fragile or expensive, the same arithmetic argues for failing closed and accepting the outage instead.

• The counter store slows down: every request waits on its check, so the gateway sheds load by failing open where the policy allows, rather than queueing checks behind a struggling store
• The counter store is unreachable: the failure policy decides alone — requests pass unchecked under fail open, or are blocked under fail closed, until the store returns
• One identity floods the gateway: the flood is refused at the gateway from the first request past the limit, so app servers see none of it and other identities are unaffected
• A gateway machine dies: its in-flight requests fail, but no counts are lost, because no count ever lived on the machine; the remaining gateways keep checking against the shared store
• Clocks disagree between machines: window boundaries and refill maths drift slightly per machine, which is one more reason the count itself lives in one store rather than being reconstructed per gateway

limit = 3
counts = {}
arrivals = ["ana", "ana", "ana", "ana", "ana", "ben", "ben"]

for user in arrivals:
    counts[user] = counts.get(user, 0) + 1
    print(user, "accepted" if counts[user] <= limit else "rejected")

print("ana count", counts["ana"])
print("ben count", counts["ben"])

Fairness is the failure mode that never announces itself as a failure. One user sends well past the personal limit in the model while another sends a little, and the heavy user is refused while the lighter user still passes in full. Without per identity counts, the heavy user's flood would consume shared capacity and the light user's ordinary requests would fail alongside it. The per user limit converts a potential outage for everyone into a refusal for exactly the identity causing it.

Trade-offs

The decisions and their costs

limit = 3
counts = {}
arrivals = [("ana", False), ("ana", False), ("ana", False), ("ana", False), ("ana", True), ("ben", True)]

for user, limiter_down in arrivals:
    if limiter_down:
        print(user, "limiter down, passed by fail open")
        continue
    counts[user] = counts.get(user, 0) + 1
    if counts[user] <= limit:
        print(user, "status 200")
    else:
        print(user, "status 429, retry after 30 seconds")

print("ana counted", counts.get("ana", 0))

The closing model runs the finished design as one small story. Requests are counted in a shared per user tally, the over-limit request is refused with status 429 and a wait hint, and then the limiter goes down and requests pass by failing open. Each beat of the run is one of the decisions in the table, exercised in sequence: the scope, the refusal contract, and the failure policy, all visible in a handful of arrivals.

Taken together, the design spends a fast shared store and a synchronous check on every request, and buys exact per identity limits that no routing accident can dilute and no single heavy user can turn into everyone's outage. The finished shape clarifies the ask, checks each request in the gateway, counts in one shared store with atomic increases, and answers over-limit requests with a clear refusal that says when to try again. A quieter service can give pieces of that back — a fixed window instead of a log, a private count instead of a shared store — as long as it names the burst or the inexactness it is accepting in exchange.

Common questions

How does a token bucket rate limiter work?

A token bucket is a store of tokens that fills at a steady rate and pays one token for each request. At the worked limit the bucket refills at 100 tokens per minute up to a capacity of 100 tokens, so a caller who has been quiet can spend a saved balance on a burst, while a caller who keeps arriving finds only the tokens the refill has supplied. A request that finds a token passes; a request that finds the bucket empty is rejected until a new token arrives, which holds the long-run pace to the refill rate without demanding an even rhythm.

Why can a fixed window pass 200 requests in a few seconds?

A fixed window counts inside a period with a set start and end, and the count resets at each new period. A caller can send 100 requests at the end of one minute and 100 more at the start of the next, and each half is legal inside its own window, so 200 requests pass inside a few seconds even though the stated limit is 100 requests per minute. A sliding window closes the hole by counting over the most recent stretch of time instead of a fixed period, at the cost of storing one time per recent request.

Where should rate limiter counters live?

The counters live in one shared counter store that every gateway machine reads and updates, keyed per identity so users never share a count. Sharing matters because a private count on each gateway would let an identity multiply its limit by spreading requests across machines. The update matters just as much: the increase must be atomic, performed by the store as one single step, because the read-then-write style loses updates under concurrency — two machines adding one each to a count of 100 leave the store at 101, not 102.

What happens when the rate limiter itself goes down?

The design chooses in advance between failing open and failing closed. Failing open passes requests without a limit check while the limiter is down, keeping the service reachable at the price of an unguarded interval: an outage of 2 minutes can let one user send 200 requests past a failed limiter at the worked limit. Failing closed blocks requests instead, holding the guard at the price of the service itself, and suits limits that protect something too fragile to expose even briefly.

Related designs: the API gateway placement and the shared store return in Design a URL Shortener, Design a Chat System, Design a News Feed, Design Autocomplete, Design a Notification System and Design a Key-Value Store, and the capstone, Design a Social App, places a rate limiter of exactly this shape at its front door. Each of those designs feels the same pressure from the other side: a limiter is only worth designing well because the services behind it are worth protecting.

The estimate at a glance