Design Ad Click Aggregation

The ask

Ad click aggregation counts clicks on advertisements and groups the counts for reporting and payment. Advertisers pay per click, so the counts are money.

The raw events arrive as a continuous flow, day and night. Reports group the flow by advertiser, campaign, ad, and minute.

A total that is too high overcharges the advertiser. A total that is too low underpays the platform and the publisher.

Exact counting is therefore the core requirement. Speed matters, but a fast wrong total is worthless.

Estimate the scale

The ad system's impressions run at 10,000,000,000 a day.

10,000,000,000 impressions / 86,400 seconds = 115,741 impressions per second.

The click rate is 1% of impressions. One impression in a hundred becomes a click.

10,000,000,000 impressions x 1% = 100,000,000 clicks a day.

100,000,000 clicks / 86,400 seconds = 1,157 clicks per second.

Aggregation runs per minute. A day holds 1,440 minutes, from 24 hours x 60 minutes.

100,000,000 clicks / 1,440 minutes = about 69,444 clicks land in each minute's count.

One raw click record takes 200 bytes as a design figure. Raw records stay small, yet their daily total still grows fast.

100,000,000 clicks x 200 bytes = 20 GB of raw click records a day.

API and data

Record click takes a click ID, an ad ID, a campaign ID, and a timestamp. The click ID is created once, when the click first happens.

Get totals takes an advertiser, a time range, and a grouping. It returns stored totals, never a fresh scan of raw clicks.

A click record holds the click ID, the ad, the campaign, the publisher page, and the time. The record is small and never changes after it lands.

An aggregate record holds one count for one ad for one minute. A second record rolls minute counts into one hourly count per campaign.

With 100,000 campaigns as the design figure, stored hourly totals stay compact. The roll-up writes far less than the raw flow.

100,000 campaigns x 24 hours = 2,400,000 stored hourly totals a day.

High-level design

Clicks land first in an append-only click log. The log is the source of truth, and nothing is counted before it is logged.

Workers read the log in order and update per-minute counters. Each counter belongs to one ad and one minute.

Counters are sharded by ad ID. One ad's clicks always reach the same shard, so its count has one writer path.

A roll-up job folds finished minute counts into hourly and daily totals. Reports read only the rolled-up totals.

The raw log is kept after counting. Any disputed total can be recounted from the log, click by click.

Stream processing

Stream processing means computing on events as they arrive, one after another, without waiting to collect a large batch. The click counters work exactly this way.

Each click is handled once, in seconds. It is logged, checked, and added to its minute counter.

The alternative is a nightly batch over the whole day's clicks. A batch delays every report by hours and rescans data the stream already handled.

Stream processing keeps reports about a minute behind reality. Advertisers watch a campaign spend its budget during the day, not after it.

The stream never stops, so workers track their position in the log. A restarted worker resumes from its last position instead of starting over.

Deep dive: duplicates and late clicks

Networks retry, and workers crash after counting but before recording progress. The same click can therefore arrive at a counter twice.

Exact counting survives retries through the click ID. Each minute's counter keeps the set of click IDs it has counted, and a repeated ID adds nothing.

The counted-ID set lives only while the minute is open and shortly after. Old sets are dropped once the minute's recount window has passed.

Late clicks are clicks that arrive after their minute has closed. A slow phone can deliver a click minutes after the tap.

A grace period keeps a closed minute adjustable for a short time. Late clicks inside the grace period update the minute and its roll-ups.

A click later than the grace period becomes a correction record. Reports show the correction on the day it arrived, linked to the day of the click.

Payment uses the corrected totals only. Raw clicks, counted IDs, and corrections together make every paid total auditable.

Fraud clicks and filtering

Some clicks are not people. Automated programs click ads to drain an advertiser's budget or to earn fake publisher income.

Filtering runs before counting for payment. Clicks from known automated sources are marked invalid and kept out of the paid totals.

Invalid clicks stay in the raw log, marked as invalid. Deleting them would destroy the evidence an advertiser needs during a dispute.

Reports show valid and invalid counts side by side. The advertiser sees what was filtered, not just a smaller total.

When parts fail

A counter worker that crashes stops at its last log position. Another worker takes over from that position, and click IDs stop any double count.

A full shard delays its ads' reports, not the whole system. Other shards keep counting their own ads.

A lost aggregate is rebuilt from the raw log. The log is kept long enough to recount any period still open to disputes.

Reports degrade to older totals during a failure. Showing a slightly old exact total beats showing a fresh guess.

Evaluation and trade-offs

Per-minute exact counting trades storage and care for trust. Counted-ID sets and corrections cost real space, and they make every paid total provable.

The per-minute grain trades freshness against volume. Finer grains would multiply aggregate records without helping advertisers decide faster.

Keeping the raw log trades storage for recount power. At 20 GB a day, the log is cheap insurance against counting bugs.

The design fits because the flow is steady and the grain is fixed. About 69,444 clicks land in each minute's count, and sharding by ad spreads them evenly.

Scale of the click stream