Design Ad Click Prediction

The ask

An advertiser offers an ad for a single slot on a page. Several ads compete for that slot.

Click-through rate is clicks divided by impressions. An impression is one showing of an ad.

The model predicts that rate for one person, one ad and one page. The prediction arrives before the page finishes loading.

A good prediction earns money from clicks that truly happen. A bad prediction shows ads nobody wants.

Estimate the scale

The service handles 10,000,000,000 ad impressions a day.

10,000,000,000 ÷ 86,400 = 115,741 impressions per second.

One impression is scored in about 10 milliseconds. Scoring sits inside page loading, so the budget allows no slow steps.

Suppose 2 of every 100 impressions earn a click. Two per hundred is 0.02 as a rate.

10,000,000,000 × 0.02 = 200,000,000 clicks a day.

Data and features

Each impression becomes one training example. The example records the person, the ad, the page and the moment.

Some features describe the ad itself. Examples are its topic, its advertiser and its past click rate.

Other features describe the person and the page. Examples are the device, the place and the page topic.

The label is simple. The label says clicked or not clicked for that impression.

Delayed labels are known answers that arrive late. A person may click an ad long after seeing it, so a fresh impression has no final label yet.

Recent examples therefore look worse than they are. Their clicks have simply not arrived yet.

High-level design

An ad request arrives at an app server. The server gathers the candidate ads for the slot.

A feature lookup fills in the stored facts for each candidate. The lookup reads from the feature store.

Scoring combines the bid with the predicted rate. The bid is what an advertiser pays for one click.

2 × 0.05 = 0.1.

A bid of 2 units times a predicted rate of 0.05 gives that score of 0.1. The highest score wins the slot.

The winning ad is shown. Every impression and click is logged for later training.

Training the model

Training uses 30 days of logged impressions.

30 × 10,000,000,000 = 300,000,000,000 examples in one training set.

Training waits for late clicks before it trusts recent days. Cutting the wait short teaches the model that fresh ads fail.

The model learns which feature patterns go with clicks. Rare ads get little evidence, so their estimates stay rough.

A new model is checked offline before it serves. Offline checks compare its predictions against logged outcomes.

Serving predictions

Serving reads the person's stored features at request time. The values must match the values training used, or training-serving skew quietly ruins accuracy.

The model scores each candidate inside the budget. Slow candidates are dropped rather than delaying the page.

Calibration means a predicted rate means what it says. Groups predicted at the same rate should click at that rate in real life.

10,000 × 0.05 = 500 expected clicks.

Poor calibration breaks the auction. Bids are multiplied by the rate, so an inflated rate overcharges advertisers.

Scores are logged with the impression. The log later shows whether each score was honest.

Deep dive: delayed labels

A click is not instant. A person may open the page, read for a while, and click much later.

Some clicks arrive through later visits or another device. Matching them back to the right impression takes time.

Training on today's data today would mark those pending clicks as misses. The model would learn to fear new ads and busy hours.

One fix waits a fixed window before an example enters training. Examples inside the window stay out until their labels settle.

Another fix trains on older complete days and treats recent days with care. The gap is accepted as the cost of honest labels.

Evaluation and trade-offs

Offline evaluation measures prediction error on held-back days. It also checks calibration group by group.

Online experiments give a new model a small share of impressions. Clicks and earnings are compared against the current model.

Accuracy pulls against speed. A larger model may predict better and still miss the scoring budget.

Freshness pulls against completeness. Training sooner uses fresher data with labels still missing.

The design favours honest rates over clever ones. The auction trusts the rate with real money.

Scale at a glance