Training Pipelines and Serving

From data to a live model

The pipeline runs the same steps every time. It takes a training set, trains a model, checks the model and releases it. Recorded steps make a run repeatable months later.

Every run produces a new model version. The version records its training set, its settings and its check scores. A bad release can then be traced and replaced.

Automation runs the steps in order. A failed check stops the line. No model reaches users by passing silently around a failed check.

Training at scale

One training set holds 1,000,000,000 examples. Training reads 1,000,000 examples per second.

1,000,000,000 / 1,000,000 = 1,000 seconds for one pass over the data.

One full pass is called an epoch. 1,000 seconds is about 17 minutes per epoch, since 1,000 / 60 = 17 when rounded.

Three epochs take 3 x 1,000 = 3,000 seconds. 3,000 / 60 = 50 minutes of reading alone.

A checkpoint is a saved copy of the model's state during training. A machine failure restarts from the last checkpoint, not from the beginning. Checkpoints turn a long run from fragile to routine.

Training-serving skew

Training-serving skew is a mismatch between the data a model learns from and the data it receives when it answers live requests. The model was prepared for one world and meets another.

A common cause is two computations of the same feature. Training computes a weekly count one way. The live path computes it another way, perhaps over a different window.

The shared feature store is the main defence. One definition feeds both sides. The remaining defence is a check that compares live inputs against training inputs and alarms on differences.

Model serving

Model serving is running a trained model so it can answer live requests. The serving path loads one model version, gathers the features and returns a prediction inside a latency budget.

Copies of the model run on many machines behind a load balancer. Traffic growth adds copies. A failing copy leaves rotation while the rest keep answering.

A new version starts on a small share of traffic. Its errors and latency are compared against the current version. Only a clean comparison earns it the full traffic.

Batch prediction

Batch prediction is computing predictions ahead of time for many cases at once and storing the answers for later use. A nightly job can score every user once. The live path then reads a stored answer instead of running the model.

Batch prediction suits answers that change slowly and requests that cannot wait. It wastes work on cases nobody asks about. Live serving suits answers that depend on the moment, at the price of running the model on the request path.

Many systems mix the two. Slow-changing parts of the answer are precomputed. The final step runs live with fresh facts.

Retraining

Retraining is training the model again on newer data so it can learn how the world has changed. New items, new behaviour and new seasons all leave the old model behind.

Retraining runs on a schedule, or when live quality falls below a stated line. Each run follows the same pipeline and the same checks as the first release. A retrained model is a new version, never a quiet patch.

The old version stays available during the changeover. If the new version misbehaves live, traffic returns to the old one in minutes. Rollback is part of the design, not an improvisation.

Failures and trade-offs

Bad data poisons retraining quietly. A broken source can teach the new version a wrong pattern that checks on the same broken data cannot see. Input checks compare each new training set against the last good one.

Bigger models train slower and serve slower. The pipeline makes the cost visible per version. A small quality gain that doubles serving cost is a trade the numbers must justify.

Freshness trades against stability. Retraining daily follows change closely and ships more chances to break. Retraining monthly is calmer and staler. The schedule follows how fast the underlying behaviour moves.

Training arithmetic