Machine Learning System Design

What makes ML different

Machine learning is a way of building programs that learn patterns from past examples. The examples come first. The program's behaviour follows from them.

An ML model is the trained program that carries the learned patterns. Training data is the set of past examples the program learns from.

A label is the known right answer attached to one example. A prediction is the model's answer for a new case.

Data is therefore a design surface, not a detail. Weak examples produce a weak model, however strong the servers are.

The ask

Start with the decision the system supports. A spam filter blocks mail. A scoring system orders items for a screen.

Ask who uses the answer and how fast it must arrive. Ask what a wrong answer costs. A missed fraud and a missed song carry very different costs.

Ask what the system may never do. Some answers need a person to approve them. Some cases must fall back to a safe default.

Estimate the scale

Put rough numbers on traffic and data early. The numbers decide whether one machine is enough or a fleet is needed.

Say a product sends 10,000,000 prediction requests a day.

10,000,000 / 86,400 = 116 requests per second on average.

Peak traffic sits well above the average. The design states its peak assumption plainly and sizes for it.

Data and measured facts

Name where the examples come from. User actions, uploaded files and business records are common sources. Each source has gaps and delays.

A measured fact is one recorded value about a case. Examples are its size, its age and its source. The model learns from measured facts paired with labels.

Ask how labels are produced and checked. Wrong labels teach wrong patterns. A small set of carefully checked labels can beat a large careless set.

The model and training

Training is the process that adjusts the model until its predictions match the labels well. The model is one component in the system, not the whole system.

Start with a simple model. A simple model exposes data problems early. A complex model hides them behind its own failures.

Keep some examples aside for checking. The model never learns from the checking examples. Honest scores come only from examples the model has not memorised.

Serving predictions

Serving is the live path that takes a request, gathers the measured facts and returns a prediction. The path runs under a latency budget.

Cache answers that repeat. Precompute answers that change slowly. Both moves cut cost and latency.

Plan the fallback before launch. A slow model, a missing fact or an unsure model should trigger a safe default answer, not an error page.

Evaluation and trade-offs

Check the model on held-back examples before release. Then watch its live answers against real outcomes. Both checks answer different questions.

Accuracy trades against latency and cost. A slightly weaker model that answers in time often serves users better than a stronger model that answers late.

An ML design is judged as a whole system. Data, model, serving path and checks all carry part of the result.

The interview steps