Design a Data Infrastructure for ML

What the infrastructure does

A data pipeline is a chain of steps that moves data from its source to a place where it can be used. Each step changes the data a little. Cleaning, joining and storing are common steps.

A feature is one measured fact fed to a model. The number of items a user bought last week is a feature. So is the hour of the day.

The infrastructure serves two masters. Training needs large, correct sets of past examples. Live answers need fresh values, fast.

Estimate the scale

The worked design has 100,000,000 users. Each user produces 100 events a day.

100,000,000 x 100 = 10,000,000,000 events a day.

10,000,000,000 / 86,400 = 115,741 events per second.

One event takes 500 bytes. 10,000,000,000 x 500 = 5,000,000,000,000 bytes, which is 5 TB of raw events a day.

Raw events are kept for 90 days. 5 x 90 = 450 TB of raw storage.

Where data comes from

Events stream in from apps and servers as they happen. Business records arrive in daily loads from other systems. Uploaded files arrive whenever users send them.

Every source lands in raw storage first, unchanged. Raw records can be replayed when a later step turns out to be wrong. Processed records cannot be unprocessed.

Each record carries the time it happened and the time it arrived. The two times differ when devices work offline or loads run late. Training cares about the time it happened.

Cleaning and checking

Raw data contains duplicates, missing values and broken records. Cleaning removes duplicates and repairs or drops broken records. The rules are written down and versioned like any other part of the system.

Checks run at every step. Counts that jump without a cause stop the pipeline. Bad records are set aside for inspection, never silently mixed in.

Personal data needs care at this stage. Fields the model does not need are removed early. Access to raw records is limited and logged.

Labelling

Human labelling is the work of people reading examples and attaching the right label to each one. Some labels come from user behaviour instead. A purchase after a click is a label no person had to write.

One labeller labels 1,000 examples a day. Say the goal is 1,000,000 labelled examples.

1,000,000 / 1,000 = 1,000 labeller-days.

A team of 50 labellers finishes in 1,000 / 50 = 20 days. The same team labels 50 x 1,000 = 50,000 examples a day.

Quality is checked by overlap. Two labellers label some of the same examples. Disagreements go to a senior labeller, and the guidance is updated so the same disagreement stops recurring.

Building the training set

A training set is the collection of examples, with their features and labels, used to train one model version. It is a snapshot. Later corrections create a new snapshot, never a quiet edit of the old one.

The snapshot is split by time. Older examples train the model. Newer examples check it. A time split mimics the real task, which is always predicting the future from the past.

Near-duplicates must not straddle the split. The same user or the same item on both sides inflates the check scores. The live scores then disappoint.

Failures and trade-offs

Sources change without warning. A renamed field or a new app version can halve the event count overnight. Count checks catch the break before training consumes it.

Late data is a standing condition, not an exception. The pipeline accepts late records and rebuilds the affected days. Training snapshots state their cut-off time.

Keeping everything costs storage. Deleting early costs the ability to replay. The worked design keeps raw events for 90 days and processed data far longer, because processed data is smaller and already cleaned.

Scale of the worked design