Feature Stores

What a feature store is

A feature store is a shared store that holds features under one definition each, ready for training and for live answers. Teams register a feature once. Every model then reads the same values.

Without a shared store, each team computes its own copy of the same feature. The copies slowly fall out of step with each other. Models trained on one copy meet another copy in live use, and quality falls for reasons nobody can see.

The store also records where each feature came from. Source, owner and update schedule sit beside the values. A wrong value can be traced to its source.

Estimate the scale

The worked design holds features for 200,000,000 users. Each user has 1,000 stored features.

200,000,000 x 1,000 = 200,000,000,000 stored feature values.

One stored value takes 8 bytes. 200,000,000,000 x 8 = 1,600,000,000,000 bytes, which is 1.6 TB in the online store.

One live read fetches one user's values. 1,000 x 8 = 8,000 bytes, which is 8 KB per read.

Two stores, one definition

The offline store holds feature history. Training reads it in large scans that build training sets covering months of examples. Slow, cheap storage suits it.

The online store holds the latest value of each feature for each user. Live reads fetch one user's values in a few milliseconds. Fast, costlier storage suits it.

Both stores serve the same registered definition. One computation writes both stores. Two computations would recreate the mismatch the store exists to prevent.

Point-in-time correctness

Point-in-time correctness means a training example carries only the feature values that were known at the moment the example happened. Nothing from later may slip in.

Take an example from a purchase on Monday. The feature counting purchases in the last week must be computed as it stood on Monday. A value computed on Friday includes purchases the model could not have known about.

Future values in training make scores look excellent. The live model never sees the future, so live quality falls far below the training scores. The offline store prevents this by keeping dated history and joining each example to the values in force at its moment.

Freshness and cost

Features go stale at different speeds. A user's age changes yearly. A user's last action changes by the second. The update schedule follows the speed of the underlying fact.

Frequent updates cost compute and writes. Infrequent updates cost accuracy. Each feature's schedule is a stated choice, recorded beside its definition.

Stored values carry their age. A live read can see that a value is old. The caller then decides whether to use it, refresh it or fall back to a default.

Failures and trade-offs

A broken update job leaves values frozen. Freshness checks alarm when a feature stops moving. The model keeps answering with old values until the job is fixed, so the alarm matters.

A store outage must not take answers down with it. Callers fall back to default values or to a cached copy. Quality drops, but the product stays up.

One shared store couples many teams to one system. That coupling is the price of one shared definition. The alternative, private copies with quiet mismatches, costs more in lost quality.

The two stores