Embeddings and Vector Search

What an embedding is

An embedding is a list of numbers that captures what an item is like, so items alike in meaning or appearance get lists that sit close together. A model produces the list from the item's content, such as its text or its image.

No single number in the list has a name a person would recognise. The pattern across all the numbers carries the meaning. Closeness between two lists is the useful signal.

One embedding holds 1,000 numbers. Each number takes 4 bytes. 1,000 x 4 = 4,000 bytes, which is 4 KB per embedding.

Nearest neighbours

A nearest neighbour is the item whose list sits closest to the list being compared. Closeness is measured as a distance between the two lists. Small distance means the items are alike.

A photo of a red chair sits close to other chairs and far from houses. The distances do the sorting. No keyword needs to match.

Search by embedding therefore finds items alike in meaning, not just items sharing words. A query about seating can find a chair whose description never uses that word.

Vector search

Vector search is the search for the nearest neighbours of one embedding inside a large collection of embeddings. The collection lives in a vector index, a store organised for this search.

The query is turned into an embedding by the same model that embedded the collection. The index compares the query list against the stored lists. The closest few come back as the answer.

Filters often follow the search. Price, place or availability cut the neighbours down to items the user may actually choose. The search finds alike items. The filters enforce the rules.

The index at scale

Take a collection of 10,000,000 items. At 4 KB per embedding, the vectors alone take 10,000,000 x 4,000 = 40,000,000,000 bytes.

40,000,000,000 / 1,000,000,000 = 40 GB of vectors.

An exact scan compares the query against every list. The scan is simple and always right, but its cost grows with every item added. Large collections cannot afford it on a live path.

An approximate index groups the lists so a search checks only the likely groups. A few true neighbours may be missed. In exchange, the search returns in milliseconds instead of seconds.

Keeping the index fresh

New items are embedded and added as they arrive. Deleted items are removed, or filtered out at read time until removal catches up. A stale index returns items that no longer exist.

The embedding model version matters. Lists from two different model versions do not sit in the same space, so their distances mean nothing. A new model version means re-embedding the whole collection and swapping the index in one move.

The swap needs both indexes alive at once for a while. Storage cost doubles briefly. The alternative, mixing versions in one index, corrupts every distance in it.

Failures and trade-offs

Approximate search trades a little accuracy for a lot of speed. The trade is tuned per product. A safety-related search tunes toward accuracy. A browsing search tunes toward speed.

Embeddings inherit the gaps of their training data. Items unlike anything in training get poor lists and poor neighbours. Checking neighbour quality per category of item exposes the gaps.

Meaning-based search and keyword search fail in different ways. Many systems run both and merge the answers. Each covers the other's blind spots.

Exact search against an approximate index