Design Visual Search

The ask

A shopper uploads a photo of an item, such as a shoe or a lamp.

The service returns the same item when it sells it. It returns near matches when it does not.

A filter is a hard rule a match must pass. Filters narrow the matches by colour, size or stock status.

A photo search must feel fast, because shoppers compare it with typing.

Estimate the scale

The catalogue holds 1,000,000,000 photos. An embedding is a list of numbers that captures what a photo shows.

Photos that look alike get lists that sit close together. Each embedding takes 4,000 bytes.

1,000,000,000 × 4,000 bytes = 4,000,000,000,000 bytes. That is 4 TB of vectors.

100,000,000 visual queries arrive a day. 100,000,000 ÷ 86,400 = 1,157 queries per second, rounded to the nearest whole number.

Peak load is set at 3 times the average for the design. 100,000,000 × 3 ÷ 86,400 = 3,472 queries per second at peak, rounded.

Data and features

Each training example pairs a photo with its known matches. The known matches are its labels, the right answers for that photo.

Pairs come from shopper behaviour, such as clicks and purchases after a search.

Hard pairs matter most, such as two chairs that differ only in leg shape.

A feature is one measured fact about a photo, such as its colour mix. The model learns most facts itself from the picture.

A data pipeline moves new catalogue photos into the store as items are added or changed. Each new photo gets an embedding and a place in the vector index.

The vector index is the store that finds the closest embeddings fast.

High-level design

An app server receives the photo and checks its size and type.

A model turns the photo into an embedding. The list uses the same format as the catalogue lists.

A nearest neighbour is one of the closest embeddings in the index.

The vector index returns the top 100 neighbours; filters and a second, heavier scoring pass cut them to the top 20 shown.

Blob storage holds the catalogue photos themselves. A cache holds results for popular photos, so a repeat search can skip the index.

A queue holds new catalogue items until a worker builds their embeddings.

Training the model

The model learns from pairs of photos that should sit close or far apart.

Training moves matching embeddings closer together. It moves non-matching embeddings further apart.

Clicks can mislead, because a shopper may click a poor match out of curiosity.

People check a held-back set of matches by hand, so the test stays honest.

A new model replaces the old one only when it wins on that held-back set.

Serving predictions

Serving uses the same photo preparation as training.

A different preparation would shift every embedding. That mismatch is training-serving skew, and it spoils the matches.

The vector index is sharded across machines. Each shard holds one slice of the catalogue.

Each shard searches its own slice, and a merger keeps the best 100 overall.

If one shard is slow, the answer uses the other shards and still shows 20 matches.

A new embedding model needs a fresh index built from its embeddings. The new pair takes over only when the fresh index is complete.

Deep dive: similar listings

A listing page can show other listings that look like the one on screen.

The listing's own photo gives the query embedding, with no upload needed.

Attribute filters then narrow the neighbours by size, place and condition.

The listing set changes fast, as items sell and new ones appear.

Sold items must leave the index quickly, or shoppers tap dead listings.

The scoring pass weighs photo likeness against distance and freshness. The core stays unchanged, because only the index and the filters differ.

Deep dive: landmark recognition

Landmark recognition names a known place in a traveller's photo.

The catalogue is fixed and small next to the shop catalogue. It holds one entry per known place.

Each place holds several photos, taken from different angles and in different light.

The same vector search returns the closest places. The top match names the place.

The service sets a closeness bar for naming a place. A photo under the bar gets no name.

A wrong name is worse than no name. New places join rarely, so the index rebuild is cheap and can run on a schedule.

Evaluation and trade-offs

Offline evaluation uses held-back photos with human-checked matches. It counts how often a true match appears in the top 20 shown.

Online experiments compare click and purchase rates between two models.

A model can win offline and lose online, because shopper taste is part of the truth.

Bigger embeddings capture more detail and cost more storage and search time.

A heavier scoring pass lifts match quality and adds delay.

The design favours match quality inside the funnel, because only 100 items reach the heavy pass.

Visual search at a glance