Design a Price Tracking Service

The ask

A price tracking service watches prices so buyers do not have to. A user watches a product, and the service reports every price drop and rise.

The service watches prices at many stores for the same product. A drop at any watched store is news for the user.

Price history is part of the answer. A drop means little without the prices that came before it.

The service reports prices only. Setting prices belongs to the stores, and their pricing choices stay outside the design.

Estimate the scale

The service tracks 100,000,000 products, the size of the store's catalogue. Each product is checked once an hour as the design choice.

100,000,000 products x 24 checks = 2,400,000,000 checks a day.

2,400,000,000 checks / 86,400 seconds = 27,778 checks per second.

Checks are spread evenly across each hour. Even spread keeps the fleet and the stores under a steady load.

100,000,000 checks / 60 minutes = about 1,666,667 checks start in each minute.

One check downloads 10 KB as a design figure. The fetched pages look small one at a time and enormous in total.

2,400,000,000 checks x 10 KB = 24 TB of pages fetched a day.

API and data

Watch product takes a user ID and a product link. It returns the current price and starts the hourly checks.

Get price history takes a product ID and a time range. It returns the stored price points, oldest first.

A product record holds the store, the product link, and the last seen price. A price point holds the product, the price, and the time it was seen.

A watch record holds the user, the product, and a target price. The target price is optional and sharpens the alert when the user sets one.

Price points are written only when the price changes. An unchanged check costs a fetch and no storage row.

High-level design

A scheduler holds the next check time for every tracked product. Due checks enter a queue, spread across the hour by their scheduled times.

Fetcher workers take checks from the queue and download the product page. Each fetcher handles one store's pages at a polite, limited rate.

A reader extracts the price from the fetched page. Pages from one store share a layout, so one reader serves every product from that store.

The seen price is compared with the last stored price. A changed price writes a price point and wakes the alert path.

The alert path finds the watchers of the product and sends each one a notification. Watchers with a target price are alerted when the new price meets the target.

Deep dive: checking without harming stores

Politeness means the checker limits its load on each store it visits. The crawler's politeness rule applies here at full strength.

Checks for one store are capped per second, however many products that store carries. A store with a small site feels a small, steady load.

A store that slows down gets fewer checks, not more retries. Backing off protects the store and keeps the fleet free for healthy stores.

Failed checks retry later inside the same hour. A store that stays down is marked stale, and its prices are labelled with their last seen time.

Products watched by many users are checked first when the fleet falls behind. Popular products drive most alerts, so they hold their hourly rhythm longest.

Unwatched products can slip to a slower rhythm under load. Their history gains gaps before popular products gain any.

Changes and history

Most checks find no change. As a design figure, 5% of products change price on a given day.

100,000,000 products x 5% = 5,000,000 price changes found a day.

Only changes write price points, so history storage follows the changes, not the checks. The 24 TB fetch stream narrows to a small stored stream.

A price that flickers up and down is stored at every sighting. The history shows the flicker honestly instead of smoothing it away.

A missing product is a state, not a price of zero. The record marks the product gone, and alerts say the product vanished rather than dropped.

When parts fail

A fetcher failure delays its checks inside the queue. Other fetchers take later checks, and the delayed ones run when capacity returns.

A reader failure on a changed store layout produces wrong prices. Sudden impossible prices are held for checking instead of alerting users.

A scheduler failure pauses new checks while stored history stays readable. Users see labelled stale prices, never invented ones.

Evaluation and trade-offs

Hourly checks trade freshness for a fetch load the stores can bear. A watched price can be almost an hour old, and the history says exactly how old.

Storing only changes trades completeness for size. The service cannot prove a price held between two sightings, only that both sightings agreed.

Per-store politeness caps trade coverage for goodwill. Stores that welcome the checker stay checkable, and blocked checking helps nobody.

The design fits because the work is steady and splittable. Checks spread by product and by store, at 27,778 per second, all day.

Scale of the checking