Design a News Aggregator

The ask

A source is a publisher feed or page that the service checks for new reports. Readers follow sources and subjects.

The service collects new reports soon after publication. Readers see one stream ordered by time and grouped by story.

Many sources report the same story. The stream shows one entry per story, with the sources listed.

Readers search past reports and open the original at the source. The service stores text for grouping and search.

Estimate the scale

The service polls 100,000 sources.

Each source is polled every 15 minutes, and 15 minutes is 900 seconds.

100,000 / 900 = about 111 polls per second.

New reports arrive at 10,000,000 a day.

10,000,000 / 86,400 = 116 reports per second.

API and data

Reader operations list the stream, list one story group, search by words and follow a source or subject.

A source record holds the feed address, the last poll time and the last seen identity. Poll health is stored with the source.

A report record holds the source, the headline, the published time, the text and a story group identity.

A content fingerprint is a short value computed from a report's text. Equal values mark equal text.

High-level design

A poll scheduler queues sources by their next due time. Fetch workers poll the due sources.

A processing worker computes the fingerprint, checks for duplicates and assigns a story group. Clean reports enter the report store.

A read service serves the stream from a cache. Search runs over an index of headlines and text.

Polling and reading are separate paths. A polling backlog never slows the reader stream.

Deep dive: grouping one story

Exact copies share one fingerprint and collapse at once. Wire copy republished by many outlets is the common case.

Fresh reports of one story differ in wording. Grouping then compares headline words, named people and places, and the published window.

A group keeps the earliest report as its lead entry. Later sources join the group's source list.

A wrong grouping is visible and fixable. Editors and reader reports can split a group, and the split is stored as a rule.

Polling and freshness

Fast sources earn the stated polling rhythm. Quiet sources keep the same rhythm in the base design, for simplicity and fairness.

A failed poll is retried with growing gaps. A source that keeps failing is marked unhealthy and polled less often.

A poll that finds nothing costs one small response. Feeds that report their last change let the worker skip unchanged sources.

The reader stream

The stream orders story groups by their newest report. A fresh report lifts its whole group.

Followed subjects filter the stream at read time. One stored stream serves every reader's filter.

The stream page is cached in short slices. New groups enter at the head as slices refresh.

Evaluation and trade-offs

A news aggregator is a polling engine feeding a grouped stream. Fingerprints catch copies, and grouping catches retellings.

Faster polling freshens the stream and loads small sources. The fixed rhythm keeps that load predictable.

Aggressive grouping hides distinct reports. Loose grouping floods the stream with copies of one story.

Storing full text powers search and grouping. Storage grows daily, so old reports move to cheaper storage.

Aggregator scale