Design Trending Topics

The ask

Trending topics tell readers what the world just started talking about. The list answers with phrases, ranked by how fast their use is rising.

A trend is a rise, not a total. The most-used phrases overall are often steady background words that nobody calls news.

The list refreshes as fast as conversation moves. A list built from yesterday's posts describes yesterday.

The service tracks the top 100 phrases per window as the design choice. Readers see the head of the list, and the head must be right.

Estimate the scale

The social system's posts run at 500,000,000 a day. Every post is a possible carrier of a rising phrase.

500,000,000 posts / 86,400 seconds = 5,787 posts per second.

The window is 15 minutes. Fifteen minutes holds 900 seconds, from 15 x 60.

5,787 posts per second x 900 seconds = 5,208,300 posts per window.

Each window therefore carries millions of posts and far more phrases. Counting everything exactly, forever, is not a workable shape.

API and data

Get trends takes a region and a language. It returns the ranked phrases for the current window in that region and language.

A post record holds the text, the author, the region, the language, and the time. The trend counters read posts as they arrive.

A phrase count holds one phrase's count in one minute bucket for one region and language. Buckets are the raw material of every window.

A baseline holds one phrase's normal count per window. Baselines are built from many earlier windows at the same time of day.

A trend record holds the phrase, its window count, its baseline, and its rise. The record explains its own ranking.

High-level design

Posts enter an append-only post log. Trend workers read the log as a stream and pull phrases from each post.

Phrases are counted with approximate counting, region by region and language by language. The sketch keeps the counting memory small at full post speed.

Counts land in one-minute buckets. A window count is the sum of the buckets the window covers, so a new window needs no recount of old posts.

At each minute, the window slides forward. The newest bucket joins, the oldest bucket leaves, and candidate phrases are rescored.

Candidates with the highest rises get an exact count over the window's posts. The published list is ordered by those exact counts and rises.

Readers are served from a cache holding the current list per region and language. One list build serves every reader until the next minute turns.

Deep dive: what counts as trending

A rise compares the window count with the baseline for the same phrase. A phrase used twice its normal level is rising, whatever its total.

Raw volume alone picks the wrong winners. Everyday words and daily greetings would hold the list forever at their steady totals.

A minimum count guards the other end. A phrase jumping from one use to four uses has risen sharply and still matters to almost nobody.

The ranking score therefore joins both facts. A phrase must clear a real volume floor and rise far above its baseline.

Events explain most real trends. A goal, a storm, or a launch can lift a phrase within minutes. The window catches the lift while it is still news.

The same phrase can trend in one region and stay flat in another. Regions keep separate counts, baselines, and lists for exactly this reason.

Manipulation and spam

A trend is valuable attention, so coordinated accounts try to manufacture one. Many new accounts post the same phrase in the same minutes.

The counters weigh who is speaking. Posts from accounts created moments earlier count less toward a trend than posts from settled accounts.

Identical text posted many times counts once per account toward the phrase. Copying one message a thousand times is one voice, not a crowd.

A phrase pushed by a narrow cluster rises with a warning attached. Human reviewers can hold the phrase from the list while they check the cluster.

Removed spam posts leave the counts at the next window slide. The short window limits how long manufactured volume can haunt the list.

When parts fail

A worker failure leaves a gap in one region's buckets. That region's list is marked stale rather than silently built from partial counts.

A cache failure pushes readers to the last built lists. Trend lists tolerate a minute of staleness far better than an outage.

A post log backlog delays every trend together. The lists keep their window labels, so readers see which window a list truly describes.

Evaluation and trade-offs

A 15-minute window trades stability for speed. Real events surface within minutes, and the list also twitches more than a daily list would.

Approximate counting trades a small error for a workable memory. Exact recounts of the leading candidates keep the published order honest.

Rise against a baseline trades simplicity for meaning. A plain most-used list is easier to build and far less worth reading.

The design fits because the window does the forgetting. Old posts slide out every minute, and 5,208,300 posts per window stay countable as a stream.

Scale of the trend window