Design Content Moderation

The ask

A service receives photos, videos, text and live streams from its users. Some items break the service rules.

Content moderation is deciding which items may stay and which must come down. Every new item needs that decision.

Decisions must be fast for live items. A harmful stream harms viewers while it runs.

Decisions must also be fair. Removing honest speech angers the people the service needs.

Estimate the scale

The service receives 1,000,000,000 new items a day.

1,000,000,000 ÷ 86,400 = 11,574 items per second.

People cannot check that flow item by item. The model must decide almost everything alone.

The model decides 99% of items alone.

1,000,000,000 × 99 ÷ 100 = 990,000,000 items decided by the model each day.

Data and features

Features depend on the item type. Text gives words, images give visual patterns, and video gives both.

Other features describe the account and the setting. Examples are account age, past removals and audience size.

Labels come from past decisions. A label says allowed, removed or limited for one item.

Reports from users add fresh labels. A report is a hint, not proof, and reports can be gamed.

Rules change as harms change. Training data must track the current rules, not old habits.

High-level design

An uploaded item enters a queue. Workers pull items from the queue for checking.

Cheap checks run first. Known harmful files are matched by their content fingerprint.

The model then scores the item for each harm category. Examples are spam, violence and sexual content, and each category carries its own threshold.

Clear passes are published at once. Clear violations are removed at once.

Human review is a check by a trained person. Uncertain items go to human review, where a person sees the item and its context.

The person's decision is final for that item. The decision also becomes a new label.

Training the model

Training data comes from past human decisions. Those decisions carry the rules as people applied them.

Disagreements between reviewers are valuable. They mark the borderline cases the model finds hardest.

New harms appear with new slang and new events. Fresh labelled examples are gathered for them quickly.

Offline evaluation measures each category on held-back items of its own. A strong average can hide a failing category.

Errors in training data teach errors at scale. A wrong rule, once learned, repeats on millions of items.

Serving predictions

Serving runs at two speeds. Uploads are checked before publication, and live streams are checked while they run.

A fast model screens every item. A heavier model rechecks items near a threshold.

Scores near a threshold go to human review. Scores far from a threshold are decided by the model alone.

Removed items can be appealed. An appeal goes to human review with the original context attached.

Every decision is logged. Logs feed training, audits and appeals.

Deep dive: hate speech

Hate speech is harmful content aimed at people for who they are. It is one category of harmful content on the same pipeline.

A text model handles this category. The model reads words and their order rather than pixels.

Context decides many cases. A quoted insult inside a news report differs from the same words aimed at a person.

Slang, spelling tricks and coded words shift constantly. Attackers adapt their spelling to dodge the model.

Reclaimed words add a harder edge. A word used inside a community can wound when aimed from outside.

Uncertain hate speech cases go to human review. Reviewers see the whole conversation, not one line alone.

Evaluation and trade-offs

1% of items go to human review.

1,000,000,000 ÷ 100 = 10,000,000 items a day.

One reviewer clears 1,000 items a day.

10,000,000 ÷ 1,000 = 10,000 reviewers.

Speed pulls against care. Faster queues protect viewers and rush the hardest decisions.

Strictness pulls against free expression. Removing more protects some readers and silences others.

Scale at a glance