Design Pastebin

The ask

A paste is one stored block of text with its own link. The owner creates a paste and shares the link.

A reader opens the link and sees the text. Reading needs no editing rights.

A paste may carry an expiry time. A paste may also be unlisted, so only the link reaches it.

Writes are modest and reads are heavy. The design is shaped by that gap.

Estimate the scale

10,000,000 pastes are created a day.

10,000,000 / 86,400 = 116 pastes per second.

The average paste is 10 KB.

10,000,000 x 10 KB = 100 GB of new pastes a day.

Reads run at 1,000,000,000 a day.

1,000,000,000 / 86,400 = 11,574 reads per second.

API and data

The create operation takes the text, an expiry choice and a visibility choice. The response returns the short link.

The read operation takes the short code in the link. The delete operation takes the code and the owner's delete key.

A paste record holds the short code, the creation time, the expiry time and the visibility. The record also holds a reference to the stored text.

A short code is a brief string that names one paste. The URL shortener's code shape is the same idea.

High-level design

A write service creates the short code and stores the text in blob storage. The paste record goes into a database sharded by code.

A read service looks up the code and returns the text. Hot pastes are served from a cache in front of storage.

An expiry worker removes pastes whose time has passed. A removed paste returns a clear gone response.

Codes and storage

Codes are generated before use and handed out one by one. A handed-out code is never reused for another paste.

The code names the shard and the record. A read needs no search across shards.

Text lives in blob storage because reads fetch it whole. The database keeps only the small record.

Deep dive: one paste read by everyone

A shared link can turn one paste into a hot paste within minutes. Every reader asks for the same bytes.

The cache holds hot pastes in memory. Repeated reads end at the cache and never reach storage.

A cache miss fetches the text once and fills the cache. Many waiting readers share that one fetch.

A deleted or expired hot paste is removed from the cache first. Stale copies must not outlive the paste.

Expiry, privacy and abuse

Expiry is checked on read as well as by the expiry worker. An expired paste is never served from a stale record.

Unlisted pastes are hidden from listings and search. The link itself is the only path in.

Create requests are rate limited per address. Size limits keep one paste from becoming file storage.

Reports on public pastes go to a review queue. Confirmed harmful pastes are removed and their codes are retired.

Evaluation and trade-offs

Pastebin is a read-heavy store behind short links. The cache and the code lookup carry the design.

Blob storage keeps large text cheap. Small records keep code lookups fast.

Unlisted links are convenient and weak as privacy. Anyone holding the link can read the paste.

Expiry frees storage and breaks old links. The gone response keeps that break honest.

Pastebin scale