Design a Data Export System

The ask

An owner asks for a copy of everything the account holds. The copy covers content, uploads, messages and settings.

The system gathers the data, builds an archive and sends a download link. The owner downloads the archive within a set window.

Exports are rare for one account and heavy in total. Live traffic must not suffer while exports run.

An archive holds personal data. Only the verified owner may request one or use its link.

Estimate the scale

100,000 export requests arrive a day.

100,000 / 86,400 = about 1 export request per second.

The average account holds 5 GB of data.

100,000 x 5 GB = 500,000 GB of export data a day.

500,000 / 1,000 = 500 TB of export data a day.

API and data

The request operation names the account and the data groups wanted. The response returns an export identity.

A status operation returns the state of one export. States are gathering, packaging, ready and expired.

An export record holds the account, the requested groups, the state and the archive size. The record also holds the link expiry time.

A manifest lists every part of the archive and its checksum. The manifest travels with the archive.

High-level design

A request service checks ownership and places an export job on a queue. The request returns at once.

Gather workers pull each data group for the account into a staging area. Each group is gathered by the service that owns it.

A packaging worker builds the archive in the staging area. A link service then issues the download link.

Deep dive: gathering across stores

Account data is spread across many stores. The job tracks one slice per store and marks each slice done as it lands.

A snapshot point marks which data belongs to the export. Changes after that point are left out, so the archive is one consistent moment.

A crashed job resumes from its completed slices. Finished slices are never gathered twice.

Checksums prove each slice arrived whole. A slice with a bad checksum is gathered again before packaging.

Packaging and delivery

Large archives are split into parts. A failed download restarts one part, not the whole archive.

The download link lives 7 days. After that window the staged archive is deleted and a fresh export is needed.

Downloads are served from object storage near the owner. Export traffic stays off the live app servers.

Privacy and load control

Requesting an export requires a fresh login. The link carries a long random token and works only inside its window.

Every request, download and expiry is logged for the account. The owner can see and cancel a pending export.

Export workers run under a strict share of each store's capacity. Live reads always win over export reads.

Evaluation and trade-offs

A data export system is a queued gathering job with a guarded delivery step. Consistency comes from the snapshot point.

Staging doubles storage for the data being exported. The link window bounds how long that cost lasts.

Resumable slices cost bookkeeping per job. Without them, one crash would restart a multi-hour gather.

Low priority keeps exports invisible to live users. Export waits grow on busy days, and the status page says so plainly.

Export scale