Design an Object Store

The ask

The ask is storage for whole objects at very large scale. Photos, backups, and videos are typical objects.

Callers store an object under a name and fetch it by that name. Nobody edits part of an object in place.

Durability matters most. Losing an object is the failure to prevent.

Objects range from tiny to huge. The design treats every object the same way.

Estimate the scale

The store holds 10,000,000,000 objects. The average object takes 1 MB.

10,000,000,000 objects x 1 MB = 10,000,000,000 MB = 10 PB of logical data.

New objects arrive at 100,000,000 a day.

100,000,000 objects / 86,400 seconds = 1,157 new objects per second.

100,000,000 objects x 1 MB = 100 TB of new logical data a day.

API and data

Put takes a name and the object's bytes and stores the object. Get takes a name and returns the whole object.

Delete takes a name and removes the object. A list operation returns names under a shared prefix.

Each object carries small facts about itself. Size, a checksum, and the storage locations ride along.

A checksum is a short code that exposes damage in stored data. A damaged fragment fails its checksum and counts as lost.

Names live in a metadata store. The metadata store is a small database that maps each name to its fragment locations.

High-level design

A request reaches a gateway server first. A gateway server is the entry point that checks the name and finds the object's record.

Writes split the object into fragments. Fragments spread across many storage servers.

Reads gather enough fragments and rebuild the object. The caller sees one whole object, never the fragments.

Storage servers hold fragments on disks. Disks are cheap and large, and objects are read far less often than cached data.

Deep dive: erasure coding

Erasure coding is a protection method that splits data into fragments and adds parity fragments computed from them.

A parity fragment is an extra fragment computed from the data fragments. It stands in for a lost fragment during a rebuild.

Each object becomes 10 data fragments and 4 parity fragments. 10 + 4 = 14 fragments per object.

Any 10 of the 14 fragments rebuild the object. Four fragments can be lost without losing the object.

14 fragments / 10 data fragments = 1.4 times the storage. 10 PB x 1.4 = 14 PB stored.

100 TB of new data a day x 1.4 = 140 TB stored a day.

Rebuilds and repairs

Disks fail every day at this scale. The system watches fragment counts for every object.

A lost fragment triggers a rebuild. Surviving fragments recreate the missing one on a fresh server.

Repairs run in the background at a calm pace. A repair flood must never crowd out caller reads.

Checksums expose silent damage. A damaged fragment is treated as lost and rebuilt.

Deletes and space

A delete first marks the object's record as removed. Callers can no longer fetch the object.

Space returns when the fragments are removed in the background. Batching removals keeps a delete cheap.

A removed name can be reused. A reused name points at the new object only.

Consistency choices

A finished put is durable before success is reported. The caller never hears success for fragments not yet stored.

A read right after a put returns the new object. Name records update in one place, so reads agree.

Lists of names can lag slightly. A brand-new name may take a moment to appear in a listing.

Evaluation and trade-offs

Erasure coding buys durability at 1.4 times the storage. Full copies would cost more space for similar safety.

Rebuilds cost computation and network traffic. That cost lands on repairs, not on normal reads.

Whole-object storage stays simple for callers. Partial edits stay impossible, and callers replace objects instead.

The store keeps whole objects safe across many disks. Safety, not speed, is the point of the design.

Storage in numbers