Design a Distributed File System

The ask

The ask is storage for very large files shared by many machines. Analysis jobs read whole files, often in order.

Files are written once and read many times. Small edits in the middle are rare.

Machines are ordinary and failures are normal. The design must keep files whole despite constant small failures.

One cluster stores 100 PB. No single machine comes close to that size.

Estimate the scale

Files are cut into blocks of 128 MB. A block is one fixed-size piece of a file.

1 GB = 1,000 MB of file.

1,000 MB / 128 MB a block rounds up to 8 blocks for a 1 GB file.

Each block is stored 3 times on different machines.

8 blocks x 3 copies = 24 stored block copies for that file.

API and data

Callers open a file by its path. A path is the name that places a file inside folders.

Read takes a position and a length and returns bytes. Write appends data at the end of a file.

The system records facts about every file. Name, size, and the list of its blocks form the record.

Block locations change as machines come and go. The record is updated whenever a copy moves.

High-level design

A metadata server holds the record of every file. The record lists each block and the machines holding its copies.

Data servers hold the blocks themselves. Many data servers can share one cluster.

A caller asks the metadata server for block locations first. The caller then talks straight to the data servers.

Data flows machine to machine, never through the centre. The centre stays a directory, not a bottleneck.

Writing a file

The writer asks for space for new blocks. The metadata server names three machines for each block.

The writer sends each block to the first machine. That machine passes the block along to the next copy holder.

Copies confirm receipt back along the chain. The writer reports success only after the copies confirm.

A failure mid-write leaves one block short of copies. The shortfall is visible in the record and gets repaired.

Reading a file

The reader fetches the block list for the file. Blocks are read in order, each from its nearest copy.

Reading straight from data servers spreads the load. No central machine touches the file bytes.

A checksum rides with each block, a short code that exposes damage. A damaged block is detected on read and fetched from another copy.

Whole-file reads run at disk and network speed. Small random reads are not the strength of the design.

Deep dive: keeping blocks whole

Machines report their blocks at startup and at intervals. Reports let the metadata server spot missing copies.

A block under its copy target gets a fresh copy. A data server with spare room receives it.

Over-copied blocks lose their extra copies. Balance, not maximum copies, is the goal.

Copying competes with caller traffic for the network. Repairs run at a limited pace during busy hours.

When machines fail

A data server that stops reporting loses its place. Its blocks still exist on the other copy holders.

The metadata server is a single point of worry. Its records are copied to a standby machine that can take over.

Checks that a server answers run from the metadata server. Missed answers start the copy repair, not panic.

Losing every copy of one block loses that part of one file. Three copies on separate machines make that event rare.

Evaluation and trade-offs

Large blocks suit large files and long reads. Small files waste record space and crowd the metadata record.

One metadata server simplifies every decision. Its memory bounds how many files the cluster can hold.

Appends are easy and safe for writers at the end of a file. Editing the middle of a file is not offered.

Files live whole across many ordinary machines. Ordinary failures stay ordinary events, not disasters.

One file in numbers