Design File Sync

Clarify the ask

The service promise is fixed before any machinery is chosen.

The core promise is simple. A file saved on one device appears on the other devices without manual copying.

A device is a phone, a laptop, or a tablet. One person may use several devices in one day.

Sync is short for synchronisation. It means keeping copies of the same file equal to each other.

Estimate the scale

The estimate uses the figures fixed for this problem.

10,000,000 active users × 5 changed files a day = 50,000,000 file changes a day.

50,000,000 changes ÷ 86,400 seconds = 579 changes per second on average.

An average file is 1 MB. Most changes are therefore small, but a few files are far larger.

The average rate looks gentle. Peaks, large files, and reconnecting devices make the real load spikier.

Name the parts

A client app is the software on a device that watches the synced folders. It notices each new or changed file.

A sync service is the server side that receives changes. It tells the other devices that a new version exists.

A metadata store is a database that holds file details, not file content. Details include the name, the owner, and the current version.

Blob storage holds the file content itself. Blob storage is storage for large unnamed pieces of data.

Name the hard parts

Two devices can change the same file while both are offline. The design must not silently lose either change.

A large file should not travel in full after a tiny edit. Cutting files into pieces avoids that waste.

Sharing adds another hard part. A shared file must reach the right people and nobody else.

Sync, conflicts, storage, offline work, and trade-offs each answer one of these hard parts.

What success looks like

Success means no saved change is lost. A slower sync is acceptable, and a lost file is not.

Success also means the copies agree in the end. Every device should reach the same latest version.

File sync keeps the same files current on every device a person uses. That promise, kept at this scale, is the whole design.

The high-level design

The high-level design shows the whole sync flow, from the watching client to the notified devices.

The client app watches the synced folders. A change starts the sync flow on that device.

The client cuts the changed file into chunks. It then asks the service which chunks the service still needs.

The client uploads only the missing chunks. The service updates the metadata and notifies the other devices.

Each other device downloads the missing chunks. It rebuilds the file in the right order.

How a chunk is named

A chunk name is a checksum of the chunk content. A checksum is a short value worked out from data, so changed data gets a different value.

The same content always gives the same checksum. Two identical chunks therefore share one name on every device.

The metadata store keeps the ordered list of chunk names for each file version. The order rebuilds the file.

A chunk is one fixed-size piece of a file. The name makes each piece recognisable without sending the piece itself.

The chunk arithmetic

An average file is 1 MB. It fits inside one chunk, so most files are a single chunk.

A 10 MB file needs 3 chunks. Two chunks hold 4 MB each and the last chunk holds the rest.

20 MB ÷ 4 MB = 5 chunks.

100 MB ÷ 4 MB = 25 chunks.

Only the missing chunks travel. A small edit in a large file can therefore cost one chunk of upload.

One copy of each chunk

The service stores each different chunk once. Many files can point at the same stored chunk.

A copied file needs no new storage at first. Its metadata simply lists the same chunk names.

This sharing of chunks is called deduplication in plain terms here. The design stores one copy and references it many times.

Deleting a file removes its metadata first. A background job later removes chunks that no file references.

Where chunking strains

An edit near the start of a file can shift every later byte. The chunk boundaries then move and many chunks look new.

This design accepts that cost. The shifted chunks upload again, and correctness stays simple.

Tiny files strain the design in the other direction. Metadata can cost more effort than the single chunk it describes.

A failed upload leaves the old version in place. The new version becomes current only when every chunk has arrived.

A deep dive on conflicts

Conflicts are the hardest sync problem. Vector clocks are the tool for spotting them.

A vector clock tracks who wrote each version, and how many times.

A vector clock is a list of counters, one counter per writer. Each writer raises its own counter on each save.

Every saved file version carries its vector clock. The clock travels with the metadata for that version.

How the comparison works

One clock is ahead of another when every counter is at least as high. At least one counter must also be strictly higher.

An ahead clock means a later version. The later version replaces the earlier one with no conflict.

Two clocks can each be ahead in a different counter. Neither version then came after the other.

Versions that neither follow nor precede each other are called concurrent. Concurrent versions of one file are a conflict.

A worked conflict

A report starts at one shared version on a laptop and a phone. Both devices then go offline.

The laptop saves a new draft and raises the laptop counter. The phone saves different edits and raises the phone counter.

Both devices reconnect and upload. Each clock is ahead in its own counter and behind in the other.

The service compares the clocks and sees concurrency. It keeps both versions instead of picking a silent winner.

How a conflict is resolved

The design keeps the first version under the file name. It keeps the second version as a clearly marked conflicted copy.

The person who owns the file compares the copies. The person keeps, merges, or deletes them by hand.

Last-write-wins is a rule that keeps the newest write and drops the rest. This design rejects it for file content.

The reason is silent loss. A clock time can look newer while the other version holds the only copy of real work.

Where clocks strain

A shared file with many writers carries a long clock. The clock grows with the number of writers, not with the file size.

An old client may send a version with no clock. The service treats that version with care and never assumes it is the latest.

A replaced device starts with a fresh counter. The service gives it a new writer entry rather than reusing the old one.

A conflict is two versions changed from the same earlier version. The clock comparison is how the design knows, instead of guessing.

Design the data model

The data model holds file details, chunk lists, folders, and sharing grants.

A file record holds the file name and the folder. It also holds the owner and the current version clock.

The record holds the ordered list of chunk names. That list is the recipe that rebuilds the file.

A folder record holds its name and its parent folder. Folders form a tree that the client shows to the user.

The main operations

The create operation takes a file name, a folder, and a chunk list. It returns the new file record.

The list-changes operation takes a device marker and returns newer versions. The marker keeps the answer to one page at a time.

The share operation takes a file, a person, and a role. It returns a grant that the service can check and revoke.

The revoke operation takes a grant and ends it. Access stops from that moment on every device.

The storage arithmetic

This design assumes each user stores 2 GB on average. That figure is a stated assumption of this design.

10,000,000 users × 2 GB = 20,000,000 GB of user data.

20,000,000 GB is 20 PB of user data before deduplication.

The design assumes one quarter of all chunks are duplicates of other chunks. Deduplication therefore leaves 15 PB of different data.

The design keeps 3 copies of each different chunk for safety. 15 PB × 3 copies = 45 PB stored in total.

How sharing works

The owner can read, change, share, and delete.

An editor can read and change.

A viewer can read only.

A grant is a stored record that names a person, a file or folder, and a role. The service checks the grant on every open.

A shared folder grants access to everything inside it. New files in the folder inherit the folder grant.

A share link is a grant tied to a hard-to-guess address instead of a named person. Links are weaker, so this design gives links an expiry time.

Sharing is a controlled grant to another named person. The grant, not the file copy, is what the owner controls.

Where storage and sharing break

Losing the metadata is worse than losing one chunk. Chunks without their ordered lists cannot rebuild any file.

The metadata store therefore gets the strongest backup in the design. Its copies sit in more than one data center.

A forwarded share link can reach unintended people. Expiry and revoking limit that damage.

A removed editor may keep an old downloaded copy. The design controls future access, not copies already taken.

A deep dive on offline work

Offline work and delta sync decide what a reconnecting device sends.

An offline device keeps working on local copies. Local means held on the device itself, with no network needed.

Each offline save joins a pending list on the device. The pending list holds the changed files in save order.

Take a person who edits 20 files offline. At an average of 1 MB a file, the pending list holds 20 MB of changed files.

How reconnecting works

The device reconnects and sends its pending list in order. Order matters because later saves may build on earlier ones.

The service checks the version clock of each incoming save. A save based on the current version is accepted at once.

A save based on an older version may be a conflict. The conflict rules then apply.

The device clears each pending save only after the service confirms it. A crash during upload therefore loses nothing.

The delta arithmetic

Delta sync is sending only the parts of a file that changed. Chunks make those parts easy to name.

A 100 MB file has 25 chunks at 4 MB per chunk.

A one-line edit changes the text inside one chunk. The other chunks keep their checksums and stay on the server.

The device sends the 1 changed chunk, which is 4 MB of data.

100 MB − 4 MB = 96 MB saved on that edit. The device sends 25 times less data than a full upload.

When the saving shrinks

An average file is 1 MB and fits in one chunk. A one-line edit in such a file still sends about 1 MB, the whole chunk.

The saving grows with file size. Large files gain the most because one edit touches a small share of their chunks.

A packed or encrypted file changes almost everywhere after one small edit. Delta sync then sends nearly the whole file.

Encryption here means scrambling content so only a key holder can read it. This design encrypts in transit and leaves chunking on plain content.

Where offline sync breaks

The server may no longer hold the base version a delta was built against. The device then falls back to a full upload.

A very long offline period can build a long pending list. The device syncs in save order and shows progress as it goes.

Two offline devices can edit the same file for days. Delta sync moves the changes cheaply, and the conflict rules decide the outcome.

Evaluate the design

The evaluation starts from the opening estimate.

The design still rests on the opening estimate. It handles 579 changes per second on average, with peaks above that rate.

Each choice trades one good property against another. The verdicts state what this design picks and why.

Chunk size, both ways

Small chunks send less data after a small edit. A 1 MB chunk sends 1 MB where a 16 MB chunk sends 16 MB.

Small chunks also multiply the bookkeeping. A 100 MB file needs 100 chunks at 1 MB per chunk, and 7 chunks at 16 MB per chunk.

This design keeps chunks at 4 MB. That size keeps a 100 MB file at 25 chunks and a small edit at 4 MB sent.

The middle size is the balance this design chooses.

Conflicts, both ways

Keeping a conflicted copy is simple and safe. Nobody loses work, and a person decides which copy survives.

Automatic merging would feel smoother when it works. A wrong merge can corrupt a file in ways nobody notices at once.

This design keeps conflicted copies. It trades a little user effort for a guarantee against silent loss.

Consistency, both ways

Consistency here means whether all copies agree on the current version. The metadata needs one agreed answer to that question.

Chunk content can arrive more loosely. A missing chunk only delays one file, and the version does not advance without it.

This design therefore guards the small metadata strictly. It lets the large chunk transfers stay simple and restartable.

The wrap-up

The design syncs changes, not whole folders. Chunks, checksums, and version clocks do the heavy lifting.

The main weakness is the shifted-chunk edit. An early edit can force many chunks to upload again.

File sync keeps the same files current on every device a person uses. This design keeps that promise without silent loss, and that is the verdict.

The ask in short