Design a Distributed Logging System

The ask

Every server writes log lines as it works.

A log line is one short record of one thing that happened.

One request can leave lines on many servers.

Reading the lines server by server is slow and painful.

The system gathers all lines into one store that can be searched.

Estimate the scale

The fleet is the same 10,000 servers the monitoring system watches.

Each server writes 1,000 log lines per second.

10,000 servers x 1,000 lines = 10,000,000 lines per second.

One line takes 200 bytes.

10,000,000 lines x 200 bytes = 2,000,000,000 bytes per second.

2,000,000,000 bytes per second = 2 GB per second.

2 GB per second x 86,400 seconds = 172,800 GB a day.

172,800 GB a day is about 173 TB a day.

API and data

Each line carries the time it was written.

Each line carries the server name and the service name.

Each line carries a level, like information or error.

Each line carries the message itself.

A request carries one request name through every server it touches.

That name ties the lines of one request together.

Each line also carries a sequence number from its source.

High-level design

A small agent runs on every server.

The agent reads new lines and sends them onward in batches.

Batches enter a message queue first.

The queue smooths bursts from the fleet.

Writers take batches from the queue and store the lines.

A search front reads the store and answers questions about the lines.

Gathering the lines

Log aggregation is the gathering of log lines from many sources into one place.

Aggregation lets one search cover the whole fleet.

An agent keeps unsent lines on its local disk.

A brief outage downstream therefore loses nothing.

Batches are confirmed only after the queue holds them.

An unconfirmed batch is sent again.

A line sent twice is stored once. Its source and sequence number name it.

Searching the store

Most searches name a span of time and a service.

The store is split by time so old spans are skipped fast.

An index on the request name finds one request at once.

Full word search over every line is offered but costs more.

Recent lines are kept in hot storage for fast search.

About 173 TB a day x 7 days = about 1.2 PB of hot storage.

Older lines move to cheaper storage or are dropped under set rules.

Deep dive: one ordered stream per source

Order is kept inside one source, not across all sources.

Each source numbers its own lines with a sequence number.

The numbers grow by one for each new line from that source.

Stored lines are read back in that number order for the source.

A gap in the numbers shows a missing line.

Clocks differ across servers, so times alone cannot order two sources.

Lines from two sources are placed by time only roughly.

The log is the record

Event sourcing is the pattern where the stored log is the record and current state is rebuilt by reading the log from the start.

The log here works the same way. Lines are added and never changed.

What a server did can be rebuilt by reading its lines in order.

Floods and failures

A sick server can write lines far faster than a healthy one.

Agents cap the rate they send from any one server.

Extra lines wait on local disk up to a set size.

Past that size the oldest unsent lines are dropped and the drop is counted.

The queue absorbs a burst while writers catch up.

The store keeps copies so one dead machine loses no stored lines.

Evaluation and trade-offs

One store for all lines makes hard questions answerable.

The price is moving and storing a huge stream every day.

Hot storage is fast and costly. Cheap storage is slow to search.

Seven days hot is the balance the design takes.

Exact order across sources is not promised.

Exact order inside each source is promised and kept by its numbers.

Scale at a glance