Time-Series Data

Define the problem

Monitoring produces a particular shape of data. Machines report measurements, the measurements are never edited afterwards, and nobody ever asks for one measurement by its own key.

Reads ask about ranges. What did memory do in the last hour, and how does this week compare with last week.

A general database spends much of its machinery on updates, transactions and single-row lookups. Time-series data uses almost none of it, and arrives at a rate that punishes waste.

Estimate the volume

A fleet has 10,000 servers, and each server reports 100 metrics, as stated demo figures.

One metric from one server is one series. The fleet therefore has 10,000 × 100 = 1,000,000 series.

Each series is sampled every 10 seconds. The fleet produces 1,000,000 series ÷ 10 seconds = 100,000 samples per second.

Over a day that is 100,000 samples per second × 86,400 seconds = 8,640,000,000 samples.

At 16 bytes per stored sample, one day of raw data is 8,640,000,000 × 16 bytes = 138,240,000,000 bytes, about 138 GB.

Keeping a year at that rate is out of the question. The design must spend detail to buy time, which is where downsampling comes in.

Storing by time

Writes are appends. Samples land in time order and are never changed, so storage is organised as time ranges, one after another.

Partitioning by time range is the natural shard key here. A query for the last hour touches only the newest ranges, and old ranges sit untouched on cheaper storage.

Whole ranges expire together. Dropping a month of old data is deleting its ranges, not hunting through live data for old rows.

Recent ranges stay on fast disks for dashboards. Older ranges move to slower, cheaper storage, because they are read rarely and in bulk.

Downsampling

Downsampling is the practice of keeping older data at lower resolution. Many samples are combined into one summary value, such as the average, the highest and the lowest for each minute.

Combining 10-second samples into one value per minute shrinks the data sixfold. The day's 8,640,000,000 samples become 8,640,000,000 ÷ 6 = 1,440,000,000 summary values.

Stored at 16 bytes each, the summarised day takes 1,440,000,000 × 16 bytes = 23,040,000,000 bytes, about 23 GB instead of about 138 GB.

The tiers run on a schedule. Raw samples are kept for days, minute summaries for months, and hour summaries for years.

Each tier answers the questions its age actually gets. Nobody debugging a live incident needs last year's data at 10-second detail.

Where it breaks

Summaries hide spikes. A five-second burst vanishes inside a one-minute average, so the summaries keep the highest value as well as the average.

Machine clocks disagree. A sample stamped by a slow clock lands in the wrong minute, and late samples arrive after their range was already summarised.

Labels can explode the series count. A label is a tag on a series, and a label with millions of values, like a user id, creates millions of series and sinks the store.

The guard is discipline about labels. Series stay countable when labels come from small sets, like server names and regions.

Trade-offs and wrap-up

Every retention choice trades three things. Resolution, retention and storage cost pull against each other, and the tiers are how a design buys all three partly.

Against a general database, the specialised store gives up flexible queries. In return it swallows a write rate and a data volume the general store cannot hold.

The main idea is that time is the index. Time-series data is written by time, read by time and deleted by time, and downsampling lets it age gracefully instead of growing forever.

Keeping less as data ages