Design a Monitoring and Alerting System

The ask

Thousands of servers run the wider system.

People cannot watch each server by hand.

Metrics are the measured numbers a server reports over time, like processor use or requests per second.

An alert is a warning sent to a person when a metric stays outside its allowed range.

Alerting is the sending of alerts under clear rules about when to send and when to stay quiet.

The system collects the metrics, stores them and runs the alerting rules.

Estimate the scale

The fleet is 10,000 servers.

Each server reports 100 metrics.

10,000 servers x 100 metrics = 1,000,000 series.

A series is one metric for one server over time.

One sample is taken every 10 seconds for each series.

1,000,000 series / 10 seconds = 100,000 samples per second.

One sample takes 16 bytes.

100,000 samples x 16 bytes = 1,600,000 bytes per second.

100,000 samples x 16 bytes x 86,400 seconds = about 138 GB a day.

API and data

A server sends a sample with four parts.

The parts are the server name, the metric name, the time and the value.

Extra labels can name the region or the service.

The stored form is time-series data.

A read request names a series and a span of time.

A read returns the samples in that span, or a summary of them.

High-level design

A small agent runs on every server.

The agent takes samples and sends them to collectors.

Collectors check each sample and pass it to the store.

The store keeps the samples by series and by time.

A rule checker reads the fresh samples on a steady rhythm.

The rule checker hands firing alerts to a sender.

The sender delivers each alert to the owning team.

Collecting and storing

Agents send samples. Collectors do not have to ask for them.

An agent keeps a short buffer of samples on its own server.

The buffer covers a brief collector outage.

Samples are written in time order inside each series.

Writes in time order stay fast even at this rate.

The store is split by series across many machines.

Deep dive: from a bad value to a warning

A single high sample does not raise an alert.

Brief spikes are normal in a busy fleet.

A rule names a metric, a limit and a span of time.

The alert fires only when the metric stays past the limit for the whole span.

Each alert names the server, the metric and the current value.

Repeated alerts for the same trouble are grouped into one.

A known maintenance window can silence a rule for its span.

Too many alerts teach people to ignore alerts.

Quiet rules are therefore part of correctness, not polish.

Keeping the cost down

Full-resolution samples are kept for 30 days.

About 138 GB a day x 30 days = about 4.1 TB at full resolution.

Downsampling keeps fewer points for old data by storing averages over longer spans.

Old data is read for trends, not for single seconds.

Averages serve that reading at a far smaller size.

When the watcher fails

The store keeps copies of its samples on other machines.

A dead collector loses only the samples in flight to it.

Agent buffers cover that short gap.

The watcher itself is watched from a separate small setup.

A watcher that depends on the fleet it watches can fail with that fleet.

Evaluation and trade-offs

Short sample gaps give fresh alerts but cost storage and traffic.

Long spans before firing cut false alerts but delay real ones.

The design samples often and fires slowly.

That pairing keeps the data sharp and the warnings worth reading.

The main risk is noise. The rules, not the store, carry that risk.

Scale at a glance