Metrics monitoring and alerting

01 · CONTEXT AND REQUIREMENTS

What a metrics platform needs to do

A metrics monitoramento platform collects performance data from servers and services, stores everything as time series, shows it on dashboards, and fires alerts when thresholds are exceeded. Datadog, Prometheus with Grafana, and CloudWatch are well-known examples.

There are four functional requirements: ingest metrics from the services; query and visualize them on dashboards, with filters, aggregations, and time ranges; define alert rules with thresholds over time windows; and notify whoever needs to know by email, Slack, or PagerDuty.

Log search and aggregation, distributed tracing, and machine-learning anomaly detection are out of scope. Each one is a separate problem.

02 · SCALE ESTIMATION

Five million points per second

Hello Interview proposes monitoring 500 thousand servers. If each one emits 100 metric points every 10 seconds, the peak is 5 million points per second. Each point is small (timestamp, value, and labels), somewhere between 100 and 200 bytes, but at this volume raw ingestion reaches about 1 GB per second.

The other non-functional requirements pull in different directions. Dashboards should respond in seconds, even over days or weeks. Alerts should be evaluated in under a minute between metric emission and firing. The system should stay available, tolerating eventual consistency on dashboards but with reliable alert evaluation. And late or out-of-order data must be handled tolerantly.

Under a minute seems slow for alerts, but you generally only see an event after accumulating enough data, which is why alerts usually use moving averages or trends.

03 · INITIAL DESIGN

Collect, store, query

Start with the simplest path: a collector receives the points and writes them to a time-series database, and a query service reads that database for the dashboards.

1. The monitored services deliver their points to the collector, which receives or scrapes them.

2. The collector writes each point to the time-series database, with timestamp, value, and labels.

3. Whoever opens a dashboard makes a query with filters, aggregation, and time range; the query service receives it.

4. The query service reads the series in the requested range and returns the already aggregated result.

The data model is the time-series model: each metric is stored with the instant it was recorded and with key-value labels. It is the Prometheus model, which serves as the reference here.

There are two ways to collect. In the pull model, the monitoring server scrapes the targets over HTTP, and the targets are discovered through service discovery or static configuration. Push also exists: Prometheus supports it through an intermediary gateway.

04 · INGESTION AND STORAGE

Absorbing spikes without losing points

Ensuring high availability during spikes and failures is one of the deep dives that Hello Interview lists for this problem.

The initial design writes straight from the collector to the database, so a spike or a database failure goes directly to ingestion. A queue between the two decouples the two ends.

1. The points reach the load balancer, which spreads them across several collector instances.

2. The collector validates the point and publishes it to the queue, without waiting for the write to finish.

3. The writers consume the queue in batches, at the pace the database can handle.

4. Each batch is written to the time-series database, with the point's own timestamp.

5. If the database becomes unavailable for a while, the queue accumulates the points and the collectors keep accepting data; the writers resume when the database comes back.

The queue trades a bit of latency for slack: it decouples whoever receives from whoever writes and gives time to reprocess. Since the timestamp comes from the point and not from the moment of arrival, a late point still lands in the right place in the series. The cost is that the queue enters the alert latency budget.

The biggest danger for storage is labels. Each unique combination of labels is a new series and can greatly increase the stored data, which is why the Prometheus documentation says not to use labels for high-cardinality dimensions, such as user IDs. Hello Interview also lists cardinality explosion as a deep dive.

As for the database, Prometheus is a good example of how a time-series database handles continuous writes: ingested samples are grouped into two-hour blocks, the current block stays in memory and is protected by a write-ahead log, and the initial blocks are compacted in the background into larger blocks.

For dashboards over weeks of data, the server can also run rules that aggregate and write new series from existing ones. Hello Interview treats low-latency queries over weeks of data as one of the deep dives.

This suggests pre-aggregating: store, besides the raw points, summarized versions at larger intervals, and let long dashboards read the summarized ones. In exchange for more storage, the query no longer scans the raw points.

05 · ALERTS

From rule to notification

An alert rule is a condition over a time window. Hello Interview's example is to alert if p99 latency exceeds 500 ms for 5 minutes. In Prometheus, the for clause makes the server wait for a duration before counting the alert as firing, and an alert that is already active but has not yet fired stays in the pending state. keep_firing_for, in turn, keeps the alert active for some time after the condition stops holding, to avoid flapping alerts.

1. The evaluator loads the alert rules: the threshold, the window, and who should be notified.

2. On each cycle, the evaluator queries the database for the series in each rule's window.

3. When the condition is true, the alert becomes pending and only fires if it holds for the required time.

4. Once firing is confirmed, the evaluator hands the alert to the notifier, which groups and removes duplicates.

5. The notifier alerts the on-call person through the configured channel, such as email, Slack, or PagerDuty.

Prometheus's Alertmanager is a reference for the last stage: it deduplicates, groups, and routes the alerts. Grouping gathers similar alerts into a single notification, which matters in a large failure, when many alerts fire together. Inhibition suppresses notifications of some alerts when others are already firing.

Reducing alert latency below one minute is another Hello Interview deep dive. The source itself notes that, when you want to fire as early as possible, the alert is usually built in a particular way.

06 · LIMITS AND EVOLUTION

What was left out

Multi-region replication was left out of scope because it adds complexity.

If the product grows, multi-region is a natural next step, along with more summary layers for long queries. It is also worth monitoring the monitoring itself, and not making it depend on the same components it watches.

Another limit: for full accuracy, such as per-request billing, Prometheus is not a good choice. Monitoring metrics serve to understand system health, not to bill.