Distributed logging

01 · CONTEXT AND REQUIREMENTS

What a distributed logging system needs to do

A distributed logging system collects the records produced by many services on many machines, stores those records and lets the team search them. The design brings together the stages of collection, transport, storage and query. It is a good exercise in monitoring and observability.

A log is a textual record with a timestamp, which may or may not be structured; the OpenTelemetry documentation recommends the structured format. Structured, here, means having a consistent schema or well-defined typed fields that downstream processing can rely on.

The functional requirements are to receive logs from many services, allow search by time range and filters, and trigger alerts from the logs. Among the non-functional ones, these drive the design: not losing logs silently, absorbing write spikes, keeping storage cost under control and isolating each team's data.

02 · COLLECTION AT THE EDGE

An agent close to the service

The OpenTelemetry documentation generally recommends running a collector alongside the service. According to it, the collector lets the service hand off data quickly and takes care of retries, batching, encryption and even filtering of sensitive data.

The same collector reads logs from any file and offers features to parse different formats, and deploying it as a general-purpose logging agent is usually the first step for those adopting OpenTelemetry. OpenTelemetry also automatically correlates logs and traces.

In Loki, the split is similar: the agent scrapes the logs, turns them into streams by adding labels and sends them through an HTTP API. A stream is a set of logs that share the same labels.

03 · DURABLE TRANSPORT

A queue between those who write and those who store

Between the agents and storage, a durable message queue can be added, which absorbs the difference in pace between those who produce and those who write.

Kafka, for example, has automatic failover to replicas when a cluster server fails, and only considers a message committed when all in-sync replicas have applied it to their log.

The publisher may or may not wait for this acknowledgment, trading latency for durability. On the read side, the consumer controls its position in the log and can rewind to re-read data; each partition is consumed by exactly one consumer within each consumer group, which, according to the documentation, gives a combination of ordering and scalability.

If the consumer processes the messages and crashes before saving its position, the new process receives already-processed messages again. This sequence corresponds to at-least-once semantics in case of consumer failure.

1. Each service writes its records and the local agent collects them and hands them onward.

2. The agent publishes the records to the durable queue, which decouples producers from storage.

3. The ingestion consumer reads from the queue at its own pace and saves the position it has read.

4. The records are written to long-term storage.

5. The investigator opens the search panel and the query service searches the storage.

6. In parallel, the alert evaluator runs repeated queries over the same logs.

04 · WRITE PATH

Distributor, hash ring and ingesters

As a concrete reference, Loki is a horizontally scalable, highly available, multi-tenant log aggregation system. It does not index the content of the logs, only metadata in the form of each stream's labels, and stores the compressed data in chunks in object storage.

The distributor is the first step of the write path: it ensures that data arrives according to the specification, limits the ingestion rate per tenant and is a stateless component. A load balancer is needed in front of the distributors.

Each stream belongs to a tenant and has a unique set of labels. Distributors use consistent hashing, together with the replication factor, to choose the ingesters that receive that stream.

The distributor tries to write the same data to all the chosen ingesters. With the write acknowledged by two of three ingesters, the loss of one is tolerated, but not of two. If fewer than two writes succeed, the distributor returns an error and the write is retried.

Replication is not alone: the ingester's WAL complements it, and data is only lost with significant failures in both mechanisms, for example several ingesters dying and losing or corrupting their disks.

The ingester persists the data and sends it to long-term storage, and on queries it returns the recent data that is in memory. A chunk is closed, compressed and marked read-only when it reaches the configured capacity or when too much time passes without receiving an update.

1. The agent sends the stream over HTTP and the load balancer delivers it to a distributor.

2. The distributor validates the stream and applies the tenant's ingestion limit.

3. By the stream's hash, the distributor chooses the ingesters and tries to write the same data to all of them.

4. With at least two of the three ingesters acknowledging, the write is accepted; with fewer than that, the distributor returns an error and the write is retried.

5. If one ingester fails, the loss of one is tolerated, but not of two.

6. The ingesters compress the chunks and send them to object storage.

05 · READ PATH

Split, parallelize and deduplicate

According to the Loki documentation, queries should be directed to the query frontend instead of the queriers. The frontend splits large queries into smaller ones, runs them in parallel on the queriers and merges the results.

The frontend also schedules queries fairly across tenants, preventing one tenant from taking down the others. With multi-tenancy, each tenant's data and requests are isolated from the others.

The queriers query all the ingesters for in-memory data before falling back to backing storage. The querier may receive duplicate data and deduplicates it internally.

For log queries, the frontend caches only empty results of quantized time ranges. The documentation says this is more efficient than caching actual results because log queries have a result limit.

1. The query goes to the query frontend, not directly to the queriers.

2. The frontend checks the results cache and splits what is missing into subqueries.

3. The queriers run the subqueries in parallel.

4. The queriers query all the ingesters for recent in-memory data.

5. Then, the queriers turn to object storage for the rest.

6. The frontend merges the partial results; duplicates coming from replication have already been eliminated by the queriers.

06 · RETENTION AND COST

How long to keep and where

Because Loki indexes only the labels and stores highly compressed chunks, the index is small, which simplifies operation and reduces cost. In return, the documentation says a good set of labels is the key to efficient queries.

In Elasticsearch, ILM automates the management of time-based indices, such as logs and metrics. It can create a new write index when the current one reaches a certain size, document count or age, move indices to cheaper hardware and delete old ones to enforce retention standards.

In Loki, retention is applied by the compactor, which accepts policies per tenant or per stream, and, without enabling it, logs sent to Loki stay forever.

07 · FAILURES AND TRADE-OFFS

What can go wrong

If an ingester process crashes abruptly, data not yet sent to storage may be lost. Replication and WAL are mitigations that complement each other.

On the transport side, the consumer's at-least-once semantics means the same record may be processed again after a crash. On the publisher side, it is possible to choose between waiting for the acknowledgment or not.

Loki lets several tenants share a single instance, with isolated data and requests.