Distributed message queue (Kafka-style)

01 · CONTEXT AND REQUIREMENTS

Why a distributed queue exists

A message queue separates the one who produces from the one who consumes. The producer hands off the message and moves on; the consumer fetches at its own pace. This buffer in the middle is what lets two systems evolve, scale, and fail independently.

Kafka's design started from very concrete requirements: high throughput for high-volume event streams, such as log aggregation; tolerance for large backlogs, for periodic loads coming from offline systems; and low-latency delivery for traditional messaging uses.

1. The producer hands the message to the queue and moves on; it does not wait for processing.

2. The queue stores the messages in arrival order. This buffer is what absorbs spikes and backlog.

3. The consumer fetches messages when it has capacity; if it falls behind, the queue accumulates instead of losing messages.

02 · THE LOG

An append-only log on disk

The core idea is to store the queue as a log: messages are always appended to the end of a file. Kafka's documentation describes that a persistent queue can be built on simple reads and appends to files, and that in this structure all operations are O(1) and reads do not block writes.

Disk seems slow because of random access. The documentation measures the difference: in the setup it cites, linear writes reach about 600 MB/s, versus about 100 kB/s for random writes, a difference of more than 6000 times. That is why a well-used disk can be as fast as the network, and Kafka relies heavily on the file system.

Two tricks complete the gain. First, batching: the producer gathers messages, compresses the batch, and the server appends it all at once, which turns bursts of random writes into linear writes. Second, the operating system's page cache: with up-to-date consumers, reads are served from the cache and the disk is barely read.

03 · PARTITIONS AND CONSUMERS

Scaling reads without losing order

A single log does not scale beyond one machine. The way out is to split the topic into partitions, each one an independent log. The producer client chooses the partition, at random or by a semantic function over the key, and total order exists within each partition, not across them.

Consumers organize themselves into groups. Within a group, each partition is read by exactly one consumer at a time; thus, a group's maximum parallelism is limited by the number of partitions.

1. The producer client decides which partition each message goes into, at random or by a function over a key.

2. Each partition is an ordered log. Order exists within the partition, not across partitions.

3. Within the group, each partition has a single consumer at a time; one consumer can hold several partitions.

4. The consumer stores only one integer per partition, the offset. Falling behind is harmless, and going back to an old offset allows reprocessing.

The read position is just one integer per partition, the offset of the next message. The consumer reports the offset on each request and receives a chunk of the log from that point. Because it pulls the data, a slow consumer just lags and catches up later, and it can also go back to an old offset and reconsume.

04 · REPLICATION AND FAILURES

Surviving a broker failure

Each partition is replicated across several brokers, and the unit of replication is the partition itself. Every write goes to the leader; followers pull the log from the leader and apply it to their own. Kafka assumes replication by default: a topic without replicas is just a topic with a replication factor of one.

The leader tracks the set of in-sync replicas (ISR). A message is only considered committed when all replicas in that set have applied it. With this model and f+1 replicas, the topic tolerates f failures without losing committed messages; a majority quorum would require three copies for one failure and five for two.

1. The producer writes directly to the partition leader, with no routing layer in between.

2. Followers pull the log from the leader and apply the messages to their own log.

3. Only in-sync replicas count toward committing the message. A follower that lags too far can be removed from that set.

4. The leader goes down. Writes to that partition have nowhere to go until there is a new leader.

5. The controller notices the loss of the broker's session, and a follower from the in-sync set is elected. It already has all the committed messages.

6. Producer and consumer query the metadata, discover the new leader, and resume work.

05 · DELIVERY AND RETENTION

Duplicates, loss, and what to keep

Acknowledging delivery runs into a dilemma. If the broker marks the message as consumed when it delivers it and the consumer fails, the message is lost. If the consumer processes it and fails before acknowledging, the message is consumed twice. On the producer side, resending after an error can write the message again.

Kafka's default is to deliver at least once; at most once is an option. That is why the consumer needs to tolerate repetition: the classic design is to make it an idempotent operation, for example by discarding an identifier already seen.

Finally, keeping everything forever is not feasible. Simple retention discards old data after a fixed period or when the log reaches a size limit. When only the latest state of each key matters, log compaction keeps at least the last value of each key within a partition.