Papers
Kafka: a Distributed Messaging System for Log Processing
Jay Kreps, Neha Narkhede, Jun Rao · NetDB 2011, 2011
The problem: log data became part of the product, not just analytics
At internet companies like LinkedIn, the volume of "log" data — click events, page views, likes, searches, plus operational metrics from every machine — is orders of magnitude larger than the volume of the product's "real" data. What changed is that this stream stopped serving only offline reports: it began feeding production features such as search relevance, recommendations, ad targeting, abuse detection, and activity feeds. This creates a new requirement: the same data must reach both offline workloads (loaded into a data warehouse or Hadoop) and online consumers with a delay of a few seconds. Traditional enterprise messaging systems (such as JMS-compatible queues) were a poor fit because they prioritized rich, transactional delivery guarantees that are unnecessary for logs — losing an occasional pageview event is not serious — and they were not designed for the required throughput or for partitioning data across multiple machines. The existing specialized log aggregators (such as Facebook's Scribe or Cloudera's Flume), in turn, handled the offline load into a data warehouse well but did not offer a real-time consumption API equivalent to that of a messaging system. Kafka was born to fill exactly that gap: a single piece of infrastructure able to serve both offline and online consumption of the same data stream.
Core architecture: topics, partitions, brokers, and the pull model
In Kafka, a stream of messages of a particular type is called a topic. Producers publish messages to a topic, which are stored on a set of servers called brokers; consumers subscribe to one or more topics and pull the messages from the brokers, rather than waiting for the broker to push them — a deliberate choice, since the pull model lets each consumer pull data at the rate it can process, without being flooded, and makes it easy to "rewind" reading to an earlier offset. To allow horizontal scaling, each topic is split into multiple partitions, and each partition is physically stored as a simple log: a set of segment files to which the broker only does sequential appends. Each message has no explicit ID — it is identified by its offset (position) within that partition's log, which avoids maintaining expensive index structures. Consumers organized into consumer groups divide a topic's partitions among themselves: each partition is consumed by exactly one consumer in that group at a time, which eliminates the need for coordinated locking between consumers of the same partition. Coordination between brokers and consumers (who owns which partition, which brokers are alive) is done in a decentralized way, using the ZooKeeper coordination service, instead of a central master node.
'Unconventional' design decisions: simplicity in exchange for throughput
The paper describes several deliberately simple choices to maximize throughput. First, Kafka avoids caching messages in the application itself and relies on the operating system's page cache, since producer and consumer access the segment files sequentially — this avoids duplicating the buffer in memory and keeps the cache 'warm' even if the broker restarts. Second, to transfer data from a file on disk to the network, Kafka uses the sendfile system call, which avoids unnecessary intermediate copies between kernel and application. Third, the broker is 'stateless' with respect to consumption: it does not keep track of which messages each consumer has already read — it is the consumer itself that maintains its offset. This simplifies the broker, but creates a problem: how do you know when to delete a message if the broker does not know who has already consumed it? The solution was a time-based retention policy (for example, keeping messages for a fixed period, such as seven days) rather than one based on read acknowledgment — an explicit trade-off of per-message delivery guarantees for operational simplicity. A welcome side effect of this choice is that the consumer can deliberately go back to an earlier offset and reprocess data, which is useful for fixing logic errors in the consuming application.
The guarantee the paper assumes: ordering only within the partition, 'at-least-once' delivery
The Kafka described in the paper guarantees only 'at-least-once' delivery: most of the time each message arrives exactly once at each consumer group, but if a consumer fails without a clean shutdown, the consumer that takes over its partitions may reprocess messages already delivered after the last committed offset — leaving it to the application to implement its own deduplication if this is a problem. The authors justify this choice by saying that 'exactly-once' delivery would normally require two-phase commits, a cost they judged unnecessary for the logs domain. Another explicit, limited guarantee: messages from the same partition are delivered to the consumer in the order they arrived, but there is no ordering guarantee between messages from different partitions of the same topic. At the time of the paper, Kafka also had no built-in replication between brokers — if a broker went down and its storage was permanently damaged, that broker's unconsumed messages would be lost; the authors themselves cite this as future work. This set of choices illustrates the paper's central trade-off: giving up strong guarantees and rich features (such as multi-queue transactions) in exchange for much higher throughput and an implementation that is simpler to operate at scale.
Legacy: from LinkedIn internal tool to standard of streaming platforms
At the time of the paper, Kafka was already processing a few hundred gigabytes of new data per day and about a billion messages per day on LinkedIn's infrastructure, with end-to-end latency of about ten seconds between an event being generated and being available for offline consumption. The paper's own comparative experiments showed Kafka publishing tens to hundreds of thousands of messages per second (depending on the producer batch size), clearly outperforming established messaging systems of the time such as ActiveMQ and RabbitMQ in the tests run by the authors, largely because of batching, the leaner storage format, and the use of sendfile. The authors themselves point to next steps: adding native replication between brokers and stream-processing capabilities (such as aggregations over time windows) directly on top of Kafka. These two points would, in the following years, become pillars of the ecosystem that grew around the project (partition replication between brokers and a stream-processing layer), consolidating the paper's central idea — a distributed, partitioned, persistent log as the transport layer — as one of the most widely used patterns today for connecting systems within event-driven architectures.
See the ideas at work
Practice the designs that use this paper in the System Design school.