Concepts
Partitioned commit log
Intuition
Think of a minutes notebook where each new line is numbered and always written at the end, never erased or edited in the middle — anyone who wants to know what happened just has to look from a given line number and keep reading forward. Now imagine that, instead of a single notebook, you use several notebooks in parallel, each handling a slice of the subject, so that many people can write and read at the same time without fighting over the same notebook.
How it works
In Kafka, a stream of messages of one type is a topic, and each topic is divided into multiple partitions to allow horizontal scaling. Each partition is physically stored as a simple log — a set of segment files to which the broker only does sequential appends, never writes or edits in the middle. A message has no explicit ID assigned by the system: it is identified by its offset, its position within that partition's log, which avoids maintaining expensive index structures. This append-only log structure brings two direct advantages: first, I/O is sequential for both producers and consumers, which is orders of magnitude faster on disk than random I/O, and lets Kafka lean on the operating system's page cache instead of maintaining its own cache; second, because data is not deleted when read (retention is by time, not by read acknowledgment), any consumer can go back to an earlier offset and reprocess the history. The number of partitions in a topic determines the degree of parallelism possible: within a consumer group, each partition is consumed by exactly one consumer at a time, so more partitions allow more consumers to process the same topic in parallel, at the cost that ordering is guaranteed only within each partition, never across different partitions of the same topic.
In practice
This design is the foundation of any modern event streaming platform (Kafka and its descendants) used to decouple producers and consumers in event-driven architectures, ingestion pipelines into a data warehouse/Hadoop, or near-real-time activity feeds; choosing the number of partitions of a topic is literally choosing how much consumption parallelism that stream will support.
Use it in a design
Practice the cases that rely on this concept in the System Design school.