Skip to main content

Concepts

Pull-based consumption model

Intuition

It is the difference between a waiter who keeps piling dishes on you at the kitchen's pace, even if you have not finished the previous one, and a buffet where you walk to the table yourself and serve yourself at your own pace, remembering how far you have already eaten. In the second case nobody drowns you in food, and if you want to have a dish again that has already passed, you just go back there.

How it works

In Kafka, consumers subscribe to topics and pull messages from the brokers, instead of the broker pushing messages to them as soon as they arrive. The broker is 'stateless' with respect to consumption: it does not keep track of which messages each consumer has already read; it is the consumer itself that maintains and advances its offset within each partition. This choice solves the classic problem of a push model, in which the broker has to guess the right sending rate for each consumer: sending too fast overwhelms a slow consumer, and sending too slowly wastes the capacity of a fast one. With pull, each consumer fetches data at the speed it can process, naturally controlling its own backpressure, and can even ask for larger batches of messages at once to amortize the network cost (batching), which raises throughput. The explicit trade-off is latency: if no new data is available when the consumer asks, it has to poll repeatedly (or wait in a long-poll call) instead of being notified instantly as it would be in a pure push. A welcome side effect of the consumer holding its own offset is the ability to deliberately 'rewind' to an earlier offset and reprocess messages already read, something that becomes much harder when the broker is the one deciding and pushing what has already been delivered.

In practice

This model is the reason event streaming systems like Kafka favor throughput and replay capability over minimal per-message latency, and it is the right choice when consumers have very different processing speeds from one another or need to reprocess history; systems that prioritize immediate delivery latency above all else (such as push notifications) tend to prefer a push model over this one.

Use it in a design

Practice the cases that rely on this concept in the System Design school.