Real-time chat

01 · CONTEXT

What a real-time chat needs to solve

A chat delivers messages between people in seconds, tracks who is online, stores the history, and notifies those who are away. The three hard parts are keeping a large number of connections open, getting the message to the right server, and losing nothing when the recipient is offline. Reference interview scripts start from a scale of tens of millions of daily active users, and products like Facebook Messenger and WhatsApp are cited at tens of billions of messages per day.

In this study, communication uses WebSocket, routing depends on load balancing and the history requires choosing and partitioning a database. Presence and ID generation appear in the text, without a page of their own in the track.

02 · CONNECTION

Why chat is not an ordinary HTTP API

HTTP is client-initiated, so the server has no way to push a new message on its own. Querying the server periodically (polling) can be expensive, depending on the frequency. WebSocket solves this with a bidirectional, persistent connection, opened once and used in both directions.

The consequence of keeping connections open is that the chat server stops being a stateless service: it needs to know who is connected to it. In the reference architectures, the login, profile, and similar services remain stateless, and the chat service is the only stateful one.

03 · DELIVERY

Getting the message to the right server

With many chat servers, the sender and recipient are almost never on the same one. A simple design starts with a single server and soon runs into routing: each message has to be taken to the server where the recipient is connected. One option is a pub/sub channel between the servers.

1. The sender keeps a persistent connection; the load balancer pins it to a chat server, which becomes the entry point for their messages.

2. Before trying to deliver, the server writes the message to the history and to the inbox. If delivery fails later, nothing is lost.

3. Only then does it publish the message to the pub/sub channel. This channel does not guarantee delivery, which is acceptable because the write has already happened.

4. The server where the recipient is connected receives the publication. Without this step, sender and recipient on different servers would never meet.

5. Server B pushes the message over the open connection, without the client needing to ask.

The subtle point is the order of operations: the message is written to the inbox and the history before being published. The pub/sub described in the reference delivers at most once and does not guarantee delivery, which is only acceptable because the write has already been done.

04 · PRESENCE AND OFFLINE

Who is online and what to do when no one answers

Presence is maintained by liveness signals: while the client is connected, it or the server emits periodic heartbeats. The intervals vary by design; one reference cites a client signal every few seconds, another, server pings every few tens of seconds. If the signals stop beyond the tolerance, the user is shown as offline, which keeps a brief network drop from flipping the status all the time.

1. While the recipient is connected, periodic liveness signals (heartbeat) keep their presence as online.

2. The recipient's connection drops and the heartbeats stop. After the tolerance time, presence marks them as offline.

3. Seeing that the recipient is not online, the server keeps the message in the inbox until they return.

4. In parallel, a queue feeds the worker that fires the push notification, so the user knows a message is waiting.

5. When the device reconnects, it syncs what was held back, in ID order.

For offline recipients, the references converge on two pieces: hold the message on the server until the user returns and fire a push notification. One of them considers push the most important third-party integration for a chat app.

05 · ORDER AND SCALE

Ordering, storing, and growing

For ordering, the creation timestamp is not enough, because two messages can be created at the same time. The reference calls for a unique, time-sortable identifier, so that new rows have larger IDs than old ones. Another reference treats the subject more pragmatically: users prefer to see new messages as soon as possible over having guaranteed order, so global ordering is not pursued directly.

For the history, one of the references recommends a key-value store for ease of horizontal scaling and low access latency, and the other chooses DynamoDB for the same kind of performance and scalability.

1. Each server holds many open connections, so the fleet grows in number of machines as the base of connected users grows.

2. A chat server goes down and takes with it the connections it was holding. The diagram highlights the server that went down and the load balancer, which is impacted.

3. The history is the only durable state in the system, which is why it is the one that gets partitioned to keep up with the writes of all conversations.

For the connection fleet, one reference estimates that each connection takes a few kilobytes of memory, so about ten gigabytes on a single machine would be enough to hold all the connections in the reference scenario. The other recalls that WhatsApp served one to two million users per host, and that even so this requires hundreds of chat servers. The aggregate volume, in a scenario of two hundred million simultaneously connected users, reaches the order of tens of thousands of messages per second.