Slack: real-time messaging (gateways, channel servers, Flannel)

01 · CONTEXT

Real-time messages for lots of people

A work app needs to deliver each message to whoever is online in the channel, fast and anywhere in the world.

At Slack, every client keeps a websocket persistent connection to receive real-time events and keep its state.

Slack's messaging infrastructure is made of four core services, all written in Java: Channel Servers, Gateway Servers, Admin Servers and Presence Servers. According to the post itself, the servers handle tens of millions of channels per host and tens of millions of connected clients, and deliver messages around the world in 500 ms.

02 · THE FOUR SERVICES

Who does what

The GS is the interface between clients and the CSs. Unlike the other servers, it is deployed in several geographic regions, which lets the client connect quickly to a host in the nearest region. For region failures there is a draining mechanism that moves users from a bad region to the nearest good one.

Presence, the green dots in the app, comes from the PSs: users are hashed to individual PSs and the client makes the queries over the websocket, with the GS as a proxy. The client only receives presence notifications for a subset of users visible on screen.

03 · CHANNEL SERVERS AND THE RING

Every channel has an owner

Each CS holds in memory part of the history of the channels assigned to it. The channel ID is hashed and mapped to a single server, by consistent hashing. At peaks, about 16 million channels are served per host; and "channel" is an abstract term whose ID is assigned to an entity such as a user, team, company, file, huddle or regular channel.

The channels of a single team are spread across all CSs. The CHARMs manage the ring and replace unhealthy CSs; a new CS is ready to serve in under 20 seconds. When a CS is replaced, users of its channels see elevated delivery latency for under 20 seconds. Each ring is defined and managed by the CHARMs, and the AS and CSs discover the configuration by querying Consul.

1. The ring is defined and managed by the CHARMs, and the Admin Server discovers it by querying Consul.

2. The channel ID is hashed and mapped to a single Channel Server.

3. When a CS is not healthy, the CHARM replaces it quickly; users of its channels see elevated delivery latency for under 20 seconds.

04 · CLIENT CONNECTION

From the app to the websocket

At startup, the client fetches the token and the data to set up the websocket from the Webapp. It opens the connection in the nearest edge region, and Envoy, used at Slack as a load balancer and TLS terminator, forwards the request to the GS.

1. The client asks the Webapp for the token and the websocket connection data.

2. The client opens the websocket in the nearest edge region and Envoy forwards it to the Gateway Server.

3. The GS fetches the user's data from the Webapp, including their channels, and sends the first message to the client.

4. By consistent hashing, the GS asynchronously subscribes to the Channel Servers that host those channels.

With that, the client is ready to send and receive messages in real time.

05 · MESSAGES AND EVENTS

From sending to everyone else's screen

Sending a message starts with a call to the Webapp API. The Webapp hands it to the AS, which looks at the channel ID, finds the CS through the ring and routes it to the responsible CS. The CS sends it to every GS in the world subscribed to the channel, and each GS delivers it to each subscribed connected client.

1. Client A calls the Webapp API to send the message.

2. The Webapp passes the message to the AS, which finds the channel's CS through the ring and routes it there.

3. The CS sends the message to every GS subscribed to the channel, in any region.

4. Each GS delivers the message to the clients connected and subscribed to the channel.

Besides messages, there are events: any real-time update that changes the client's state. There are hundreds of types, such as reactions, favorites added and members joining a channel, and they follow a journey similar to the message's. Reminders, scheduled messages and calendar events tend to occur at the top of the hour, which explains regular traffic spikes.

Transient events are not persisted in the database and follow a slightly different flow; typing in a channel is an example. In that case, the client sends over the websocket to the GS, which routes to the CS through the ring.

1. Client A, when typing, sends the event over the websocket to its GS.

2. The GS looks at the channel ID and routes the event to the CS responsible for it on the ring.

3. The CS sends the event to all GSs subscribed to the channel, which pass it on to the subscribed clients.

06 · FLANNEL

Edge cache for app startup

Before Flannel, the server sent the client a full snapshot of the team's data through the rtm.start call and the client opened a WebSocket for events. The client used that data to initialize itself and opened a WebSocket for the event stream. In very large teams, connection time got long and client memory grew; in addition, reconnection storms, such as an entire office connecting at once, are resource-intensive.

Flannel is an application-level caching service, developed in-house and deployed at the edge points of presence. At client startup it caches data on users, channels and bots and offers on-demand query APIs; it also anticipates data, such as that of a mentioned user, and sends it ahead of the message to spare the client a lookup.

1. The client connects to Flannel.

2. Flannel gathers the full initialization data and opens a WebSocket to Slack's servers in the main AWS region, to consume real-time events.

3. Flannel returns to the client a reduced version of the initialization data, with which it initializes itself.

4. By consistent hashing, users from the same team and the same network region are directed to the same Flannel instance.

Flannel uses consistent hashing to choose the host the user connects to, keeping team affinity: users from the same team and the same network region go to the same instance, for cache efficiency. New or recently disconnected users are served straight from the cache, which reduces the impact of reconnection storms on the backend servers. According to the post, at peak Flannel served 4 million simultaneous connections and 600 thousand client queries per second; the size of the initialization data was reduced 7 times for a team of 1.5 thousand users and 44 times for one of 32 thousand.

As future work, the post proposes moving event fanout to Flannel, sending one copy per Flannel server, and moving clients to a pub/sub model; with presence updates already migrated, the presence events received by clients were reduced by a factor of 5.

07 · WHAT TO TAKE INTO AN INTERVIEW

Questions the design answers

A useful outline for a real-time chat is to split the responsibilities: one layer holds the client connections, another decides who owns each channel, and a third delivers to whoever is subscribed. It is worth justifying each split and saying what happens when a piece is replaced.