Segment: goodbye, microservices

01 · CONTEXT

What Segment's pipeline does

Segment's customer data infrastructure receives events, JSON objects generated by web and mobile apps with information about users and their actions, and forwards them to partner APIs, called destinations. According to the team, that is hundreds of thousands of events per second and more than a hundred types of destinations.

At the start the architecture was simple: an API received the events and published them to a distributed message queue. When consuming from the queue, the customer's settings decided which destinations received each event, and the event was then sent to each of them. Anyone using the product only needed to send the event to one endpoint, instead of building several integrations.

1. Anyone using Segment sends the event to a single endpoint, instead of building an integration for each destination.

2. The ingestion API publishes the event to a distributed message queue.

3. When consuming from the queue, the customer's settings are checked to find out which destinations should receive the event.

4. The event is sent to the API of each selected destination.

When a delivery fails, the system sometimes retries later, but not every failure deserves a retry. Errors the destination could accept without any change to the event, such as HTTP 500, rate limit, and timeout, are retryable. Errors such as invalid credentials or missing required fields will never be accepted and are not retryable.

02 · THE SINGLE-QUEUE PROBLEM

One slow destination delays everyone

In this version, a single queue held both the newest events and those that had already gone through several attempts, from all destinations. The result was head-of-line blocking: if one destination became slow or went down, the retries flooded the queue and delayed all destinations.

1. Imagine a destination starts timing out on every request.

2. Every event that fails goes back to the queue for another attempt, and a large backlog forms.

3. The sudden growth of the queue exceeds the capacity to scale, and the newest events suffer delays.

4. Delivery times for all destinations increase because of one destination's problem.

The team describes that the systems scaled automatically as load increased, but the sudden increase in queue depth exceeded the capacity to scale. Because customers depend on timely delivery, increased wait at any point in the pipeline was not acceptable.

03 · THE MICROSERVICES SOLUTION

Isolating each destination worked, and then it charged a price

To fix the blocking, the team created a service and a queue for each destination. A router process receives the incoming events and distributes a copy to each selected destination. That way, only the queue of the destination with the problem builds up, and the others are unaffected. This architecture in the style of microservices, according to the team, isolated the destinations from one another, which was crucial when a destination had problems.

1. A router receives the incoming events.

2. The router distributes a copy of the event to the queue of each selected destination.

3. Destination X starts having problems, and its service's deliveries fail.

4. Only destination X's queue builds up; the queues of the other destinations are unaffected.

Each destination API uses its own request format, so each destination needs code to translate the event. Initially all of that code lived in one repository, and a broken test made the tests of every destination fail. In response, each destination's code was moved to its own repository.

With growth came more than 50 new destinations and more than 50 repositories, and the team created shared libraries for common transformations and functionality, such as HTTP request handling. But testing and deploying changes to those libraries impacted every destination, and their versions started to diverge across the codebases.

There was also the load problem: some services handled few events per day and others thousands per second, and each needed its own combination of CPU and memory, which made tuning autoscaling more art than science. The team added an average of three destinations per month, and the operational cost grew linearly with each destination.

04 · CONSOLIDATION

From more than 140 services to one

The first item on the list was to consolidate the more than 140 services into a single service. The cost of managing them was a heavy tax on the team, and it was common for the on-call engineer to be paged by load spikes.

There was an obstacle: with one queue per destination, each worker would have to check all the queues, a complexity the team did not want. That inspired Centrifuge, which would replace the individual queues and send the events to the single service.

In the monorepo, the team committed to a single version for each of the 120 distinct dependencies, updating each destination as it was moved. According to the team, a solid test suite was needed to put everything in one repository.

The original reason for splitting into repositories was to isolate test failures, but that proved to be a false advantage. The main cause of the failures was the HTTP requests to the destinations' endpoints during tests, and some destinations took up to 5 minutes to run theirs. With the full suite able to take up to an hour, the team built the Traffic Recorder: on the first run it records requests and responses to a file and, on later runs, replays them. After that, the tests for all of the more than 140 destinations ran in milliseconds.

1. The code of all destinations now lives in a single service, with a single version of each dependency.

2. Centrifuge replaces the individual queues and sends the events to the single service.

3. The service delivers the event to each destination from a large pool of workers.

4. On the other hand, a bug in one destination that brings down the service brings down all destinations.

05 · RESULT AND TRADEOFFS

What the monolith gave and what it cost

With everything in one service, an engineer can deploy in a matter of minutes, and it is no longer necessary to deploy more than 140 services to change a shared library. As a measure of velocity, the team cites 32 improvements to the shared libraries with microservices and 46 a year after the change.

In operations, the mix of CPU-intensive and memory-intensive destinations made scaling easier, and the large worker pool absorbs load spikes, so the team stopped being paged for low-volume destinations.

Before splitting a system into services, it is worth asking who will pay the operational cost of each additional service, and whether the team has the testing and deployment tooling for batch changes.

Definitions used by the text: microservices are server applications built by combining many small, single-purpose network services; in a monolith, a lot of functionality lives in a single service, tested, deployed, and scaled as one unit.