Fury Streams: distributing permits so the destination doesn't go down

01 · CONTEXT

A destination that does not scale

The streams service of the Fury platform, an internal tool at Mercado Libre for developers, carries real-time information from many sources to destinations such as queues, key-value stores, and relational databases. The account mentions more than 30 million messages, coming from more than 12 thousand sources and connected to more than 15 thousand destinations.

The problem comes from a common design: the application processes information and hands the result to a complementary service, such as a database or an external application. Faced with a spike, it opts for scale horizontally. The challenge appears when the volume exceeds the destination's capacity, which may be a legacy database or an external application outside the team's control, and then scaling the destination is not viable.

1. Several application instances process information and send the result to a complementary service.

2. The destination has a capacity that cannot be expanded by scaling horizontally.

3. When the combined volume of the instances exceeds the destination's capacity, the limit is exceeded, and the number of instances fluctuates all the time.

The exercise's question is: if the destination handles 100 thousand requests per minute, how does the application keep the total below that with a changing number of instances? In the real service, delivery came from more than 800 instances that fluctuated constantly.

Describing the impact of the problem on the Streams team, the account mentions about 600 alerts in the last months of 2023, each requiring on average 30 minutes of a person's attention.

02 · FIRST ITERATION

Split the permits equally

In the first iteration, the team balanced the load per instance based on how many were active, and divided the total permits equally. With a destination of 100 thousand requests per minute and ten active instances, each received 10 thousand permits.

The system had three pieces: a message queue, a database, and an additional application that compiles and stores the data, the worker-balancer. Each time a new instance started processing, it sent a heartbeat through the queue. The heartbeat was written to the database with a limited time to live (TTL). Then, the worker application queried the balancer to find out how many instances were alive and split the permits proportionally.

1. Each instance that starts processing sends a heartbeat through the message queue.

2. The worker-balancer compiles the signals and writes them to the database with a limited TTL.

3. The worker application queries the balancer to find out how many instances are alive and splits the permits proportionally.

4. With the split distributed, the destination is not overloaded by excess requests.

The weak point is the premise: the strategy assumed every instance would run the same processing load, which does not always happen. The account points to load balancer performance and traffic distribution as sources of disparity, and notes that different tasks take different amounts of time: an invoice with thousands of lines takes longer than one with a few.

The account's numerical example shows the effect. With 100 thousand requests per minute and two instances, A and B, each receives 50 thousand. If A only processes 20 thousand and B processes 80 thousand, the equal-split model lets 70 thousand be processed in total, and 30 thousand per minute go unprocessed, creating lag. The destination is protected, but the application's capacity is underused.

03 · SECOND ITERATION

Report usage without getting in the way of the work

With these conclusions, the team restructured the application to report each host's availability and details of each one's current load state. The worker-balancer was adjusted to analyze this information and act as the load distribution center. The data sent includes the instance identifier, the timestamp, and the assigned permits.

Sending synchronized information during the work is neither practical nor possible, because it would require more computation and get in the way of processing. The team then adopted an approach similar to monitoring libraries such as DataDog or NewRelic: an in-memory data structure, with an asynchronous queue, that compiles, consolidates, and sends the information in an asynchronous thread.

In the flow, processing details and permits accumulate in an in-memory buffer; a dedicated thread compresses and flushes the buffer every minute; and the compressed information is dispatched to the message queue.

1. Each instance accumulates processing details and permits in an in-memory buffer.

2. A dedicated thread compresses and flushes the buffer every minute and dispatches the content to the message queue.

3. The balancer receives from the queue and does a batch upsert into the database, writing according to each tuple's timestamp, which relieves the queue and removes the dependency on message order.

4. The balancer evaluates the distribution every minute: the surplus permits of a host that uses less are reallocated to another host that uses all of its own.

5. The evaluated distribution is stored persistently in long-term storage, Amazon S3.

6. When a host distribution request arrives, the file is read and the result returns to the client with minimal response time.

04 · DISTRIBUTION AND AVAILABILITY

Store proportions and serve ready-made files

The distribution stores each host's percentage of permits, not an absolute value, which avoids reprocessing everything when the number of permits changes. With 100 thousand requests per minute split 80% to host A and 20% to B, the proportion holds even if the total rises to 200 thousand permits, with a quick effect on the instances.

The algorithm is simple: if a host uses fewer permits than it received, the surplus is reallocated to another host that uses all of its own. The evaluation runs minute by minute, which allows continuous adjustments until reaching the ideal distribution.

Querying each instance's permits on every request would be slow, so the distribution processing happens every minute and the result is stored in Amazon S3. When a request arrives, the file is read and returned with minimal response time.

For persistence, the balancer receives the data from the queue and does a batch UPSERT, writing only if the tuple's timestamp is older than the new element's. This relieves the queue and removes the dependency on message order. The elements have a TTL of three minutes, useful for example when a host stops.

05 · RESULT AND LIMITS

What changed and what to take into the interview

The account presents the restructuring as the result of several iterations, with a 95% drop in lag alerts and a friendlier environment for those who develop on the platform.

The Fury Streams service also appears in another article by the team, about the migration to virtual threads. In it, the pusher component reads from Kinesis streams and delivers to several destinations, decoupling consumers from any specific technology, and uses thousands of threads to parallelize consumption and delivery. To restrict concurrent tasks, the article suggests replacing the bounded thread pool with a semaphore-based method.

The two articles deal with limits at different levels: the semaphore restricts concurrency within a process, while distributed permits limit throughput against the destination by summing across all instances.