KVS: key-value as a platform product (642 million operations per minute)

01 · CONTEXT

The Mercado Pago digital wallet

Mercado Pago, the fintech arm of Mercado Libre, is an ecosystem of thousands of microservices that process financial transactions. One of the payment methods is account balance, the money the user keeps in the digital wallet.

Before debiting, the system must guarantee that there is a balance, in a naturally concurrent environment. That is why the critical path of these flows implements the TCC protocol (Try-Confirm/Cancel), with the amount reserved in a first step and the transfer in a later step.

1. The flow starts with the creation of the reservation: an amount is set aside for the transaction and, if this step fails, the transaction is aborted immediately.

2. With the reservation created, the user receives confirmation on the device, but the operation is not finished yet.

3. Later, another microservice releases the reservation, debits the source account, and credits the recipient in the same database transaction.

The company's relational database supports these and other critical services.

02 · THE PROBLEM

When the single database becomes the bottleneck

With ever heavier loads, the relational database became an obstacle to the wallet's horizontal scalability. The movements table, which holds the double-entry postings, grew eightfold in four years.

Growth went from about 12 TB in 2021 to about 100 TB at the end of 2024, with almost 700 million new rows per day. The team came within a few days of a hard limit: the table's tablespace was about to reach the maximum number of datafiles, and the situation was averted with an urgent migration to a partitioned table.

03 · BENCHMARK

How to choose the new database

The team evaluated three candidates in a benchmark that simulated the loads of the key services: a well-known NoSQL database and two newer NewSQL options.

The decision went to NoSQL, for its performance, its market presence, and the team's experience, relying on optimistic concurrency control because of the lack of explicit locking.

04 · MIGRATION

Two phases and two synchronized databases

The plan was split into two phases. Because it is a hypercritical service, the migration was done in stages. In the first, the reservation creation service was moved to the new storage, with the goal of increasing resilience to relational database degradation. In the final phase, the remaining services would be migrated.

The new reservation service operates exclusively on NoSQL. The challenge became keeping consistency between two stores, since account information would be distributed across both.

To synchronize, they used change data capture (CDC) on both databases, in both directions. They added two synchronization API calls: an asynchronous, best-effort one, triggered by each service to update balances, and another for when a service needs reservations that are not in the relational database.

Using features of the Fury platform, the team built an audit subsystem that records an event log for every API call, with the entity's before and after states in both databases, errors, and HTTP data. Fury was also used in the stress tests.

1. In the first phase, the new reservation service operates exclusively on the NoSQL database.

2. To keep the two databases consistent, change capture runs in both directions: what changes in one is passed on to the other.

3. The remaining services only migrate in the final phase; for when a service needs reservations that are not in the relational database, there is a synchronization call that guarantees the data immediately.

4. In the stress test, 30 TPS on one account, with items holding hundreds of pointers to reservations not yet replicated, were enough to generate hot-key throttling errors.

5. Preventive step: when detecting an account with at least 500 reservations not yet replicated to the relational database, the system triggers a best-effort synchronization.

6. Reactive step: upon hitting a throttling error, the account is temporarily blocked and the creation of new reservations pauses, to drain the item and resume safely.

05 · RESULTS

What the stress test showed

The test raised the payment volume on a single source account in steps of 1, 5, 10, 20, and 30 TPS, for three minutes each. Contention did not improve much when the synchronization stimuli were gradually turned off, which showed the need for an efficient synchronization strategy.

In the reviewed tests, there was an improvement of about 8% to 12% in the total reservations generated at each TPS level. In return, the new service saw an increase of about 33% in average response time.

According to the text, this trade brought much greater stability and predictability in the response time percentiles, something essential for a critical financial service. What remains is to fully redo the data model when moving off the relational database.