Mercado Pago digital account: TCC reservations and moving off the relational database

01 · CONTEXT

Reserve before moving the money

Mercado Pago is the fintech arm of Mercado Libre and accepts, among several payment methods, account money: funds a person keeps in the digital wallet.

Whenever someone pays with account money, making sure the balance exists is crucial. To do this safely in a naturally concurrent environment, the critical path of these flows uses the protocol TCC (Try-Confirm/Cancel).

The flow has three stages. When the reservation is created, an amount is set aside for the transaction, and if this step fails the transaction is aborted immediately. Once the reservation call succeeds, the person receives the confirmation on the device, but the operation is not finished yet. Later, another microservice releases the reservation, debits the source account, and transfers the amount to the recipient in the same database transaction, creating double-entry movements between the accounts involved.

1. The reservations API sets the amount aside for the transaction; if this step fails, the transaction is aborted immediately.

2. Once the reservation call succeeds, the person receives the confirmation on the device, but the operation is not finished yet.

3. Later, another microservice releases the reservation, debits the source, and transfers to the recipient in the same database transaction, creating double-entry movements.

4. The relational database underpins these flows and other critical services.

02 · THE PROBLEM

The table that never stopped growing

Mercado Pago's growth brought challenges mainly in its interaction with the relational database, which underpins these and other critical services. The movements table, where the double-entry movements are stored, grew about eightfold in four years: from roughly 12 TB in 2021 to about 100 TB by the end of 2024, adding close to 700 million rows per day, according to the article.

The limit became concrete: only a few days remained before hitting a hard database limit, because the tablespace for that table was close to the maximum number of datafiles allowed. To escape it, the team urgently migrated to a partitioned table, without users noticing the transition.

There was also a concurrency problem. To preserve data integrity, the relational database uses pessimistic locking, which serializes transactions to avoid problems such as the lost update. It is effective, but it can degrade performance under heavy load and still requires additional strategies to handle hot users, which complicates scalability.

03 · BENCHMARK

Choosing the database by measuring

The team built a benchmark to simulate the load of the main services, with read-heavy, write-heavy, and mixed scenarios. It compared a well-known NoSQL database and two newer NewSQL options against the current relational database, migrating anonymized production data to each one. The choice of storage type is the subject of database selection.

The NoSQL database was chosen for its performance, its market presence, and the team's experience. With no explicit locking, the team adopted optimistic concurrency control, which gave satisfactory results. The current relational database outperformed all candidates in the benchmarks, and the stated need was for horizontal scalability strategies.

04 · MIGRATION

Two phases and two databases in parallel

The migration was split into two phases. In the first, the reservation creation service moved to the new storage. The team wanted to gain resilience against degradations of the relational database and to test the viability of NoSQL with real production traffic. In the final phase, the remaining services would migrate to the same technology.

The goal of the first phase was to decouple account-money operations from the relational database. To do this, a new reservations service was created, operating exclusively on NoSQL. The central challenge became keeping the consistency between the two stores, since the account information would be distributed across both.

1. In the first phase, only reservation creation changes place: a new service operates exclusively on NoSQL.

2. The remaining services are left for the final phase; until then, the account information is distributed across the two databases.

3. Each database's CDC captures changes and passes them to the other, in both directions.

4. Each service also triggers a best-effort asynchronous sync to update balances, and a second call covers reservations still missing from the relational database.

To monitor the health of the new service, the team built an audit subsystem aimed at giving complete traceability of every action on every account and reservation. It writes event records with the information of each API call, including the entity's before and after states in both databases, errors, and HTTP data.

05 · HOT ACCOUNT

Shielding the account against throttling

The team tested concurrency by gradually increasing the payment volume on a single source account, measuring performance at 1, 5, 10, 20, and 30 transactions per second, for three minutes at each level. Contention did not improve significantly as the sync triggers were turned off, which showed the need for an efficient synchronization strategy.

An operational limit of the NoSQL engine also emerged: write throughput and the size of the items written govern resource consumption. A throughput of 30 TPS, with account items holding hundreds of pointers to reservations not yet replicated to the relational database, was enough to generate hot-key throttling errors.

1. With high throughput on one account and account items carrying hundreds of pointers to unreplicated reservations, NoSQL started returning hot-key throttling errors.

2. In the preventive step, when it detects an account with at least 500 reservations not yet replicated to the relational database, the service triggers a best-effort sync.

3. In the reactive step, when a throttling error occurs, the account is temporarily blocked and the creation of new reservations is paused.

4. The goal is to empty the account item right away so that reservations for that account can be safely resumed.

In the redone stress tests, the team reports an improvement of about 8% to 12% in the total reservations generated at each TPS level.

06 · COSTS AND NEXT STEPS

The price paid and what is left

In performance, the new service saw an increase of approximately 33% in average response time. In exchange, the team says it gained much greater stability and predictability in the response-time percentiles, something essential for a critical financial service. The article also acknowledges that the service is open to optimizations.

The work is not finished. The data model will have to be completely redone because of the move from relational to non-relational, and pieces of business logic that, over the years, came to depend on quirks of the relational database will have to be untangled.