VTEX Jul 12, 2023: circuit breakers that don't close on their own

01 · CONTEXT

A platform that depends on a replicated database

VTEX's public post-mortem is the source of this case. On July 12, 2023, at 20:29 UTC, one of the company's replicated, high-availability database systems suffered severe service degradation and did not recover automatically. The described effect was intermittent errors and high latency across the overall experience, with a partial platform outage.

The sales flow, product indexing, and the admin environment were severely affected.

1. Merchants and shoppers reach the platform through the services.

2. The services query a replicated, high-availability database, which is a shared dependency.

02 · THE TIMELINE

From the degraded database to recovery

According to the post-mortem, at 20:29 UTC the database degraded and did not recover automatically. At 21:15 UTC the initial problem was remediated manually. At 22:00 UTC the response team was still investigating the cascading effects of the failure and applying remediations manually. At 23:03 UTC VTEX validated that the remediation actions had been effective, with stable recovery.

1. At 20:29 UTC the replicated database suffers severe service degradation and does not recover automatically.

2. For users, the effect is intermittent errors and high latency, with a partial platform outage.

3. At 21:15 UTC the initial problem is remediated manually.

4. Several service circuit breakers opened and did not close automatically after the initial problem was resolved.

5. At 22:00 UTC the team was still investigating the cascading effects and applying remediations manually.

6. At 23:03 UTC VTEX validated that the remediation actions were effective, with stable recovery.

03 · THE BREAKER

The protection that does not come back on its own

According to the post-mortem, circuit breakers exist to avoid complete unavailability and, as a side effect, a subset of the platform kept operating.

1. The degraded replicated database is at the origin of the impact described in the post-mortem.

2. The sales flow was severely affected.

3. Product indexing and the admin environment were also severely affected.

04 · THE ACTIONS

What VTEX committed to changing

The post-mortem lists actions to avoid similar incidents. Among them are reassessing the size of the failure domain associated with the replicated, high-availability database system, investigating and fixing the cause of the severe service degradation in the database system, and investigating and fixing the bug in the circuit breakers that did not close properly after the database problem was resolved.

05 · IN THE INTERVIEW

Questions to bring to your design

When proposing circuit breakers, ask who closes the breaker: does it test the dependency automatically and periodically, or does it depend on a manual action? Also ask how you would know, through monitoramento and alerts, that a breaker is still open after the dependency came back. Finally, ask how many customers and functions share the same dependency, because that defines the reach of a failure in the disponibilidade.