Payment system

01 · CONTEXT

Moving money does not tolerate a middle ground

The classic exercise asks that a merchant be able to start a charge of a specific amount against a customer and track its state (pending, success, or failure). The payment itself is made by an external provider, the PSP: the customer enters the details on the merchant's site, which passes them to the PSP, and the PSP returns the result.

The exercise's reference load is about 10,000 transactions per second at peak, and the requirement that weighs most on the design is not throughput: it is guaranteeing durability and auditability, without losing transaction data even in case of failure. This requirement explains the case's three decisions: idempotent operations to tolerate repetition, filas to absorb the PSP's asynchronous nature, and an immutable ledger to prove what happened.

02 · IDEMPOTENCY

Repeating the call without charging twice

The central danger is described bluntly by those who operate a PSP: calling the operation twice by accident would leave the customer charged twice. And repetition is inevitable, because the call can succeed and the connection can drop before the server notifies the client. The way out is to make the operation safe to repeat, with an idempotency key generated by the client.

The mechanism: the client resends the same request with the same identifier, the server correlates that identifier with the request's state on its side and, if the operation had already succeeded, responds with the cached result. In Stripe's API the result of the first execution is stored and returned on repeats, including when it was a server error; in addition, the idempotency layer compares the received parameters with those of the original request and rejects the call if they differ, which prevents misuse of the same key.

1. The client generates a unique key for this payment intent and sends it along with the charge request.

2. The payment service queries the database: if the key already exists, it returns the stored result; if not, it records the key before talking to the PSP.

3. Only then does the service ask the PSP for the charge, passing along the same key so that repetition is also safe in that stretch.

4. The PSP executes the charge, but the connection drops before the response arrives: the service does not know whether the money moved.

5. The client retries with the same key. The service recognizes the key, checks the state, and returns the result, without creating a second charge.

Retrying without discipline also has a cost: if all clients try again at the same time, the server receives a synchronized wave. That is why the wait between attempts grows exponentially and gets a random component (jitter) per client, to spread the retries out over time.

03 · ASYNCHRONOUS

The PSP notifies later: webhooks and queue

Part of a payment's outcome arrives later, through events: the customer's bank confirms the charge, the customer disputes it, a recurring payment happens. The merchant receives these events at a webhook endpoint and needs to treat them as an asynchronous flow, not as a continuation of the original call.

Stripe's documentation is explicit about three properties of this delivery. First, the same event can arrive more than once, and the defense is to record the IDs of events already processed. Second, delivery order is not guaranteed, so handling cannot depend on a sequence; when an object is missing, the way out is to fetch it from the API. Third, the endpoint should respond with success quickly, before any heavy logic, and process the events through an asynchronous fila, at a rate the system can sustain. If delivery fails, the PSP retries for several days with increasing waits.

1. The PSP sends an event (payment confirmed, failed, or disputed) to the merchant's webhook endpoint.

2. The endpoint validates the event's origin, puts it on the queue, and responds with success immediately, without touching the database.

3. The queue decouples the PSP's pace from the pace of our processing: a spike of events becomes waiting, not an outage.

4. The consumer checks whether the event ID has already been processed; if so, it discards it; if not, it applies the effect and records the ID in the same transaction.

5. If the consumer crashes midway, the message was not acknowledged and stays on the queue; when it comes back, it reprocesses, and the ID check prevents the duplicate effect.

04 · LEDGER

The record nobody rewrites

Payment state (pending, success, failure) answers "where is this charge". The question "where did the money go" calls for a different record: a ledger, a sequence of entries that only grows. Stripe describes its own as an immutable log of events and the system of record for all financial data: what has been posted cannot be erased or altered, and that immutability is what makes it possible to audit and replay any data at any time. An error is fixed with a new compensating entry, never by editing the old one.

Double-entry accounting gives the ledger a verifiable rule: every entry takes from one account and puts into another for the same amount, so the whole always balances. The same Stripe text uses this construction as the basis for a mathematical proof of correctness. Writing the entry together with the state change in a single database transaction keeps the two views consistent inside our system; between us and the PSP there is no shared transaction, and that is where reconciliation comes in.

05 · RECONCILIATION

Proving both sides agree

Even with idempotency and a queue, two independent systems can diverge: a lost event, a delayed entry, a refund that only one side saw. Reconciliation is the periodic check that compares our ledger with the PSP's records and surfaces every difference. Stripe describes the same idea internally: it measures what fraction of the ledger is properly zeroed out in steady state and uses explicit data completeness checks to guard against cases where something stops being recorded.

1. The reconciliation job reads the period's entries from our ledger.

2. In parallel, it loads the records the PSP publishes for the same period.

3. Each payment is matched by identifiers; anything that exists on only one side, or has a different amount, becomes a discrepancy.

4. Discrepancies go to monitoring and become correction work via compensating entries, never by editing history.