Webhook delivery (retries, signing, DLQ)

01 · CONTEXT AND REQUIREMENTS

What it means to deliver a webhook

In Stripe's documentation, the user registers an HTTPS endpoint and, after that, Stripe pushes data in real time when events happen in the account. The delivery uses HTTPS and carries a JSON payload with the event information.

For the sender, which sends webhooks to many customers, the problem boils down to questions of requirements gathering: what to do when the destination fails, how the destination knows the event is legitimate, what happens if the same event arrives twice, and how the customer finds out what happened with each delivery. The following sections cover each one.

02 · CONTRACT WITH THE DESTINATION

What counts as success

The contract of API with the destination is short: Stripe expects the handler to quickly return a success code (2xx), before any complex logic that could cause a timeout. Redirects also count as failures.

For the sender, this defines what the delivery worker must treat as a failure and reschedule: any response that is not the expected success, including no response at all.

03 · DELIVERY, RETRY, AND DLQ

From the first version to the queue-based pipeline

The first version is naive: the service that generates the event calls the customer's endpoint itself, within the same request.

1. The service generates the event and calls the customer's endpoint directly, waiting for the response.

2. If the destination is slow or down, the service gets stuck on that call and there is nowhere to store the event to try again.

The way out is to decouple. Stripe describes the behavior a sender needs to reproduce: in live mode, automatic retries continue for up to three days with exponential backoff; deliveries created in sandbox are retried three times over a few hours. If the destination is disabled or deleted at retry time, future attempts for that event are prevented.

1. The service writes the event to the delivery queue and continues the main flow without waiting for the customer.

2. A delivery worker takes the event from the queue, signs the request, and POSTs it to the customer's endpoint.

3. When the destination returns success, the worker marks the delivery as completed in the delivery log.

4. When the destination fails or does not respond, the worker returns the event to the redelivery queue, to be tried again later.

5. The redelivery queue returns the event to the worker after a wait, and the cycle repeats.

6. When the attempts run out, the event goes to the dead-letter queue, where it is kept for analysis and manual resend.

Besides the automatic retries, Stripe allows manual resends: through the Dashboard, up to fifteen days after the event was created; through the CLI, up to thirty days. Manually resending an event that did not arrive does not cancel the automatic retries, even if the resend returns a 2xx code.

04 · SIGNATURE AND SECURITY

How the destination knows the event is legitimate

Stripe's documentation warns that, without verification, an attacker could send fake events to the endpoint to trigger actions such as releasing orders, granting access, or altering records. The protection described is to sign every event, with the signature in the Stripe-Signature header.

Each endpoint has its own secret, and the destination needs the raw request body to verify the signature. The timestamp goes in the header and is verified by the signature itself, and the application can reject a payload whose timestamp is too old; by default, Stripe's libraries accept a five-minute difference. On each new delivery attempt, Stripe generates a new signature and a new timestamp.

05 · DUPLICATES, ORDERING, AND THE RECEIVER

Repeated delivery, no guaranteed order

The endpoint may receive the same event more than once. The defense described is to record the IDs of events already processed and not process again those already on record. In some cases, two separate Event objects are generated and sent; to identify these duplicates, the documentation uses the ID of the object in data.object together with the event type.

Stripe does not guarantee delivery in the order the events were generated, and the destination must not depend on a specific order. Instead of using the created field to decide the order or whether something has already been processed, the recommendation is to track event IDs. If an event arrives first, you can fetch the missing objects through the API.

On the receiver side, the documentation recommends processing events with a fila that is asynchronous. A large spike of deliveries, such as at the start of the month when all subscriptions renew, can overload the endpoint's hosts, and the queue lets you process events at the pace the system can sustain.

1. The endpoint receives the POST, verifies the signature using the raw body, and returns success right away.

2. The endpoint just puts the event on the queue and responds; the heavy logic is left to the worker.

3. A worker consumes from the queue at the pace the customer's system can sustain.

4. The worker checks the ID log: if the event has already been processed, it ignores it; if not, it processes it and records the ID.

If the sender may repeat deliveries, the receiver needs to be idempotente. Document this for customers and offer the event ID as a deduplication key.

06 · OBSERVABILITY AND LIMITS

What the customer needs to see, and what is left out

Stripe shows, per endpoint, a deliveries tab with the list of events and whether each one is delivered, pending, or failed. When opening an event, the customer sees the HTTP status of the attempt and the time of pending deliveries. Registration also has a limit: up to 16 registered event destinations.

Two design decisions deserve their own justification in the interview: the scheduling of redeliveries, that is, when each event goes back to the queue, and a send limit per destination, so that one slow customer does not consume all the workers.