Webhook Reliability Engineering
Prerequisite: API Design
Your payment processor sends a payment.completed webhook. Your handler credits the customer’s account. The processor’s retry logic fires because your 200 response arrived 50ms after its 30-second timeout. The webhook arrives again. Your handler credits the customer’s account again. The customer now has double their money, and your ledger is wrong.
Webhooks are the standard mechanism for asynchronous event delivery in financial systems. Stripe sends roughly 4 billion webhook events per month. Every payment processor, banking API, and identity verification provider uses them. The delivery model is fundamentally unreliable — HTTP over the public internet, fire-and-forget semantics, no transaction boundary between sender and receiver — and you are building financial logic on top of it.
Delivery Guarantees: At-Least-Once or Nothing
No webhook provider guarantees exactly-once delivery. The delivery model is at-least-once: the provider sends the event, waits for a 2xx response, and retries if it doesn’t get one. Duplicates are an expected operating condition, not an edge case.
| Provider | Retry attempts | Retry window | Timeout |
|---|---|---|---|
| Stripe | Up to 16 | 72 hours | 20 seconds |
| Plaid | Up to 7 | 24 hours | 15 seconds |
| Modern Treasury | Up to 20 | 7 days | 30 seconds |
| Adyen | Up to 10 | 24 hours | 30 seconds |
Duplicates arrive for three reasons. First, your handler succeeded but the response was lost — network failure, load balancer timeout, TCP reset. The provider retries because it never saw the 2xx. Second, the provider’s delivery infrastructure has its own retry semantics — message queues, worker restarts, partition rebalancing — that can re-enqueue an event that was already delivered. Third, the provider intentionally sends the same logical event multiple times: a payment status change might generate both a payment.updated and a payment.completed event that carry overlapping information.
A webhook handler that is not idempotent will eventually create or destroy money. The duplicate is not a bug in the provider. It is a design property of the delivery model.
Idempotent Receivers
Every webhook event carries a unique identifier. Stripe uses evt_1MqLkZ2eZvKYlo2C0NXIMR8O. Plaid uses wh_1234abcd. Your handler must record this identifier before processing the event and reject any event it has already seen.
BEGIN;
INSERT INTO webhook_events (event_id, provider, received_at, status)
VALUES ($1, $2, NOW(), 'processing')
ON CONFLICT (event_id) DO NOTHING;
-- If the insert affected 0 rows, this is a duplicate
-- RETURNING or checking affected rows tells you whether to proceed
-- ... process the event ...
UPDATE webhook_events SET status = 'completed' WHERE event_id = $1;
COMMIT;The idempotency check and the business logic must share a transaction boundary. If you check for duplicates, process the event, and then record the event ID in separate transactions, a crash between processing and recording produces a state where the event was processed but not recorded — the next retry will process it again.
For systems where the webhook handler writes to a different datastore than the idempotency table (e.g., the handler calls an external API), use a two-phase approach: record the event as processing before doing anything, execute the side effect, then mark it completed. On retry, check the status — if it’s processing, the previous attempt may have crashed mid-execution and you need recovery logic, not a simple skip.
| Strategy | Duplicate protection | Crash safety | Complexity |
|---|---|---|---|
| Event ID in same transaction | Full | Full | Low |
| Event ID in separate table, same DB | Full | Partial (gap between check and record) | Low |
| Event ID in Redis + business in Postgres | Partial (Redis eviction) | Low | Medium |
| Idempotency key forwarded to downstream API | Full (if downstream is idempotent) | Full | Medium |
Ordering Is Not Guaranteed
Webhook events for the same resource can arrive out of order. A payment.updated event sent at T=1 and a payment.completed event sent at T=2 may arrive at T=2 and T=1 respectively. The provider’s delivery infrastructure processes events through distributed workers with independent retry queues. Ordering is best-effort at best.
This breaks any handler that applies state transitions sequentially. If your handler processes payment.completed (setting status to completed) and then processes payment.updated (setting status to pending), the final state is wrong.
Two approaches handle this.
Timestamp-based last-write-wins. Each webhook carries a timestamp or a sequence number from the provider. Your handler only applies the update if the incoming event is newer than the last event processed for that resource.
UPDATE payments
SET status = $2, updated_at = $3
WHERE id = $1 AND updated_at < $3;If the update affects zero rows, the event is stale — a newer event was already processed.
Fetch-on-receive. Treat the webhook as a notification, not as data. When you receive a payment.updated event, ignore the payload and fetch the current state from the provider’s API. The API always returns the latest state, regardless of what order events arrived. This adds one API call per webhook but eliminates ordering concerns entirely.
See API Design. Stripe explicitly recommends the fetch-on-receive pattern in their documentation. The webhook tells you something changed. The API tells you what the current state is.
Signature Verification
Every webhook provider signs events with HMAC-SHA256 or a similar scheme. The provider includes a signature header computed from the raw request body and a shared secret. Your handler must verify the signature before processing the event.
import hmac
import hashlib
def verify_stripe_signature(payload: bytes, signature: str, secret: str) -> bool:
timestamp, sig = parse_signature_header(signature)
signed_payload = f"{timestamp}.{payload.decode()}"
expected = hmac.new(
secret.encode(), signed_payload.encode(), hashlib.sha256
).hexdigest()
if not hmac.compare_digest(expected, sig):
return False
# Reject events older than 5 minutes to prevent replay attacks
if abs(time.time() - int(timestamp)) > 300:
return False
return TrueSkipping signature verification exposes you to two attacks. First, an attacker sends a forged payment.completed event with a valid-looking structure. Your handler credits the customer without a real payment. Second, a replay attack: an attacker captures a legitimate webhook and resends it. Timestamp validation in the signature check prevents this — Stripe includes a timestamp in the signed payload, and you reject events older than 5 minutes.
| Provider | Signature scheme | Header | Replay protection |
|---|---|---|---|
| Stripe | HMAC-SHA256 with timestamp | Stripe-Signature |
Timestamp in signed payload |
| Plaid | HMAC-SHA256 | Plaid-Verification |
JWT with expiry |
| Modern Treasury | HMAC-SHA256 | X-Signature |
Timestamp header |
| Adyen | HMAC-SHA256 | X-HMAC-Signature |
Nonce in payload |
Parse the raw request body, not a deserialized-and-reserialized version. JSON serialization is not round-trip stable — field ordering, whitespace, and Unicode escaping can change between deserialization and reserialization, producing a different byte sequence and a failed signature check.
Retry Logic and Backoff
When your handler returns a non-2xx status, the provider retries with exponential backoff. Stripe’s schedule: 5 seconds, 5 minutes, 30 minutes, 2 hours, 5 hours, 10 hours, then roughly every 12 hours up to 72 hours.
Your handler should return the correct status code to signal its intent.
| Status code | Meaning | Provider behavior |
|---|---|---|
| 200–299 | Processed successfully | No retry |
| 400 | Bad request (malformed payload) | Some providers stop retrying |
| 401/403 | Auth failure (bad signature) | Stop retrying |
| 429 | Rate limited | Retry with longer backoff |
| 500–599 | Server error | Retry with backoff |
| Timeout | No response within window | Retry with backoff |
Return 200 as early as possible. Accept the event, enqueue it for processing, and return 200 before doing any business logic. If your handler performs database writes, API calls, or ledger updates synchronously before responding, you risk hitting the provider’s timeout window. A handler that takes 25 seconds to credit an account and then returns 200 at second 25 will be retried by a provider with a 20-second timeout — your system processed the event, but the provider thinks it failed.
POST /webhooks/stripe → 200 (within 2 seconds)
└─ Enqueue to internal processing queue
└─ Worker: validate, deduplicate, process, recordThe internal queue gives you control over retry semantics, concurrency, and error handling. The webhook endpoint becomes a fast ingestion layer.
Dead Letter Queues
Events that fail processing after your internal retry budget belong in a dead letter queue, not in an infinite retry loop. A webhook event that fails because a dependent service is down will succeed when the service recovers. A webhook event that fails because the payload references a customer that does not exist in your system will fail forever.
Classify failures before deciding what to retry.
| Failure type | Example | Action |
|---|---|---|
| Transient | Database connection timeout | Retry with backoff |
| Dependency | Downstream API returns 503 | Retry with longer backoff |
| Data integrity | Referenced customer not found | Dead letter queue |
| Logic error | Unhandled event type | Dead letter queue |
| Poison message | Malformed JSON, invalid schema | Dead letter queue, alert |
Dead letter queue events require a resolution workflow: manual review, automated reprocessing after a fix, or explicit discard with an audit record. For financial events, discarding without review is not an option — a dead-lettered payment.completed event might represent real money that your system has not yet recorded.
Monitor the dead letter queue depth. A growing DLQ means your system is falling behind on event processing. Set alerts at thresholds: 10 events triggers investigation, 100 events triggers incident response, 1,000 events means something structural is broken.
When Webhooks Carry Financial Events
The failure modes above are inconveniences in a social media notification system. In a financial system, they create money, lose money, or produce audit findings.
Double-credit. A duplicate payment.completed event processed twice credits the customer twice. The money in your ledger exceeds the money in your bank account.
Missed event. A payment.failed event that your system never processes leaves a customer’s account in a pending state indefinitely. Depending on your flow, the customer either cannot use their funds or believes they were charged when they were not.
Stale state. An out-of-order refund.created event processed after a refund.completed event resets the refund status to created. The customer contacts support because their refund appears stuck.
Forged event. A payment.completed event with a valid structure but no signature verification credits a customer with no corresponding bank movement. The reconciliation break surfaces at end-of-day — or end-of-month, if your reconciliation cadence is that slow.
Every webhook handler in a financial system must be idempotent, must verify signatures, should treat webhooks as notifications rather than authoritative data, and must have a dead letter path with human review. The webhook is a trigger. The provider’s API is the source of truth. Your ledger is the system of record. The webhook connects them, and the connection is unreliable by design.