Idempotency at Scale
Prerequisite: API Design, Distributed Systems
Your payment API receives a POST to create a $500 transfer. The request succeeds — the entry posts, the balance updates, the webhook fires. But the client never receives the 200 response. A load balancer timeout, a dropped TCP connection, a mobile app backgrounded at the wrong moment. The client retries. Without idempotency, your system processes the transfer again. The customer’s account is now $1,000 lighter, and your support queue has a ticket that ends with “I was charged twice.”
The idempotency problem is not “how do you deduplicate requests.” It is how you deduplicate requests across unreliable networks, distributed services, and retry windows measured in days — without turning your API into a bottleneck.
The Idempotency Key Contract
The client generates a unique key — typically a UUID v4 — and sends it in a request header. The server stores the key alongside the response. If the same key arrives again, the server returns the stored response without re-executing the operation.
POST /v1/transfers
Idempotency-Key: 7a3b8c9d-4e5f-6a7b-8c9d-0e1f2a3b4c5d
Content-Type: application/json
{"from": "acct_001", "to": "acct_002", "amount": 50000, "currency": "USD"}First request: execute the transfer, store the response keyed by the idempotency key, return 200. Second request with the same key: look up the stored response, return it verbatim. No transfer created, no balance changed, no webhook fired.
The contract has three requirements:
- The key is client-generated. The server cannot generate idempotency keys — it would need to identify “same request” before having a key, which is the problem idempotency solves.
- Same key, same parameters. If a client sends the same key with different request bodies, the server must reject it. Allowing parameter changes on a reused key turns idempotency into an aliasing mechanism.
- The response is identical. The retry must return exactly the response the original request produced — same status code, same body, same headers. A 201 on the first call and a 200 on the retry is a contract violation that breaks client state machines.
Stripe processes over a billion API requests per month. Every mutating endpoint requires an idempotency key. When a key is reused with different parameters, Stripe returns a 400 with an explicit error: the key was already used for a different request. This catches a common client bug — reusing keys across logically distinct operations.
Key Generation
UUID v4 is the default. 122 bits of randomness yields a collision probability of 1 in 2.7 × 10^18 for a billion keys — effectively zero. The key must be generated before the first attempt and reused on all retries of the same logical operation.
The failure mode is generating a new key per retry. If the client creates a fresh UUID on each attempt, every retry looks like a new request. This is not an edge case — it is the single most common idempotency bug in client implementations.
| Key strategy | Collision risk | Replay correctness | Common mistake |
|---|---|---|---|
| UUID v4 per operation | ~0 | Correct if reused on retry | New UUID per attempt |
| Deterministic hash of params | ~0 | Correct | Hash doesn’t include timestamp — two identical transfers days apart collide |
| Client-provided order ID | Depends on client | Correct | Client reuses order IDs across different operations |
| Sequential integer | None | Correct | Gaps in sequence cause false-positive dedup on unrelated requests |
Deterministic keys — derived from hashing the request parameters — solve the “new key per retry” problem but introduce a different one. Two $500 transfers from account A to account B on the same day produce the same hash. You need enough entropy in the hash input to distinguish intentionally identical operations from retries. Including a client-generated nonce in the hash input resolves this, at which point you have effectively reinvented UUID v4 with extra steps.
Storage and Lookup
The idempotency store maps keys to responses. Every mutating request checks this store before executing.
CREATE TABLE idempotency_keys (
key TEXT PRIMARY KEY,
request_hash BYTEA NOT NULL,
status TEXT NOT NULL DEFAULT 'processing',
response JSONB,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);The write path is a conditional insert: INSERT ... ON CONFLICT (key) DO NOTHING. If the insert succeeds, the key is new — proceed with execution. If it conflicts, the key exists — return the stored response.
The critical race condition is two identical requests arriving simultaneously. Both check the store, both find no key, both proceed to execute. The conditional insert prevents this at the database level — only one insert succeeds. But the loser must not execute the operation. The pattern:
BEGIN;
INSERT INTO idempotency_keys (key, request_hash, status)
VALUES ($1, $2, 'processing')
ON CONFLICT (key) DO NOTHING;
-- Check if we won the insert
SELECT status, response FROM idempotency_keys WHERE key = $1;
-- If status = 'processing' and response IS NULL: we won, execute
-- If status = 'complete' and response IS NOT NULL: return stored response
-- If status = 'processing' and response IS NULL but we didn't insert: another request is in-flight, return 409
COMMIT;The “in-flight” state matters. When request A is still processing and request B arrives with the same key, B should not wait indefinitely. Return 409 Conflict and let the client retry after a delay. Stripe returns 409 for in-flight collisions with a Retry-After header.
See API Design. The idempotency key is an API-level concern. The underlying services — ledger, payment processor, notification — do not need to know about it. The API layer handles deduplication; the services handle execution.
TTL and the Retention Window
Idempotency keys cannot live forever. A key stored indefinitely means the store grows without bound — at 1 million API requests per day, that is 365 million rows per year of key-response pairs.
| TTL | Store size (1M requests/day) | Retry coverage |
|---|---|---|
| 1 hour | ~42K rows | Covers network retries, not human retries |
| 24 hours | ~1M rows | Covers same-day retries and most automated retry policies |
| 7 days | ~7M rows | Covers weekend retry scenarios |
| 30 days | ~30M rows | Covers billing dispute windows |
Stripe uses a 24-hour TTL. After 24 hours, the key expires and the same key can be reused — but since keys are UUIDs, reuse is effectively impossible. The 24-hour window covers the vast majority of retry scenarios: client-side retry loops (seconds to minutes), queue-based retry policies (minutes to hours), and manual retries by engineers debugging a failed request (same day).
The danger of a short TTL is a retry arriving after expiration. If the key has been garbage-collected and the client retries with the same key, the server treats it as a new request and executes again. For financial operations, this means a second charge. The TTL must be longer than the longest plausible retry window for your use case.
For payment operations with chargeback windows of 60–120 days, the idempotency key’s TTL is not the only deduplication layer. The ledger itself provides a second check — the transfer’s own unique identifier (not the idempotency key) prevents duplicate entries regardless of whether the idempotency key has expired.
Distributed Idempotency
In a monolithic service with a single database, idempotency is a single-table lookup. In a microservice architecture, a payment request might traverse an API gateway, a payment orchestrator, a ledger service, and a bank integration service. Each service has its own database. The idempotency key lives at the API layer — but what about the downstream services?
The failure mode: the API layer accepts a request, forwards it to the ledger service, which posts the entry. Then the call to the bank integration service fails. The API layer retries the entire flow. The ledger service receives a second request for the same entry. Without its own deduplication, it posts a duplicate.
Three patterns handle this:
Key propagation. The API layer generates a deterministic sub-key for each downstream call: hash(idempotency_key + service_name). Each service uses its sub-key for its own idempotency check. The same API-level retry produces the same sub-keys, and each service deduplicates independently.
Orchestrator ownership. A single orchestrator service owns the entire flow’s idempotency. It records each step’s completion status in its own database. On retry, it skips completed steps and resumes from the point of failure. AWS Step Functions and Temporal use this pattern — the workflow engine is the single source of truth for “what has already happened.”
Event-driven deduplication. Each service consumes events with a unique event ID. The consumer tracks processed event IDs and skips duplicates. Kafka consumer offsets provide at-least-once delivery; the consumer’s deduplication table provides exactly-once processing.
| Pattern | Complexity | Consistency guarantee | Operational cost |
|---|---|---|---|
| Key propagation | Low | Each service independently consistent | N idempotency stores to monitor |
| Orchestrator ownership | Medium | Orchestrator is single source of truth | Orchestrator is a SPOF |
| Event-driven dedup | High | Eventually consistent | Event ID tracking per consumer |
“Exactly-once” in distributed systems is a property of the outcome, not the delivery. The network delivers at-least-once. Your application makes it appear exactly-once through idempotent handlers. Any claim of “exactly-once delivery” is either redefining the term or hiding the deduplication layer.
The Database Transaction Boundary
The idempotency key insert and the business operation must share a transaction boundary. If they do not, a crash between “record the key” and “execute the operation” produces a key with no corresponding action — the operation never happened, but the key prevents it from ever happening.
BEGIN;
-- Step 1: Claim the idempotency key
INSERT INTO idempotency_keys (key, request_hash, status)
VALUES ($1, $2, 'processing')
ON CONFLICT (key) DO NOTHING;
-- Step 2: Execute the operation (same transaction)
INSERT INTO entries (account_id, amount, ...) VALUES (...);
UPDATE accounts SET balance = balance + $amount WHERE id = $account_id;
-- Step 3: Record the response
UPDATE idempotency_keys
SET status = 'complete', response = $response
WHERE key = $1;
COMMIT;If the transaction commits, both the key and the operation are recorded atomically. If it rolls back, neither is recorded, and the retry will succeed. There is no window where the key exists without the operation, or the operation exists without the key.
When the business operation spans multiple databases — the idempotency key is in Postgres, the ledger entry is in DynamoDB, the notification is in SQS — you cannot use a single database transaction. The orchestrator pattern becomes necessary: record intent in one database, execute steps idempotently, and reconcile on failure.
Idempotency Beyond the API
Idempotency is not only an API concern. Every system boundary where a message crosses an unreliable channel needs it.
Webhook delivery. Your system sends a webhook to a merchant. The merchant’s server is slow — your HTTP client times out after 30 seconds. You retry. The merchant received both deliveries and processes both. Every webhook must carry a unique event ID. The receiver deduplicates on event ID before processing.
Queue consumers. An SQS consumer processes a payment message, posts the entry, and crashes before acknowledging the message. SQS re-delivers it. The consumer must check whether the entry already exists before posting again. The entry’s own unique identifier — not the SQS message ID — is the deduplication key, because SQS can assign different message IDs to redeliveries of the same logical message.
Batch processing. A nightly settlement job processes 50,000 entries. It crashes at entry 30,001. On restart, it must skip the first 30,000 without re-posting them. Each entry in the batch carries a deterministic identifier derived from the batch ID and the entry’s position. The ledger rejects duplicates on this identifier.
Cron jobs. A daily interest calculation runs at midnight. The job completes but the completion flag fails to persist. The scheduler fires it again. Without idempotency, interest is applied twice. The job writes a completion record keyed by the calculation date. The second run finds the record and exits.
The pattern is always the same: assign a unique identifier to the logical operation before the first attempt, and check for that identifier before executing. The identifier’s scope — request, event, batch entry, job run — changes. The mechanism does not.