API Design When Money Moves

Prerequisite: API Design

Your payment API receives a POST /transfers request. The client sends $500 from account A to account B. Your server processes the transfer, debits A, credits B, and begins generating the 201 Created response. The TCP connection drops before the response reaches the client. The client sees a timeout. It retries. Your server processes the transfer again — debits A a second time, credits B a second time. Account A is now short $500, account B has $500 too many, and your ledger has two entries for a transfer that should have happened once.

Every API that moves money must answer this question before writing any business logic: what happens when the client retries a request that already succeeded?

Idempotency Keys

An idempotency key is a client-generated identifier that tells the server “this is the same request you may have already processed.” The server uses it to return the original response instead of executing the operation again.

POST /transfers
Idempotency-Key: txn_a1b2c3d4e5f6
Content-Type: application/json

{
  "from_account": "acct_001",
  "to_account": "acct_002",
  "amount": 50000,
  "currency": "USD"
}

The server stores the idempotency key alongside the request hash and the response. On retry, it looks up the key, verifies the request body matches, and returns the stored response without re-executing.

CREATE TABLE idempotency_keys (
  key         TEXT PRIMARY KEY,
  request_hash BYTEA NOT NULL,
  response    JSONB,
  status      TEXT NOT NULL DEFAULT 'processing',
  created_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
  expires_at  TIMESTAMPTZ NOT NULL DEFAULT now() + INTERVAL '24 hours'
);

Stripe stores idempotency keys for 24 hours. Modern Treasury stores them for the lifetime of the request. The TTL decision depends on your retry window — if clients retry within minutes, 24 hours is generous. If your system supports scheduled payments that may be retried days later, the window must be longer or the key must be derived from the payment’s own identity.

An idempotency key that only checks the key without verifying the request body is a footgun. If a client reuses a key with different parameters — different amount, different recipient — the server must reject the request, not silently return the old response. Stripe returns 422 Unprocessable Entity when the request body doesn’t match the stored key.

Key Generation

The client generates the key, not the server. This is not a style choice — it’s a correctness requirement. If the server generates the key and returns it in the response, a network failure before the response arrives means the client never receives the key and has no way to deduplicate on retry.

UUIDs (v4) are the common choice. The collision probability of two UUIDv4 values across 1 billion keys is approximately 1 in 10^18. For payment APIs, that margin is sufficient.

Some systems derive idempotency keys from business identifiers — payment:{order_id}:{attempt} — which makes keys predictable and debuggable. The tradeoff is that clients must maintain their own sequence counters, and key reuse across unrelated operations becomes a failure mode if the derivation scheme is ambiguous.

Request Deduplication at the Server

The idempotency key table is the deduplication layer. The implementation has three states:

State	Meaning	On retry
`processing`	Request received, execution in progress	Return `409 Conflict` or block until original completes
`completed`	Request executed, response stored	Return stored response
`failed`	Request failed, no side effects	Allow re-execution with same key

The processing state handles the race condition where two instances of the same request arrive simultaneously. The first instance inserts the key with processing status using an INSERT ... ON CONFLICT DO NOTHING (Postgres) or a conditional put (DynamoDB). If the insert succeeds, this instance owns the key and proceeds with execution. If it fails, another instance already owns it.

INSERT INTO idempotency_keys (key, request_hash, status)
VALUES ($1, $2, 'processing')
ON CONFLICT (key) DO NOTHING
RETURNING key;

If the RETURNING clause returns no rows, the key already exists. Read the existing row: if completed, return the stored response. If processing, either return 409 or poll until the original completes. If failed, allow re-execution — update the status back to processing and proceed.

The critical invariant: the idempotency key insert and the business operation must share a transaction boundary. If they don’t — if you insert the key first, then execute the transfer in a separate transaction — a crash between the two leaves a key in processing state with no way to complete or fail it. Stale processing keys require a reaper process that detects keys stuck in processing beyond a timeout (typically 60 seconds) and marks them failed.

Partial Failure Semantics

A transfer that debits account A and credits account B is two database writes. If the debit succeeds and the credit fails, you have an inconsistent ledger. The API must define what the client sees and what the server does.

Single-database transactions are the simplest solution. Both the debit and credit happen in one database transaction. Either both succeed or neither does. The idempotency key insert lives in the same transaction. This works when both accounts are in the same database — which is true for most ledger architectures.

BEGIN;
INSERT INTO idempotency_keys (key, request_hash, status) VALUES ($1, $2, 'processing');
INSERT INTO entries (account_id, amount, ...) VALUES ('acct_001', -50000, ...);
INSERT INTO entries (account_id, amount, ...) VALUES ('acct_002', 50000, ...);
UPDATE idempotency_keys SET status = 'completed', response = $3 WHERE key = $1;
COMMIT;

When accounts span databases or services, you need a saga. The API returns 202 Accepted immediately with a transfer ID. The client polls for completion or receives a webhook when the saga resolves. The response contract changes from synchronous success/failure to asynchronous eventual resolution.

Pattern	Response	Failure mode	Client experience
Single transaction	`201 Created`	Atomic rollback	Synchronous success/failure
Saga with polling	`202 Accepted`	Compensating actions	Poll until `completed` or `failed`
Saga with webhook	`202 Accepted`	Compensating actions	Receive async notification

The API must never return 200 OK or 201 Created for a transfer that has not fully completed. If the debit succeeded but the credit is pending, the response is 202 Accepted with a status field — not 201 Created with a caveat in the body. The HTTP status code is the contract. A client that checks response.status_code == 201 and proceeds to ship an order must be able to trust that the money moved.

See API Design. The handbook rule — never return 200 with an error body — is non-negotiable for payment APIs. A 200 that means “we received your request but the payment failed” will cost someone money.

Response Contracts for Money

Payment API responses carry information that clients use to make financial decisions. The response contract must be unambiguous about the state of the money.

Every transfer response includes at minimum:

id — Server-generated, globally unique, immutable. The canonical reference for this transfer.
status — One of a closed set: pending, posted, failed, returned. Never free-text.
amount and currency — Echo back exactly what the server executed, not what the client requested. If the server applied fees or FX conversion, the response shows the actual amounts.
created_at — Server timestamp in UTC. The client’s clock is irrelevant.

Stripe’s API returns the full object on every mutation — create, update, and retrieve all return the same shape. This means the client never needs to make a separate GET request after a POST to learn what happened. Modern Treasury follows the same pattern. The cost is larger response payloads (typically 1–3 KB per transfer object). The benefit is that every response is self-contained.

Status Transitions

Transfer statuses follow a state machine. The API documentation must publish the valid transitions, and the server must enforce them.

stateDiagram-v2
    [*] --> pending
    pending --> posted : settlement confirmed
    pending --> failed : rejected by processor
    posted --> returned : chargeback or reversal
    failed --> [*]
    returned --> [*]

A transfer in posted status cannot transition to pending. A transfer in failed status cannot transition to posted. These constraints are not suggestions — they are invariants that clients build automation around. If your API allows a transfer to move from failed to posted because a retry succeeded after the initial failure, clients that took action on the failed status (refunding a customer, canceling an order) now have inconsistent state.

Webhook Reliability

Webhooks notify clients about asynchronous state changes — a transfer that moved from pending to posted, a payment that was returned, an account that was frozen. The delivery semantics determine how much engineering the client must do.

At-least-once delivery is the only practical guarantee. Exactly-once delivery across a network boundary is impossible without cooperation from the receiver. Your system will deliver some webhooks more than once — after a timeout, after a deployment, after a retry — and the receiver must handle duplicates.

Each webhook carries an event_id that the receiver uses for deduplication, and an idempotency_key if the event corresponds to a specific API request. The receiver stores processed event IDs and skips duplicates.

Retry Strategy

Failed webhook deliveries (non-2xx response or timeout) retry with exponential backoff. Stripe retries up to 30 times over approximately 3 days. Modern Treasury retries with 1-minute, 5-minute, 30-minute, 2-hour, and 24-hour intervals.

Attempt	Delay	Cumulative
1	Immediate	0
2	1 min	1 min
3	5 min	6 min
4	30 min	36 min
5	2 hours	2h 36m
6	24 hours	26h 36m

After exhausting retries, the webhook is dead-lettered. The client must poll the API to discover events they missed, or your system surfaces dead-lettered webhooks in a dashboard for manual replay.

Ordering

Webhooks arrive out of order. A transfer.posted event may arrive before the transfer.created event if the first delivery attempt for created times out and retries while posted delivers on the first try. The receiver must handle this — either by ignoring events for unknown resources (and catching up on the next poll) or by buffering events and reordering by sequence number.

Stripe includes a created timestamp on each event. Modern Treasury includes a sequence counter. Neither guarantees delivery order matches creation order. The receiver’s deduplication logic must be order-independent.

Signature Verification

Every webhook request includes a cryptographic signature that proves it came from your system, not from an attacker who discovered the endpoint URL. HMAC-SHA256 over the raw request body with a shared secret is the standard.

import hmac, hashlib

def verify_webhook(payload: bytes, signature: str, secret: str) -> bool:
    expected = hmac.new(
        secret.encode(), payload, hashlib.sha256
    ).hexdigest()
    return hmac.compare_digest(expected, signature)

Use hmac.compare_digest (constant-time comparison), not ==. A timing side-channel on signature comparison leaks the secret one character at a time. At webhook-level traffic volumes the attack is impractical, but the fix is one function call, and the vulnerability is permanent if ignored.

The receiver must verify the signature against the raw request body, not a parsed-and-reserialized version. JSON serialization is not deterministic — key ordering, whitespace, and Unicode escaping vary across parsers. Parse after verification, never before.

Error Responses That Don’t Lie

Payment API errors must distinguish between “you made a mistake” (client error, do not retry) and “something went wrong on our end” (server error, retry is appropriate).

Status	Meaning	Client action
`400`	Malformed request — bad JSON, missing field, invalid type	Fix the request, do not retry as-is
`402`	Insufficient funds or payment declined	Inform the user, do not retry without user action
`409`	Idempotency key conflict — concurrent duplicate request	Wait and retry, or fetch the existing resource
`422`	Valid JSON but business rule violation — amount exceeds limit, account frozen	Fix the input, do not retry as-is
`429`	Rate limited	Retry after `Retry-After` header
`500`	Server error — safe to retry with the same idempotency key	Retry with exponential backoff
`503`	Service temporarily unavailable	Retry after `Retry-After` header

The distinction between 400, 402, and 422 matters because retry logic is automated. A client that retries a 422 (account frozen) wastes resources and may trigger rate limiting. A client that does not retry a 500 (transient server error) leaves a payment unprocessed. The HTTP status code is the machine-readable signal. The response body provides the human-readable explanation.

Never return 200 with {"error": "insufficient funds"}. The client’s HTTP library sees 200 and reports success. The error handling code never runs. The payment appears to succeed in the client’s logs. The reconciliation break surfaces hours or days later.