Hot Accounts and Contention
Prerequisite: The Ledger
Your settlement account processes 2,000 incoming entries per second at end-of-day. Each entry takes a row-level lock on the account’s balance row, holds it for the duration of a transaction that includes an entry insert and a balance update, and releases it on commit. The second writer waits. The third writer waits behind the second. By the time writer 2,000 arrives, the lock queue is deep enough that your p99 write latency has climbed from 12ms to 4 seconds, and your application’s connection pool is exhausted.
The account is not broken. The database is not misconfigured. The account is hot — it receives writes at a rate that serializes on a single row, and serialization under load produces a queue.
What Makes an Account Hot
Most accounts in a ledger are cold. A consumer checking account receives a few transactions per day. A small-business operating account might see a hundred. These accounts never contend because two writes arriving in the same millisecond is unlikely.
Hot accounts are different. They receive a disproportionate share of a system’s total write volume, and that volume concentrates on a single database row.
| Account type | Why it’s hot | Typical write rate |
|---|---|---|
| Omnibus wallet | Pools funds for thousands of users; every deposit and withdrawal touches it | 500–5,000 entries/sec |
| Fee collection account | Every transaction in the system generates a fee entry | 1x transaction volume |
| Settlement account | Batch settlements from payment processors land in bursts | 1,000–10,000 entries/sec in bursts |
| Pooled funding account | Aggregates customer funds for a single banking partner | Proportional to total customer activity |
| Suspense account | Holds unmatched or in-flight funds pending resolution | Spiky, correlated with system issues |
A system with 100,000 customer accounts and 5 operational accounts can have 80% of its write contention concentrated on those 5 accounts. The long tail is cold. The head is on fire.
See The Ledger. Double-entry means every entry touches at least two accounts. A customer deposit creates an entry on the customer’s cold account and an entry on the omnibus hot account. The hot account is the bottleneck even when the customer account is not.
Pessimistic Locking and the Write Queue
The standard pattern for maintaining a running balance is to lock the account row, insert the entry, update the balance, and commit.
BEGIN;
SELECT balance FROM accounts WHERE id = $1 FOR UPDATE;
INSERT INTO entries (account_id, amount, ...) VALUES ($1, $2, ...);
UPDATE accounts SET balance = balance + $2 WHERE id = $1;
COMMIT;FOR UPDATE acquires an exclusive row-level lock. Every concurrent transaction targeting the same account blocks on this lock until the holding transaction commits or rolls back. On a cold account, this is invisible — the lock is never contended. On a hot account, the lock creates a serial queue.
The throughput ceiling is determined by the lock hold time. If the transaction takes 5ms (network round trips, entry insert, balance update, commit), the theoretical maximum is 200 writes per second to that account. Faster transactions raise the ceiling; more concurrent writers don’t — they just lengthen the queue.
| Lock hold time | Max throughput (single account) | Queue depth at 1,000 writes/sec |
|---|---|---|
| 2ms | 500 writes/sec | 2 |
| 5ms | 200 writes/sec | 5 |
| 10ms | 100 writes/sec | 10 |
| 50ms | 20 writes/sec | 50 |
Under sustained load, the queue depth grows linearly until writers start timing out. Those timeouts surface as failed transactions, which the application retries, which adds more writers to the queue.
Optimistic Concurrency and the Retry Storm
Optimistic concurrency control replaces the exclusive lock with a version check at commit time. Read the account’s current version, do the work, and attempt the update only if the version has not changed.
BEGIN;
SELECT balance, version FROM accounts WHERE id = $1;
-- application checks balance, prepares entry
INSERT INTO entries (account_id, amount, ...) VALUES ($1, $2, ...);
UPDATE accounts
SET balance = balance + $2, version = version + 1
WHERE id = $1 AND version = $3;
-- if UPDATE affected 0 rows: rollback and retry
COMMIT;When the UPDATE matches zero rows, the version has changed — another transaction committed first. The application rolls back and retries from the beginning: re-read the balance, re-validate, re-insert, re-attempt the update.
Under low contention, this outperforms pessimistic locking. No writer blocks — they all proceed in parallel, and the loser retries. Under high contention, the retry rate climbs. With N concurrent writers, approximately N−1 fail on each round. Those N−1 retry, contend again, and approximately N−2 fail. The total number of database operations grows superlinearly with the number of concurrent writers.
| Concurrent writers | Expected retries per successful write | Total DB operations per success |
|---|---|---|
| 2 | 0.5 | 3 |
| 5 | 2.5 | 7 |
| 10 | 5.5 | 13 |
| 50 | ~25 | ~52 |
At 50 concurrent writers, each successful write consumes roughly 52 database operations. The retries themselves become the dominant load on the database. You have replaced a lock queue with a retry storm, and the retry storm consumes more total resources than the queue did.
Optimistic concurrency is not a scaling strategy for hot accounts. It trades latency (waiting in a lock queue) for throughput waste (retrying failed transactions). On cold accounts, the tradeoff is favorable. On hot accounts, the retry amplification makes it worse than pessimistic locking.
Partitioned Writes
Split the hot account into N sub-accounts. Instead of one omnibus wallet, create omnibus_01 through omnibus_16. Route each incoming write to a randomly selected partition. The account’s true balance is the sum of all partitions.
-- Write: route to a random partition
-- Application computes partition_id once per write:
-- 'omnibus_' || lpad((floor(random() * 16) + 1)::text, 2, '0')
INSERT INTO entries (account_id, amount, ...)
VALUES ($partition_id, $1, ...);
UPDATE accounts
SET balance = balance + $1
WHERE id = $partition_id;
-- Read: sum all partitions
SELECT SUM(balance) FROM accounts
WHERE id LIKE 'omnibus_%';With 16 partitions and uniformly distributed writes, each partition receives 1/16th of the traffic. The per-partition throughput ceiling is unchanged, but the aggregate throughput is 16x higher. A hot account that maxed out at 200 writes/sec with a 5ms lock hold time now handles 3,200 writes/sec across partitions.
| Partitions | Aggregate throughput (5ms lock) | Read cost |
|---|---|---|
| 1 | 200 writes/sec | 1 row |
| 4 | 800 writes/sec | 4 rows |
| 16 | 3,200 writes/sec | 16 rows |
| 64 | 12,800 writes/sec | 64 rows |
The costs appear on the read path and in balance management.
Read amplification. A balance check that was one row is now N rows. For 16 partitions this is negligible. For 256 partitions it starts to matter, though a covering index keeps the scan fast.
Debit routing. Credits can go to any partition. Debits cannot — you must route a debit to a partition with sufficient balance. Random routing for debits fails when the selected partition has a zero balance while others hold funds. The routing logic must either check balances before selecting a partition (adding a read before the write) or use a deterministic routing strategy that keeps balances roughly even.
Rebalancing. Over time, partitions accumulate uneven balances. Some partitions are flush, others are nearly empty. A background rebalancing job moves funds between partitions with internal ledger entries. The rebalancing entries are real ledger entries — they appear in the audit trail — but they represent internal housekeeping, not customer activity. Your reconciliation process must distinguish the two.
Google’s Cloud Spanner documentation specifically recommends this pattern for financial systems with hot rows. Their “Hot Rows, Cool Solutions” guidance describes partitioning a single counter across multiple rows to avoid write-throughput bottlenecks on a single split.
Batching and Coalescing
Instead of writing each entry individually, collect entries into a batch and apply them in a single transaction. A batch of 100 entries takes one lock acquisition instead of 100.
BEGIN;
SELECT balance FROM accounts WHERE id = $1 FOR UPDATE;
INSERT INTO entries (account_id, amount, ...)
VALUES
($1, $2, ...),
($1, $3, ...),
($1, $4, ...),
-- ... up to batch_size entries
;
UPDATE accounts
SET balance = balance + $batch_total
WHERE id = $1;
COMMIT;A batch of 100 entries with a 10ms transaction time achieves 10,000 entries/sec through a single account — 100x the throughput of individual writes with the same lock hold time. The entries are real individual entries in the ledger; only the lock acquisition is shared.
The tradeoff is latency. Each entry must wait for the batch to fill (or for a timeout to flush a partial batch). A batch window of 50ms means every entry waits up to 50ms before processing begins. A batch window of 500ms means higher throughput but half-second minimum latency.
| Batch window | Batch size (at 1,000 entries/sec) | Effective throughput | Added latency |
|---|---|---|---|
| 10ms | 10 | 1,000/sec | 10ms |
| 50ms | 50 | 5,000/sec | 50ms |
| 200ms | 200 | 20,000/sec | 200ms |
Modern Treasury uses this approach on their async write path. Entries that do not require real-time balance checks route to an SQS queue. A consumer batches queued entries and applies them in bulk. The target is all queued entries processed within 60 seconds; actual p90 is 1 second.
The batch approach works for entries that tolerate latency — settlement postings, fee accruals, interest calculations. It does not work for entries that require immediate balance feedback, such as a payment authorization that must confirm available funds before proceeding.
See Balance Derivation at Scale. Modern Treasury’s hybrid sync/async model splits the write path by consistency requirement. Synchronous writes take locks for immediate balance checks. Asynchronous writes batch for throughput. The caller decides which path to use.
Eliminating Contention by Design
Some systems avoid the hot-account problem entirely by changing the execution model.
TigerBeetle processes all operations on a single thread in a deterministic state machine. There is no per-account contention because there is no concurrent access — every operation runs sequentially. Under heavy contention on a single account, Postgres throughput degrades from 15,000 TPS to approximately 3,200 TPS. TigerBeetle maintains a stable 42,000 TPS regardless of account distribution. The contention penalty is zero because the execution model has no locks.
Fragment uses DynamoDB conditional writes for the transactional (current) balance. A conditional write is a compare-and-swap: read the current balance, compute the new balance, and write it only if the stored value has not changed. Under contention, the conditional write fails and the application retries — but DynamoDB’s single-digit-millisecond write latency means the retry window is small. At 14,578 entries/sec with a p95 write latency of 69ms, the conditional write path handles contention without explicit locking.
Modern Treasury separates hot-path from cold-path at the API level. The caller specifies whether an entry requires synchronous balance locking. Entries that need real-time balance feedback take the synchronous path with Postgres row locks. Entries that do not — settlement postings, batch imports, reconciliation adjustments — route through the async queue. This design acknowledges that most entries on a hot account do not need the balance checked before posting. Separating the paths means only the entries that need locks compete for them.
| System | Contention strategy | Hot account throughput | Tradeoff |
|---|---|---|---|
| Postgres (pessimistic) | Row-level locks | ~200/sec per account (5ms lock) | Queue depth under load |
| Postgres (optimistic) | Version check + retry | Degrades superlinearly | Retry storm |
| Partitioned writes | N sub-accounts | N × single-account throughput | Debit routing, rebalancing |
| Batching | Collect and flush | batch_size × single-account throughput | Added latency |
| TigerBeetle | Single-threaded state machine | 42,000 TPS stable | No SQL, co-processor model |
| Fragment | DynamoDB conditional writes | 14,578 entries/sec | No ad-hoc queries |
| Modern Treasury | Sync/async split | 4,800 entries/sec | Async entries delayed up to 60s |