Ledger Drift Runbooks

Prerequisite: The Ledger, Incident Response

Your reconciliation job fires at 2:17 AM. The alert says the customer funds sub-ledger is $43,891.22 higher than the sum of its entries. You have two questions: is money missing, or is the ledger wrong? The answer determines whether this is a data incident or a financial incident — and you have about 30 minutes to figure out which before the next automated sweep moves money based on that incorrect balance.

Ledger drift is the gap between what your ledger says and what is true. “True” might be the bank’s actual balance, the on-chain token supply, or the mathematical sum of your own entries. Drift is not a bug category — it is a symptom. The runbook’s job is to diagnose the cause, contain the blast radius, and restore agreement between systems before the drift compounds.

Classifying the Drift

Not all drift is equal. The first step in any runbook is classification, because the response differs by orders of magnitude.

Class	Definition	Example	Severity
Accounting drift	Ledger disagrees with its own entries	Running balance ≠ SUM(entries)	Critical — data integrity violation
Internal drift	Two internal systems disagree	Sub-ledger total ≠ general ledger	High — reconciliation break
External drift	Internal ledger disagrees with external source	Ledger balance ≠ bank statement	High — potential money movement error
Cross-domain drift	Fiat ledger disagrees with on-chain state	Reserve balance ≠ token supply	Critical — solvency implication

Accounting drift means your own data is inconsistent. This is always a bug — entries are the source of truth, and a stored balance that disagrees with SUM(entries) is wrong by definition. Every other class requires investigation to determine which side is correct.

Do not attempt to fix drift before classifying it. A premature compensating entry on an accounting drift masks the bug. A premature adjustment on an external drift creates a second discrepancy when the bank’s delayed transaction arrives.

Runbook 1: Accounting Drift

Trigger: running_balance ≠ SUM(entries) WHERE account_id = X

This is the most urgent class. Your balance derivation is broken, which means every balance-dependent operation — spending checks, settlement calculations, regulatory reporting — is operating on wrong data.

Step 1: Freeze the account. Prevent new entries from posting to the affected account. If your system supports account-level holds, apply one. If not, disable the flows that write to this account.

-- Verify the drift
SELECT
  a.balance AS stored_balance,
  SUM(e.amount) AS derived_balance,
  a.balance - SUM(e.amount) AS drift
FROM accounts a
JOIN entries e ON e.account_id = a.id
WHERE a.id = $1
GROUP BY a.id, a.balance;

Step 2: Find the divergence point. Walk backward through the entry sequence. The drift originated at the entry where the running balance first disagreed with the cumulative sum.

-- Find the first divergent entry
WITH running AS (
  SELECT
    id,
    amount,
    created_at,
    SUM(amount) OVER (ORDER BY sequence_id) AS cumulative,
    balance_after
  FROM entries
  WHERE account_id = $1
  ORDER BY sequence_id
)
SELECT * FROM running
WHERE cumulative != balance_after
ORDER BY created_at
LIMIT 1;

Step 3: Identify the cause. Common causes, in order of frequency:

Cause	Signature	Fix
Concurrent write without serialization	Two entries with the same `balance_after`	Enforce serializable isolation or row-level locking
Partial transaction	Entry inserted but balance update failed	Wrap both in the same database transaction
Manual adjustment	Direct UPDATE on balance column	Revoke write access, add audit trigger
Snapshot corruption	Stale snapshot used for derivation	Rebuild snapshot from entries

Step 4: Correct the balance. Set the stored balance to the derived value. Log the correction as an administrative event with the incident ID, the old value, and the new value.

Step 5: Verify downstream. Any system that cached or propagated the incorrect balance — settlement calculations, regulatory reports, customer-facing displays — must be recalculated from the corrected state.

Runbook 2: Internal Drift (Sub-Ledger vs General Ledger)

Trigger: SUM(sub_ledger_balances) ≠ general_ledger_balance for a given account class.

The general ledger is a derived view of the sub-ledgers. When they disagree, either a sub-ledger posted an entry that didn’t propagate to the GL, or the GL received an entry that doesn’t exist in any sub-ledger.

Step 1: Isolate the break. Determine which sub-ledger is off.

SELECT
  sub_ledger_id,
  SUM(balance) AS sub_total,
  gl.balance AS gl_balance,
  SUM(balance) - gl.balance AS break_amount
FROM sub_ledger_accounts sla
CROSS JOIN general_ledger gl
WHERE gl.account_class = $1
GROUP BY sub_ledger_id, gl.balance
HAVING SUM(balance) != gl.balance;

Step 2: Check the propagation pipeline. If the GL is updated asynchronously (CDC, event stream, batch job), the break may be a timing issue. Check the pipeline lag.

Pipeline type	Expected lag	Break threshold
Synchronous (same transaction)	0	Any nonzero break is a bug
CDC / streaming	< 30 seconds	Break persisting > 5 minutes
Batch (hourly/daily)	Up to batch interval	Break persisting past next batch

If the break is within the expected lag window and shrinking, it is a timing break — not a data integrity issue. Monitor it through the next pipeline cycle.

Step 3: Reconcile entries. If the break persists past the lag window, compare entry-level detail between the sub-ledger and GL. Look for:

Missing entries: Present in sub-ledger, absent in GL (or vice versa). The propagation mechanism dropped the event.
Duplicate entries: Same entry posted twice to the GL. Deduplication failed.
Amount mismatches: Entry exists in both but with different amounts. Transformation error in the pipeline.

Step 4: Resolve. For missing entries, replay the event or create a manual GL entry referencing the sub-ledger entry ID. For duplicates, reverse the duplicate. For amount mismatches, determine which side is correct (the sub-ledger entry is authoritative — it’s closer to the source transaction) and adjust the GL.

See Sub-Ledger Architecture. Reconciliation breaks between sub-ledgers and the GL are expected signals in a multi-book architecture. A break is an alarm — it tells you the propagation pipeline needs attention, not that the ledger is fundamentally broken.

Runbook 3: External Drift (Ledger vs Bank)

Trigger: internal_ledger_balance ≠ bank_statement_balance for a given bank account.

This is the most common drift class in production. Banks and your ledger process events at different speeds, apply different cutoff times, and report with different granularity.

Step 1: Establish the comparison window. Bank statements have cutoff times (often 5:00 PM or midnight local time). Your ledger is continuous. Compare balances at the bank’s cutoff, not at the current moment.

Step 2: Account for in-transit items.

Item type	Direction	Effect
Outgoing payments submitted, not yet debited	Ledger has deducted, bank has not	Ledger < bank
Incoming payments received by bank, not yet booked	Bank has credited, ledger has not	Ledger < bank
Returns in transit	Bank has reversed, ledger has not	Ledger > bank
Holds and pending authorizations	Bank has reserved, ledger may not	Varies

adjusted_ledger = ledger_balance
  + pending_outgoing       -- ledger deducted, bank hasn't
  - pending_incoming       -- bank credited, ledger hasn't
  + returns_in_transit     -- bank reversed, ledger hasn't

if adjusted_ledger == bank_balance:
    drift is timing-only, no action needed

Step 3: Investigate residual drift. If the adjusted balance still doesn’t match, the drift is real. Common causes:

Bank fees or interest not yet booked in your ledger. Banks post fees on their own schedule — monthly maintenance fees, wire fees, interest credits.
Rejected transactions the bank processed differently than expected. A wire that your system marked as sent but the bank rejected.
Manual bank operations — customer-initiated wires, regulatory holds, or fraud freezes applied directly at the bank.

Step 4: Book the adjustments. Each identified item becomes a ledger entry with a reference to the bank statement line. Do not create a single “adjustment” entry for the aggregate difference — each item must be individually traceable.

Runbook 4: Cross-Domain Drift (Fiat vs On-Chain)

Trigger: fiat_reserve_balance ≠ total_token_supply beyond configured tolerance.

This is the most operationally complex drift class. Two fundamentally different systems — a bank account and a blockchain — must agree on the same number.

Step 1: Establish current state on both sides.

on_chain_supply  = totalSupply() across all chains
fiat_reserves    = sum(bank_balances) + sum(money_market_positions)
in_transit_mints = mints requested, fiat received, tokens not yet minted
in_transit_burns = burns confirmed on-chain, fiat not yet disbursed

Step 2: Compute adjusted positions.

expected_supply = fiat_reserves - in_transit_burns + in_transit_mints
drift = on_chain_supply - expected_supply

A positive drift means more tokens exist than reserves back them — this is the critical direction. A negative drift means reserves exceed supply — capital is stranded but solvency is not at risk.

Drift direction	Severity	Immediate action
Supply > reserves (positive)	Critical	Pause minting, investigate
Supply < reserves (negative)	Medium	Investigate, no emergency action
Within tolerance (±0.01%)	Normal	Log, continue monitoring

Step 3: Trace the source. Check the mint/burn event log against the fiat movement log. Every mint should have a corresponding fiat deposit. Every burn should have a corresponding fiat disbursement.

-- Find unmatched mints (tokens exist without fiat backing)
SELECT m.tx_hash, m.amount, m.chain, m.timestamp
FROM mint_events m
LEFT JOIN fiat_deposits f
  ON m.reference_id = f.reference_id
WHERE f.id IS NULL
  AND m.timestamp > NOW() - INTERVAL '7 days';

Step 4: Remediate by cause.

Orphaned mint (fiat deposit failed after mint): Burn the unbacked tokens or expedite the fiat deposit.
Orphaned burn (fiat disbursement failed after burn): Process the fiat disbursement manually.
Chain reorg invalidated a burn: The tokens are back in circulation. Re-execute the burn after finality.
Stale oracle / delayed bank reporting: Wait for the next reporting cycle; log the timing break.

Cross-domain drift with supply exceeding reserves is a solvency event. Even a $1 discrepancy in the wrong direction means your attestation report cannot state that tokens are fully backed. Escalate immediately — do not wait for the next reconciliation cycle.

Post-Incident Verification

After resolving any drift, run the full verification checklist before declaring the incident resolved.

Check	Query	Pass criteria
Account balance integrity	`running_balance = SUM(entries)` for affected accounts	Zero drift
Sub-ledger roll-up	`SUM(sub_ledger) = GL` for affected account classes	Zero drift
External reconciliation	Adjusted ledger = bank statement	Within tolerance
Cross-domain reconciliation	`token_supply = adjusted_reserves`	Within tolerance
Hash chain integrity	Verify chain from last known-good checkpoint	No breaks
Downstream recalculation	Recompute any reports or settlements that used incorrect data	Results match corrected state

Log the incident with: root cause, drift amount, duration (when drift started to when it was resolved), blast radius (which accounts, which downstream systems), and the corrective entries created. This log is audit evidence — regulators and auditors will ask for it during examination.

The most dangerous drift is the one nobody notices. Continuous reconciliation — running these checks every 60 seconds instead of once a day — converts a 2 AM emergency into a 2 PM alert that gets resolved before end-of-day processing. The runbooks stay the same. The urgency drops by an order of magnitude.