Cross-Service Saga Orchestration

Prerequisite: Distributed Systems

Your payment flow debits a customer wallet, reserves inventory in a separate service, and initiates a bank transfer through a third-party API. The bank transfer times out after 30 seconds. Your system retries. The retry succeeds — but so did the original. Two bank transfers execute. The customer’s account is debited once, but money left twice. Your ledger and the bank disagree by $4,200, and the reconciliation break won’t surface until tomorrow’s batch run.

Distributed transactions don’t work when the participants are independent services with separate databases and external APIs. The saga pattern replaces a single ACID transaction with a sequence of local transactions, each paired with a compensating action that semantically undoes it if a later step fails. The compensation is a new forward action, not a rollback — both the original and the compensation appear as entries in the ledger.

The Anatomy of a Financial Saga

A saga is a directed sequence of steps. Each step performs a local transaction and, on success, advances the saga to the next step. If any step fails, the saga executes compensation for every previously completed step in reverse order.

A payment disbursement saga has four steps:

Step	Action	Compensation
1. Debit customer	Debit $500 from wallet	Credit $500 back to wallet
2. Reserve funds	Place hold in treasury account	Release hold
3. Initiate transfer	Submit ACH origination to bank	Submit return/reversal to bank
4. Confirm settlement	Mark payment as settled	Mark payment as failed, credit customer

Each step’s compensation is a new transaction. Step 3’s compensation — submitting a return to the bank — is not “undoing” the transfer. It is a separate financial operation with its own settlement timeline, fees, and failure modes.

Compensation is not rollback. A compensating action for a bank transfer takes 1–3 business days to settle. During that window, the customer’s funds are in flight. Your system must track compensations as first-class operations with their own state machines, not as cleanup callbacks.

Orchestration vs Choreography

The saga steps must be coordinated. Two patterns exist.

Choreography relies on events. Each service publishes an event when its step completes. The next service subscribes to that event and begins its step. No central coordinator — the workflow emerges from the event chain.

WalletService → "wallet.debited" → TreasuryService → "funds.reserved" → BankService → "transfer.initiated"

Orchestration uses a central coordinator. The orchestrator calls each service in sequence, tracks the saga’s state, and triggers compensation on failure.

Orchestrator → debit(wallet) → reserve(treasury) → initiate(bank)
              ← success        ← success           ← failure
              → release(treasury) → credit(wallet)

Aspect	Choreography	Orchestration
Visibility	Workflow is implicit in event subscriptions	Workflow is explicit in orchestrator code
Debugging	Reconstruct flow from distributed event logs	Read orchestrator state and history
Coupling	Services know about downstream events	Services know only their own API
Failure handling	Each service handles its own compensation triggers	Orchestrator owns all compensation logic
Ordering	No guaranteed ordering across event consumers	Orchestrator enforces step sequence

Choreography works for fan-out after money has safely landed — sending notifications, updating analytics, triggering downstream workflows. Orchestration is the right choice for the money-moving path, where step ordering matters, compensation must be coordinated, and a human needs to see the workflow state in one place.

See Distributed Systems. The handbook draws this line explicitly: orchestration for the critical path, choreography for the fan-out after the critical path completes.

State Transition Enforcement

Every saga instance has a current state, and every step defines a valid from-state and to-state. The orchestrator must reject transitions that violate the expected sequence.

VALID_TRANSITIONS = {
    "created":         ["debiting"],
    "debiting":        ["debited", "failed"],
    "debited":         ["reserving", "compensating_debit"],
    "reserving":       ["reserved", "compensating_debit"],
    "reserved":        ["initiating_transfer", "compensating_reserve"],
    "initiating_transfer": ["transfer_initiated", "compensating_reserve"],
    "transfer_initiated":  ["settled", "compensating_transfer"],
    "compensating_transfer": ["compensating_reserve"],
    "compensating_reserve":  ["compensating_debit"],
    "compensating_debit":    ["compensated"],
}

def transition(saga, to_state):
    if to_state not in VALID_TRANSITIONS.get(saga.state, []):
        raise InvalidTransition(saga.id, saga.state, to_state)
    saga.previous_state = saga.state
    saga.state = to_state
    saga.updated_at = now()

The state machine prevents invalid transitions — you cannot jump from created to settled, and you cannot compensate a step that hasn’t executed. Persist the state to a durable store (Postgres, DynamoDB) before executing each step. If the orchestrator crashes and restarts, it reads the last persisted state and resumes from there.

The state transition must be idempotent. If the orchestrator retries a transition because it crashed after executing the step but before persisting the state, the step’s action must produce the same result. This is where idempotency keys at each service boundary become mandatory — the saga passes a unique key for each step invocation, and the downstream service deduplicates on that key.

Saga Timeouts

How long do you wait before declaring a step failed? Too short, and you compensate for operations that will eventually succeed. Too long, and customer funds are locked while the system waits.

Step type	Reasonable timeout	Why
Internal service call	5–10 seconds	Service is in your infrastructure; slow response indicates a problem
Bank API (ACH origination)	30–60 seconds	External API with variable latency
Bank settlement confirmation	24–72 hours	ACH settles next business day; wires settle same day
Cross-border transfer	1–5 business days	Correspondent banking adds intermediaries

Short-lived steps (internal calls, API submissions) use synchronous timeouts. The orchestrator waits for a response and treats a timeout as a failure. Long-lived steps (settlement confirmation) use asynchronous callbacks. The orchestrator persists its state and waits for a webhook or a polling result. A separate timeout monitor checks for sagas stuck in a waiting state beyond their expected duration.

SELECT saga_id, state, updated_at
FROM sagas
WHERE state IN ('initiating_transfer', 'transfer_initiated')
  AND updated_at < now() - interval '72 hours';

Stuck sagas surface in an operations dashboard. Some require manual intervention — a bank transfer that neither confirmed nor failed needs a human to check the bank’s portal and update the saga state. Automated timeout-to-compensation works for steps with clean failure semantics. It does not work when the step’s outcome is genuinely unknown.

A timeout does not mean the step failed. It means you don’t know. Compensating an operation that actually succeeded creates a duplicate — the original transfer completes, and the compensation transfer also executes. For bank transfers, this means money moves twice. The safe response to an unknown state is to investigate, not to compensate automatically.

Compensation When Money Has Moved

Compensation for a ledger debit is straightforward — credit the amount back. Both entries appear in the ledger, the balance is correct, and the audit trail shows what happened. The compensation is instant, atomic, and under your control.

Compensation for an external transfer is different. The money has left your system. You cannot credit the customer’s wallet until you confirm the original transfer can be reversed or returned. The compensation has three phases:

1. Request reversal. Submit a return or cancellation to the bank or payment processor. For ACH, this is a reversal within the 5-business-day window. For a wire, this requires contacting the receiving bank directly — there is no automated reversal mechanism.

2. Wait for confirmation. The reversal follows its own settlement cycle. ACH returns settle in 1–2 business days. Wire recalls may take days or weeks, and the receiving bank is not obligated to return the funds.

3. Complete compensation. Once the reversal settles and funds return to your account, credit the customer’s wallet and transition the saga to its terminal compensated state.

During the wait, the saga sits in a compensating_transfer state. The customer’s funds are neither in their wallet nor confirmed returned. Your system must communicate this state clearly — “refund in progress” with an estimated timeline — and your operations team must monitor compensation sagas that exceed expected durations.

Transfer type	Compensation mechanism	Expected timeline
Internal ledger	Reverse entry	Instant
ACH credit	ACH return (R06)	1–2 business days
Wire transfer	Manual recall request	1–10 business days
Card payment	Refund via acquirer	5–10 business days
Stablecoin transfer	On-chain reverse transfer	Seconds to minutes

Dead Letter Queues for Failed Steps

When a saga step fails and the failure is not retryable — a validation error, a permanently declined transaction, a compliance block — the step’s message lands in a dead letter queue (DLQ). The DLQ serves three purposes.

Preservation. The original message is stored with its full context: saga ID, step number, request payload, error response, timestamp. No data is lost. An operator can inspect the failure, fix the underlying issue, and replay the message.

Isolation. Failed messages do not block the processing of other sagas. Without a DLQ, a poison message — one that fails every retry — blocks the queue for all subsequent messages.

Alerting. DLQ depth is a leading indicator of systemic problems. A sudden spike in DLQ entries for the bank transfer step signals an API outage or a configuration change. Monitor DLQ depth per step type, not just aggregate depth.

The replay path from a DLQ must respect idempotency. If a message is replayed after the saga has already compensated, the downstream service must recognize the duplicate and reject it. The saga orchestrator must also validate that replaying the step is consistent with the saga’s current state — replaying a transfer initiation for a saga that has already compensated creates a new untracked transfer.

DLQ message → operator reviews → fix root cause → replay to original queue
                                                 → OR manually transition saga state

Financial DLQs require a retention policy aligned with your audit requirements. SOC 2 and regulatory examinations may ask for evidence of failed operations and their resolution. Retain DLQ messages for at least the same period you retain ledger entries — typically 7 years for financial records.

Workflow Engines

Building a saga orchestrator from scratch — state persistence, timeout management, retry logic, compensation sequencing — is substantial engineering. Workflow engines encapsulate these concerns.

Engine	Model	Durability	Latency overhead
Temporal	Durable execution, replay from event history	Event-sourced, survives crashes	5–15ms per workflow task
AWS Step Functions	JSON state machine (ASL)	Managed, exactly-once for Standard	50–100ms per state transition
Conductor (Netflix)	JSON workflow definition, worker polling	Persistent task queue	10–30ms per task
Custom (Postgres + workers)	Application code + state table	Postgres transactions	1–5ms per transition

Temporal’s durable execution model is particularly suited to financial sagas. A workflow function looks like normal sequential code — debit, reserve, transfer — but the runtime persists the execution state after each step. If the worker crashes, a new worker picks up the workflow and replays the event history to reconstruct the state. The code does not need explicit state persistence or retry logic.

The tradeoff is operational complexity. Running a Temporal cluster (or paying for Temporal Cloud) adds infrastructure. A custom orchestrator on Postgres is simpler to operate but requires you to build timeout monitoring, state machines, and retry logic yourself. The decision depends on your saga count. A system running 100 sagas per day does not need Temporal. A system running 100,000 sagas per day with dozens of step types and multi-day settlement windows does.

Regardless of engine, every saga must be observable. Log every state transition with the saga ID, the step, the from-state, the to-state, and the timestamp. Expose saga counts by state in your monitoring system. Alert on sagas stuck in non-terminal states beyond their expected timeout. The operations team should be able to answer “how many sagas are currently waiting for bank confirmation?” with a single dashboard query.