Event Mesh Design for Financial Services

Prerequisite: Event Streaming

Your ledger service publishes a payment.completed event. Two consumers process it — one credits the merchant’s balance, the other triggers the settlement file. A Kafka broker restarts during a rebalance, and the merchant balance consumer re-reads an offset it already processed. The credit posts twice. Your ledger now shows $14,200 in the merchant account instead of $7,100, and because the event carried enough authority for the consumer to act, the damage is done before your reconciliation job catches it four hours later.

Event-driven architecture in fintech is not a pattern choice — it is the nervous system connecting every ledger write, compliance check, and settlement action. The failure modes are denominated in dollars.

Topic Granularity

The first design decision is how many topics you need and what each one represents. Too few topics and consumers waste cycles filtering events they don’t care about. Too many and operational overhead — monitoring, ACLs, retention policies — grows linearly with topic count.

The financial event domain has a natural structure: each bounded context owns a small set of topics representing its state transitions.

Domain	Topics	Events per topic
Payments	`payments.commands`, `payments.events`	`initiated`, `authorized`, `captured`, `failed`, `refunded`
Ledger	`ledger.entries`	`entry.posted`, `entry.voided`
Compliance	`compliance.events`	`kyc.completed`, `kyc.failed`, `sanctions.cleared`, `sanctions.hit`
Settlement	`settlement.events`	`batch.opened`, `batch.closed`, `file.submitted`, `file.acknowledged`

Each topic carries events for one domain, and each event type is distinguished by a type field in the payload — not by a separate topic. A payments.events topic with five event types is operationally manageable. Fifty topics with one event type each is not.

Naming convention

Use a three-segment naming pattern: {domain}.{category}.{version}.

payments.events.v1
ledger.entries.v1
compliance.screening.v1
settlement.batches.v1

The version suffix is not optional. When you need a breaking schema change, you publish to payments.events.v2 and run both topics in parallel during migration. Without a version in the topic name, you are forced into backward-compatible-only schema evolution forever — which works until it doesn’t.

Partitioning Strategy

Kafka guarantees ordering within a partition but not across partitions. For financial events, ordering per entity is the minimum guarantee your consumers need. A payment’s authorized event must arrive before its captured event — at least within the same consumer instance.

Partition by entity ID: account ID for ledger events, payment ID for payment events, customer ID for compliance events.

partition_key = event.account_id   // ledger events
partition_key = event.payment_id   // payment events
partition_key = event.customer_id  // compliance events

This guarantees that all events for a single payment, account, or customer are processed in order by a single consumer instance. Cross-entity ordering is not guaranteed — and should not be. If your consumer requires ordering across entities, your domain model has a missing aggregate.

The partition count determines your maximum consumer parallelism. For a payment events topic processing 50,000 events per second, 64 partitions yield roughly 780 events/sec per partition — well within a single consumer’s capacity. Over-partitioning (512 partitions for a 1,000 events/sec topic) wastes broker resources and increases rebalance time.

Events/sec	Recommended partitions	Events/partition/sec
1,000	8–16	62–125
10,000	32–64	156–312
50,000	64–128	390–780
200,000	128–256	780–1,562

Changing the partition count after creation redistributes keys across partitions. Events for the same account may land on a different partition than their predecessors, breaking ordering for in-flight processing. Increase partitions only during a maintenance window with all consumers drained.

Schema Evolution

Every financial event is a contract. Downstream consumers — ledger writers, compliance checkers, settlement generators — depend on the shape of the payload. A field rename breaks them. A removed field breaks them. Even a field type change from string to int breaks them.

Use a schema registry (Confluent Schema Registry, AWS Glue, or Apicurio) with Avro or Protobuf schemas. JSON Schema works but lacks the compact wire format that matters at high throughput.

The evolution rules for financial events are stricter than for general-purpose systems:

Change	Allowed?	Why
Add optional field	Yes	Existing consumers ignore it
Add required field	No	Breaks all existing consumers
Remove field	No	Consumers relying on it fail
Rename field	No	Semantically identical to remove + add
Change field type	No	Deserialization fails
Add enum value	Yes (backward)	Only if consumers have a default case

AsyncAPI specifications document your event contracts. Each topic gets an AsyncAPI channel definition with the schema, the partition key, and the semantic meaning of the event.

channels:
  payments/events/v1:
    subscribe:
      message:
        schemaFormat: 'application/vnd.apache.avro+json;version=1.9.0'
        payload:
          type: record
          name: PaymentEvent
          fields:
            - name: payment_id
              type: string
            - name: type
              type:
                type: enum
                symbols: [initiated, authorized, captured, failed, refunded]
            - name: amount
              type: { type: fixed, size: 16, logicalType: decimal, precision: 18, scale: 2 }
            - name: currency
              type: string
            - name: occurred_at
              type: { type: long, logicalType: timestamp-millis }

When you need a breaking change, create a new version of the topic, deploy consumers that read both, and drain the old topic. Never mutate a live schema in a way that violates the evolution rules above.

Dead Letter Queues and Replay

A consumer fails to process an event. The schema is valid, the message deserialized correctly, but the downstream database rejected the write — a foreign key violation, a constraint check, a transient network failure. The consumer has three choices: retry in place (blocks the partition), skip and lose the event, or route to a dead letter queue (DLQ).

For financial events, the DLQ is the only acceptable option. Lost events mean lost money or broken audit trails.

payments.events.v1          →  consumer  →  success
                                         →  payments.events.v1.dlq (on failure)

Each DLQ event carries the original payload, the failure reason, the attempt count, and the timestamp of the last attempt.

The replay strategy matters. Replaying a failed payment.captured event is not the same as processing a new capture — the payment may have already been partially settled, the merchant may have already been credited by a compensation process, or the event may represent a state that no longer exists. Every DLQ replay must be idempotent, and the consumer must check current state before acting on a replayed event.

DLQ depth is a monitoring metric. A DLQ with zero messages is healthy. A DLQ accumulating 50 messages per hour indicates a systemic issue — a bad deployment, a schema mismatch, a downstream outage. Alert when the DLQ growth rate exceeds your baseline, not when it reaches an absolute count.

See Idempotency at Scale. Every event consumer that writes to a ledger or triggers a payment must be idempotent. The DLQ replay path is the most common source of duplicate processing — and the easiest to miss in testing.

Choreography vs Orchestration

A multi-step financial workflow — customer pays, merchant is credited, settlement file is generated, bank is notified — can be coordinated in two ways.

Choreography lets each service react to events from the previous step. The payment service publishes payment.captured. The ledger service consumes it, posts entries, and publishes entry.posted. The settlement service consumes entry.posted and adds it to the current batch. No central coordinator exists.

Orchestration uses a central workflow engine (Temporal, AWS Step Functions, a custom state machine) that explicitly calls each step in sequence, handles retries, and manages compensation.

Dimension	Choreography	Orchestration
Visibility	Distributed — requires correlation IDs to trace	Centralized — the orchestrator holds the full workflow state
Failure handling	Each service handles its own retries and DLQ	Orchestrator manages retries, timeouts, and compensation
Coupling	Services only know about events, not about each other	Orchestrator knows the full workflow graph
Latency	Lower — no coordinator hop	Higher — each step routes through the orchestrator
Debugging	Hard — reconstruct the flow from multiple topic logs	Easy — query the orchestrator for workflow state

For financial systems, the pattern is: choreography for fan-out after money has moved, orchestration for the critical path where money is moving.

The payment capture → ledger entry → balance update path is orchestrated. One coordinator owns the sequence, handles failure at any step, and runs compensation if step 3 fails after step 2 succeeded. You cannot afford to discover that the ledger entry posted but the balance update didn’t by correlating events across three topics.

The post-settlement fan-out — notify the merchant, send the receipt, update the dashboard, trigger the reconciliation — is choreographed. Each consumer acts independently on the settlement.completed event. If the receipt email fails, the merchant’s balance is unaffected.

Exactly-Once Semantics

Kafka’s exactly-once semantics (EOS), introduced in KIP-98 and available since Kafka 0.11, guarantees that a produce-consume-produce cycle executes exactly once. The producer writes to a topic, the consumer reads from that topic and writes to another topic, and EOS ensures the read and write are atomic — if the consumer crashes after the write but before committing the offset, Kafka rolls back the write on recovery.

This works when both sides of the transaction are Kafka topics. It does not work when the consumer’s side effect is a database write, an API call, or a ledger entry.

When your consumer reads a payment.captured event and credits a merchant’s ledger balance, the side effect is a Postgres transaction. Kafka’s EOS guarantees the offset commit and the downstream topic write are atomic. It says nothing about the Postgres write. If the consumer crashes after the Postgres commit but before the Kafka offset commit, the next consumer instance re-reads the event and credits the merchant again.

The solution is not Kafka-level EOS. The solution is consumer-level idempotency.

BEGIN;

INSERT INTO processed_events (event_id, topic, partition, offset_val, processed_at)
VALUES ($1, $2, $3, $4, now())
ON CONFLICT (event_id) DO NOTHING;

-- Only proceed if the insert succeeded (event not already processed)
INSERT INTO entries (account_id, amount, event_id, ...)
SELECT $5, $6, $1, ...
WHERE EXISTS (
  SELECT 1 FROM processed_events
  WHERE event_id = $1 AND processed_at = now()
);

COMMIT;

The processed_events table is the idempotency store. The event ID is the natural deduplication key. Both the deduplication check and the ledger write happen in the same database transaction. If the consumer crashes after this transaction commits, the re-read event hits the ON CONFLICT clause and the ledger write is skipped.

This pattern — database-level idempotency wrapping the side effect — is the only reliable exactly-once guarantee when the side effect leaves Kafka’s transactional boundary. Kafka’s EOS is a building block, not a complete solution.

Guarantee level	What it covers	What it doesn’t cover
At-most-once	Consumer commits offset before processing	Lost events on crash
At-least-once	Consumer commits offset after processing	Duplicate processing on crash
Kafka EOS	Atomic offset commit + downstream topic write	Side effects outside Kafka (DB, API, ledger)
Consumer idempotency	Deduplication at the side-effect boundary	Requires per-consumer implementation

“We use Kafka exactly-once” is not a complete answer to “can a payment event credit a merchant twice?” Kafka’s EOS covers the offset-to-topic path. The offset-to-database path requires your own idempotency layer. Every financial event consumer must implement both.

Retention and Compaction

Financial event topics carry data that may be subject to regulatory retention requirements. A payment.captured event is part of the transaction audit trail. Deleting it after Kafka’s default 7-day retention may violate recordkeeping obligations.

Two strategies:

Long retention. Set retention to match your regulatory requirement — 5 years for BSA/AML, 7 years for SOX. At 50,000 events/day averaging 1KB each, 7 years is roughly 128GB per topic. Kafka handles this, but your consumers must handle reprocessing from arbitrary offsets without re-executing side effects.

Event archival. Consume events into a durable store (S3 with Object Lock, a data warehouse) and keep Kafka retention at 30–90 days for operational replay. The archive is the regulatory record; Kafka is the operational bus.

Log compaction — Kafka’s strategy of keeping only the latest value per key — is appropriate for state topics (current account balance, current KYC status) but never for event topics. Compacting a payment events topic deletes intermediate states (authorized, captured) and keeps only the latest (settled), destroying the audit trail.

Operational Metrics

Monitor four signals per topic:

Metric	Healthy	Investigate	Alert
Consumer lag (messages behind)	< 1,000	1,000–10,000	> 10,000
DLQ growth rate	0/hour	1–10/hour	> 10/hour
End-to-end latency (produce to consume)	< 500ms p99	500ms–2s p99	> 2s p99
Schema validation failures	0	1–5/day	> 5/day

Consumer lag is the leading indicator. A consumer falling behind on a payment events topic means payment processing is backing up — which surfaces to customers as delayed transactions before any downstream system raises an alarm.