Financial Data Lineage

Prerequisite: Auditability

An auditor asks where the $4,217,893.41 on line 12 of your call report came from. You trace it to a dbt model that aggregated three tables. One table was populated by a Kafka consumer. The Kafka topic was fed by a microservice that read from two upstream services. One of those services was decommissioned six months ago and its data was backfilled from a CSV export. The auditor asks you to prove the CSV matched the original source. You cannot.

Data lineage in analytics is a convenience — it helps engineers debug pipelines. Data lineage in financial reporting is a regulatory obligation. Every number on a regulatory filing must trace back to its source transaction through every transformation, and the trace itself must be reproducible. BCBS 239 (Principles for Effective Risk Data Aggregation) requires that firms demonstrate “the ability to trace back to the source data” for all risk reports. The cost of failing that trace is not a stale dashboard. It is a Matter Requiring Attention from your examiner.

What Lineage Means in Finance

Data lineage tracks three relationships: where data came from (provenance), what happened to it (transformation), and where it went (propagation). In a financial system, each relationship carries a different burden of proof.

Relationship	Analytics lineage	Financial lineage
Provenance	Which table, which column	Which source system, which transaction, which timestamp
Transformation	SQL query or function name	Exact logic applied, version of the logic, inputs and outputs
Propagation	Which dashboards consume this	Which regulatory reports, which customer statements, which audit exhibits

Analytics lineage answers “what feeds this chart.” Financial lineage answers “can you prove this number is correct, and can you reproduce it as of the date it was filed.”

Financial lineage has legal weight. A regulatory report filed with a number you cannot trace is a number you cannot defend in an examination. The lineage trail is not metadata — it is evidence.

Table-Level vs Column-Level

Table-level lineage tracks which tables feed which downstream tables. It answers “this report reads from daily_balances, which reads from entries and accounts.” For analytics, this is sufficient. For regulatory reporting, it is not.

Column-level lineage tracks individual fields through transformations. The total_deposits column on the call report came from SUM(entries.amount) where entries.entry_type = 'deposit' and entries.booking_date BETWEEN $start AND $end, joined to accounts on account_id where accounts.account_type IN ('checking', 'savings').

-- Column-level lineage must capture that this specific
-- aggregation produced this specific report field
SELECT
  SUM(e.amount) AS total_deposits  -- → Call Report Schedule RC, line 12
FROM entries e
JOIN accounts a ON e.account_id = a.id
WHERE e.entry_type = 'deposit'
  AND a.account_type IN ('checking', 'savings')
  AND e.booking_date BETWEEN '2026-07-01' AND '2026-09-30';

The transformation is not just “sum the deposits.” It is the exact filter criteria, the exact join condition, the exact date range, and the exact version of the query that ran. If the query logic changed between Q2 and Q3, the lineage must show which version produced which filing.

Transformation Logging

Every transformation that touches data destined for a regulatory report must be logged with enough detail to reproduce the output from the same inputs.

A transformation log entry captures five fields:

Field	Purpose	Example
run_id	Unique execution identifier	run_2026Q3_callreport_001
transform_version	Git hash or semantic version of the transformation code	a3f7b2c
input_snapshot	Hash or pointer to the input data at execution time	s3://snapshots/entries_20261001T000000Z.parquet
output_snapshot	Hash or pointer to the output	s3://snapshots/call_report_rc_20261001.parquet
parameters	Runtime configuration that affected the output	{"reporting_date": "2026-09-30", "entity": "bank_charter_001"}

The critical discipline: inputs must be snapshotted or content-hashed before the transformation runs, not after. If the source table receives new rows between your snapshot and your computation, the lineage is broken — you cannot reproduce the output from the recorded input.

dbt captures model-level lineage automatically through its DAG. Each dbt run produces a manifest.json and run_results.json that record which models executed, their SQL, and their dependencies. For table-level lineage, this is sufficient. For column-level lineage, dbt’s built-in support (introduced in dbt 1.6) traces column-level relationships through ref() calls and SQL parsing. The limitation: dbt’s column-level lineage relies on static SQL analysis and does not capture dynamic SQL, Jinja-templated column names, or transformations that happen outside dbt (Python models, external ETL).

The OpenLineage Standard

OpenLineage is an open standard for lineage event collection. Every job execution emits three events: START, COMPLETE, and FAIL. Each event includes input datasets, output datasets, and facets — structured metadata about the transformation.

{
  "eventType": "COMPLETE",
  "eventTime": "2026-10-01T04:15:22.000Z",
  "job": {
    "namespace": "regulatory_reporting",
    "name": "call_report_schedule_rc"
  },
  "inputs": [{
    "namespace": "warehouse",
    "name": "entries",
    "facets": {
      "columnLineage": {
        "fields": {
          "total_deposits": {
            "inputFields": [
              {"namespace": "warehouse", "name": "entries", "field": "amount"}
            ],
            "transformationType": "AGGREGATION",
            "transformationDescription": "SUM(amount) WHERE entry_type = 'deposit'"
          }
        }
      }
    }
  }],
  "outputs": [{
    "namespace": "regulatory",
    "name": "call_report_q3_2026"
  }]
}

Marquez is the reference implementation for storing and querying OpenLineage events. It provides a REST API and a UI for exploring lineage graphs. A typical deployment processes 10,000–50,000 lineage events per day for a mid-size financial institution with 200–500 data pipeline jobs.

The value of a standard: OpenLineage events can be emitted from Spark, Airflow, dbt, Flink, and custom jobs using the same schema. A regulatory report that combines data from a Spark aggregation and a dbt model has a single, unified lineage graph — not two disconnected tool-specific views.

Lineage-Aware Testing

Standard data testing validates the output: row counts, null checks, value ranges. Lineage-aware testing validates the path: did the output arrive through the expected transformations from the expected sources?

Three categories of lineage tests:

Provenance tests. Assert that a report’s inputs are the expected source tables. If call_report_schedule_rc suddenly reads from a new table that was not in the approved lineage, the test fails before the report is filed. This catches accidental dependency changes introduced during refactoring.

def test_call_report_provenance():
    lineage = get_lineage("call_report_schedule_rc")
    expected_sources = {"entries", "accounts", "account_types"}
    actual_sources = {inp.name for inp in lineage.inputs}
    assert actual_sources == expected_sources, (
        f"Unexpected sources: {actual_sources - expected_sources}"
    )

Transformation version tests. Assert that the transformation logic has not changed since the last approved version. If the SQL for computing total_deposits was modified, the test flags it for re-validation before the report runs. This prevents silent logic changes from affecting filed numbers.

Reproducibility tests. Re-run a transformation against its recorded input snapshot and verify the output matches the recorded output hash. If the output differs, either the transformation logic changed (check the version) or the input was not properly snapshotted (check the content hash). Run these quarterly or after any pipeline change that touches regulatory data paths.

See Auditability. Lineage-aware testing extends the auditability principle from individual ledger entries to entire data pipelines. The same “can you prove it” discipline applies at every layer.

Analytics Lineage vs Regulatory Lineage

The tooling overlaps. The requirements do not.

Dimension	Analytics lineage	Regulatory lineage
Retention	Until the dashboard is decommissioned	5–7 years minimum (matches report retention)
Reproducibility	Nice to have	Mandatory — must reproduce any filed number
Granularity	Table or model level	Column level with transformation logic
Change control	Informal — update and redeploy	Formal — changes require validation before production
Audit trail	Git history	Git history + transformation logs + input snapshots
Failure mode	Stale chart	Regulatory finding, restatement, enforcement action

A common architectural mistake: building one lineage system and assuming it covers both use cases. The analytics team wants fast iteration — deploy a new model, see the lineage update. The regulatory team wants controlled change — no model touching a regulatory report can change without validation. These are opposing requirements.

The separation pattern: tag every dataset and transformation with a regulatory_scope flag. Datasets and transformations in scope for regulatory reporting are subject to change control, transformation logging, and input snapshotting. Everything else follows the analytics path with lighter-weight lineage. The tagging must be enforced in CI — a PR that modifies a regulatory-scoped model without updating the lineage metadata and triggering a re-validation pipeline is blocked.

Scaling Lineage Collection

A financial institution running 500 pipeline jobs per day generates approximately 1,500 OpenLineage events (start, complete/fail for each). At column-level granularity across 2,000 tables with an average of 40 columns each, the lineage graph has roughly 80,000 nodes and 200,000 edges. Marquez handles this comfortably on a single Postgres instance.

At 5,000 jobs per day — typical for a larger institution with real-time streaming pipelines alongside batch — event volume reaches 15,000 per day and the graph grows to 500,000+ edges. At this scale, query latency on the lineage graph becomes the bottleneck. Marquez supports Postgres for storage but benefits from connection pooling and read replicas for lineage graph traversal queries.

The operational concern is not storage (lineage events are small — 1–5 KB each) but query depth. An auditor’s question — “trace this number back to its source” — requires a graph traversal that may span 8–12 hops across batch jobs, streaming consumers, and API ingestion. Each hop requires a join against the lineage event table. Materializing common lineage paths as pre-computed views reduces query time from seconds to milliseconds for frequently audited report lines.

Lineage collection must not slow the pipeline it monitors. OpenLineage emitters are designed as fire-and-forget: the lineage event is sent asynchronously after the job completes. If the lineage backend is down, the pipeline continues — the lineage gap is logged and backfilled. A lineage system that blocks production data pipelines has inverted its priorities.