Model Risk Management

No direct handbook prerequisite. This chapter covers the regulatory framework that governs how financial institutions develop, validate, and monitor models — a discipline that sits above any specific model or technique.

Your credit decisioning model approves a $400M loan portfolio over six months. An examiner asks to see the model’s validation report. You don’t have one — the data science team built the model, tested it internally, and deployed it. The examiner issues a Matter Requiring Attention. Your institution has 90 days to demonstrate a functioning model risk management framework or face restrictions on model-dependent activities. The $400M portfolio stays on the books, but you cannot originate new loans until the framework is in place.

Model risk management is not about building better models. It is about proving — to regulators, auditors, and your own board — that every model in production has been independently validated, is performing as expected, and has a defined owner who is accountable when it stops working.

What “Model” Means to a Regulator

SR 11-7 defines a model as “a quantitative method, system, or approach that applies statistical, economic, financial, or mathematical theories, techniques, and assumptions to process input data into quantitative estimates.” This definition is deliberately broad. It covers machine learning classifiers, but it also covers spreadsheets, rule engines, and any system that transforms inputs into outputs used for decision-making.

System	Model under SR 11-7?	Why
XGBoost fraud classifier	Yes	Statistical method producing risk scores
FICO score threshold rule	Yes	Quantitative method for credit decisions
Excel interest rate calculator	Yes	Mathematical model for pricing
Deterministic routing logic (if amount > $10K, use wire)	No	No statistical/mathematical estimation
Sanctions screening fuzzy matcher	Yes	Algorithmic method producing match scores
Hardcoded fee schedule	No	Lookup table, no estimation

The distinction matters because every system classified as a model requires inventory registration, documentation, independent validation, and ongoing monitoring. Misclassifying a model as “just business logic” creates regulatory exposure. Overclassifying deterministic logic as a model wastes validation resources.

A spreadsheet that a risk analyst maintains on their desktop and uses to price loans is a model under SR 11-7. The regulatory framework does not distinguish between a Python microservice and an Excel workbook — both require the same governance controls.

The Three Lines of Defense

SR 11-7 and OCC 2011-12 mandate a separation of responsibilities across three lines:

First line: model developers. The team that builds, calibrates, and maintains the model. They own the implementation, the training pipeline, and the production deployment. They write the model documentation and conduct initial testing. They are accountable for the model’s day-to-day performance.

Second line: model validators. An independent team — organizationally separate from the first line — that challenges the model’s assumptions, tests its performance on holdout data, and assesses its limitations. “Independent” means they do not report to the same executive as the model developers. A data scientist validating their own team’s model does not satisfy the requirement.

Third line: internal audit. Evaluates whether the model risk management framework itself is functioning. Audit does not validate individual models — they verify that the validation process was followed, that the model inventory is complete, and that findings from validation are tracked to resolution.

Line	Role	Reports to	Frequency
First	Build, deploy, monitor	Business line / CTO	Continuous
Second	Validate, challenge, approve	CRO / independent risk	Annual + material changes
Third	Audit the framework	Board audit committee	Annual

The independence requirement is the constraint that catches most fintech companies. A startup with 20 engineers and no dedicated model validation team does not meet the standard. Regulators expect the validation function to exist before models go into production, not after the first examination finding.

The Model Inventory

Every model in production, in development, or recently retired must appear in a centralized inventory. The inventory is the single source of truth for “what models does this institution operate?”

Each entry includes:

model_id: MDL-2024-0047
name: "Real-time fraud scoring model v3.2"
type: "ML classifier (gradient boosted trees)"
tier: 1
owner: "Fraud Engineering — Jane Chen"
validator: "Model Risk — External firm (Deloitte)"
status: "production"
deployed: "2024-03-15"
last_validated: "2024-02-28"
next_validation_due: "2025-02-28"
materiality: "high"
dependencies:
  - "Feature store v2 (online serving)"
  - "Transaction enrichment pipeline"
risk_rating: "high"
findings_open: 2
findings_closed: 14

The tier classification drives the validation cadence and depth:

Tier	Criteria	Validation frequency	Validator requirement
Tier 1 (critical)	Revenue > $50M impact, regulatory mandate, or credit/market risk	Annual, full scope	Independent team or external firm
Tier 2 (significant)	Revenue $5M–$50M impact, operational risk	Every 18 months	Independent team
Tier 3 (low)	Limited impact, supplementary models	Every 24 months	Peer review acceptable

A large bank maintains 200–500 models in its inventory. A fintech with lending, fraud, and compliance products typically has 15–40. The inventory must be auditable — an examiner will ask for it on day one.

Model Documentation

SR 11-7 requires documentation sufficient for “a knowledgeable third party to understand the model’s operations and limitations and reconstruct it.” In practice, this means:

Development documentation (first line produces):

Problem statement and business context
Data sources, features, and feature engineering logic
Model architecture and hyperparameter choices, with rationale
Training methodology (cross-validation strategy, holdout periods)
Performance metrics on training, validation, and out-of-time test sets
Known limitations and conditions under which the model will degrade
Implementation details (serving infrastructure, latency requirements)

Validation documentation (second line produces):

Scope of validation (what was tested, what was excluded and why)
Independent performance assessment on holdout data
Sensitivity analysis (how do outputs change when inputs shift?)
Benchmarking against alternative approaches
Conceptual soundness review (do the model’s assumptions hold?)
Findings classified by severity (high, medium, low)
Compensating controls for accepted limitations

The documentation standard is not a formality. During an examination, regulators will read the validation report. If the report says “the model performs well” without quantitative evidence — confusion matrices, calibration curves, population stability indices — the finding writes itself.

Independent Validation

Validation is the core regulatory requirement. A model that has not been independently validated is, in the regulator’s view, an uncontrolled risk.

The validation covers three dimensions:

Conceptual soundness. Are the model’s assumptions reasonable? Does a linear model applied to a nonlinear relationship make sense? Is the feature set plausible — does it include variables with a causal or empirical relationship to the target, or is it fitting on noise?

Outcomes analysis. Does the model perform as expected on data it has not seen? The validator runs the model on a holdout dataset (ideally out-of-time, not just out-of-sample) and compares predicted outputs to actual outcomes.

Validation Performance Summary — Fraud Model v3.2
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
                    Development    Validation (OOT)
AUC-ROC             0.943          0.931
Precision @ 5% FPR  0.72           0.68
KS Statistic        0.81           0.77
PSI (vs. training)  —              0.08 (stable)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
PSI > 0.25 triggers mandatory revalidation.

Process verification. Was the model developed following documented procedures? Is the training data representative? Are there data quality issues that could affect outputs? Is the implementation consistent with the documented specification — does the production code match what was validated?

The validator issues findings. High-severity findings require remediation before the model can be approved for production. Medium-severity findings require a remediation plan with a deadline. Low-severity findings are tracked and addressed during the next model update cycle.

Champion/Challenger Governance

When a new model version is ready to replace the production model, the transition requires governance — you cannot silently swap a model that makes credit decisions affecting thousands of customers.

The champion/challenger framework runs both models simultaneously:

Component	Champion	Challenger
Status	Production, making decisions	Shadow, scoring only
Traffic	100% of decisions	100% scored, 0% acted upon
Duration	Until replaced	Minimum 30–90 days shadow period
Approval	Previously validated	Must pass validation before promotion

The challenger scores every transaction the champion scores. The outputs are logged but not used for decisions. After the shadow period, the model risk team compares performance:

Does the challenger outperform on the primary metric (AUC, precision at target FPR)?
Does it maintain performance across all population segments (no disparate impact)?
Does it degrade gracefully under distribution shift?
Are there edge cases where the challenger performs significantly worse?

Promotion from challenger to champion requires sign-off from the model risk team (second line), the business owner (first line), and — for Tier 1 models — a model risk committee or CRO.

See Fraud Model Deployment and Model Monitoring and Drift. Champion/challenger is the governance framework. Shadow scoring and gradual rollout are the engineering implementation. The governance says when promotion is allowed. The engineering determines how promotion is executed.

Ongoing Monitoring vs. Periodic Validation

SR 11-7 distinguishes between ongoing monitoring (continuous, first-line responsibility) and periodic validation (scheduled, second-line responsibility).

Ongoing monitoring (first line, continuous):

Population Stability Index (PSI) on input features — drift beyond 0.25 triggers investigation
Model performance metrics tracked daily or weekly (precision, recall, false positive rate)
Input data quality checks (missing values, out-of-range values, schema violations)
Latency and availability monitoring for serving infrastructure
Alert thresholds with escalation procedures

Periodic validation (second line, scheduled):

Full revalidation on fresh out-of-time data
Reassessment of conceptual soundness given market or regulatory changes
Review of all monitoring alerts and their resolution
Updated benchmarking against alternative approaches
Review of model limitations in light of new evidence

The monitoring catches gradual degradation between validations. The validation provides a comprehensive independent assessment. Neither substitutes for the other. A model that passes its annual validation but shows PSI > 0.25 in month seven needs attention now, not at the next validation cycle.

BCBS 239 and Data Governance

Basel Committee’s BCBS 239 (Principles for effective risk data aggregation and risk reporting) intersects with model risk management at the data layer. Models are only as reliable as the data they consume. BCBS 239 requires:

Accuracy and integrity. Data feeding models must be reconciled to authoritative sources. A fraud model trained on transaction data that doesn’t match the ledger is trained on wrong data.
Completeness. Missing data must be identified, quantified, and addressed. A model that silently drops 3% of transactions because of null values in a required field is producing biased outputs.
Timeliness. Data must be available within the latency budget. A real-time fraud model that uses features computed from yesterday’s batch is not real-time.
Lineage. Every data element feeding a model must be traceable to its source. When an examiner asks “where does this feature come from?”, the answer must be specific — not “the data warehouse.”

For engineering teams, BCBS 239 compliance means the feature pipeline is subject to the same data governance controls as regulatory reporting. Feature definitions, transformation logic, and data sources must be documented and version-controlled. A change to a feature’s computation logic triggers model revalidation.

The Examination

An OCC or Fed examiner assessing model risk management will request, at minimum:

The model inventory — complete, current, with tier classifications
Validation reports for all Tier 1 models — within their validation cycle
Open findings tracker — with remediation timelines and accountability
Ongoing monitoring dashboards — evidence that models are being watched
Model risk policy — approved by the board, reviewed annually
Evidence of independence — organizational charts showing separation between first and second lines

The examination is not a one-time event. For banks, it occurs annually. For fintechs with OCC charters or state banking licences, the cadence varies but the expectation is the same. The cost of a mature model risk management function — dedicated validation staff, tooling, external validation for critical models — runs $500K–$2M annually for a mid-size institution. The cost of an MRA (Matter Requiring Attention) or MRIA (Matter Requiring Immediate Attention) — restricted activities, remediation projects, board-level reporting — is higher.