Feature Stores in Production

Note: No direct handbook prerequisite. This chapter covers feature engineering infrastructure for real-time financial ML systems.

Your fraud model catches 94% of fraudulent transactions in offline evaluation. In production, the same model catches 81%. The model didn’t change — the features did. During training, transaction_count_30d was computed from a complete historical table with perfect hindsight. In production, the same feature is served from a cache that was last refreshed 47 minutes ago. The model learned patterns against accurate data and now makes decisions against stale data.

The gap between offline and online model performance is almost always a feature store problem. The model is a function of its inputs. If the inputs differ between training and serving — in freshness, in computation logic, or in the data they had access to — the model’s behavior diverges from its evaluation.

The Dual-Compute Problem

A feature store serves two environments with fundamentally different constraints: batch (offline) and real-time (online). Batch features power model training, backtesting, and reporting. Online features power real-time inference — the 45ms fraud decision, the 250ms credit check.

Dimension	Offline store	Online store
Latency	Minutes to hours	< 10ms p99
Freshness	Hours to days	Seconds to minutes
Compute	Full table scans, joins, window functions	Pre-computed lookups
Storage	Data warehouse (BigQuery, Snowflake, Redshift)	Key-value store (Redis, DynamoDB, Bigtable)
Query pattern	Arbitrary SQL, point-in-time joins	Single-key GET by entity ID
Consistency	Eventual, batch-updated	Read-after-write for critical features

The architecture challenge is ensuring both paths produce identical results for the same entity at the same point in time. If avg_transaction_amount_90d returns $342.17 during training but $338.92 during serving for the same customer at the same timestamp, you have training-serving skew.

Training-Serving Skew

Skew has three sources, and all three are common in financial feature stores.

Logic skew. The offline pipeline computes avg_transaction_amount_90d with a SQL window function. The online pipeline computes it by maintaining a running average in Redis, updated on each transaction event. The two implementations use different rounding, different null handling, or different definitions of “90 days” (calendar days vs business days). The numbers diverge.

# Offline: SQL window function
SELECT AVG(amount) OVER (
  PARTITION BY customer_id
  ORDER BY created_at
  RANGE BETWEEN INTERVAL '90 days' PRECEDING AND CURRENT ROW
) AS avg_transaction_amount_90d
FROM transactions;

# Online: streaming update (different logic)
new_avg = (old_avg * old_count + new_amount) / (old_count + 1)
# Missing: expiry of transactions older than 90 days
# Missing: null handling for reversed transactions

The fix is a single feature definition that generates both the batch SQL and the streaming computation. Feast, Tecton, and Feathr all support this pattern — define once, materialize to both stores.

Data skew. The offline store has access to corrected, reconciled data. A chargeback filed on day 45 updates the transaction record retroactively. The model trains on the corrected version. In production, the online store sees the original transaction — no chargeback yet. The feature values are correct in both environments, but they represent different points in the data’s lifecycle.

Timing skew. The offline store computes features at training time using data that was available only after the prediction would have been made. The classic look-ahead bias: training a fraud model on features that include “was a chargeback filed for this transaction” — information that arrives 30–60 days after the transaction.

Point-in-time correctness is not optional for financial models. A fraud model trained with look-ahead bias will overfit to future information it will never have in production. The resulting production accuracy drop is typically 5–15 percentage points — enough to either miss fraud or generate thousands of false positives per day.

Point-in-Time Joins

A point-in-time join retrieves feature values as they existed at a specific historical moment — not their current values, not their corrected values. For a transaction at 2:14:37 PM on March 12, the join returns the feature values that were available at exactly 2:14:37 PM on March 12.

Standard SQL joins don’t enforce this constraint. A regular left join on customer_id returns the latest feature row, which may include data from after the prediction timestamp. A point-in-time join adds a temporal constraint: return the most recent feature row where feature_timestamp <= event_timestamp.

SELECT
  t.transaction_id,
  t.customer_id,
  t.amount,
  t.created_at,
  f.avg_transaction_amount_90d,
  f.transaction_count_30d,
  f.max_single_transaction_7d
FROM transactions t
ASOF JOIN customer_features f
  ON t.customer_id = f.customer_id
  AND t.created_at >= f.feature_timestamp;

DuckDB, QuestDB, and kdb+ support ASOF JOIN natively. For Postgres, you simulate it with a lateral join:

SELECT t.*, f.*
FROM transactions t
LEFT JOIN LATERAL (
  SELECT *
  FROM customer_features f
  WHERE f.customer_id = t.customer_id
    AND f.feature_timestamp <= t.created_at
  ORDER BY f.feature_timestamp DESC
  LIMIT 1
) f ON true;

The lateral join is expensive — it runs a subquery per row. At 10 million training examples with 50 features across 5 feature tables, this becomes a multi-hour computation. Tecton and Feast pre-sort features by entity and timestamp, then merge with a single-pass algorithm that runs in O(n log k) where k is the number of feature tables.

Online Serving Architecture

The online store is a key-value lookup. Given an entity key (customer_id: 12345), return the latest feature vector. The latency budget for fraud detection is 5–10ms for feature retrieval out of a total 45ms inference budget.

Component	Latency budget
Feature retrieval	5–10ms
Model inference	15–25ms
Policy rules	5–10ms
Network overhead	5–10ms
Total	45ms

Redis is the most common online store for financial feature serving. A single Redis node handles 100,000+ GET operations per second with sub-millisecond p50 latency. A MGET for 50 features completes in 1–2ms. Redis Cluster distributes across nodes for higher throughput.

DynamoDB is the alternative when you need durability guarantees. A strongly consistent GetItem returns in 2–5ms. BatchGetItem for 50 features across multiple tables runs 5–10ms. The tradeoff: DynamoDB is durable (data survives node failures without replication lag risk), but 3–5x slower than Redis for the same operation.

Feature request flow:

Transaction event
  → Feature service (gRPC, 1ms)
    → Redis MGET for 50 features (2ms)
    → Compute 3 real-time features from event data (1ms)
    → Assemble feature vector (< 1ms)
  → Model service (gRPC, 1ms)
    → Inference (15ms)
  → Response (< 1ms)

Total: ~21ms

Feature Freshness and Materialization

Not every feature needs real-time freshness. Financial feature stores typically serve three freshness tiers:

Tier	Freshness	Update mechanism	Examples
Real-time	Seconds	Stream processing (Flink, Spark Streaming)	Transaction velocity, current session behavior
Near-real-time	Minutes	Micro-batch or CDC	Rolling 24h transaction count, account balance
Batch	Hours to daily	Scheduled pipeline (Airflow, dbt)	90-day averages, credit bureau features, demographic enrichment

Real-time features are computed on the event stream itself. When a transaction arrives, a Flink job updates transaction_count_1h by incrementing a windowed counter in state. The updated value is written to the online store within seconds.

Near-real-time features use change data capture. A Debezium connector captures ledger entry inserts, streams them through Kafka, and a consumer updates rolling aggregates in Redis. Staleness is bounded by consumer lag — typically 5–30 seconds under normal load, potentially minutes during backlog processing.

Batch features run on a schedule. A dbt model computes avg_transaction_amount_90d nightly and writes the results to both the offline warehouse and the online store. These features are stale by up to 24 hours, which is acceptable for slow-changing attributes but dangerous for features that capture recent behavioral shifts.

The freshness tier determines the detection window for fraud. A batch-updated velocity feature with 24-hour staleness means a burst of 50 fraudulent transactions in an hour looks identical to normal behavior until tomorrow’s pipeline runs. Real-time velocity features catch the burst on the third or fourth transaction.

Feature Monitoring

Feature stores fail silently. A broken pipeline doesn’t throw an error at inference time — it serves stale data. The model continues returning scores, but the scores are wrong because the inputs are wrong.

Monitor four signals:

Freshness. Track the age of the most recent feature value per entity group. Alert when any feature tier exceeds its SLA. A batch feature older than 26 hours (24-hour schedule + 2-hour grace) indicates a failed pipeline.

Distribution shift. Compare the serving distribution against the training distribution. Kolmogorov-Smirnov test or Population Stability Index (PSI) on a rolling window. PSI > 0.1 warrants investigation; PSI > 0.25 indicates the feature has shifted enough to degrade model performance.

PSI range	Interpretation	Action
< 0.1	No significant shift	None
0.1–0.25	Moderate shift	Investigate, monitor closely
> 0.25	Significant shift	Retrain model, audit feature pipeline

Null rate. Track the percentage of null or default values per feature. A sudden spike in nulls — from 0.1% to 15% — usually means an upstream data source went down. The model fills nulls with defaults, which are almost certainly wrong for the specific entity.

Serving latency. Track p50, p95, and p99 latency for feature retrieval. A p99 that climbs from 5ms to 40ms means 1% of fraud decisions are being made with incomplete feature vectors (because the feature service timed out and the model fell back to defaults). At 100,000 transactions per day, that’s 1,000 decisions on degraded inputs.

Feast, Tecton, and Build vs Buy

The build-vs-buy decision for feature stores is a function of feature count, freshness requirements, and team size.

Dimension	Feast (open source)	Tecton (managed)	Custom
Setup cost	2–4 engineer-weeks	Days (SaaS)	3–6 engineer-months
Online store	Redis, DynamoDB (BYO)	Managed DynamoDB	Whatever you build
Offline store	BigQuery, Snowflake, Redshift	Managed Spark/Rift	Your warehouse
Point-in-time joins	Built-in	Built-in, optimized	You implement
Stream features	Limited (experimental)	Native Flink integration	You build and maintain
Monitoring	Basic metrics	Built-in drift detection	You build
Cost at 1,000 features	$5K–$15K/mo (infra)	$20K–$50K/mo	$8K–$20K/mo (infra) + eng time

Feast is the starting point. It handles offline-to-online materialization, point-in-time joins, and feature serving with a well-defined SDK. Its limitation is real-time features — stream processing integration is still maturing.

Tecton is the managed option for teams that need sub-second feature freshness with guaranteed point-in-time correctness. Stripe’s ML infrastructure team published that their feature platform serves 50+ features per prediction at p99 < 10ms across hundreds of models. Tecton’s architecture mirrors this pattern.

Custom feature stores make sense when your feature computation is deeply intertwined with your domain — when the feature logic is inseparable from the ledger, the payment state machine, or the compliance engine. The cost is maintenance: schema evolution, backfill orchestration, monitoring, and the ongoing work of keeping offline and online stores in sync.

The decision that matters most is not which platform to use. It is whether your online and offline feature computation produces identical results for the same entity at the same timestamp. Every other feature store capability — monitoring, versioning, access control — is secondary to that invariant. If the training features and serving features diverge, the model’s production performance is unpredictable regardless of the infrastructure behind it.