Market Data Infrastructure

New domain. No handbook prerequisite — this chapter is self-contained.

Your trading system subscribes to three exchanges for BTC/USD quotes. At 14:32:07.003 UTC, Coinbase publishes a price of $67,241.50. At 14:32:07.004 UTC — one millisecond later — Kraken publishes $67,238.20. Your system processes the Kraken update first because of a network path difference, executes a buy at $67,238.20, then receives the Coinbase update showing the price was already $3.30 higher. The trade was correct at the time of execution. The P&L calculation that uses the Coinbase price as the market reference says you bought below market. The one that uses Kraken says you bought at market. The difference is which update your system ingested first, and whether your normalization layer timestamps by source time or arrival time.

Market data infrastructure is the plumbing that determines whether your prices are right, your order books are current, and your historical analysis reflects what actually happened — not what your system happened to see.

The Data Types

Market data arrives in three forms, each with different volume characteristics and freshness requirements.

Data type	What it contains	Update frequency	Typical message size
Trade ticks	Executed trades: price, quantity, aggressor side, timestamp	Per-trade (1K–100K/sec per venue)	40–80 bytes
Order book updates	Price level changes: add, modify, cancel at each level	Per-event (10K–1M/sec per venue)	30–60 bytes
Order book snapshots	Full depth at a point in time	Periodic (every 100ms–1s)	2–50 KB depending on depth

Trade ticks are the smallest and most manageable. A busy crypto exchange produces 5,000–20,000 trade messages per second. The CME during peak hours produces roughly 50,000 trade messages per second across all products.

Order book updates are the firehose. Nasdaq’s ITCH feed produces up to 20 million messages per day for US equities — peaks of 500,000 messages per second during market open. Each message represents a single order event: add, execute, cancel, or replace. Reconstructing the order book from these events requires maintaining state for every active order.

Order book snapshots are the shortcut. Instead of processing every incremental update, subscribe to periodic snapshots of the full book. You lose sub-millisecond granularity but eliminate the state management complexity. Most crypto exchanges offer both: a WebSocket stream of incremental updates and a REST endpoint for snapshots.

Distribution: Multicast vs Unicast

Traditional exchanges distribute market data via UDP multicast. A single stream is published once and received by all subscribers on the network segment. The exchange does not maintain per-subscriber state, does not retry dropped packets, and does not slow down for lagging consumers.

Exchange → Multicast Group (239.x.x.x)
              ├── Subscriber A (co-located)
              ├── Subscriber B (co-located)
              └── Subscriber C (co-located)

Multicast is efficient — the exchange sends each packet once regardless of subscriber count. The tradeoff is reliability. UDP provides no delivery guarantee. A dropped packet means a missed order book update, which means your local book state diverges from reality. Recovery requires either requesting a snapshot from the exchange (adding 10–50ms of staleness) or maintaining a secondary feed and cross-referencing.

CME’s MDP 3.0 protocol uses UDP multicast with two redundant feeds (A and B) on separate network paths. Subscribers process both and arbitrate — if a sequence number gap appears on feed A, they substitute from feed B. The arbitration logic adds complexity but eliminates most single-packet losses.

Crypto exchanges use TCP-based WebSocket connections — unicast per subscriber. Each subscriber gets its own connection with TCP’s delivery guarantees: ordered, reliable, with backpressure. The tradeoff is scalability. An exchange with 10,000 subscribers maintaining 10,000 WebSocket connections has a very different infrastructure profile than one multicasting to the same 10,000.

Protocol	Delivery	Ordering	Recovery	Latency
UDP multicast	Best-effort	Sequenced (app layer)	Snapshot request or redundant feed	Lowest (no handshake)
TCP WebSocket	Guaranteed	TCP-ordered	Built-in (TCP retransmit)	Higher (head-of-line blocking)
FIX/FAST	Guaranteed (TCP)	Application-sequenced	Gap fill request	Moderate

TCP’s reliability comes with head-of-line blocking. A single retransmitted packet stalls every subsequent message in the stream. Under congestion, a WebSocket feed can deliver a burst of stale updates all at once — your system processes them in order, but the “current” price at the end of the burst may be 50–200ms old.

Normalization

Your system receives data from Coinbase (JSON over WebSocket), Binance (JSON over WebSocket with different field names), CME (binary SBE over multicast), and Nasdaq (binary ITCH over multicast). Each has a different message format, different field semantics, and different timestamp conventions.

Normalization maps venue-specific formats into a canonical internal representation. The canonical message contains the fields your system needs regardless of source: instrument identifier, price, quantity, side, timestamp, and venue.

@dataclass
class NormalizedTrade:
    instrument: str      # "BTC-USD" (canonical, not venue-specific)
    price: Decimal       # always in quote currency
    quantity: Decimal    # always in base currency
    side: Side           # BUYER_AGGRESSOR or SELLER_AGGRESSOR
    source_time: int     # exchange timestamp (nanoseconds since epoch)
    arrival_time: int    # local receipt timestamp (nanoseconds)
    venue: str           # "coinbase", "binance", "cme"
    sequence: int        # venue-assigned sequence number

The critical field is the dual timestamp. source_time is when the exchange says the event occurred. arrival_time is when your system received it. The difference between these two values is your effective latency to that venue, and it varies by milliseconds to hundreds of milliseconds depending on network topology.

Normalization traps that create production bugs:

Trap	Example	Consequence
Timestamp precision	Coinbase sends microseconds, Binance sends milliseconds	Cross-venue ordering wrong by up to 999 microseconds
Quantity semantics	One venue reports base quantity, another reports quote quantity	Position calculations off by price factor
Price representation	Tick-based (CME) vs decimal (crypto)	Comparison and spread calculations break
Instrument naming	“BTC-USD” vs “BTCUSD” vs “XBT-USD”	Aggregation across venues fails silently
Side convention	Taker side vs maker side vs aggressor side	Trade flow analysis inverted

Build a normalization test suite that feeds recorded raw messages from each venue through the normalizer and verifies the canonical output against manually verified expected values. Run this suite on every deploy. A normalization bug that swaps bid and ask is invisible until a trading strategy loses money on every trade.

The Firehose Problem

A single Nasdaq ITCH feed produces peaks of 500,000 messages per second. Subscribing to 5 venues at similar rates means processing 2.5 million messages per second. Each message must be parsed, normalized, and delivered to downstream consumers — order book builders, analytics engines, trading strategies — within microseconds if you are latency-sensitive, or within low milliseconds if you are not.

The processing pipeline has three stages, each with a latency budget:

Stage	Budget (latency-sensitive)	Budget (analytics)
Wire → parse	< 1 μs	< 100 μs
Parse → normalize	< 500 ns	< 50 μs
Normalize → deliver	< 500 ns	< 1 ms

For latency-sensitive systems, the standard approach is a single-threaded event loop with no allocation on the hot path. LMAX’s Disruptor pattern — a lock-free ring buffer — is the canonical architecture. A single thread reads from the network, parses the binary message in place (no copy), writes the normalized event to the ring buffer, and downstream consumers read from the ring buffer at their own pace. No locks, no allocation, no garbage collection pauses.

For analytics systems that need all the data but can tolerate milliseconds of latency, a multi-threaded pipeline with bounded queues works. Kafka or Redpanda as the message bus provides durability, replay, and decoupling between producers and consumers. Throughput at this tier is 1–5 million messages per second per partition with p99 latency of 2–10ms.

The gap between “analytics-grade” and “trading-grade” market data is not a quality difference — it is a latency difference measured in microseconds. Both systems must process every message correctly. The analytics system processes them 1,000x slower.

Order Book Construction

An order book is a stateful projection of the event stream. Each event — add, modify, cancel, execute — updates the local representation. A missed or out-of-order event produces a book that diverges from the exchange’s actual state.

class OrderBook:
    def __init__(self):
        self.bids: SortedDict = SortedDict()  # price → quantity (descending)
        self.asks: SortedDict = SortedDict()  # price → quantity (ascending)
        self.sequence: int = 0

    def apply(self, event: BookEvent):
        if event.sequence != self.sequence + 1:
            raise SequenceGap(expected=self.sequence + 1, got=event.sequence)
        side = self.bids if event.side == Side.BID else self.asks
        if event.type == EventType.ADD or event.type == EventType.MODIFY:
            side[event.price] = event.quantity
        elif event.type == EventType.CANCEL:
            side.pop(event.price, None)
        self.sequence = event.sequence

The sequence check is the integrity mechanism. If a gap appears — event 1,001 arrives after event 999 with 1,000 missing — the book is unreliable. The recovery procedure depends on the venue: request a snapshot and replay from the snapshot’s sequence number forward, or discard the book and rebuild from a fresh snapshot.

Crossed book detection is the runtime health check. If the best bid exceeds the best ask in your local book, the book is corrupted — real exchanges do not allow crossed books (arbitrage would clear them instantly). A crossed book means your event processing has a bug, a missed message, or a race condition in your update logic.

Historical Storage

Storing tick data at scale is a time-series problem with financial-specific constraints: nanosecond timestamps, exact decimal precision, and queries that span both narrow time ranges (reconstruct the book at 14:32:07.003) and wide aggregations (VWAP for the last 24 hours).

Database	Strength	Throughput (writes)	Query model
kdb+	Fastest for tick data, column-oriented, in-memory	10M+ rows/sec	q (proprietary language)
QuestDB	SQL interface, column-oriented, designed for time-series	2–4M rows/sec	SQL (with extensions)
TimescaleDB	Postgres extension, full SQL, hypertables	500K–1M rows/sec	Standard SQL
ClickHouse	Column-oriented OLAP, strong aggregation	1–3M rows/sec	SQL (with extensions)
Parquet on S3	Cheapest for cold storage, columnar, immutable	Batch only	Athena/Spark/DuckDB

kdb+ dominates institutional finance — Goldman Sachs, Morgan Stanley, and most major banks run kdb+ for tick storage and analytics. The q language is terse to the point of opacity, but the performance is unmatched: a full day of US equities tick data (2–3 billion rows) can be queried in seconds. The licence cost reflects this: $100K+/year per core.

QuestDB and TimescaleDB offer the SQL interface that kdb+ lacks. TimescaleDB is a Postgres extension — your existing Postgres tooling, ORMs, and operational knowledge transfer directly. The throughput ceiling is lower, but for crypto market data (millions of rows per day, not billions), it is sufficient.

The storage cost calculation drives the architecture. A single crypto exchange produces roughly 500 million trade and order book events per day. At 100 bytes per event after compression, that is 50 GB/day or 18 TB/year from one exchange. Five exchanges over three years: 270 TB. Hot storage (SSD-backed database) at $0.10/GB/month costs $27,000/month. Cold storage (Parquet on S3) at $0.023/GB/month costs $6,210/month. The tiering strategy — keep recent data hot, age to cold storage on a rolling window — is not optional at this scale.

Clock Synchronization

Market data timestamps are only meaningful if the clocks generating them are synchronized. Your system’s arrival timestamps are generated by your local clock. The exchange’s source timestamps are generated by the exchange’s clock. PTP (Precision Time Protocol, IEEE 1588) achieves sub-microsecond synchronization between co-located systems. NTP achieves 1–10ms synchronization over the public internet.

If your system is co-located with the exchange and uses PTP, the difference between source time and arrival time measures actual network propagation plus processing — useful data. If your system is in a different data center using NTP, the difference includes up to 10ms of clock skew — noise that masks the signal.

For cross-venue analytics, clock skew between exchanges matters. If Exchange A and Exchange B both claim an event happened at 14:32:07.003 UTC but their clocks are 2ms apart, your cross-venue order flow analysis has a 2ms error bar on every comparison. There is no way to correct for this externally — you can only bound the error by knowing each exchange’s synchronization methodology and adding the appropriate uncertainty to cross-venue calculations.