Latency-Sensitive Systems

Prerequisite: Distributed Systems

Your matching engine processes 50,000 orders per second with a median latency of 8 microseconds. The dashboard looks healthy. Then a market-moving event hits, order volume spikes 5x, and your p99.9 jumps to 12 milliseconds — 1,500x slower than the median. The orders that arrived during that spike were filled at stale prices because your system could not process them before the market moved. The median tells you what happens when nothing interesting is happening. The tail tells you what happens when it matters.

Latency-sensitive systems are not about making the average case faster. They are about eliminating the variance that turns a fast system into an unpredictable one under load.

Median vs Tail Latency

Median latency (p50) describes the typical experience. Tail latency (p99, p99.9, p99.99) describes the worst experience that still happens regularly. A system processing 100,000 requests per second at p99.9 = 10ms means 100 requests per second experience that 10ms delay. At trading speeds, those 100 requests represent real money lost to stale fills.

Percentile	Meaning at 100K req/sec	Affected requests
p50 (median)	Half are faster	50,000/sec
p99	1 in 100 is this slow	1,000/sec
p99.9	1 in 1,000	100/sec
p99.99	1 in 10,000	10/sec

Tail latency has distinct causes from median latency. The median is dominated by the hot path — instruction cache hits, branch predictor accuracy, memory locality. The tail is dominated by system interference — garbage collection pauses, context switches, page faults, NUMA cross-socket access, thermal throttling. Optimizing the hot path improves the median. Eliminating system interference improves the tail.

Never report median latency alone for a system that moves money or fills orders. A matching engine with 5μs median and 50ms p99.9 is a 50ms matching engine for the orders that matter most — the ones arriving during volatility.

The Measurement Discipline

You cannot optimize what you cannot measure at the resolution your system operates. A trading system operating at microsecond latencies requires nanosecond-precision measurement. System.currentTimeMillis() has millisecond granularity — three orders of magnitude too coarse.

// rdtsc: CPU timestamp counter, ~1ns resolution
static inline uint64_t rdtsc() {
    uint32_t lo, hi;
    asm volatile("rdtsc" : "=a"(lo), "=d"(hi));
    return ((uint64_t)hi << 32) | lo;
}

The x86 rdtsc instruction reads the CPU’s timestamp counter with roughly 1-nanosecond resolution and costs approximately 20 cycles (7ns on a 3GHz processor). clock_gettime(CLOCK_MONOTONIC) on Linux provides nanosecond-resolution timestamps via the vDSO — no system call overhead, approximately 20ns per call.

Method	Resolution	Cost	Use case
rdtsc	~1ns	~7ns	Inline microbenchmarks
clock_gettime (vDSO)	1ns	~20ns	Application timestamps
gettimeofday	1μs	~1μs (syscall)	Legacy, avoid
System.nanoTime() (JVM)	1ns	~25ns	JVM applications

Measurement introduces overhead. A 20ns timestamp call on a 5μs operation adds 0.4% overhead per measurement point. Instrument the critical path with no more than 3-5 measurement points, and use hardware performance counters (perf, Intel VTune) for detailed profiling in non-production environments.

Coordinated omission — a term coined by Gil Tene — is the measurement error that occurs when your load generator slows down in response to server slowness. If the server takes 100ms to respond and your load generator waits for the response before sending the next request, you never measure the latency that queued requests would have experienced. HdrHistogram and wrk2 correct for this. Most benchmarking tools do not.

Kernel Bypass: Removing the OS from the Data Path

The Linux kernel networking stack adds 5-15μs of latency per packet traversal. For a system targeting sub-microsecond wire-to-wire latency, the kernel is the bottleneck.

Traditional path:    NIC → driver → kernel → socket → userspace    (~10μs)
Kernel bypass:       NIC → userspace (DPDK/RDMA)                   (~1μs)

DPDK (Data Plane Development Kit) maps NIC queues directly into userspace memory. The application polls the NIC in a tight loop — no interrupts, no context switches, no kernel involvement. A DPDK poll-mode driver receives packets in 1-2μs compared to 10-15μs through the kernel stack.

The cost is a dedicated CPU core per NIC queue. DPDK’s poll loop consumes 100% of a core regardless of traffic volume. A system with 4 NIC queues dedicates 4 cores to packet reception, even during idle periods. DPDK also bypasses the kernel’s TCP/IP stack — you either reimplement TCP in userspace (expensive) or use UDP-based protocols.

io_uring provides a middle ground. It eliminates system call overhead through a shared submission/completion ring buffer between userspace and kernel but still uses the kernel’s networking stack. Latency improvement over traditional epoll: 30-50%, not 10x. io_uring is the right choice when you need lower latency than epoll but cannot commit dedicated cores to polling.

Approach	Latency	CPU cost	TCP support	Complexity
epoll + kernel stack	10-15μs	Normal	Full	Low
io_uring	5-8μs	Normal	Full	Medium
DPDK	1-2μs	Dedicated cores	Must reimplement	High
RDMA	<1μs	Dedicated cores	No (custom protocol)	Very high

Lock-Free Data Structures

Mutex contention is the most common source of tail-latency spikes in multi-threaded systems. A thread that acquires a mutex forces every other thread targeting the same data to block. Under contention, the worst-case wait time is proportional to the number of contending threads multiplied by the lock hold time.

Lock-free data structures use atomic compare-and-swap (CAS) operations instead of mutual exclusion. No thread ever blocks — a failed CAS retries immediately without yielding the CPU.

// Lock-free SPSC (single-producer, single-consumer) ring buffer
typedef struct {
    uint64_t head __attribute__((aligned(64)));  // producer writes
    uint64_t tail __attribute__((aligned(64)));  // consumer writes
    void* buffer[RING_SIZE];
} spsc_ring;

bool push(spsc_ring* r, void* item) {
    uint64_t h = __atomic_load_n(&r->head, __ATOMIC_RELAXED);
    if (h - __atomic_load_n(&r->tail, __ATOMIC_ACQUIRE) >= RING_SIZE)
        return false;  // full
    r->buffer[h & (RING_SIZE - 1)] = item;
    __atomic_store_n(&r->head, h + 1, __ATOMIC_RELEASE);
    return true;
}

The LMAX Disruptor — the architecture behind one of the highest-throughput financial exchanges — uses a ring buffer with mechanical sympathy: cache-line-aligned entries, single-writer principle, and sequence barriers instead of locks. LMAX processes 6 million orders per second on a single thread with a mean latency of 1 microsecond.

Cache-line alignment matters. The __attribute__((aligned(64))) on head and tail ensures they occupy separate 64-byte cache lines. Without this, a write to head invalidates the cache line containing tail (false sharing), forcing a cache coherence round trip that costs 40-70ns on a modern multi-socket system.

The single-writer principle is the same insight behind TigerBeetle’s architecture. Eliminating concurrent access to shared state removes lock contention and cache-coherence overhead simultaneously. See Balance Derivation at Scale for how TigerBeetle applies this to financial transactions.

Memory Architecture

A cache miss to main memory costs 60-100ns. A cache miss to a remote NUMA node costs 100-300ns. For a system operating at microsecond latencies, memory access patterns determine performance more than algorithmic complexity.

Access	Latency	Relative cost
L1 cache hit	~1ns	1x
L2 cache hit	~4ns	4x
L3 cache hit	~12ns	12x
Local DRAM	~65ns	65x
Remote NUMA	~120ns	120x
Page fault (4KB)	~10μs	10,000x
THP page fault (2MB)	~2ms	2,000,000x

Huge pages eliminate TLB misses. The standard 4KB page size means a 1GB working set requires 262,144 TLB entries. A typical L1 dTLB holds 64 entries. TLB misses trigger page table walks costing 10-30ns each. 2MB huge pages reduce the entry count by 512x; 1GB huge pages reduce it by 262,144x.

NUMA pinning ensures a process accesses only local memory. On a two-socket system, accessing memory attached to the remote socket crosses the QPI/UPI interconnect. numactl --membind=0 --cpunodebind=0 pins both CPU and memory to socket 0.

Memory-mapped I/O (mmap) maps files directly into the process address space. Reads and writes become pointer dereferences — no read()/write() system calls. For market data replay and historical tick access, mmap with MAP_POPULATE (prefault all pages) and madvise(MADV_SEQUENTIAL) eliminates page fault stalls during access.

FPGA Acceleration

An FPGA (field-programmable gate array) processes data in hardware — no instruction fetch, no branch prediction, no cache hierarchy. A market data parser implemented in FPGA decodes a wire-format message in 50-200 nanoseconds, compared to 1-5 microseconds in optimized C.

Operation	CPU (optimized C)	FPGA	Speedup
Market data decode	1-5μs	50-200ns	10-25x
Order validation	500ns-2μs	100-500ns	3-5x
Risk check	1-10μs	200ns-1μs	5-10x
TCP offload	5-15μs	1-3μs	3-5x

The tradeoff is development cost and flexibility. FPGA development requires hardware description languages (Verilog, VHDL) or high-level synthesis (HLS). A market data parser in C takes days to write and modify. The same parser in Verilog takes weeks, and changing the wire format requires re-synthesis. Xilinx Alveo U50 cards cost $3,000-5,000; development boards and toolchains add $10,000+ per developer.

FPGAs are deployed where latency is a competitive advantage measured in nanoseconds — typically market-making and high-frequency trading. For fraud scoring at 45ms or payment processing at 200ms, FPGA acceleration does not change the outcome.

Co-location and Network Topology

Physical distance determines minimum latency. Light in fiber travels at approximately 200,000 km/s (two-thirds the speed of light in vacuum). New York to Chicago is 1,145 km by fiber — 5.7ms minimum one-way latency. Co-locating your server in the exchange’s data center reduces this to the length of a cross-connect cable: 1-10 meters, under 100 nanoseconds.

Route	Distance (fiber)	Minimum latency (one-way)
Same rack	2m	10ns
Same data center	200m	1μs
NY ↔︎ Chicago	1,145km	5.7ms
NY ↔︎ London	5,600km	28ms
NY ↔︎ Tokyo	15,000km	75ms

Exchange co-location services (NYSE Mahwah, CME Aurora, NASDAQ Carteret) charge $5,000-15,000/month per rack for proximity. The value proposition: if your strategy’s edge erodes at 10μs of additional latency, $15,000/month for a cross-connect is the cheapest optimization available.

Within the data center, network topology matters. A layer-2 switch adds 300ns-2μs of latency. Kernel bypass (Solarflare OpenOnload, Mellanox VMA) removes kernel overhead on the last hop. Direct cross-connects between servers eliminate the switch entirely.

JIT Compilation: When It Helps and When It Hurts

JVM-based systems (Java, Kotlin, Scala) use just-in-time compilation — the JVM interprets bytecode initially, then compiles hot methods to native code after observing execution patterns. The compiled code is often faster than ahead-of-time compiled C because the JIT can optimize based on runtime branch profiles.

The problem is when compilation happens. JIT compilation during a latency-sensitive operation produces a spike. The C2 compiler in HotSpot can stall for 10-100ms while compiling a complex method. If that method is in your order processing path, you get a 100ms latency spike on the first invocation pattern the JIT has not seen before.

# Force ahead-of-time compilation before serving traffic
java -XX:+TieredCompilation \
     -XX:CompileThreshold=1 \
     -XX:+PrintCompilation \
     -jar trading-engine.jar --warmup

Warmup is the mitigation. Before accepting production traffic, replay a representative set of order types to trigger JIT compilation of all hot paths. LMAX runs a 30-second warmup sequence that exercises every order type, every validation path, and every state transition. After warmup, the JIT has compiled everything — subsequent executions run native code with no compilation pauses.

Garbage collection is the other JVM latency tax. A G1GC pause on a 4GB heap runs 10-50ms. ZGC and Shenandoah target sub-millisecond pauses but add 5-15% throughput overhead. The LMAX Disruptor avoids GC entirely by pre-allocating all objects at startup and reusing them — no allocation on the hot path means no garbage to collect.

GC	Typical pause	Throughput cost	Use case
G1GC	10-50ms	Low	General purpose
ZGC	<1ms	5-15%	Low-latency services
Shenandoah	<1ms	5-10%	Low-latency services
No allocation (object pooling)	0ms	Development cost	Extreme latency sensitivity

The choice between JVM and native code (C/C++/Rust/Zig) is not about peak throughput — it is about tail-latency guarantees. A well-tuned JVM system with warmup and ZGC delivers p99 under 1ms. A system that cannot tolerate any GC pauses — even sub-millisecond — must either eliminate allocation entirely or use a non-GC language.