Ultra-low-latency C++20 stat-arb engine with a full Python research stack — cointegration (Engle-Granger + Johansen), Kalman-filtered hedge ratio, HMM regime detection, and walk-forward backtesting.
Statistical arbitrage — cointegration-based pair selection, a Kalman-filtered hedge ratio that adapts online, regime-gated signal generation, and an expanding-window walk-forward backtester with multiple-testing correction (Deflated Sharpe Ratio, White's Reality Check) — is core quant/finance methodology. But the discipline transfers well beyond trading: the same walk-forward-not-in-sample validation rigor is what any tech/AI role needs when a model's claimed accuracy has to survive deployment on genuinely out-of-sample data, and the same nanosecond-level, branchless, lock-free systems-engineering techniques in the C++ execution layer (RDTSC timing, SPSC rings, atomic BBO caching) generalize to any latency-critical pipeline, in finance or elsewhere.
| Component | Latency | Throughput |
|---|---|---|
| Signal → Order (end-to-end) | 9.81 ns (RDTSC) | 33.8M signals/s |
| Risk Check (branchless) | 1.92 ns | 521.8M checks/s |
| SPSC Ring Push/Pop | 4.97 ns | 201.3M ops/s |
| Fill → Position Update | 1.40 µs | 714K fills/s |
Target metrics: Sharpe ≥ 3.0 | Max DD ≤ 5% | 93%+ signal coverage
Benchmark hardware not documented in the original write-up. Re-run on a modest 4-core 2.5GHz shared VM in this pass produced the same order of magnitude (e.g. BM_SignalToOrder ~20.5ns vs. the 9.81ns above), consistent with weaker/shared hardware rather than a discrepancy — the numbers above are kept as the original dedicated-hardware measurement.
Two-layer design — Python research stack feeds signals into a C++ execution engine:
┌─────────────────── Research Layer (Python) ───────────────────┐
│ Binance WebSocket → Feature Engineering → Cointegration │
│ Engle-Granger / Johansen → Kalman Hedge Ratio → HMM Regime │
│ Walk-Forward Backtest (252d train / 63d refit) → Signal Gen │
└────────────────────────────────────────────────────────────────┘
↓ signals
┌─────────────────── Execution Layer (C++) ─────────────────────┐
│ Signal → Risk Manager (branchless) → SPSC Ring → Order Router │
│ Position Manager → SimExchange → Fill Callback │
└────────────────────────────────────────────────────────────────┘
Engle-Granger two-step (ADF with AIC lag selection) and Johansen VECM for multi-asset spreads. Rolling window classification tracks regime transitions and triggers model refit when p-value exceeds threshold.
Time-varying hedge ratio estimated via a state-space model. Noise covariances Q and R initialised via Expectation-Maximisation (Rauch-Tung-Striebel smoother) and updated online, giving a hedge ratio that adapts to structural breaks without look-ahead bias.
Two-state Gaussian HMM (trending / mean-reverting) implemented from scratch in NumPy. Baum-Welch EM for parameter estimation, Viterbi decoding for state sequence. Regime posterior used to gate signal generation — only trade when posterior P(mean-revert) > 0.6.
Expanding-window backtester: 252-day burn-in, 63-day refit cadence. All models (cointegration, Kalman, HMM) refit per fold. Transaction costs modelled via realistic bid-ask spread and market impact.
Sharpe, Sortino, Calmar, Deflated Sharpe Ratio (DSR), and White's Reality Check for multiple-testing correction. All computed on out-of-sample fold returns.
- RDTSC instead of
chrono::now()— eliminates 20–50 ns VDSO overhead, enabling accurate sub-10 ns measurement of the signal-to-order path. - Branchless risk check with bitwise
&— single cycle vs. potential branch misprediction cost; position limits enforced without a conditional jump. - SPSC ring with cached head — producer caches the consumer head pointer locally, removing an atomic read from the hot path entirely.
- Lock-free design — zero mutex / condvar overhead in the order-routing hot path; all coordination via acquire-release semantics on the ring buffer.
CMakeLists.txt lives at the repo root (not inside cpp/) — build from there:
cmake -B build -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_FLAGS="-march=native"
cmake --build build -j$(nproc)
./build/bm_arbpip install -r requirements.txt
python python/backtest/walk_forward.py
streamlit run python/viz/dashboard.pypython3 tools/gen_graphs.pyC++: x86-64 with AVX2, GCC 12+, CMake 3.20+, google-benchmark
Python: numpy, pandas, scipy, statsmodels, streamlit, matplotlib
- Built a sub-10 ns signal-to-order C++20 execution engine using RDTSC timing, branchless risk checks, and a lock-free SPSC ring buffer, achieving 33.8M signals/s end-to-end throughput.
- Implemented a full statistical arbitrage research stack (Engle-Granger + Johansen cointegration, Kalman-filtered hedge ratio with EM noise estimation, 2-state HMM regime detection) with a walk-forward backtester achieving Sharpe 3.2 and max drawdown under 5%.
- Eliminated 20–50 ns VDSO overhead by replacing
std::chrono::now()with direct RDTSC reads, and removed branch misprediction cost from the risk path via bitwise branchless logic, cutting round-trip latency from ~30 ns to 9.81 ns.
The Sharpe 3.2 result above was measured on real historical crypto data fetched via python/data/binance_client.py, which this environment cannot re-fetch to independently re-verify (no live network access to a market-data vendor in this sandbox). Rather than leave that unverified or silently re-derive a different number and present it as equivalent, here is what is independently verifiable in this repo, run in this pass:
python -m python.backtest.walk_forward ships its own __main__ demo, which runs the full six-fold expanding-window backtester end to end on synthetic price data (_generate_synthetic_prices) — deliberately constructed with strongly mean-reverting AR(1) residuals (phi=0.85) "so EG/Johansen tests will pass," per the function's own docstring. Run on this machine:
Completed 6 folds
Fold Period Sharpe MaxDD Trades WinRate
0 2022-12-21 -> 2023-03-17 4.11 -0.26% 4 75.0%
1 2023-03-20 -> 2023-06-14 3.05 -0.15% 3 100.0%
2 2023-06-15 -> 2023-09-11 5.46 -0.18% 6 100.0%
3 2023-09-12 -> 2023-12-07 4.18 -0.20% 2 100.0%
4 2023-12-08 -> 2024-03-05 4.34 -0.20% 3 100.0%
5 2024-03-06 -> 2024-04-19 7.58 -0.42% 4 100.0%
Overall Walk-Forward Performance
sharpe: 4.5444 sortino: 7.1552 calmar: 21.2487
max_drawdown: -0.0042 n_trades: 22 win_rate: 0.9545
This confirms the pipeline (cointegration -> Kalman -> HMM -> walk-forward -> cost model) runs correctly end to end and produces internally consistent fold-by-fold metrics — it does not confirm the Sharpe 3.2 real-market number, and should not be read as one. The synthetic data's near-perfect win rate and low trade count are exactly what you'd expect from residuals engineered to mean-revert easily; a real backtest on genuinely noisy market data is a harder problem than this smoke test, which is why the two numbers differ and neither should be substituted for the other.