Public engineering docs for BT, a Python system I built to test whether a public weather-model ensemble can find mispriced contracts on Kalshi's daily high-temperature markets. It ran unattended for four months, settled 7,440 signals across five cities, and produced a clear answer: no. The model is badly overconfident, the market price predicts outcomes better than the model does, and the paper simulation loses money at every threshold. The source is private; this repo documents the engineering and the results.
| Question | Does a 31-member GFS ensemble, read as a probability distribution, beat Kalshi's price on KXHIGH* contracts after fees? |
| What I built | Full pipeline: forecast → bracket probabilities → fee-adjusted edge → Kelly sizing → risk gate → order execution → settlement → out-of-fold calibration and paper simulation. ~13K LOC Python, 945 tests, scheduled by launchd. |
| What happened | 337 unattended scans over 4 months. 7,440 signals on 2,472 contracts, all settled against the official NWS climate record. No orders placed: live trading was gated on calibration, and calibration failed. |
| Result | Raw model Brier 0.187 vs market price 0.120 on the same contracts. Out-of-fold isotonic calibration fixes the reliability curve but adds no skill. Paper P&L is negative at every threshold, raw or calibrated. |
| Status | Paused as a research result. The pipeline is healthy again after a September fix series (below) and still scans daily to grow the dataset. |
The negative result is the useful part. The pipeline did its job: it measured the strategy honestly and refused to trade on it. Getting to an honest measurement took three data bugs, also documented below.
Every scan window (four per day), for five cities (NYC, Chicago Midway, Denver, Miami, LAX):
- Pull the 31-member GFS ensemble from Open-Meteo for the contract day.
- Fetch the live
KXHIGH*market list from Kalshi and parse each bracket (B73.5,T82, etc.) into a temperature interval. - Compute
p_model= fraction of ensemble members landing in the bracket. - Compute fee-adjusted edge against the market price.
- Persist every bracket evaluation as a signal, edge or not, so calibration sees the full distribution.
- After settlement, reconcile each signal against the NWS Daily Climate Report for that station and record the outcome.
- Report reliability curves, Brier score, and a paper simulation. Trading is gated on this report.
7,440 settled signals, 5 cities, 83 contract days, 2,472 distinct contracts. Base rate of a bracket resolving YES: 16.7%.
| Predicted bucket | n | Mean predicted | Realized hit rate |
|---|---|---|---|
| 0.00 – 0.17 | 5,223 | 0.02 | 0.14 |
| 0.17 – 0.33 | 661 | 0.26 | 0.21 |
| 0.33 – 0.50 | 548 | 0.42 | 0.26 |
| 0.50 – 0.67 | 406 | 0.58 | 0.24 |
| 0.67 – 0.83 | 272 | 0.74 | 0.19 |
| 0.83 – 1.00 | 330 | 0.92 | 0.24 |
Overconfident at both ends. Brackets the model prices near zero hit 14% of the time. Brackets it prices above 83% hit 24% of the time. Reading ensemble-member frequency directly as probability ignores that the members share systematic bias on any given day, so the spread badly understates true uncertainty.
| Brier | ECE | |
|---|---|---|
| Raw model | 0.187 | 0.169 |
| Isotonic-calibrated model, out-of-fold (leave one contract day out) | 0.136 | 0.002 |
On the 2,777 signals that had a two-sided order book:
| Brier | |
|---|---|
| Raw model | 0.251 |
| Calibrated model | 0.204 |
| Kalshi market price | 0.120 |
Calibration makes the model honest, but it does so by squeezing every forecast into roughly 0.13 to 0.46. The calibrated model barely separates brackets. The market price beats both versions by a wide margin. There is no information here that the market doesn't already have.
YES-side trades only, one contract per signal at the executable ask, Kalshi taker fee 0.07 × P × (1−P), over signals with a valid book:
| Probabilities | Edge threshold | Trades | Distinct contracts | Net return per $1 staked |
|---|---|---|---|---|
| Raw | 0.05 | 740 | 542 | −$0.087 |
| Raw | 0.10 | 617 | 465 | −$0.112 |
| Raw | 0.20 | 467 | 361 | −$0.104 |
| Calibrated | 0.05 | 1,115 | 844 | −$0.420 |
| Calibrated | 0.10 | 740 | 606 | −$0.273 |
Negative everywhere. The calibrated model does worse because it lifts long shots just enough to clear the threshold. Trades on the same contract across scan windows are correlated, no fills or slippage are modeled, and only the YES side is testable historically because the bid wasn't recorded until September. None of that changes the sign.
The first version of this analysis, on 1,299 Chicago-only outcomes, showed a positive simulated return. Every cent of it was an artifact.
- Wrong weather station. The settlement fetcher passed the NWS forecast-office code (
LOT,BOU,OKX…) where the product endpoint expects the climate-station code (MDW,DEN,NYC…). Four cities never settled at all. Chicago settled against the Romeoville office's own report, not Midway. 246 of the 1,299 stored outcomes were wrong. - Phantom market prices. The stored market price was the raw YES ask. On an empty book Kalshi returns 100, so 4,663 of 7,440 signals were recorded as "market says 100%" when the market said nothing. A bot that trades against that "price" looks brilliant on paper and gets no fills in reality.
- Retired forecast model. Open-Meteo dropped the ECMWF model id the blend adapter used; the blend had been silently failing and falling back to GFS-only.
The fixes: station-keyed fetch, a backfill command that resettles every signal from the IEM archive of NWS climate reports (validated against Kalshi's official results on 2,724 overlapping contracts, 0 disagreements), mid-price capture with a book-validity flag, and the new model id.
- The settlement-source guard worked. In August Kalshi changed the resolution text on every series to cite The Weather Company. The bot checks the market's rules text against the expected station on every startup, detected the mismatch, and aborted rather than settle against an unverified source. It stayed halted for six weeks until I updated the expected tokens. The guard was right; the missing alarm on the abort loop was a monitoring gap.
- Four months unattended. 337 scheduled scans, heartbeat monitoring, log rotation, NTP drift checks, and rate-limit backoff on a laptop.
- Idempotent, auditable state. Every evaluation is a row in SQLite and an event in an append-only JSONL journal. All of the analysis above was reconstructed from that data after the fact, including catching the bugs.
Kalshi's taker fee is quadratic in price:
fee = 0.07 * P * (1 - P)
Edge deducts it before anything else:
edge = p_model - p_market - fee(p_market)
Sizing is fractional Kelly with hard caps that the strategy cannot raise:
kelly_fraction = f * (edge / (price * (1 - price)))
position_usd = bankroll * min(kelly_fraction, max_position_pct)
Calibration is the gate. A model that says 74% and wins 19% is not edge; it is a bug with a P&L attached.
launchd timers (4 scan windows/day)
│
▼
┌────────────────────┐ ┌─────────────────────┐
│ forecast adapter │ │ market adapter │
│ Open-Meteo GFS │ │ Kalshi REST, RSA-PSS│
│ 31-member ensemble │ │ signed requests │
└─────────┬──────────┘ └──────────┬──────────┘
└────────────┬─────────────┘
▼
┌───────────────────────┐
│ signal engine │
│ bracket parse → p │
└───────────┬───────────┘
▼
┌───────────────────────┐
│ edge + fee (mid-price)│
└───────────┬───────────┘
▼
┌───────────────────────┐
│ risk gate │
│ caps · kill file · │
│ calibration pass │ ← never passed; no orders placed
└───────────┬───────────┘
▼
┌───────────────────────┐
│ executor (maker-first)│
└───────────┬───────────┘
▼
┌───────────────────────┐
│ SQLite + JSONL journal│
└───────────┬───────────┘
▼
┌───────────────────────┐
│ settlement (NWS CLI / │
│ IEM archive backfill) │
│ → isotonic + paper sim│
└───────────────────────┘
A second pipeline (Phases 12–19) adapts the same shape to sports moneylines using de-vigged sharp-book consensus as the model. It is built and tested but was only run for a single afternoon; I have no results to report for it and don't claim any.
| Concern | Choice | Why |
|---|---|---|
| Runtime | Python 3.11, uv |
Stdlib plus httpx, cryptography, numpy, scikit-learn |
| Kalshi client | Hand-rolled, RSA-PSS (SHA-256, MGF1, salt = digest length) | Community SDKs are auto-generated and churn; auditability matters when orders are automatic |
| Forecasts | Raw HTTP to Open-Meteo | One less dependency |
| Settlement | NWS CLI product (live), IEM CLI archive (backfill) | Same underlying record; IEM gives a full year per station in one call |
| Storage | SQLite (8 schema migrations) + append-only JSONL | ACID state, replayable audit log, single file |
| Calibration | scikit-learn IsotonicRegression, leave-one-day-out |
No in-sample numbers reported, ever |
| Scheduling | launchd (macOS) / systemd (Linux) | Process supervision belongs to the OS |
| Testing | pytest + respx, 945 tests | Recorded-response fixtures; tests cannot reach the real exchange |
| Quality | ruff, mypy --strict, gitleaks pre-commit |
|
| Config | pydantic-settings + TOML | Fail fast on invalid config, exit code 2 |
- Trading gated on calibration, not backtest P&L. The gate never passed, so the executor never ran against a real book. That is the design working.
- Hard caps live in the risk gate, not the strategy. Per-position USD, per-position % of bankroll, daily loss.
- Kill file. A
.killfile in the runtime directory halts order placement within one scan. - Settlement-source assertion. Rules text is checked against the expected station token every startup.
- Tests are air-gapped from the exchange via recorded fixtures.
- Audit the data before believing the P&L. The first simulated return had the wrong sign because of two fields.
- Ensemble frequency is not probability. Members share bias; the spread understates uncertainty. And even after calibration, the market already knew everything the ensemble knew.
- Report out-of-fold numbers only. In-sample isotonic calibration would have looked perfect and meant nothing.
- Guards need alarms. A guard that halts the system correctly and silently is half a guard.
- Time zones are the top bug source. NWS reports on local standard time year-round. Store UTC, render local, settle on LST, assert at every boundary.
Not more of the same data. The GFS ensemble alone is dominated by the price. Things that might contain information the market lacks: a proper multi-model blend with per-station bias correction, intraday METAR updates close to settlement, or a station-specific downscaling model. Each is a new hypothesis, not a tuning of this one. The pipeline is ready to test any of them the same way.
kwx/
auth/ Kalshi RSA-PSS signing
forecast/ Open-Meteo GFS + ECMWF ensemble adapters, METAR
markets/ Kalshi REST market data, startup sanity guard
signals/ bracket parsing, probability distribution
risk/ caps, kill switch, calibration gate
exec/ order lifecycle
settle/ NWS CLI fetch + parse, IEM archive backfill, reconciler
calib/ reliability curves, Brier, ECE, isotonic report + paper sim
backtest/ historical replay harness
storage/ SQLite migrations, JSONL journal
ops/ heartbeat, daily summary
sports_*/ sports pipeline (Phases 12–19)
odds/ de-vig engine, sharp-consensus client
tests/ 945 tests
deploy/ launchd plists, systemd units
docs/ runbook, key rotation
Happy to walk through the code, the data, or the analysis on request.
Author: Luke Hanna · lllukehanna@gmail.com · Los Angeles