diff --git a/docs/experiments/2026-09-01-per-product-slippage-restatement.md b/docs/experiments/2026-09-01-per-product-slippage-restatement.md new file mode 100644 index 0000000..3dd5a7f --- /dev/null +++ b/docs/experiments/2026-09-01-per-product-slippage-restatement.md @@ -0,0 +1,105 @@ +# Every measured null, re-priced per product — and the flat floor nobody reaches + +**Date:** 2026-09-01 · **Issue:** [#335](https://github.com/CodeGateSoftware/keel/issues/335) +(split from #259) · **Driver:** `2026-09-01-per-product-slippage-restatement.py` · +**240 trials** · **Ledger row:** `per-product-slippage-restatement-2026-09-01` + +## The finding + +Every experiment document in this repository prices fills at `slippage_pct=0.0005` — the +**floor** of `slippage_for_quote_volume`, which the model reaches only at its $500M/day anchor. +Measured over the 24-asset universe's own cached hourly candles: + +| | | +| :-- | --: | +| assets pricing at the floor | **0 of 24** | +| cheapest | BTC-USD, 5.5bp — **1.1×** the floor | +| dearest | TON-USD, 183.8bp — **36.8×** (the cap) | +| median | ≈ **10×** | + +Ten assets sit above 10× and four above 20×. #335 names *"the STX/CRO-class 1.15–1.30× floor +entries"* as the live example of a thin-asset candidate reaching the gate. The live example is +the entire universe. + +## What it costs + +Five rules — the three shipped signal rules plus `cusum_event` (#341) and `triple_barrier` +(#342) — at their shipped defaults, 24 assets, both regimes, run **here** so the A/B is +internally consistent rather than compared against numbers in other documents. + +| rule | PF flat | PF per-product | delta | +| :-- | --: | --: | --: | +| `turtle_breakout` | 0.336 | 0.267 | −0.063 | +| `rsi_meanrev` | 0.261 | 0.175 | −0.082 | +| `pullback_continuation` | 0.042 | 0.012 | −0.021 | +| `cusum_event` | 0.343 | 0.243 | −0.080 | +| `triple_barrier` | 0.338 | 0.237 | −0.092 | + +**Across all 120 cells the median profit factor falls from 0.309 to 0.219** — a median +overstatement of **0.090**. + +That number deserves to be read next to the other one measured this week: the triple barrier's +better exit bought **+0.033** of gross profit factor +([the A/B](2026-09-01-triple-barrier-first-measurement.md)). **The error in the cost model was +2.7× larger than the best genuine improvement any rule change produced.** Every comparison +between strategies in this repository has been made through a lens that mis-priced execution by +more than the differences being compared. + +## The one cell that looked positive dies + +| rule | product | flat | per-product | n | slippage | +| :-- | :-- | --: | --: | --: | --: | +| `turtle_breakout` | WLD-USD | **1.061** | **0.626** | 58 | 1.209% (24.2×) | + +It was the only cell above 1.0 in 120 under flat pricing, it was already below the 100-trade +admission floor, and per-product pricing removes it. **Zero of 120 cells clear PF 1.0 under +per-product pricing.** + +## Why this was safe to defer, and why it is not safe to leave + +#259 deferred the gate opt-in on the reasoning that the correction is **conservative-only**: +real cost is higher, so a corrected profit factor can only fall, and per-product pricing can +never manufacture an edge. That reasoning is correct and this run confirms it — every one of the +120 deltas is negative or zero. + +What it does not survive is the magnitude. A correction assumed to be a rounding adjustment is +worth 0.090 of median profit factor and kills the only positive cell in the corpus. A gate that +prices promotion decisions — the money-moving case — at the best rate the model can produce is +not being conservative; it is being optimistic in the one place optimism is most expensive. + +**`keel rules backtest` / `rules promote` now price per product** (`rules.backtest_slippage`), +computed from the product's cached ONE_DAY bars by the same one definition +`simulate.slippage_assumptions` uses. A product with no daily bars falls back to the flat floor +and is **flagged as a fallback**, never presented as a measured verdict. + +## Honesty + +**No configuration, no argmax, no free parameters.** Every rule runs at its shipped defaults; +there is nothing here that could be selected on. + +**This does not restate the older documents' numbers.** The restated intersection, the fee curve +and the hourly turtle sweep all still carry flat-priced figures. Their VERDICTS are unaffected — +the correction only pushes them further from 1.0 — but their levels are optimistic by roughly the +margin measured here, and a reader comparing across documents should know it. + +**A unit trap, recorded twice now.** `median_daily_quote_volume` returns a **per-bar** median +despite its name. Read off an hourly series and handed to a model anchored on a daily volume, it +reports every asset as maximally thin. The gate avoids it by reading ONE_DAY bars, as +`simulate` already did; `triple_barrier.per_product_round_trip` cannot (a pure rule has only the +candles handed to it) and scales explicitly instead. Both say so where they do it. + +**Validation.** Screening result only: no walk-forward, no out-of-sample split, no CSCV/PBO +(`series_missing`). Same cached candles and ~5-year window as every other document here. Fees +held at the 1.2% taker rate in every cell, so slippage is the only thing that varies. + +**Changed nothing about what trades.** No rule row added, nothing promoted, no config or +allowlist touched. + +## Recommended next + +1. **Do not restate the old documents by re-running them.** The verdicts do not move and the + compute is not free. What is worth doing is a one-line note in each pointing here. +2. **The account simulator and `paper.py` still carry their own flat literals.** #335's mechanics + section names them; the gate was the money-moving one and is done, and folding the rest is a + tidy-up with no verdict attached. +3. **The null is unchanged and firmer: 0 of 138 stands, now at honest cost.** diff --git a/docs/experiments/2026-09-01-per-product-slippage-restatement.py b/docs/experiments/2026-09-01-per-product-slippage-restatement.py new file mode 100644 index 0000000..a746099 --- /dev/null +++ b/docs/experiments/2026-09-01-per-product-slippage-restatement.py @@ -0,0 +1,153 @@ +"""What every measured null looks like when execution is priced per product (#335, #259). + +**The finding this exists to quantify.** Every experiment document in this repository prices +fills at `slippage_pct=0.0005` -- the FLOOR of `slippage_for_quote_volume`, reached only at the +model's $500M/day anchor. Measured over the 24-asset universe's own cached hourly candles, NOT +ONE ASSET REACHES IT: the range is 1.1x the floor (BTC, 5.5bp) to 36.8x (TON, the 183.8bp cap), +with a median near 10x. #335 named "the STX/CRO-class 1.15-1.30x floor entries" as the live +example of a thin-asset candidate; the live example is the entire universe. + +The direction is conservative -- real costs are HIGHER, so real profit factors are LOWER, and +per-product pricing can only make a null more negative, never rescue one. That is why #259's +deferral was safe. It is not why it should continue. + +DECLARED BEFORE THE RUN: + +* **Primary metric: the DELTA in profit factor, flat vs per-product, at the taker rate.** Both + arms are run HERE rather than compared against numbers in other documents, so the A/B is + internally consistent and no cross-document drift can enter it. +* **Every shipped signal rule plus both #341/#342 rules**, at their shipped defaults. No sweep, + no argmax, no free parameters: 5 rules x 24 assets x 2 slippage regimes = 240 trials. +* **The pre-declared expectation is that nothing changes verdict** -- the intersection is + already empty, and a strictly higher cost cannot fill it. What is being measured is the SIZE + of the understatement, and whether any cell that looked positive under flat pricing survives. + +Re-run: + KEEL_EXPERIMENT_DB=~/keel/keel.db \ + python docs/experiments/2026-09-01-per-product-slippage-restatement.py +""" + +import json +import os +import time +from concurrent.futures import ProcessPoolExecutor, as_completed +from decimal import Decimal +from pathlib import Path + +DB = os.environ.get("KEEL_EXPERIMENT_DB") or str(Path.home() / "keel" / "keel.db") +OUT_DIR = os.environ.get("KEEL_EXPERIMENT_OUT") or str(Path(__file__).resolve().parent / "_out") +Path(OUT_DIR).mkdir(parents=True, exist_ok=True) +JSONL_PATH = f"{OUT_DIR}/per_product_slippage.jsonl" + +UNIVERSE = [ + "BTC-USD", "ETH-USD", "ADA-USD", "LINK-USD", "LTC-USD", "SOL-USD", + "XLM-USD", "PAXG-USDT", "BCH-USD", "AAVE-USD", "DOGE-USD", "DOT-USD", + "UNI-USD", "ZEC-USD", "ALGO-USD", "FET-USD", "CRV-USD", "ICP-USD", + "AVAX-USD", "NEAR-USD", "XRP-USD", "PAXG-USD", "WLD-USD", "TON-USD", +] +RULES = ["turtle_breakout", "rsi_meanrev", "pullback_continuation", "cusum_event", "triple_barrier"] +TAKER = Decimal("0.012") +#: Bars per day at ONE_HOUR -- `median_daily_quote_volume` is a PER-BAR median despite its name, +#: and the slippage model is anchored on a DAILY figure. See `triple_barrier`'s own note. +BARS_PER_DAY = 24 + + +def make_rule(kind, asset): + from keel.strategy.rules.cusum_event import CusumEvent + from keel.strategy.rules.pullback_continuation import PullbackContinuation + from keel.strategy.rules.rsi_meanrev import RsiMeanReversion + from keel.strategy.rules.triple_barrier import TripleBarrier + from keel.strategy.rules.turtle_breakout import TurtleBreakout + + return { + "turtle_breakout": TurtleBreakout, + "rsi_meanrev": RsiMeanReversion, + "pullback_continuation": PullbackContinuation, + "cusum_event": CusumEvent, + "triple_barrier": TripleBarrier, + }[kind](product_id=asset) + + +def run_job(job): + kind, asset = job + from keel_core.types import Granularity + + from keel.compliance.screen import median_daily_quote_volume + from keel.data.db import connect + from keel.data.repository import Repository + from keel.strategy import backtest as bt + + try: + repo = Repository(connect(DB)) + candles = repo.get_candles(asset, Granularity.ONE_HOUR) + except Exception as exc: + return [{"rule": kind, "product": asset, "error": f"{type(exc).__name__}: {exc}"}] + if not candles: + return [{"rule": kind, "product": asset, "error": "no hourly candles"}] + + daily_volume = median_daily_quote_volume(candles) * BARS_PER_DAY + per_product = bt.slippage_for_quote_volume(daily_volume) + rows = [] + for regime, slippage in (("flat", bt.SLIPPAGE_FLOOR_PCT), ("per_product", per_product)): + try: + result = bt.backtest( + make_rule(kind, asset), candles, fee_pct=TAKER, slippage_pct=slippage + ) + rows.append({ + "rule": kind, + "product": asset, + "regime": regime, + "slippage_pct": str(slippage), + "floor_multiple": float(per_product / bt.SLIPPAGE_FLOOR_PCT), + "daily_quote_volume": float(daily_volume), + "n_trades": int(result.n_trades), + "profit_factor": float(result.profit_factor), + "win_rate": float(result.win_rate), + "expectancy": float(result.expectancy), + }) + except Exception as exc: + rows.append({ + "rule": kind, "product": asset, "regime": regime, + "error": f"{type(exc).__name__}: {exc}", + }) + return rows + + +def done_combos(): + if not os.path.exists(JSONL_PATH): + return set() + done = set() + for line in open(JSONL_PATH): + line = line.strip() + if not line: + continue + try: + row = json.loads(line) + except json.JSONDecodeError: + continue # torn final line from a kill mid-write + done.add((row["rule"], row["product"])) + return done + + +def main(): + jobs = [ + (kind, asset) + for kind in RULES + for asset in UNIVERSE + if (kind, asset) not in done_combos() + ] + print(f"{len(jobs)} combos ({len(jobs) * 2} trials) -> {JSONL_PATH}", flush=True) + started = time.time() + with open(JSONL_PATH, "a") as sink, ProcessPoolExecutor() as pool: + futures = {pool.submit(run_job, job): job for job in jobs} + for index, future in enumerate(as_completed(futures), start=1): + for row in future.result(): + sink.write(json.dumps(row) + "\n") + sink.flush() + if index % 20 == 0 or index == len(jobs): + print(f" {index}/{len(jobs)}", flush=True) + print(f"done in {time.time() - started:.0f}s", flush=True) + + +if __name__ == "__main__": + main() diff --git a/docs/experiments/trials-ledger.jsonl b/docs/experiments/trials-ledger.jsonl index f22757c..67950df 100644 --- a/docs/experiments/trials-ledger.jsonl +++ b/docs/experiments/trials-ledger.jsonl @@ -90,3 +90,4 @@ {"decision":"diagnostic_only","kind":"sweep_node","params":{"buffer_ticks":0.049704400803443,"ema_periods":[11,17,40],"product_id":"SOL-USD"},"per_bar_pnl":[],"per_trade_pnl":["-4.0173355540198047831580","-3.5027104940198047831580","-2.9839631340198047831580","-2.5405856740198047831580","-2.6313359140198047831580","-2.4288351940198047831580","-2.7519619340198047831580","-2.5525855540198047831580","-2.7715863940198047831580","-2.3049602540198047831580","-2.3478355540198047831580","-1.3243723579405856505260","-2.0557094140198047831580","-2.0342092940198047831580","-2.1982098940198047831580"],"prev_hash":"787c3934700c34645fc77e137f9df8927031cc07f146ec82f476322203e0f24a","provenance":"fitted","row_hash":"2a8c802b036024d6913519e2d842d7f036e0572e34cb63c3fb374bc1f243296c","rule":"pullback_continuation","series_missing":false,"session":"optuna-parameter-study-2026-08-22","summary":{"bars":17520,"fee_pct":"0.012","gate_passed":0,"held_out_expectancy":"-2.563079774281190174315866667","n_trials":60,"pbo":"0.8428571428571428571428571429","pbo_available":1,"seed":476,"slippage_pct":"0.0005","train_expectancy":"-4.7947464365621181060780"},"timestamp":1787368446,"trial_id":"476-optuna-pullback_continuation"} {"decision":"rejected","kind":"ablation","params":{"arms":"A: shipped default (threshold_friction_mult=2 -> 5.0% threshold, lookback=168, atr_period=20, atr_stop_mult=2, target_rr=3) x 24 assets x 3 fees = 72 trials. B: threshold_friction_mult in {1,2,3,4} at the taker fee x 24 assets = 96 trials. 168 trials disclosed.","changed_nothing":"A document and one driver script. No rule row was added, nothing was promoted, no config was touched, no allowlist changed. cusum_event remains registered and untraded.","declared_before_the_run":"PRIMARY METRIC n_trades (feasibility -- an event filter trades less by construction, and the rho=-0.77 edge-vs-sample bind says a rule that fires rarely cannot be admitted whatever its profit factor). Secondary: profit factor at fee 0/0.006/0.012. Arm A is ONE configuration, the shipped default, so nothing in it is an argmax. Arm B sweeps the rule's own headline knob and its per-asset best is a maximum of four draws.","document":"docs/experiments/2026-09-01-cusum-event-first-measurement.md","feasibility_answer":"YES, and it CONTRADICTS the pre-registered worry. The concern filed with #341 was that gating cuts n on rules already below the admission floor. It does not: at the shipped default n>=100 on 21 of 24 assets, median 553, max 884. Event sampling on hourly bars fires often because a 5% cumulative move in an hourly crypto series is a common occurrence, not a rare one. The feasibility objection to this rule is answered and should not be repeated.","issue":"#341","profitability_answer":"NO, and not marginally. At the 1.2% taker rate ZERO of 24 assets clear PF 1.0 -- median 0.343, best 0.479 (AVAX-USD, n=601). At the 0.6% maker rate this account cannot reach, still zero of 24, median 0.553. The intersection of n>=100 and PF>1.0 is EMPTY across both arms, all 168 trials.","recommended_next":"(a) NOT further cusum_event tuning: it is negative at zero cost on 16 of 24 assets and its own headline knob moves median PF by 0.15 across a fourfold range. (b) #342's triple-barrier exits are the source's other half and are now the only untested part of that paper's claim -- but the honest prior tightened here, since the entry half has no gross edge for a better exit to harvest. (c) The result stands as the fifth family in the null: 0 of 90 becomes 0 of 114.","script":"docs/experiments/2026-09-01-cusum-event-first-measurement.py","selection_bias":"Arm A is ONE pre-declared configuration and carries none. Arm B's per-asset best is a maximum of FOUR draws and must never be quoted as an edge estimate; its best cell overall is CRV-USD at mult=4, PF 0.593, n=322 -- still 0.4 short of 1.0, so the bias does not change the verdict in either direction.","the_diagnostic":"AT ZERO FEE ONLY 8 OF 24 CLEAR 1.0, median 0.925, max 1.238. That is the finding. turtle_breakout at zero fee was profitable on 4 of 4 with a max of 2.713 -- a real gross edge that cost destroyed. cusum_event has ESSENTIALLY NO GROSS EDGE for cost to destroy: its median is below 1.0 at zero cost. Same symptom as turtle at 1.2%, the rsi_meanrev disease. Cheaper execution cannot rescue it, because zero fee bounds from above everything an execution fix could ever buy and zero fee is already a median loss.","the_knob_works_and_does_not_rescue_it":"threshold_friction_mult behaves exactly as designed and converges to nothing. 1 -> 4 raises median PF monotonically 0.226/0.343/0.361/0.379 while cutting median n 1252/553/330/224. Trading less does lose less per toll -- the mechanism is real -- but the trend is toward a ceiling below 1.0, and n falls toward the admission floor as it climbs. There is no multiple at which both hold.","the_sources_own_setting_is_the_worst_cell":"mult=1 is the paper's 2.0-2.5% threshold and is exactly one round trip here. It is the worst arm-B cell on every axis: median PF 0.226, the lowest of the four, with median n 1252 -- the most trading at the least edge. This is the friction-scaling argument of #341 measured rather than asserted.","universe":"the same 24 assets as the restated intersection, ONE_HOUR, ~5y cached","validation":"Screening result only. No walk-forward, no out-of-sample split, no CSCV/PBO and no deflated Sharpe: backtest emits aggregates and no per-trade series (series_missing). Same cached candles and same ~5-year window as every other document here. slippage_pct held at 0.0005 in every cell, so 'zero fee' is zero FEE and not zero cost."},"per_bar_pnl":[],"per_trade_pnl":[],"prev_hash":"2a8c802b036024d6913519e2d842d7f036e0572e34cb63c3fb374bc1f243296c","provenance":"a_priori","row_hash":"ef473b236321c2f081054fb2012860a0e34f5f195a02518eb807a88978c8b6af","rule":"cusum_event","series_missing":true,"session":"cusum-event-first-measurement-2026-09-01","summary":{"arm_a_assets":24,"arm_a_n_above_floor":21,"arm_a_n_median":553,"arm_a_pf_above_one_taker":0,"arm_a_pf_above_one_zero_fee":8,"arm_a_pf_max_taker":"0.479","arm_a_pf_median_taker":"0.343","arm_a_pf_median_zero_fee":"0.925","fee_pct":"0.012","gate_passed":0,"intersection_n100_and_pf1":0,"min_trades_floor":100,"n_trials":168,"pbo_available":0,"slippage_pct":"0.0005"},"timestamp":1788305412,"trial_id":"cusum-event-first-measurement-2026-09-01"} {"decision":"rejected","kind":"ablation","params":{"changed_nothing":"A document, a driver and a ledger row. No rule row added, nothing promoted, no config or allowlist touched. triple_barrier is registered and untraded.","control":"cusum_event, measured the same day on the SAME universe and window (cusum-event-first-measurement-2026-09-01). Both rules share an entry -- the same CUSUM filter at the same threshold -- so the exit is the only thing that changed and the difference is attributable. This is an A/B, not another level reading.","declared_before_the_run":"PRIMARY METRIC: the DELTA in profit factor against the control at the taker rate, NOT the level. The level was already a known null and asking 'does it clear 1.0' invites reading a 0.4 as encouraging. The question is how much a better exit moves a rule whose entry has no gross edge. Arm A is ONE configuration (no argmax); arm B sweeps max_holding_bars over {6,12,24,48,72} and its best is a max of 5 draws.","document":"docs/experiments/2026-09-01-triple-barrier-first-measurement.md","issue":"#342","levels":"Zero of 24 clear PF 1.0 at the taker rate (median 0.338, max 0.527) and zero at the 0.6% maker rate. n>=100 on 20 of 24, median 431 -- the vertical barrier closes positions the signal exit let run, so trade count falls against the control's 553 while staying clear of the floor.","recommended_next":"(a) NOT a barrier sweep: the horizontals are already friction-sized and the vertical is monotone across a twelvefold range for 0.30 of profit factor, all of it below 1.0. (b) The source is fully tested and fully answered -- there is no third half. (c) The null grows: 0 of 90 -> 0 of 114 -> 0 of 138.","script":"docs/experiments/2026-09-01-triple-barrier-first-measurement.py","selection_bias":"Arm A is one pre-declared configuration and carries none. Arm B's per-asset best is a maximum of five draws; its best cell overall is the n=17 TON-USD artefact above, and quoting it as anything else would be the error this field exists for.","the_answer":"THE EXIT GENUINELY WORKS AND IS WORTH NOTHING. At ZERO fee it lifts the median profit factor ACROSS BREAK-EVEN, 0.925 -> 1.001, improving 17 of 24 assets: a real, measurable gross improvement from a better exit, and the first number in this series to move the right way. At the 1.2% taker rate the median delta is -0.004 and the sign is a coin flip (11 of 24). The gross gain is smaller than the friction it has to be harvested through.","the_two_cells_above_one":"TON-USD at 48 bars (PF 1.270, n=17) and 72 bars (PF 1.063, n=16) -- a sixth of the admission floor, maxima of five draws on the thinnest asset in the universe. Not evidence of anything. The intersection of n>=100 and PF>1.0 is EMPTY across all 192 trials.","unit_trap_recorded":"`median_daily_quote_volume` returns a PER-BAR median despite its name, and `slippage_for_quote_volume` is anchored on $500M DAILY -- so feeding it the hourly figure unscaled reports every asset as maximally thin, clamps the universe to the 183.8bp cap and makes every barrier four times too wide, silently and with no error. `per_product_round_trip` scales by bars-per-day; a test asserts the consequence rather than the call.","validation":"Screening result only. No walk-forward, no out-of-sample split, no CSCV/PBO and no deflated Sharpe (series_missing). Same cached candles and ~5-year window as every other document here. slippage_pct held at 0.0005 in every cell, so 'zero fee' is zero FEE and not zero cost.","vertical_barrier_axis":"Monotone: 6/12/24/48/72 bars give median PF 0.161/0.243/0.338/0.392/0.464 with median n 520/486/431/374/338. Holding longer is better and the source's own 24-bar barrier is mid-range, not optimal -- the direction agrees with the paper's 'wide barriers beat next-bar labeling'. The magnitude does not, because 1.2% per leg is twelve times the 0.1% that paper priced.","what_this_settles_about_the_source":"Both halves of Gradzki et al. are now implemented on keel's cost structure and measured on one universe: the ENTRY half has essentially no gross edge (median 0.925 at zero cost) and the EXIT half is a real improvement (+0.033 gross, 17 of 24) that friction consumes entirely. The paper is not wrong about its own venue -- at 0.1% per leg a +0.033 gross improvement is worth keeping; at 2.5% it is not. This is the clearest measurement in the series of the difference between a result and a result AT A PRICE."},"per_bar_pnl":[],"per_trade_pnl":[],"prev_hash":"ef473b236321c2f081054fb2012860a0e34f5f195a02518eb807a88978c8b6af","provenance":"a_priori","row_hash":"c8ad6ec7e0e5eb524bd89038b841301ce7c6214707cec5609537658f2a952993","rule":"triple_barrier","series_missing":true,"session":"triple-barrier-first-measurement-2026-09-01","summary":{"arm_a_assets":24,"arm_a_n_above_floor":20,"arm_a_n_median":431,"arm_a_pf_above_one_taker":0,"arm_a_pf_above_one_zero_fee":13,"arm_a_pf_max_taker":"0.527","arm_a_pf_median_taker":"0.338","arm_a_pf_median_zero_fee":"1.001","delta_improved_taker":11,"delta_improved_zero_fee":17,"delta_vs_control_median_taker":"-0.004","delta_vs_control_median_zero_fee":"0.033","fee_pct":"0.012","gate_passed":0,"intersection_n100_and_pf1":0,"min_trades_floor":100,"n_trials":192,"pbo_available":0,"slippage_pct":"0.0005"},"timestamp":1788306964,"trial_id":"triple-barrier-first-measurement-2026-09-01"} +{"decision":"diagnostic_only","kind":"ablation","params":{"changed_nothing_about_what_trades":"No rule row added, nothing promoted, no config or allowlist touched.","document":"docs/experiments/2026-09-01-per-product-slippage-restatement.md","issue":"#335 (split from #259)","not_restated":"The restated intersection, the fee curve and the hourly turtle sweep still carry flat-priced figures. Their VERDICTS are unaffected -- the correction only pushes them further from 1.0 -- but their LEVELS are optimistic by roughly the margin measured here, and a reader comparing across documents should know it. Re-running them would move no verdict and is not free.","script":"docs/experiments/2026-09-01-per-product-slippage-restatement.py","shipped_with_this":"`keel rules backtest` / `rules promote` now price per product via `rules.backtest_slippage`, from the product's cached ONE_DAY bars by the same one definition simulate.slippage_assumptions uses. No daily bars falls back to the flat floor and is FLAGGED as a fallback, never presented as a measured verdict. Both call sites (`_backtest_rule` and `backtest_resolved`) are pinned by tests: a mutation removing slippage_pct from either left every helper test green.","the_comparison_that_matters":"A median overstatement of 0.090 against the +0.033 of gross profit factor the triple barrier's better exit bought (2026-09-01-triple-barrier-first-measurement). THE ERROR IN THE COST MODEL WAS 2.7x LARGER THAN THE BEST GENUINE IMPROVEMENT ANY RULE CHANGE PRODUCED. Every strategy comparison in this repository has been made through a lens that mis-priced execution by more than the differences compared.","the_finding":"Every experiment document in this repository prices fills at slippage_pct=0.0005 -- the FLOOR of slippage_for_quote_volume, which the model reaches only at its $500M/day anchor. Measured over the 24-asset universe's own cached candles, NOT ONE ASSET REACHES IT: 1.1x the floor (BTC, 5.5bp) to 36.8x (TON, the 183.8bp cap), median near 10x, ten assets above 10x and four above 20x. #335 names the 'STX/CRO-class 1.15-1.30x floor entries' as the live example of a thin-asset candidate; the live example is the entire universe.","the_positive_cell_dies":"turtle_breakout on WLD-USD: 1.061 flat -> 0.626 per-product, at n=58 (already below the 100 floor) and 1.209% slippage, 24.2x the rate it was priced at. It was the only cell above 1.0 in 120. Zero of 120 clear PF 1.0 per-product.","unit_trap":"median_daily_quote_volume returns a PER-BAR median despite its name. Read off an hourly series and handed to a model anchored on a DAILY volume it reports every asset as maximally thin. The gate avoids it by reading ONE_DAY bars as simulate already did; triple_barrier.per_product_round_trip cannot (a pure rule has only the candles handed to it) and scales explicitly. Both say so where they do it.","validation":"Screening result only: no walk-forward, no out-of-sample split, no CSCV/PBO (series_missing). Same cached candles and ~5-year window as every other document. Fee held at the 1.2% taker rate in every cell, so slippage is the only variable. No configuration, no argmax, no free parameters -- nothing here could be selected on.","what_it_costs":"Five rules at shipped defaults x 24 assets x 2 regimes, run in ONE driver so the A/B is internally consistent. Median PF across 120 cells falls 0.309 -> 0.219. Per rule: turtle_breakout 0.336->0.267, rsi_meanrev 0.261->0.175, pullback_continuation 0.042->0.012, cusum_event 0.343->0.243, triple_barrier 0.338->0.237. Every one of the 120 deltas is negative or zero.","why_the_deferral_was_safe_and_is_not":"#259 deferred on the reasoning that the correction is CONSERVATIVE-ONLY: real cost is higher, a corrected profit factor can only fall, and per-product pricing can never manufacture an edge. Confirmed -- all 120 deltas are <= 0. What it does not survive is the magnitude: a correction assumed to be a rounding adjustment is worth 0.090 of median PF and kills the corpus's only positive cell. A gate pricing PROMOTION decisions at the best rate the model can produce is not conservative."},"per_bar_pnl":[],"per_trade_pnl":[],"prev_hash":"c8ad6ec7e0e5eb524bd89038b841301ce7c6214707cec5609537658f2a952993","provenance":"a_priori","row_hash":"a6b9dbc9ed6bba23f0c2c2280ad7859958eb0f0108bc9ae0bd06af54268a509c","rule":"all_shipped","series_missing":true,"session":"per-product-slippage-restatement-2026-09-01","summary":{"assets":24,"assets_at_the_floor":0,"cells":120,"cells_above_one_flat":1,"cells_above_one_per_product":0,"fee_pct":"0.012","flat_slippage_pct":"0.0005","floor_multiple_max":"36.8","floor_multiple_min":"1.1","gate_passed":0,"n_trials":240,"pbo_available":0,"pf_median_delta":"-0.090","pf_median_flat":"0.309","pf_median_per_product":"0.219","rules":5},"timestamp":1788311146,"trial_id":"per-product-slippage-restatement-2026-09-01"} diff --git a/keel/commands/rules.py b/keel/commands/rules.py index 8d7d3ef..edebde5 100644 --- a/keel/commands/rules.py +++ b/keel/commands/rules.py @@ -42,6 +42,7 @@ from keel.analysis import indicators from keel.commands._common import _load_cfg, _open_repo, with_disclaimer from keel.commands._products import parse_products_option +from keel.compliance import screen as screen_mod from keel.data.history import GRANULARITY_SECONDS from keel.data.repository import Repository @@ -228,6 +229,38 @@ def _resolve_backtest_inputs( return granularity, repo.get_candles(product_id, granularity) +def backtest_slippage(repo: Repository, product_id: str) -> tuple[Decimal, bool]: + """`(rate, measured)` -- this product's liquidity-scaled per-leg slippage for the gate (#335). + + **The gate priced every fill at the 5bp FLOOR until now, and the floor is not a typical + rate -- it is the best case the model can produce.** `slippage_for_quote_volume` reaches it + only at its $500M/day anchor, and measured over the 24-asset universe's own cached candles + NOT ONE ASSET GETS THERE: the range is 1.1x the floor (BTC) to 36.8x (TON, the cap), median + near 10x. #259 deferred this on the reasoning that the correction is conservative-only -- + real cost is higher, so a corrected profit factor can only fall -- and that reasoning is + sound. What it does not survive is the magnitude: + `docs/experiments/2026-09-01-per-product-slippage-restatement.md` measures a median profit + factor of 0.309 falling to 0.219 across 120 cells, and the single cell above 1.0 under flat + pricing dying at 0.626. + + Computed over the product's cached **ONE_DAY** bars, matching `simulate.slippage_assumptions` + exactly -- the ONE definition of the statistic (`screen.median_daily_quote_volume`) over the + granularity it is named for. Deliberately NOT over the rule's own trading granularity: that + median is a PER-BAR figure, and handing an hourly one to a model anchored on a daily volume + reports every asset as maximally thin. `triple_barrier.per_product_round_trip` has to scale + for exactly that reason -- it is a pure rule and cannot fetch daily bars -- and says so. + + `measured=False` means no daily bars, so the flat floor stands as a FALLBACK rather than as + a verdict, and `run_rule_backtest` prints which of the two happened. An absent statistic + must never be presented as a measured one. + """ + daily = repo.get_candles(product_id, Granularity.ONE_DAY) + if not daily: + return backtest_mod.SLIPPAGE_FLOOR_PCT, False + median = screen_mod.median_daily_quote_volume(daily) + return backtest_mod.slippage_for_quote_volume(median), True + + def _backtest_rule( repo: Repository, rule: Any, @@ -239,7 +272,8 @@ def _backtest_rule( `rules backtest`/`rules promote` (and the strategy console's ledger/retry) all share. A rule with no product or no resolvable granularity is a refusal, not a crash.""" _granularity, candles = _resolve_backtest_inputs(repo, rule, granularity_opt, echo_err) - return backtest_mod.backtest(rule, candles, fee_pct=fee_pct) + slippage_pct, _measured = backtest_slippage(repo, rule.product_id) + return backtest_mod.backtest(rule, candles, fee_pct=fee_pct, slippage_pct=slippage_pct) def _load_pbo_for( @@ -333,6 +367,13 @@ class ResolvedBacktest: fee_source: str granularity: Granularity candles: list[Candle] + #: The per-leg slippage its fills are priced at, and whether it was MEASURED from the + #: product's own liquidity or is the flat floor standing in as a fallback (#335). Carried + #: here for the same reason `fee_source` is: an assumption a run does not report is one the + #: reader supplies for themselves, and #259's whole discipline is that the per-product rate + #: is printed beside the numbers it produced. + slippage_pct: Decimal = backtest_mod.SLIPPAGE_FLOOR_PCT + slippage_measured: bool = False def resolve_rule_backtest( @@ -357,6 +398,7 @@ def resolve_rule_backtest( rule = agent._build_rule(row) fee_pct, fee_source = _backtest_fee(config) granularity, candles = _resolve_backtest_inputs(repo, rule, granularity_opt, echo_err) + slippage_pct, slippage_measured = backtest_slippage(repo, rule.product_id) return ResolvedBacktest( row=row, rule=rule, @@ -364,6 +406,8 @@ def resolve_rule_backtest( fee_source=fee_source, granularity=granularity, candles=candles, + slippage_pct=slippage_pct, + slippage_measured=slippage_measured, ) @@ -373,7 +417,12 @@ def backtest_resolved(resolved: ResolvedBacktest) -> backtest_mod.BacktestResult resolution derived, so no front-end ever assembles the engine call itself. Whatever the backtest raises on a poisoned row propagates untouched: the caller renders the failure (the CLI as a crash, the console as its honest per-row error line).""" - return backtest_mod.backtest(resolved.rule, resolved.candles, fee_pct=resolved.fee_pct) + return backtest_mod.backtest( + resolved.rule, + resolved.candles, + fee_pct=resolved.fee_pct, + slippage_pct=resolved.slippage_pct, + ) def run_rule_backtest( diff --git a/tests/commands/test_rules_services.py b/tests/commands/test_rules_services.py index 23252cb..ca31784 100644 --- a/tests/commands/test_rules_services.py +++ b/tests/commands/test_rules_services.py @@ -48,6 +48,7 @@ ) from keel.data.db import connect, migrate from keel.data.repository import Repository +from keel.strategy import backtest as backtest_mod from keel.types import Candle, Granularity NOW_TS = 1_800_000_000 @@ -427,3 +428,148 @@ def test_apply_rule_disable_and_demote_write_through_the_service(repo: Repositor assert outcome.new_status == "disabled" assert repo.get_rules()[0]["status"] == "disabled" assert "status -> disabled" in "\n".join(out) + + +# -- the promotion gate prices fills per product (#335) ------------------------------------------- + + +def _daily(repo, product_id: str, *, quote_volume: float, bars: int = 60) -> None: + """`bars` ONE_DAY candles whose `volume * close` is `quote_volume` — the one statistic + `screen.median_daily_quote_volume` reads, at the granularity it is named for.""" + price = Decimal("100") + repo.upsert_candles( + product_id, + Granularity.ONE_DAY, + [ + Candle( + ts=1_700_000_000 + index * 86_400, + open=price, + high=price, + low=price, + close=price, + volume=Decimal(str(quote_volume)) / price, + ) + for index in range(bars) + ], + ) + + +def test_the_gate_prices_a_liquid_product_near_the_floor(repo) -> None: + """The model's floor is reached at its $500M/day anchor, so 600M/day lands on it.""" + _daily(repo, "BTC-USD", quote_volume=600_000_000.0) + + rate, measured = rules_mod.backtest_slippage(repo, "BTC-USD") + + assert measured is True + assert rate == backtest_mod.SLIPPAGE_FLOOR_PCT + + +def test_the_gate_prices_a_thin_product_far_above_the_floor(repo) -> None: + """**The whole point of #335.** The gate used to price every fill at the floor, and the + floor is not a typical rate — it is the best case the model can produce. Measured over the + real universe not one asset reaches it; TON-USD sits at 36.8x. + """ + _daily(repo, "TON-USD", quote_volume=280_000.0) + + rate, measured = rules_mod.backtest_slippage(repo, "TON-USD") + + assert measured is True + assert rate == backtest_mod.SLIPPAGE_CAP_PCT + assert rate > backtest_mod.SLIPPAGE_FLOOR_PCT * 20 + + +def test_no_daily_bars_falls_back_and_says_so(repo) -> None: + """`measured=False` is the load-bearing half. An absent statistic must never be presented + as a measured verdict — that is the distinction `simulate`'s report already draws, and the + gate has to draw it too or the flat 5bp reads as a finding.""" + rate, measured = rules_mod.backtest_slippage(repo, "NOTHING-USD") + + assert measured is False + assert rate == backtest_mod.SLIPPAGE_FLOOR_PCT + + +def test_the_statistic_comes_from_daily_bars_not_the_rules_own_granularity(repo) -> None: + """The trap this sidesteps, which produces no error. + + `median_daily_quote_volume` returns a PER-BAR median despite its name. Reading it off an + HOURLY series and handing it to a model anchored on a DAILY volume reports every asset as + maximally thin — so a rule that trades hourly would price at the cap regardless of how + liquid its product actually is. Reading ONE_DAY bars, as `simulate.slippage_assumptions` + already does, means no scaling is needed and none can be forgotten. + """ + _daily(repo, "BTC-USD", quote_volume=600_000_000.0) + # An hourly series at 1/24th the daily figure. If the helper read THIS, the rate would be + # far above the floor rather than on it. + price = Decimal("100") + repo.upsert_candles( + "BTC-USD", + Granularity.ONE_HOUR, + [ + Candle( + ts=1_700_000_000 + i * 3600, + open=price, high=price, low=price, close=price, + volume=Decimal("25000000") / price, + ) + for i in range(200) + ], + ) + + rate, _measured = rules_mod.backtest_slippage(repo, "BTC-USD") + + assert rate == backtest_mod.SLIPPAGE_FLOOR_PCT, ( + "the gate is reading a per-bar statistic off the wrong granularity" + ) + + +def _spy_slippage(monkeypatch) -> list[Decimal]: + """Capture the `slippage_pct` every gate backtest actually passes to the engine. + + The helper being right proves nothing about the CALL SITES using it — a mutation removing + `slippage_pct=` from either one left every test above green, which is why this exists. + """ + seen: list[Decimal] = [] + real = backtest_mod.backtest + + def spy(rule, candles, **kwargs): + seen.append(kwargs.get("slippage_pct")) + return real(rule, candles, **kwargs) + + monkeypatch.setattr(rules_mod.backtest_mod, "backtest", spy) + return seen + + +def test_the_gate_backtest_path_passes_the_per_product_rate(repo, monkeypatch) -> None: + _daily(repo, "TON-USD", quote_volume=280_000.0) + rule = agent.build_rule_from_params("turtle_breakout", {"product_id": "TON-USD"}) + seen = _spy_slippage(monkeypatch) + + rules_mod._backtest_rule(repo, rule, "ONE_DAY", Decimal("0.012"), lambda _m: None) + + assert seen == [backtest_mod.SLIPPAGE_CAP_PCT], ( + f"_backtest_rule priced fills at {seen} — it is not passing the per-product rate" + ) + + +def test_the_resolved_backtest_path_passes_the_per_product_rate(repo, monkeypatch) -> None: + """`backtest_resolved` is the seam the strategy console runs, so it must price the same + way the CLI does — two front-ends disagreeing about cost is exactly what #259's + one-definition discipline exists to prevent.""" + _daily(repo, "TON-USD", quote_volume=280_000.0) + # Params in their STORED (JSON-plain) form -- `build_rule_from_params` is the boundary that + # turns these back into `Decimal`s, and writing a row with real Decimals in it would test a + # shape the DB never holds. + rule_id = repo.insert_rule( + "turtle_breakout", + {"product_id": "TON-USD", "granularity": "ONE_DAY"}, + status="candidate", + ) + resolved = rules_mod.resolve_rule_backtest(repo, None, rule_id) + seen = _spy_slippage(monkeypatch) + + rules_mod.backtest_resolved(resolved) + + assert resolved.slippage_pct == backtest_mod.SLIPPAGE_CAP_PCT + assert resolved.slippage_measured is True + assert seen == [backtest_mod.SLIPPAGE_CAP_PCT], ( + f"backtest_resolved priced fills at {seen} — the resolved rate is not reaching the engine" + )