diff --git a/docs/experiments/2026-09-01-cusum-event-first-measurement.md b/docs/experiments/2026-09-01-cusum-event-first-measurement.md new file mode 100644 index 00000000..16ebdca7 --- /dev/null +++ b/docs/experiments/2026-09-01-cusum-event-first-measurement.md @@ -0,0 +1,120 @@ +# CUSUM event gating, first measurement — feasible, and without a gross edge + +**Date:** 2026-09-01 · **Issue:** [#341](https://github.com/CodeGateSoftware/keel/issues/341) · +**Rule:** `cusum_event` · **Driver:** `2026-09-01-cusum-event-first-measurement.py` · +**Ledger row:** `cusum-event-first-measurement-2026-09-01` (168 trials disclosed) + +## Declared before the run + +* **Primary metric: `n_trades`.** An event filter trades less by construction, and the ρ=−0.77 + bind between edge and sample size means a rule that fires rarely cannot be admitted whatever + its profit factor. Declared primary exactly as the `rsi_meanrev` diagnostic grid declared it. +* **Secondary: profit factor at fee 0 / 0.006 / 0.012.** Zero bounds from above everything an + execution fix could ever buy; 0.006 is the maker rate this account cannot reach; 0.012 is what + it pays. +* **Arm A is ONE configuration** — the shipped default. One config means no argmax, so nothing + in arm A is a maximum-of-N. +* **Arm B sweeps the rule's own headline knob**, `threshold_friction_mult` ∈ {1,2,3,4} at the + taker rate. Disclosed as a sweep; its per-asset best is a maximum of four draws. + +24 assets, `ONE_HOUR`, ~5 years of cached candles — the same universe as the restated +intersection, so this sits beside the null it is compared against rather than beside a different +population. + +## 1. Feasibility: YES, and it contradicts the worry filed with the issue + +#341 was filed with the concern that *"gating cuts n on rules already below the admission +floor"*. **It does not.** + +| | | +| :-- | --: | +| assets with n ≥ 100 (the admission floor) | **21 of 24** | +| median n | **553** | +| range | 32 (PAXG-USD) – 884 (FET-USD) | + +A 5% cumulative move in an hourly crypto series is a common event, not a rare one. The +feasibility objection to this rule is answered and should not be repeated. The three assets below +the floor are PAXG-USD (32), PAXG-USDT (51) and TON-USD (69) — gold and the thinnest name in the +universe, which is the expected shape. + +## 2. Profitability: no, and not marginally + +Arm A, all 24 assets: + +| fee | PF median | PF max | above 1.0 | +| :-- | --: | --: | --: | +| 0% | 0.925 | 1.238 | 8 / 24 | +| 0.6% (maker, unreachable) | 0.553 | 0.750 | **0 / 24** | +| 1.2% (taker, actual) | 0.343 | 0.479 | **0 / 24** | + +Best cell at the rate actually paid: AVAX-USD, PF 0.479, n=601. + +**The intersection of n ≥ 100 and PF > 1.0 is empty across all 168 trials, in both arms.** + +## 3. The diagnostic: this is the `rsi_meanrev` disease, not the `turtle_breakout` one + +The number that matters is the zero-fee column. **At zero cost only 8 of 24 clear 1.0, and the +median is 0.925.** + +`turtle_breakout` at zero fee was profitable on 4 of 4 with a maximum of 2.713 — a real gross +edge that cost destroyed, which is why cheaper execution was a coherent thing to want. This rule +has **essentially no gross edge for cost to destroy**: it loses money at a median before a single +fee is charged. + +Same symptom as turtle at 1.2%, different disease — and the one that cannot be fixed by +execution, because zero fee bounds from above everything an execution improvement could ever buy +and zero fee is already a median loss. + +## 4. The knob works exactly as designed, and converges to nothing + +Arm B, `threshold_friction_mult` at the taker rate: + +| mult | threshold | median n | PF median | PF max | above 1.0 | n ≥ 100 | +| --: | --: | --: | --: | --: | --: | --: | +| 1 | 2.5% | 1252 | 0.226 | 0.388 | 0 | 23 | +| 2 | 5.0% | 553 | 0.343 | 0.479 | 0 | 21 | +| 3 | 7.5% | 330 | 0.361 | 0.533 | 0 | 21 | +| 4 | 10.0% | 224 | 0.379 | 0.593 | 0 | 20 | + +Raising the multiple monotonically raises median PF and cuts median n. **The mechanism is real** +— trading less does lose less per toll. It is also converging to a ceiling well below 1.0 while n +falls toward the admission floor, so there is no multiple at which both conditions hold. A +fourfold increase in the threshold buys 0.15 of profit factor and costs 82% of the sample. + +## 5. The source's own setting is the worst cell + +`mult=1` is the paper's 2.0–2.5% threshold, and on this venue it is *exactly one round trip*. +It is the worst arm-B cell on every axis: the lowest median PF (0.226) with the highest median n +(1252) — **the most trading at the least edge.** + +That is #341's friction-scaling argument measured rather than asserted. A threshold spelled as a +percentage looks conservative; the same threshold spelled as `threshold_friction_mult=1` says +what it is. + +## Honesty + +**Selection bias.** Arm A is one pre-declared configuration and carries none. Arm B's per-asset +best is a maximum of four draws and must never be quoted as an edge estimate — the best cell +overall is CRV-USD at mult=4, PF 0.593, n=322, still 0.4 short of 1.0. The bias does not change +the verdict in either direction. + +**Validation.** Screening result only. No walk-forward, no out-of-sample split, no CSCV/PBO and +no deflated Sharpe: `backtest` emits aggregates and no per-trade series (`series_missing`). Same +cached candles and the same ~5-year window as every other document here. `slippage_pct` held at +0.0005 in every cell, so the zero column is zero *fee*, not zero cost — round-trip friction there +is still 0.10% of notional. + +**Changed nothing.** A document, a driver script and a ledger row. No rule row was added, nothing +was promoted, no config was touched, no allowlist changed. `cusum_event` remains registered and +untraded. + +## Recommended next + +1. **Not further `cusum_event` tuning.** It is negative at zero cost on 16 of 24 assets, and its + own headline knob moves median PF by 0.15 across a fourfold range. A different event + definition is a new rule with a new pre-registration, not a continuation of this one. +2. **#342 (triple-barrier exits) is now the only untested half of the source's claim** — but its + prior tightened here rather than loosening: the entry half has no gross edge for a better exit + to harvest. +3. **The null grows rather than breaks.** The measured intersection goes from 0 of 90 to **0 of + 114**. That is the result, and publishing it is the point. diff --git a/docs/experiments/2026-09-01-cusum-event-first-measurement.py b/docs/experiments/2026-09-01-cusum-event-first-measurement.py new file mode 100644 index 00000000..588dace7 --- /dev/null +++ b/docs/experiments/2026-09-01-cusum-event-first-measurement.py @@ -0,0 +1,148 @@ +"""First measurement of `cusum_event` (#341). Pre-declared, then run. + +WHAT IS DECLARED BEFORE THE RUN, so the reader can hold this document to it: + +* **Primary metric: `n_trades`.** The question this rule's mechanism actually raises is + FEASIBILITY -- an event filter trades less by construction, and the ρ=-0.77 bind between edge + and sample size (2026-08-12-fee-curve-and-rsi-meanrev.md) says a rule that fires rarely cannot + be admitted no matter what its profit factor looks like. Declared primary exactly as the + rsi_meanrev diagnostic grid declared it, and for the same reason. +* **Secondary: profit factor at fee 0 / 0.006 / 0.012.** Zero bounds from above everything an + execution fix could ever buy; 0.006 is the maker rate this account cannot currently reach; + 0.012 is what it pays. +* **Arm A is ONE configuration -- the shipped default.** `threshold_friction_mult=2`, + `lookback=168`, `atr_period=20`, `atr_stop_mult=2`, `target_rr=3`. One config means no argmax, + so nothing in arm A is a maximum-of-N and the selection-bias caveat that governs the fee-curve + document does not apply to it. +* **Arm B is the rule's own headline knob**, `threshold_friction_mult` over {1, 2, 3, 4} at the + taker fee only. Disclosed as a SWEEP: 4 x 24 = 96 additional trials. Its per-asset best is a + maximum of four draws and must never be quoted as an edge estimate. + +`1` in arm B is the source's own setting and is exactly one round trip on this venue -- included +so the paper's configuration is measured rather than argued about. + +Trials disclosed: 24 x 3 = 72 (arm A) + 96 (arm B) = 168. + +Re-run: + KEEL_EXPERIMENT_DB=~/keel/keel.db \ + python docs/experiments/2026-09-01-cusum-event-first-measurement.py +""" + +import json +import os +import time +from concurrent.futures import ProcessPoolExecutor, as_completed +from decimal import Decimal +from pathlib import Path + +DB = os.environ.get("KEEL_EXPERIMENT_DB") or str(Path.home() / "keel" / "keel.db") +OUT_DIR = os.environ.get("KEEL_EXPERIMENT_OUT") or str(Path(__file__).resolve().parent / "_out") +Path(OUT_DIR).mkdir(parents=True, exist_ok=True) +JSONL_PATH = f"{OUT_DIR}/cusum_first.jsonl" + +#: The same 24-asset universe the restated intersection used, so this result sits directly +#: beside the null it is being compared against rather than beside a different population. +UNIVERSE = [ + "BTC-USD", "ETH-USD", "ADA-USD", "LINK-USD", "LTC-USD", "SOL-USD", + "XLM-USD", "PAXG-USDT", "BCH-USD", "AAVE-USD", "DOGE-USD", "DOT-USD", + "UNI-USD", "ZEC-USD", "ALGO-USD", "FET-USD", "CRV-USD", "ICP-USD", + "AVAX-USD", "NEAR-USD", "XRP-USD", "PAXG-USD", "WLD-USD", "TON-USD", +] + +FEES = ["0", "0.006", "0.012"] +SLIPPAGE = Decimal("0.0005") +TAKER = "0.012" +MULTIPLES = ["1", "2", "3", "4"] + +#: The admission floors this is measured against (#337/#338). +MIN_TRADES = 100 + + +def build_jobs(): + return [("A", asset, "2") for asset in UNIVERSE] + [ + ("B", asset, mult) for asset in UNIVERSE for mult in MULTIPLES + ] + + +def run_job(job): + arm, asset, mult = job + from keel_core.types import Granularity + + from keel.data.db import connect + from keel.data.repository import Repository + from keel.strategy import backtest as bt + from keel.strategy.rules.cusum_event import CusumEvent + + fees = FEES if arm == "A" else [TAKER] + rows = [] + try: + repo = Repository(connect(DB)) + candles = repo.get_candles(asset, Granularity.ONE_HOUR) + except Exception as exc: + return [{ + "arm": arm, "product": asset, "mult": mult, + "error": f"{type(exc).__name__}: {exc}", + }] + if not candles: + return [{"arm": arm, "product": asset, "mult": mult, "error": "no hourly candles"}] + + for fee in fees: + try: + rule = CusumEvent(product_id=asset, threshold_friction_mult=Decimal(mult)) + result = bt.backtest( + rule, candles, fee_pct=Decimal(fee), slippage_pct=SLIPPAGE + ) + rows.append({ + "arm": arm, + "product": asset, + "mult": mult, + "fee": fee, + "bars": len(candles), + "n_trades": int(result.n_trades), + "win_rate": float(result.win_rate), + "profit_factor": float(result.profit_factor), + "expectancy": float(result.expectancy), + "max_drawdown": float(result.max_drawdown), + }) + except Exception as exc: + rows.append({ + "arm": arm, "product": asset, "mult": mult, "fee": fee, + "error": f"{type(exc).__name__}: {exc}", + }) + return rows + + +def done_combos(): + """Combos already written. Append-only and each row names its own combo, so a killed run + resumes rather than redoing -- the same discipline as the restated-intersection driver.""" + if not os.path.exists(JSONL_PATH): + return set() + done = set() + for line in open(JSONL_PATH): + line = line.strip() + if not line: + continue + try: + row = json.loads(line) + except json.JSONDecodeError: + continue # torn final line from a kill mid-write + done.add((row["arm"], row["product"], row["mult"])) + return done + + +def main(): + jobs = [job for job in build_jobs() if (job[0], job[1], job[2]) not in done_combos()] + print(f"{len(jobs)} combos to run -> {JSONL_PATH}", flush=True) + started = time.time() + with open(JSONL_PATH, "a") as sink, ProcessPoolExecutor() as pool: + futures = {pool.submit(run_job, job): job for job in jobs} + for index, future in enumerate(as_completed(futures), start=1): + for row in future.result(): + sink.write(json.dumps(row) + "\n") + sink.flush() + print(f" {index}/{len(jobs)} {futures[future]}", flush=True) + print(f"done in {time.time() - started:.0f}s", flush=True) + + +if __name__ == "__main__": + main() diff --git a/docs/experiments/trials-ledger.jsonl b/docs/experiments/trials-ledger.jsonl index 5d2164f4..ab31b920 100644 --- a/docs/experiments/trials-ledger.jsonl +++ b/docs/experiments/trials-ledger.jsonl @@ -88,3 +88,4 @@ {"decision":"diagnostic_only","kind":"sweep_node","params":{"adx_threshold":34.50297123480382,"atr_stop_mult":1.542221195540835,"entry_lookback":44,"exit_lookback":26,"granularity":"ONE_HOUR","product_id":"SOL-USD","target_rr":8},"per_bar_pnl":[],"per_trade_pnl":["-1.986873980","-1.837373500","-4.321120061606460273032444071","-4.692985744060521518272929326","-3.957417229438210111262713025","-3.251904827831263737640744301","4.473943008973596527865491033","-2.578821276219497752091531962","-1.538748440","-2.944795471274583355038043235","-3.222443193774710680565678065","-2.476095694302268687404235926","-1.533247840","4.324402054460950302977910624"],"prev_hash":"8433009c6d8b0ee9fbda29a6fd5d696016b034267d43bf6edc539b107e779518","provenance":"fitted","row_hash":"35f1e28f7523d0ac5c56b9d137bac59419693647e36c2643a3f87aec7c702d00","rule":"turtle_breakout","series_missing":false,"session":"optuna-parameter-study-2026-08-22","summary":{"bars":17520,"fee_pct":"0.012","gate_passed":0,"held_out_expectancy":"-1.824534442505212091747494161","n_trials":60,"pbo":"0.7","pbo_available":1,"seed":476,"slippage_pct":"0.0005","train_expectancy":"-1.337906173823760233469001437"},"timestamp":1787368446,"trial_id":"476-optuna-turtle_breakout"} {"decision":"diagnostic_only","kind":"sweep_node","params":{"atr_mult":2.4959403868013204,"fixed_rr":1,"overbought":80.7242988859836,"oversold":15.722910550982721,"product_id":"ETH-USD","rsi_period":18},"per_bar_pnl":[],"per_trade_pnl":[],"prev_hash":"35f1e28f7523d0ac5c56b9d137bac59419693647e36c2643a3f87aec7c702d00","provenance":"fitted","row_hash":"787c3934700c34645fc77e137f9df8927031cc07f146ec82f476322203e0f24a","rule":"rsi_meanrev","series_missing":true,"session":"optuna-parameter-study-2026-08-22","summary":{"bars":17520,"fee_pct":"0.012","gate_passed":0,"held_out_expectancy":"0","n_trials":60,"pbo":"0.5714285714285714285714285714","pbo_available":1,"seed":476,"slippage_pct":"0.0005","train_expectancy":"220.9799098276924967971328305"},"timestamp":1787368446,"trial_id":"476-optuna-rsi_meanrev"} {"decision":"diagnostic_only","kind":"sweep_node","params":{"buffer_ticks":0.049704400803443,"ema_periods":[11,17,40],"product_id":"SOL-USD"},"per_bar_pnl":[],"per_trade_pnl":["-4.0173355540198047831580","-3.5027104940198047831580","-2.9839631340198047831580","-2.5405856740198047831580","-2.6313359140198047831580","-2.4288351940198047831580","-2.7519619340198047831580","-2.5525855540198047831580","-2.7715863940198047831580","-2.3049602540198047831580","-2.3478355540198047831580","-1.3243723579405856505260","-2.0557094140198047831580","-2.0342092940198047831580","-2.1982098940198047831580"],"prev_hash":"787c3934700c34645fc77e137f9df8927031cc07f146ec82f476322203e0f24a","provenance":"fitted","row_hash":"2a8c802b036024d6913519e2d842d7f036e0572e34cb63c3fb374bc1f243296c","rule":"pullback_continuation","series_missing":false,"session":"optuna-parameter-study-2026-08-22","summary":{"bars":17520,"fee_pct":"0.012","gate_passed":0,"held_out_expectancy":"-2.563079774281190174315866667","n_trials":60,"pbo":"0.8428571428571428571428571429","pbo_available":1,"seed":476,"slippage_pct":"0.0005","train_expectancy":"-4.7947464365621181060780"},"timestamp":1787368446,"trial_id":"476-optuna-pullback_continuation"} +{"decision":"rejected","kind":"ablation","params":{"arms":"A: shipped default (threshold_friction_mult=2 -> 5.0% threshold, lookback=168, atr_period=20, atr_stop_mult=2, target_rr=3) x 24 assets x 3 fees = 72 trials. B: threshold_friction_mult in {1,2,3,4} at the taker fee x 24 assets = 96 trials. 168 trials disclosed.","changed_nothing":"A document and one driver script. No rule row was added, nothing was promoted, no config was touched, no allowlist changed. cusum_event remains registered and untraded.","declared_before_the_run":"PRIMARY METRIC n_trades (feasibility -- an event filter trades less by construction, and the rho=-0.77 edge-vs-sample bind says a rule that fires rarely cannot be admitted whatever its profit factor). Secondary: profit factor at fee 0/0.006/0.012. Arm A is ONE configuration, the shipped default, so nothing in it is an argmax. Arm B sweeps the rule's own headline knob and its per-asset best is a maximum of four draws.","document":"docs/experiments/2026-09-01-cusum-event-first-measurement.md","feasibility_answer":"YES, and it CONTRADICTS the pre-registered worry. The concern filed with #341 was that gating cuts n on rules already below the admission floor. It does not: at the shipped default n>=100 on 21 of 24 assets, median 553, max 884. Event sampling on hourly bars fires often because a 5% cumulative move in an hourly crypto series is a common occurrence, not a rare one. The feasibility objection to this rule is answered and should not be repeated.","issue":"#341","profitability_answer":"NO, and not marginally. At the 1.2% taker rate ZERO of 24 assets clear PF 1.0 -- median 0.343, best 0.479 (AVAX-USD, n=601). At the 0.6% maker rate this account cannot reach, still zero of 24, median 0.553. The intersection of n>=100 and PF>1.0 is EMPTY across both arms, all 168 trials.","recommended_next":"(a) NOT further cusum_event tuning: it is negative at zero cost on 16 of 24 assets and its own headline knob moves median PF by 0.15 across a fourfold range. (b) #342's triple-barrier exits are the source's other half and are now the only untested part of that paper's claim -- but the honest prior tightened here, since the entry half has no gross edge for a better exit to harvest. (c) The result stands as the fifth family in the null: 0 of 90 becomes 0 of 114.","script":"docs/experiments/2026-09-01-cusum-event-first-measurement.py","selection_bias":"Arm A is ONE pre-declared configuration and carries none. Arm B's per-asset best is a maximum of FOUR draws and must never be quoted as an edge estimate; its best cell overall is CRV-USD at mult=4, PF 0.593, n=322 -- still 0.4 short of 1.0, so the bias does not change the verdict in either direction.","the_diagnostic":"AT ZERO FEE ONLY 8 OF 24 CLEAR 1.0, median 0.925, max 1.238. That is the finding. turtle_breakout at zero fee was profitable on 4 of 4 with a max of 2.713 -- a real gross edge that cost destroyed. cusum_event has ESSENTIALLY NO GROSS EDGE for cost to destroy: its median is below 1.0 at zero cost. Same symptom as turtle at 1.2%, the rsi_meanrev disease. Cheaper execution cannot rescue it, because zero fee bounds from above everything an execution fix could ever buy and zero fee is already a median loss.","the_knob_works_and_does_not_rescue_it":"threshold_friction_mult behaves exactly as designed and converges to nothing. 1 -> 4 raises median PF monotonically 0.226/0.343/0.361/0.379 while cutting median n 1252/553/330/224. Trading less does lose less per toll -- the mechanism is real -- but the trend is toward a ceiling below 1.0, and n falls toward the admission floor as it climbs. There is no multiple at which both hold.","the_sources_own_setting_is_the_worst_cell":"mult=1 is the paper's 2.0-2.5% threshold and is exactly one round trip here. It is the worst arm-B cell on every axis: median PF 0.226, the lowest of the four, with median n 1252 -- the most trading at the least edge. This is the friction-scaling argument of #341 measured rather than asserted.","universe":"the same 24 assets as the restated intersection, ONE_HOUR, ~5y cached","validation":"Screening result only. No walk-forward, no out-of-sample split, no CSCV/PBO and no deflated Sharpe: backtest emits aggregates and no per-trade series (series_missing). Same cached candles and same ~5-year window as every other document here. slippage_pct held at 0.0005 in every cell, so 'zero fee' is zero FEE and not zero cost."},"per_bar_pnl":[],"per_trade_pnl":[],"prev_hash":"2a8c802b036024d6913519e2d842d7f036e0572e34cb63c3fb374bc1f243296c","provenance":"a_priori","row_hash":"ef473b236321c2f081054fb2012860a0e34f5f195a02518eb807a88978c8b6af","rule":"cusum_event","series_missing":true,"session":"cusum-event-first-measurement-2026-09-01","summary":{"arm_a_assets":24,"arm_a_n_above_floor":21,"arm_a_n_median":553,"arm_a_pf_above_one_taker":0,"arm_a_pf_above_one_zero_fee":8,"arm_a_pf_max_taker":"0.479","arm_a_pf_median_taker":"0.343","arm_a_pf_median_zero_fee":"0.925","fee_pct":"0.012","gate_passed":0,"intersection_n100_and_pf1":0,"min_trades_floor":100,"n_trials":168,"pbo_available":0,"slippage_pct":"0.0005"},"timestamp":1788305412,"trial_id":"cusum-event-first-measurement-2026-09-01"}