From af04767f8a416cf9a015c49171c9cab00c28f427 Mon Sep 17 00:00:00 2001 From: Elmehdi Aitbrahim Date: Wed, 2 Sep 2026 05:00:20 -0400 Subject: [PATCH] docs(readme): the cost-distortion finding, rendered from the ledger (#646) #646 asks the README to lead with the fee-drag benchmark. The table half shipped already; this is the finding that came out of measuring it properly. WHAT IT SAYS. Every backtest here used to price fills at the FLOOR of keel's own slippage model -- the best case the model can produce, reached only at a $500M/day anchor -- and 0 of 24 assets reach it. Re-pricing per product moved the median profit factor across 120 cells from 0.309 to 0.219. That 0.090 correction is 2.7x the +0.033 the best strategy change ever bought. WHAT IT DELIBERATELY DOES NOT SAY. Nothing about any other framework. The temptation was a headline framing crypto backtesting generally as living on fantasy friction assumptions; we have not measured anyone else's defaults, and an unsourced claim about a named third party is the exact failure this section exists to avoid making about ourselves. #646 forbids competitor naming in the asset on trademark grounds; the stronger reason is that the finding needs no help. A test asserts six framework names are absent and that the scope sentence is present. THE HEADLINE IS NARROWER THAN THE ONE PROPOSED. "Strategy optimization is irrelevant under flat slippage assumptions" generalises from one comparison on one universe at one venue. What is written instead -- "we mis-priced our own execution by more than any strategy change we ever made" -- is checkable line by line, and is the more damaging claim precisely because it is about us. The 2.7x is DERIVED at render time from two separate hash-chained ledger rows, not typed. A literal would survive either number changing underneath it, which is exactly the trap a headline ratio sets. Two rendering bugs the tests caught, both invisible in the source and obvious on the page: `textwrap` broke the source link mid-URL so it rendered as literal text, and a blank line after every paragraph turned one four-row table into four one-row tables. Tests: `tests/test_fee_reality_block.py` (+6). 6 mutants, 6 killed -- a hardcoded ratio, a dropped scope disclaimer, a named competitor, a hand-edited README number, a wrapped link, and a split table. STILL OPEN on #646: the terminal capture. The issue also asks for a ~10-second GIF of a real run with committed generation tooling; that is unbuilt and the issue stays open for it. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01NzuKAe2RVrPt9acVAWjRyL --- README.md | 37 +++++++++ scripts/render_fee_reality.py | 129 ++++++++++++++++++++++++++++++-- tests/test_fee_reality_block.py | 76 +++++++++++++++++++ 3 files changed, 234 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 170b4fe..88e9d7a 100644 --- a/README.md +++ b/README.md @@ -55,6 +55,43 @@ Source: [`docs/experiments/2026-08-12-fee-curve-and-rsi-meanrev.md`](docs/experi + + +### We mis-priced our own execution by more than any strategy change we ever made + +Every backtest in this repository used to price fills at the **floor** of its own slippage model +— the best case the model can produce, reached only at a $500M/day anchor. Measured against the +24 assets keel actually holds candles for, **0 of 24 reach it**: the range runs from 1.1× the +floor at the most liquid to 36.8× at the thinnest. + +Re-pricing every rule per product — 5 rules, 24 assets, 240 trials — moved the median profit +factor across 120 cells from **0.309** to **0.219**. The one cell that had cleared 1.0 no longer +does. + +| | profit factor | +| :-- | --: | +| the cost-model correction | **-0.090** | +| the best strategy improvement we have measured | **+0.033** | + +The error in how we priced execution was **2.7× larger** than the largest genuine gain any +change to the strategy itself produced — a better exit, measured against the same entry on the +same universe. For as long as that was true, comparing strategies meant comparing them through a +lens distorted by more than the differences being compared. + +**What this is and is not.** It is keel mis-pricing keel, on one venue, over one universe of +cached candles — found, corrected and published. It is not a claim about any other framework's +assumptions: we have not measured those, and an unsourced claim about someone else would be the +exact failure this section exists to avoid making about ourselves. + +**Where the friction actually is.** Two taker legs cost 2.4% of notional before a basis point of +slippage is counted, so on the most liquid asset the fee is the overwhelming majority of the +round trip and only at the thin end does slippage overtake it. A cheaper venue matters more than +a better model of the book — and an honest model of the book is what tells you that. + +Source: [`docs/experiments/2026-09-01-per-product-slippage-restatement.md`](docs/experiments/2026-09-01-per-product-slippage-restatement.md), rendered from the hash-chained trials ledger by `scripts/render_fee_reality.py`. + + + **The cadence problem, and the pipeline built to solve it:** the promotion gate's 100-trade floor is honest only if the sample is collectable — and at the daily clock's measured 2.15 signals per asset-year, it is 31–84 years away per asset. Waiting is not a slower diff --git a/scripts/render_fee_reality.py b/scripts/render_fee_reality.py index 5599af1..992c05c 100644 --- a/scripts/render_fee_reality.py +++ b/scripts/render_fee_reality.py @@ -27,6 +27,7 @@ import json import re import sys +import textwrap from dataclasses import dataclass from decimal import Decimal from pathlib import Path @@ -41,6 +42,16 @@ BEGIN = "" END = "" +COST_BEGIN = ( + "" +) +COST_END = "" + +#: The two ledger rows the cost-distortion block reads. Both are hash-chained, so the block +#: cannot quote a number that is not in the record of what was run. +_RESTATEMENT = "per-product-slippage-restatement-2026-09-01" +_EXIT_AB = "triple-barrier-first-measurement-2026-09-01" + #: The taker rate this account actually pays, as the fee curve's own last column. The ledger #: writes `fee_pct` as a fraction of notional; `_as_percent` is what turns it into the number an #: operator recognises from a fee schedule. @@ -176,28 +187,130 @@ def render(ledger_text: str) -> str: return "\n".join(lines) -def replace_block(readme: str, block: str) -> str: +def _summary(ledger_text: str, trial_id: str) -> dict: + """One trial's `summary`, or a loud failure. The ledger is append-only and hash-chained, so + a missing row means this renderer's expectation is stale -- never that the record is wrong.""" + for line in ledger_text.splitlines(): + if not line.strip(): + continue + row = json.loads(line) + if row.get("trial_id") == trial_id: + return row.get("summary") or {} + raise SystemExit(f"no ledger row {trial_id!r} -- the renderer's expectation is stale") + + +def render_cost_distortion(ledger_text: str) -> str: + """The block that states what keel's own cost model was worth getting wrong. + + Deliberately says nothing about any other framework. What was measured is keel mis-pricing + keel, on one universe at one venue; a claim about anyone else's assumptions would be exactly + the unsourced assertion this project refuses to make about its own results, and #646 forbids + naming a competitor in the asset regardless. + """ + restated = _summary(ledger_text, _RESTATEMENT) + exit_ab = _summary(ledger_text, _EXIT_AB) + + correction = abs(Decimal(restated["pf_median_delta"])) + improvement = Decimal(exit_ab["delta_vs_control_median_zero_fee"]) + ratio = (correction / improvement).quantize(Decimal("0.1")) + + # Paragraphs assembled as WHOLE strings, then wrapped. The first draft built them line by + # line and every interpolated number landed mid-sentence at a line break -- fine in the + # source, awkward in the rendered page, which is the only place it is read. + doc = "docs/experiments/2026-09-01-per-product-slippage-restatement.md" + paragraphs = [ + "### We mis-priced our own execution by more than any strategy change we ever made", + ( + "Every backtest in this repository used to price fills at the **floor** of its own " + "slippage model \u2014 the best case the model can produce, reached only at a " + f"$500M/day anchor. Measured against the {restated['assets']} assets keel actually " + f"holds candles for, **{restated['assets_at_the_floor']} of {restated['assets']} " + f"reach it**: the range runs from {restated['floor_multiple_min']}\u00d7 the floor " + f"at the most liquid to {restated['floor_multiple_max']}\u00d7 at the thinnest." + ), + ( + f"Re-pricing every rule per product \u2014 {restated['rules']} rules, " + f"{restated['assets']} assets, {restated['n_trials']} trials \u2014 moved the " + f"median profit factor across {restated['cells']} cells from " + f"**{restated['pf_median_flat']}** to **{restated['pf_median_per_product']}**. The " + "one cell that had cleared 1.0 no longer does." + ), + "| | profit factor |", + "| :-- | --: |", + f"| the cost-model correction | **{restated['pf_median_delta']}** |", + f"| the best strategy improvement we have measured | **+{improvement}** |", + ( + f"The error in how we priced execution was **{ratio}\u00d7 larger** than the " + "largest genuine gain any change to the strategy itself produced \u2014 a better " + "exit, measured against the same entry on the same universe. For as long as that " + "was true, comparing strategies meant comparing them through a lens distorted by " + "more than the differences being compared." + ), + ( + "**What this is and is not.** It is keel mis-pricing keel, on one venue, over one " + "universe of cached candles \u2014 found, corrected and published. It is not a " + "claim about any other framework's assumptions: we have not measured those, and an " + "unsourced claim about someone else would be the exact failure this section exists " + "to avoid making about ourselves." + ), + ( + "**Where the friction actually is.** Two taker legs cost " + f"{_as_percent(str(2 * Decimal(restated['fee_pct'])))} of notional before a basis " + "point of slippage is counted, so on the most liquid asset the fee is the " + "overwhelming majority of the round trip and only at the thin end does slippage " + "overtake it. A cheaper venue matters more than a better model of the book \u2014 " + "and an honest model of the book is what tells you that." + ), + ( + f"Source: [`{doc}`]({doc}), rendered from the hash-chained trials ledger by " + "`scripts/render_fee_reality.py`." + ), + ] + body: list[str] = [] + for index, para in enumerate(paragraphs): + # A table row is emitted verbatim and gets no blank line after it unless the NEXT + # paragraph is not a row -- a blank between rows ends the table, which is how the first + # render produced four one-row tables instead of one four-row one. + if para.startswith("|"): + body.append(para) + following = paragraphs[index + 1] if index + 1 < len(paragraphs) else "" + if not following.startswith("|"): + body.append("") + continue + # A line carrying a markdown link is never wrapped: `textwrap` will happily break inside + # the URL, and a broken link renders as literal text. + body.extend([para] if "](" in para else textwrap.wrap(para, width=96)) + body.append("") + lines = [COST_BEGIN, "", *body, COST_END] + return "\n".join(lines) + + +def replace_block(readme: str, block: str, *, begin: str = BEGIN, end_marker: str = END) -> str: """`readme` with the sentinel block swapped for `block`.""" - start, end = readme.find(BEGIN), readme.find(END) + start, end = readme.find(begin), readme.find(end_marker) if start == -1 or end == -1 or end < start: raise SystemExit( - f"README.md is missing the {BEGIN!r} / {END!r} sentinels -- refusing to guess where " + f"README.md is missing the {begin!r} / {end_marker!r} sentinels -- refusing to guess " "the benchmark belongs" ) - return readme[:start] + block + readme[end + len(END) :] + return readme[:start] + block + readme[end + len(end_marker) :] def main(argv: list[str] | None = None) -> int: parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("--write", action="store_true", help="rewrite README.md in place") args = parser.parse_args(argv) - block = render(_LEDGER.read_text(encoding="utf-8")) + ledger = _LEDGER.read_text(encoding="utf-8") + block = render(ledger) + cost = render_cost_distortion(ledger) if not args.write: print(block) + print() + print(cost) return 0 - _README.write_text( - replace_block(_README.read_text(encoding="utf-8"), block), encoding="utf-8" - ) + text = replace_block(_README.read_text(encoding="utf-8"), block) + text = replace_block(text, cost, begin=COST_BEGIN, end_marker=COST_END) + _README.write_text(text, encoding="utf-8") print(f"wrote the fee-reality block into {_README}") return 0 diff --git a/tests/test_fee_reality_block.py b/tests/test_fee_reality_block.py index 55f89be..890849b 100644 --- a/tests/test_fee_reality_block.py +++ b/tests/test_fee_reality_block.py @@ -15,6 +15,7 @@ import json import sys +from decimal import Decimal from pathlib import Path import pytest @@ -134,3 +135,78 @@ def test_the_renderer_refuses_a_readme_without_sentinels() -> None: """It will not guess where the benchmark belongs.""" with pytest.raises(SystemExit, match="sentinels"): rfr.replace_block("# keel\n\nnothing here\n", "block") + + +# -- the cost-distortion block (#646) ------------------------------------------------------------ + + +def test_the_cost_distortion_block_matches_what_the_ledger_renders() -> None: + """Same drift pin as the benchmark above, on the section that makes the claim. + + This block asserts that our own cost-model error exceeded our best strategy gain. That is a + statement about two measured numbers, and it is only worth making while both come out of the + record rather than out of someone's memory of them. + """ + assert rfr.render_cost_distortion(_LEDGER_TEXT) in _README, ( + "README.md's cost-distortion block is not what the renderer produces from the ledger. " + "Regenerate it rather than editing the numbers." + ) + + +def test_the_ratio_is_computed_not_typed() -> None: + """`2.7x` is a DERIVED figure — the correction divided by the improvement — and both sides + come from separate hash-chained ledger rows. A literal would survive either number changing + underneath it, which is exactly the failure a headline ratio invites.""" + restated = rfr._summary(_LEDGER_TEXT, rfr._RESTATEMENT) + exit_ab = rfr._summary(_LEDGER_TEXT, rfr._EXIT_AB) + correction = abs(Decimal(restated["pf_median_delta"])) + improvement = Decimal(exit_ab["delta_vs_control_median_zero_fee"]) + expected = (correction / improvement).quantize(Decimal("0.1")) + + assert f"**{expected}× larger**" in rfr.render_cost_distortion(_LEDGER_TEXT) + assert expected > 1, "the claim only holds while the correction exceeds the improvement" + + +def test_the_block_names_no_other_framework() -> None: + """#646 in terms: 'No competitor naming in the asset. Generic fee-drag math — never what + another product costs you.' + + And beyond the trademark posture: we have not measured anyone else's assumptions. An + unsourced claim about a named third party is the exact failure this section exists to avoid + making about ourselves, and it would cost more credibility than it could ever buy. + """ + block = rfr.render_cost_distortion(_LEDGER_TEXT).lower() + for name in ("jesse", "freqtrade", "hummingbot", "backtrader", "quantconnect", "zipline"): + assert name not in block, f"the block names {name!r}" + assert "not a claim about any other framework" in block + + +def test_the_block_scopes_its_own_claim() -> None: + """One venue, one universe, our own numbers. A finding stated wider than it was measured is + the thing this repository refuses to publish, and a headline ratio is exactly where that + temptation lands.""" + block = rfr.render_cost_distortion(_LEDGER_TEXT) + assert "keel mis-pricing keel" in block + assert "one venue" in block + + +def test_the_source_link_is_not_broken_by_wrapping() -> None: + """`textwrap` will happily break inside a URL, and a broken markdown link renders as literal + text — which the first render did. Every link in the block must survive on one line.""" + for line in rfr.render_cost_distortion(_LEDGER_TEXT).splitlines(): + if "](" in line: + assert line.count("](") == line.count(")"), f"link broken across lines: {line!r}" + assert ".md)" in line, f"link truncated: {line!r}" + + +def test_the_table_is_one_table() -> None: + """A blank line between rows ENDS a markdown table. The first render put one after every + paragraph uniformly and produced four one-row tables.""" + rows = [ + line + for line in rfr.render_cost_distortion(_LEDGER_TEXT).splitlines() + if line.startswith("|") + ] + assert len(rows) == 4 + block = rfr.render_cost_distortion(_LEDGER_TEXT) + assert "\n\n|" not in block.split("| :--", 1)[1], "a blank line splits the table"