Set the solver defaults to what measured best, and end the drift - #31
Merged
Merged
Conversation
adamjohnwright
force-pushed
the
config/measured-winning-defaults
branch
2 times, most recently
from
September 16, 2026 19:41
83e7d34 to
2c38439
Compare
The code defaults and docker-compose.dev.yml disagreed on four variables. Production sets no solver vars so it ran the code defaults; every benchmark ran through the dev container. specs/002 wrote its winning values into the code on 2026-09-10 and never touched the compose file, which still carried 2026-07-16 values. The tests pin the code defaults, so the suite FAILED in the project's own dev container -- and a failing top-level testset aborts the file, so 141 of 151 assertions never ran. Re-measured on 23,022 wide-curator cases, one catalog build, one variable at a time, each arm's env and catalog mount verified and its raw values checked to differ from the baseline: DS_ASSEMBLY_LIMITING 1 -> 0 -196 DS_AND_MODE hill_log -> hill_sat -90 DS_INHIBITOR_EPS 1e-3 -> 1e-12 +11 Assembly-limiting is the driver, restoring the 2026-07-16 result. Adam's hypothesis that the epsilon drove the gap does not hold: its whole range is worth 11-14 cases against a 248-case gap. Epsilon swept separately. 1e-6 and 1e-12 are BIT-IDENTICAL (same 269 changes, same +109/-98), so at or below 1e-6 it is a pure divide-by-zero guard -- confirming specs/006 FR-002. 1e-4 scores 3 better in 23,022 and is NOT taken: it still changes results, so it is still a model parameter, and picking it optimises a guard against the evaluation set. Defaults now: hill_log, assembly on, eps 1e-12, hill_sat_eps 1e-5 (inert). Compose corrected to match, so the suite passes in the dev container and production computes what the benchmark measures. Recorded honestly: specs/002 R5 was not wrong that hill_sat implements the stated AND intent -- 10x10 reads 99.98 against hill_log's 74.06. It just scores 90 worse. Why the compression helps is unexplained, so this is an empirical default and test_and_curves names its testset accordingly rather than claiming design intent. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
adamjohnwright
force-pushed
the
config/measured-winning-defaults
branch
from
September 16, 2026 19:43
2c38439 to
3480eb4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Depends on #30 (which makes the test suite run to completion); merge that first
or this branch's test changes are partly invisible.
The drift
The code defaults and
docker-compose.dev.ymldisagreed on four variables.docker-compose.prod.ymlsets no solver vars, so production ran the codedefaults while every benchmark ran through the dev container.
specs/002(2026-09-10) wrote its winning values into the code and never touched compose,
which still carried 2026-07-16 values.
The tests pin the code defaults — so the suite failed in the project's own dev
container, and since a failing top-level
@testsetaborts the file, 141 of151 assertions silently never ran.
Attribution — one variable at a time
23,022 wide-curator cases, Release97, one catalog build. Each arm's container
env and catalog mount were verified, and its raw
pred_uichecked to differfrom the baseline — a config change producing bit-identical output has not
applied.
DS_ASSEMBLY_LIMITING1 → 0DS_AND_MODEhill_log→hill_satDS_INHIBITOR_EPS1e-3 → 1e-12Assembly-limiting is the driver, restoring the result already on file from
2026-07-16. The epsilon is not: its whole range is worth 11–14 cases against a
248-case gap.
Epsilon sweep
DS_INHIBITOR_EPS1e-6 and 1e-12 are bit-identical — same 269 changes, same +109/−98. At or
below 1e-6 the epsilon has saturated into a pure divide-by-zero guard, exactly
as
specs/006FR-002 required and nobody had confirmed.1e-4 scores 3 better in 23,022 and is not taken: it still changes results,
so it is still acting as a model parameter, and picking it would optimise a
guard constant against the evaluation set.
Result
Code and compose agree, all four assertion suites pass in the dev container, and
production computes what the benchmark measures.
The tension, recorded not buried
specs/002R5 was not wrong thathill_satimplements the stated ANDintent: 10×10 reads 99.98 against
hill_log's 74.06, and a lone node at UI 50reads 50.00 against 41.43.
hill_logcompresses throughout the operating range,which by that argument is a defect.
It also scores 90 cases better. So the compression is doing useful work that
"correct" multiplication does not, and nobody has explained why — a
plausible candidate is that it damps the multi-hop amplification behind
false-change calls, but that's a hypothesis.
This is therefore an empirical default, not a principled one, and
test_and_curves.jlnames its testset accordingly instead of claiming thedefaults implement the design intent.
Full record:
specs/009-solver-defaults/research.md.🤖 Generated with Claude Code