Skip to content

Set the solver defaults to what measured best, and end the drift - #31

Merged
adamjohnwright merged 1 commit into
mainfrom
config/measured-winning-defaults
Sep 16, 2026
Merged

adamjohnwright merged 1 commit into
mainfrom
config/measured-winning-defaults

Conversation

@adamjohnwright

Copy link
Copy Markdown
Contributor

Depends on #30 (which makes the test suite run to completion); merge that first
or this branch's test changes are partly invisible.

The drift

The code defaults and docker-compose.dev.yml disagreed on four variables.
docker-compose.prod.yml sets no solver vars, so production ran the code
defaults while every benchmark ran through the dev container. specs/002
(2026-09-10) wrote its winning values into the code and never touched compose,
which still carried 2026-07-16 values.

The tests pin the code defaults — so the suite failed in the project's own dev
container
, and since a failing top-level @testset aborts the file, 141 of
151 assertions silently never ran.

Attribution — one variable at a time

23,022 wide-curator cases, Release97, one catalog build. Each arm's container
env and catalog mount were verified, and its raw pred_ui checked to differ
from the baseline — a config change producing bit-identical output has not
applied.

variable moved changed net
DS_ASSEMBLY_LIMITING 1 → 0 613 −196
DS_AND_MODE hill_loghill_sat 171 −90
DS_INHIBITOR_EPS 1e-3 → 1e-12 269 +11

Assembly-limiting is the driver, restoring the result already on file from
2026-07-16. The epsilon is not: its whole range is worth 11–14 cases against a
248-case gap.

Epsilon sweep

DS_INHIBITOR_EPS correct macro-F1 net
1e-3 19,278 0.8035
1e-4 19,292 0.8045 +14
1e-6 19,289 0.8041 +11
1e-12 19,289 0.8041 +11

1e-6 and 1e-12 are bit-identical — same 269 changes, same +109/−98. At or
below 1e-6 the epsilon has saturated into a pure divide-by-zero guard, exactly
as specs/006 FR-002 required and nobody had confirmed.

1e-4 scores 3 better in 23,022 and is not taken: it still changes results,
so it is still acting as a model parameter, and picking it would optimise a
guard constant against the evaluation set.

Result

DS_AND_MODE          = hill_log   (was hill_sat)
DS_ASSEMBLY_LIMITING = true       (was false)
DS_INHIBITOR_EPS     = 1e-12      (unchanged; compose corrected from 1e-3)
DS_HILL_SAT_EPS      = 1e-5       (unchanged; inert under hill_log)

Code and compose agree, all four assertion suites pass in the dev container, and
production computes what the benchmark measures.

The tension, recorded not buried

specs/002 R5 was not wrong that hill_sat implements the stated AND
intent: 10×10 reads 99.98 against hill_log's 74.06, and a lone node at UI 50
reads 50.00 against 41.43. hill_log compresses throughout the operating range,
which by that argument is a defect.

It also scores 90 cases better. So the compression is doing useful work that
"correct" multiplication does not, and nobody has explained why — a
plausible candidate is that it damps the multi-hop amplification behind
false-change calls, but that's a hypothesis.

This is therefore an empirical default, not a principled one, and
test_and_curves.jl names its testset accordingly instead of claiming the
defaults implement the design intent.

Full record: specs/009-solver-defaults/research.md.

🤖 Generated with Claude Code

@adamjohnwright
adamjohnwright force-pushed the config/measured-winning-defaults branch 2 times, most recently from 83e7d34 to 2c38439 Compare September 16, 2026 19:41
The code defaults and docker-compose.dev.yml disagreed on four variables.
Production sets no solver vars so it ran the code defaults; every
benchmark ran through the dev container. specs/002 wrote its winning
values into the code on 2026-09-10 and never touched the compose file,
which still carried 2026-07-16 values. The tests pin the code defaults,
so the suite FAILED in the project's own dev container -- and a failing
top-level testset aborts the file, so 141 of 151 assertions never ran.

Re-measured on 23,022 wide-curator cases, one catalog build, one variable
at a time, each arm's env and catalog mount verified and its raw values
checked to differ from the baseline:

  DS_ASSEMBLY_LIMITING 1 -> 0    -196
  DS_AND_MODE hill_log -> hill_sat  -90
  DS_INHIBITOR_EPS 1e-3 -> 1e-12    +11

Assembly-limiting is the driver, restoring the 2026-07-16 result.
Adam's hypothesis that the epsilon drove the gap does not hold: its whole
range is worth 11-14 cases against a 248-case gap.

Epsilon swept separately. 1e-6 and 1e-12 are BIT-IDENTICAL (same 269
changes, same +109/-98), so at or below 1e-6 it is a pure divide-by-zero
guard -- confirming specs/006 FR-002. 1e-4 scores 3 better in 23,022 and
is NOT taken: it still changes results, so it is still a model parameter,
and picking it optimises a guard against the evaluation set.

Defaults now: hill_log, assembly on, eps 1e-12, hill_sat_eps 1e-5
(inert). Compose corrected to match, so the suite passes in the dev
container and production computes what the benchmark measures.

Recorded honestly: specs/002 R5 was not wrong that hill_sat implements
the stated AND intent -- 10x10 reads 99.98 against hill_log's 74.06. It
just scores 90 worse. Why the compression helps is unexplained, so this
is an empirical default and test_and_curves names its testset
accordingly rather than claiming design intent.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@adamjohnwright
adamjohnwright force-pushed the config/measured-winning-defaults branch from 2c38439 to 3480eb4 Compare September 16, 2026 19:43
@adamjohnwright
adamjohnwright merged commit bd52cb8 into main Sep 16, 2026
@adamjohnwright
adamjohnwright deleted the config/measured-winning-defaults branch September 16, 2026 19:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant