From 1efbfecb7f30df659e6d7a2f5f429f2f219a9670 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 6 Sep 2026 23:27:10 +0000 Subject: [PATCH 01/39] tools: observer_gate_diff.sh normalises the exe-dir and the out-of-tree import-shadow warning before diffing (#1115) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Repro (byte-identical trees, binaries at two paths; captures via EIGS_GATE_DIFF_DIR=$S/caps): EIGS_GATE_DIFF_BIN=/home/user/EigenScript/src/eigenscript capture base ; capture base2 EIGS_GATE_DIFF_BIN=$WT/src/eigenscript capture wt EIGS_OBS_FORCE=1 EIGS_GATE_DIFF_BIN=$WT/src/eigenscript capture wtf Before (tool at a6c50fb): compare base wt -> "FAIL: SAME binary AND the same EIGS_OBS_FORCE" rc=2 (same sha at two paths was refused outright) compare base wtf -> mismatches: 9, RESULT: FAIL rc=1 — 7 of them are location: 5 x "stdlib roots '/../'" in cannot-read errors (lib/test_runner, tests/test_import, test_import_errors, test_import_toplevel_scope, test_module_scope) and 2 x the project-vs-stdlib import-shadow warning that fires only out of tree (lib/engineering, lib/linalg importing complex). After: compare base wt -> location-only differences: 7 (named), residual mismatches: 0, RESULT: LOCATION-CLEAN rc=0 compare base wtf -> location-only: 7, residual mismatches: 2 (test_chunk_verify_stack, test_desc_unrecorded_read — genuine force-arm divergences, untouched), rc=1 What changed in compare: - each arm's capture is normalised with that arm's own exe-dir (dirname of the manifest's realpath'd bin): the exe-dir string -> , and the import-shadow warning is dropped ONLY in the exact shape "using '/lib/N.eigs', shadowing '/lib/N.eigs'" (same N). Nothing else is touched; an arm whose exe-dir is '/', '.', '' or relative compares raw, announced by NOTE. - arm identity is (sha, force, path): same build at the SAME path is still refused; same build at two paths is admitted with the verdict word LOCATION-CLEAN (never PASS) because it measures location-independence, not the gate. The determinism reference must match base on all three. - `location-only differences ... : K` lists every absorbed program by name with its first raw diff line; `residual mismatches: M` is what RESULT gates on. - `selftest` subcommand (9 cases) drives the real `compare` entry point over synthetic captures: both location shapes absorbed; a different error message, a differently-named shadow, a project-file shadow, a corpus-path difference and the root-exe-dir guard each still FAIL; the two provenance refusals still fire; different builds still PASS. - slug() is pure bash (same output): the selftest went 12.6s -> 4.6s. Regression test: suite section [99q] runs the selftest and pins the tally "9 ok, 0 failed (of 9)". Planted fault (normaliser made identity): case 1 fails, [99q] goes red, and `compare base wt` reports residual mismatches: 7. Residuals (documented in the tool header): $HOME in the same error line, the corpus tree's own path, and the argv[0] fallback on platforms without /proc/self/exe are deliberately NOT normalised. Closes #1115 Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01A8rPrmr564H7yB2tQ6DwV9 --- tests/run_all_tests.sh | 29 +++ tools/observer_gate_diff.sh | 394 ++++++++++++++++++++++++++++++++++-- 2 files changed, 402 insertions(+), 21 deletions(-) diff --git a/tests/run_all_tests.sh b/tests/run_all_tests.sh index f1f154a3..8e6d6ff3 100755 --- a/tests/run_all_tests.sh +++ b/tests/run_all_tests.sh @@ -6293,6 +6293,35 @@ else fi echo "" +# [99q] Observer-gate corpus diff: location normalisation self-test (#1115). +# tools/observer_gate_diff.sh compares full-corpus captures byte-for-byte, and +# an out-of-tree baseline binary echoes its own exe-dir into two shapes of +# text (the stdlib-roots list in every "cannot read" error, and the project- +# vs-stdlib import-shadow warning that fires only out of tree). Seven programs +# mismatched on exactly those shapes across three critic rounds on #1038 and a +# clean run read as a regression. The tool now canonicalises ONLY those two +# shapes; this self-test drives the real `compare` entry point over synthetic +# captures (no corpus run) and pins that (1) both shapes are absorbed and named, +# (2) a different error message, a differently-named shadow, a project-file +# shadow, a corpus-path difference and the root-exe-dir guard each still FAIL, +# (3) the same-build-same-path and path-mismatched-reference refusals still +# fire, (4) genuinely different builds still get PASS. The case count is +# pinned (mechanical-gates §37): a self-test shrunk to one case also exits 0. +echo "[99q] Observer-gate corpus diff location normalisation (#1115)" +TOTAL=$((TOTAL + 1)) +OGD_EXPECTED=9 +OGD_OUT=$(bash "$TESTS_DIR/../tools/observer_gate_diff.sh" selftest 2>&1); OGD_RC=$? +OGD_TALLY=$(printf '%s\n' "$OGD_OUT" | sed -n 's/^SELFTEST: \([0-9]*\) ok, \([0-9]*\) failed (of \([0-9]*\))$/\1 \2 \3/p') +if [ "$OGD_RC" -eq 0 ] && [ "$OGD_TALLY" = "$OGD_EXPECTED 0 $OGD_EXPECTED" ]; then + PASS=$((PASS + 1)) + echo " PASS: exe-dir + import-shadow normalisation absorbs only the location shapes ($OGD_EXPECTED/$OGD_EXPECTED self-test cases)" +else + FAIL=$((FAIL + 1)) + echo " FAIL: observer_gate_diff.sh self-test broke or shrank (rc=$OGD_RC, tally='${OGD_TALLY:-none}', expected '$OGD_EXPECTED 0 $OGD_EXPECTED')" + printf '%s\n' "$OGD_OUT" | grep -E '^ FAIL|^SELFTEST|^FAIL' | head -8 | sed 's/^/ /' +fi +echo "" + # [99m] Lint archive symbol-collision gate (#917, hole closed by #922). # The #917 split turned lint's json_escape helper into an external symbol and # broke the static-library route for any embedder with its own json_escape. diff --git a/tools/observer_gate_diff.sh b/tools/observer_gate_diff.sh index 5c053b95..8f755976 100755 --- a/tools/observer_gate_diff.sh +++ b/tools/observer_gate_diff.sh @@ -40,9 +40,68 @@ # That arm runs the code that shipped before the gate existed, which is the # question a user actually has. # +# LOCATION NORMALISATION (#1115) — what `compare` canonicalises, and only that. +# +# An out-of-tree baseline (the flow above) runs a binary whose executable +# directory is NOT this checkout's src/, and the runtime echoes that directory +# into two pieces of text: +# 1. `eigs_file_resolve_error` (src/builtins_host.c) prints the stdlib roots +# it tried as '/../' and '/../lib/eigenscript' — +# so every "cannot read" error line carries the arm's own exe-dir. +# 2. The project-vs-stdlib import-shadow warning (src/vm.c, `import`) fires +# only when the project hit and the stdlib hit are DIFFERENT files. In +# tree, `lib/engineering.eigs` importing `complex` resolves both to this +# checkout's lib/complex.eigs and stays silent; out of tree the stdlib arm +# is /lib/complex.eigs, a different inode with the same bytes, +# and the warning fires — in that arm only. +# Bought (#1038, both blind critics, three rounds): 7 programs mismatched on +# exactly these two shapes and a clean run read as a 7-program regression. +# +# Before diffing, each arm's capture is passed through ONE normaliser driven by +# that arm's .MANIFEST (`bin=` is the binary's realpath; its dirname is the +# exe-dir the runtime echoes, because /proc/self/exe resolves the same way): +# - every occurrence of the arm's exe-dir string becomes ``; +# - a warning line of the EXACT shape +# Warning: import 'N' ... using '/lib/N.eigs', shadowing +# '/lib/N.eigs' (project-first; ...) +# — this checkout's own copy of stdlib module N shadowing the arm's copy of +# the SAME module — is dropped. Any other collision (a different module in +# the two paths, a project file outside lib/, a shadowed path not under the +# arm's tree) is left in place and mismatches. +# Nothing else is touched. Programs whose ONLY difference is one of those two +# shapes are reported by name as `location-only`, and `residual mismatches` +# (the count the RESULT gates on) is what is left after normalisation. The +# normaliser is switched OFF for an arm whose manifest exe-dir is not an +# absolute path with at least one component ('/', '.', '' would rewrite every +# line) — announced on the provenance line, and the arm then compares raw. +# +# Residuals — what compare still CANNOT canonicalise across (each is a real +# axis, not a location artifact, so it deliberately stays a mismatch): +# - $HOME: the same error line prints '$HOME/.local/lib/eigenscript', and a +# different HOME can genuinely change resolution (an installed stdlib +# there wins some lookups). Capture both arms under one HOME. +# - the corpus tree's own path: 'tried containing directory /…' +# names THIS checkout, so captures must come from one checkout (`capture` +# always cds here, so this holds unless a capture dir is copied between +# checkouts). +# - an argv[0] fallback: without /proc/self/exe the runtime echoes the +# UNRESOLVED argv[0] directory; the manifest records the realpath. A +# symlinked binary on such a platform escapes normalisation — loudly, as a +# mismatch, never silently. +# - two arms of the SAME build (same sha, same force) at different paths are +# accepted, but the verdict is `RESULT: LOCATION-CLEAN`, never PASS: that +# run proves the captures are location-independent, not that a gate did +# anything. Same build, same force AND same path is still refused. +# `selftest` (below) plants each of these shapes into synthetic captures and +# drives the real `compare` entry point: the location-only shapes must be +# absorbed, and a genuinely different error message, a differently-named +# shadow, a project-file shadow, a corpus-path difference, and the +# root-exe-dir guard must each still FAIL. +# # Usage: # tools/observer_gate_diff.sh capture