Embed observer contract: gate open by default; eval-isolation opt-in (#1038, #1028) - #1113
Merged
Conversation
Native and assembled-code embedders could update observer slots without ever calling compile_ast, leaving the #915 gate closed and returning improving=0 where the VM returns 1. Start states with recording open and permit only an initial compile verdict to close the gate. Pin default embed initialization open because a subsequently compiled module cannot classify its native caller. Keep eigs_obs_enable idempotent and retain the compiler's existing scan logic. Add eigs_set_eval_observer_isolated(int enabled) on the attached EigsState. The opt-in promises that no eval unit interrogates a binding assigned by an earlier unit, directly or through called code. Default evals continue recording across units. Isolated evals may use individual compile verdicts, with sticky obs_history_gap rejection naming EIGS_OBS_FORCE=1 before a later observer read can return incomplete history. Preserve accumulated verdicts for retained functions; registered C callbacks pin recording because their code is opaque. Keep shared observer flags atomic and recording monotonic within each unit. Enroll a hermetic C regression in the full suite and warning-flag audit. Cover native/assembled execution, low-level module compilation, all three eval arms, retained functions, C callbacks, late registration, sticky errors, force-on recovery, and independent states. Update the embed API table, observer residual, header contracts, SPEC/COMPARISON references, CHANGELOG and validation record. Validation, run sequentially in /home/jon/src/wt/es-1038: - make embed-observer-test; bash tests/test_embed_observer.sh: embed observer: 28 passed, 0 failed - G1 against origin/main cd99388: native improving=0; assembled improving=0; embed observer: 0 passed, 2 failed (exit 1). Branch --direct: both improving=1; 2 passed, 0 failed (exit 0). - G3 planted fault: remove only eval_source's missing-history rt_error; embed observer: 24 passed, 4 failed (exit 1). Restore the saved source and rebuild: 28 passed, 0 failed (exit 0). - make; cd tests; bash run_all_tests.sh: RESULTS: 4259/4259 passed, 0 failed - make asan; cd tests; ASAN_OPTIONS=detect_leaks=1 bash run_all_tests.sh: RESULTS: 4248/4248 passed, 0 failed; no leak reports; C regression 28/28. - bash tools/jit_diff.sh: jit_diff: OK (230 programs x {jit, osr} vs the interpreter; 4 arms adjudicated by replay; 0 ledgered) - make tsan; bash tests/test_tsan.sh: Results: 14 passed, 0 failed. All 13 clean programs, including test_obs_mt_race, reported no warnings; the seeded race emitted 5 warnings before its bounded 120-second timeout, proving detection remains live. - tools/observer_gate_diff.sh: 521 captured, 5 denied; compare main/branch: RESULT: PASS -- 497 programs byte-identical; 0 mismatches. 399 informative, 98 silent, 24 excluded by baseline self-difference. - EIGS_OBS_GATE_STATS=1 src/eigenscript -e $'x is 41\nprint of (x + 1)': obs-gate: unobserved <module>; output 42. - Warning audit: all 481 compile invocations across 29 targets + 7 scripts carry the three required -Werror flags. Full suites include classification, strict argument guards, suite labels and the observer gate regressions. - EMS="$PWD/build/observer-EigenMiniSat" N=5 ROWS=4 COLS=4 \ bash tools/observer_gate_measure.sh: base median: 287.78 s (289.21 288.18 287.78 285.53 285.97) gated median: 30.54 s (30.54 30.32 31.03 30.35 30.56) speedup: 9.42x RESULT: valid -- same search, same binary, interleaved. All 10 runs had identical counters: UNSAT, conflicts=9986, resolutions=33873, learnts=9985, learnt_lits=78856, peak_learnts=1595, max_level=19, decisions=15275, propagations=44166, restarts=11. Fixture cca0e1482da91ad5b2b61b3cf775a04d5a5ca4de needed only a scratch eigs.json project-root marker; solver/benchmark source was unchanged. An earlier interrupted four-pair attempt is excluded from these medians. The release binary SHA-256 remained unchanged after the sanitizer variants: 7260633c29791dd5bac01b2884b8203ffc601d2b2bc29a91c5718a3d9528384f Known pre-existing failure, reported separately as EigenScript#1112: tools/replay_diff.sh prints OK despite a SIGSEGV after the unsupported-thread replay diagnostic. The same bounded reproducer crashes 20/20 on unmodified main and 20/20 here; all recordings succeed. This is not a clean replay result. References #1038 and #1028. Closes #1038. Closes #1028. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Kpzyjv1SaLaqBf45FSFDhB
CI) The new observer-contract harness identified the CLI variant by hard-link (build/<variant>/eigenscript -ef src/eigenscript), which only exists in the Makefile objdir layout. The linux/macos CI lanes build via ./build.sh, which compiles src/eigenscript as a real file with no build/<variant> counterpart, so detection failed with "cannot identify the CLI build variant" and the [0f] section (and the [99p] child-exit ledger) went red — same class as the embed_roads fix in #1102. Default to the release variant when no hard link matches; the sanitizer suite still resolves to asan and is used there. The observer contract is variant-independent (no sanitizer semantics). Verified: normal run 28/28 via the release hard link; with all build/<variant>/eigenscript moved aside (build.sh layout simulated) the fallback prints "using release" and passes 28/28. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Kpzyjv1SaLaqBf45FSFDhB
…tion The existing C regression always entered through eigs_open, whose runtime initialization explicitly arms recording. Reverting only state.c's default therefore left all 28 checks green. Add a --raw-host arm using eigs_state_new and eigs_thread_attach, with no runtime initialization, compilation or explicit arming. It records twelve descending updates and requires both an open gate and improving=1. Keep the original eigs_open/native and assembled-code arms. Cover serialized raw first compilation followed by worker arming and atomic flag reads, and run the C contract in the TSan lane with sanitizer failures fatal. Raise the C harness's exact assertion count from 28 to 31 and require the raw-host result marker. F2's compiler clear already uses obs_flag_store, which expands to a RELEASE __atomic_store_n. Keep that operation and clarify its serialization boundary: an atomic exchange of one flag would not make the whole check-then-clear decision atomic against arbitrary concurrent native arming. In-tree initial compilation precedes execution/spawn; embed initialization consumes permission to close; opted-in eval boundaries require exclusive state access. No runtime instructions or observer scan logic changed. Clarify F3 in EMBEDDING.md: direct host predicates between isolated eval units bypass the eval guard. Recording must be arranged before relevant assignments (eigs_obs_enable for direct host work, or EIGS_OBS_FORCE=1 before the first eval); arming after assignments cannot recover missing history. Validation on fix-1038, starting clean at 7c7a11d; one heavy job at a time: - make embed-observer-test; bash tests/test_embed_observer.sh: embed observer: 31 passed, 0 failed - build/release/test_embed_observer --raw-host: raw host: obs_needed=1 improving=1 embed observer: 1 passed, 0 failed (exit 0) - cp -a the worktree to /tmp/es1038-r2-plant; change only state.c's st->obs_needed = 1 to 0; plain make; same C runner: embed observer: 30 passed, 1 failed (exit 1) Isolated --raw-host: obs_needed=0 improving=0; embed observer: 0 passed, 1 failed (exit 1). The eigs_open-path checks remain green. Reconfirmed using the exact final C source, byte-identical between the branch and planted copy. - Release full suite, once: RESULTS: 4259/4259 passed, 0 failed - ASan full suite, once, ASAN_OPTIONS=detect_leaks=1: RESULTS: 4248/4248 passed, 0 failed; no leak reports; C contract 31/31. - make tsan; bash tests/test_tsan.sh: Results: 15 passed, 0 failed 13 clean programs plus C contract 31/31; seeded race detected with 14 warnings before its bounded 120-second timeout. - Both full suites: warning audit checks all 481 compile invocations across 29 targets and seven scripts; 89 scripts recognized, six shape waivers. - EIGS_OBS_GATE_STATS=1 src/eigenscript -e $'x is 41\nprint of (x + 1)': obs-gate: unobserved <module>; output 42. Fresh baseline for the observer differential: cp -a the worktree to /tmp/es1038-r2-main, reverse git diff origin/main -- src using patch -R, then make clean and plain make. All six changed runtime files match git show origin/main:<path> byte-for-byte. Baseline source revision: cd99388 Fresh baseline binary SHA-256: 305b8c73a74e5d6478d48f0c3c20f872e97b9aec41ad4f40f8c125a2a5f83be5 Restored release binary SHA-256 (unchanged from round 1): 7260633c29791dd5bac01b2884b8203ffc601d2b2bc29a91c5718a3d9528384f Observer differential commands (run from the worktree): export EIGS_GATE_DIFF_DIR="$PWD/build/observer-r2-captures" export EIGS_GATE_DIFF_BIN=/tmp/es1038-r2-main/src/eigenscript bash tools/observer_gate_diff.sh capture main bash tools/observer_gate_diff.sh capture main2 # Reapply the runtime patch and rebuild the branch in the SAME scratch path. bash tools/observer_gate_diff.sh capture branch_samepath bash tools/observer_gate_diff.sh compare main branch_samepath Each capture: 521 programs, 5 denied. Compare exits 0: provenance: base=305b8c73a74e(force=0) ref=305b8c73a74e(force=0) gated=7260633c2979(force=0) corpus entries with captures: 520 nondeterministic under a FIXED build (excluded): 23 compared: 497 (informative: 401, silent: 96) mismatches: 0 RESULT: PASS — 497 programs byte-identical Measurement corrections and limits: - An initial compare using the original worktree's branch executable reported seven differences: two import-shadow warnings and five missing-file diagnostics depended on the different executable roots. Rebuilding the branch in the baseline's scratch path removed all seven. No output normalization or exclusion rules were changed. - The 521 attempts / 520 valid entries exposed an existing capture limitation: test_terminal.eigs reads the corpus loop's stdin, truncating the next path to ts/test_throw_unwind.eigs. All three arms captured that invalid path's error instead of tests/test_throw_unwind.eigs; comparison skips its missing capture in all arms. The 497-program claim excludes that test. Documented as a separate harness gap; observer_gate_diff.sh is unchanged this round. Final artifact: six modified files on fix-1038; git diff --check clean. No commit or push attempted: Git metadata is read-only in this session. References #1038 and #1028; closes the round-2 raw-host fixture gap. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Kpzyjv1SaLaqBf45FSFDhB
…ndary An isolated host following the documented eigs_obs_enable-before-eval recipe still lost its observer history: eval_source renewed compile permission after the arm, a read-free verdict closed recording, and direct improving(x) returned 0 with gap=0 and no error. Make the recipe work instead of replacing it with an opt-out recipe. Separate explicit host requests from internal compiler/runtime arming. The public eigs_obs_enable sets an atomic pending host request and delegates to the existing arming helper. The next eval compilation boundary consumes that request with an atomic exchange and withholds permission to close that unit. Internal arming uses eigs_obs_enable_runtime without creating a future host request. The existing gap-before-needed release stores and source scan verdict logic are unchanged. The new field owns no runtime objects and adds no tape records or per-assignment work. Add an isolated-host C fixture: opt in, arm twice, execute a read-free unit, and call observer_predicate_at directly. Require improving=1 and no gap/error, then require the following read-free unit to run unobserved. Extend the existing worker-arming fixture to read/check the new atomic flag under TSan. Keep the raw-state and eigs_open-path fixtures and all cross-unit gap checks. Document F5's host obligation: a raw state's first compile must be serialized against worker arming/execution. The atomic obs_needed store does not make the surrounding check-then-clear a transaction. Isolated eval boundaries already require exclusive state access. Remove F6's stale numeric flag count. Update the embed contract/header, changelog and validation record together. Validation started from clean fix-1038 at babd15f. One heavy job at a time: - Before the runtime change, make embed-observer-test then --isolated-host: isolated host: DIRECT improving=0 obs_needed=0 gap=0 embed observer: 3 passed, 1 failed (exit 1). Full bash tests/test_embed_observer.sh: 34 passed, 1 failed (exit 1). - After the change, plain make then bash tests/test_embed_observer.sh: embed observer: 35 passed, 0 failed (exit 0). Isolated fixture: DIRECT improving=1 obs_needed=1 gap=0; embed observer: 4 passed, 0 failed (exit 0). - Release full suite, once: (cd tests && bash run_all_tests.sh) RESULTS: 4259/4259 passed, 0 failed Every child .sh test completed; 0 nonzero child exits. Warning audit: 481 compile invocations, 29 targets, 7 scripts; recognizer: 89 scripts, 6 shape waivers. - ASan full suite, once: make asan; (cd tests && ASAN_OPTIONS=detect_leaks=1 bash run_all_tests.sh) RESULTS: 4248/4248 passed, 0 failed Every child .sh test completed; 0 nonzero child exits; no ASan/UBSan/leak diagnostic lines. C preflight and suite C contract: 35/35, leaks enabled. Same warning audit: 481 invocations / 29 targets / 7 scripts; recognizer: 89 scripts, 6 shape waivers. - make tsan; bash tests/test_tsan.sh: Results: 15 passed, 0 failed 13 clean programs; C contract 35/35; seeded race detected (16 warnings). - Restored release with make. Read-free EIGS_OBS_GATE_STATS=1 probe: obs-gate: unobserved <module>; output 42; exit 0. Release SHA-256: 11310dc5efbab32e4a3d1a4b66168de50ed923173990613790709ea69ad7bf19 Fresh same-path observer differential: - cp -a the worktree to /tmp/es1038-r3-oracle; reverse the complete git diff origin/main -- src using patch -R; verify all nine changed runtime files byte-for-byte against git show origin/main:<path>; make clean; make. Baseline source: cd99388 Fresh baseline SHA-256: 305b8c73a74e5d6478d48f0c3c20f872e97b9aec41ad4f40f8c125a2a5f83be5 - From the original worktree, set EIGS_GATE_DIFF_DIR="$PWD/build/observer-r3-captures" EIGS_GATE_DIFF_BIN=/tmp/es1038-r3-oracle/src/eigenscript and unset EIGS_OBS_FORCE, EIGS_OBS_GATE_STATS, EIGS_TRACE, EIGS_REPLAY, EIGS_JIT_OFF. Run bash tools/observer_gate_diff.sh capture main, then capture main2. Reapply the runtime patch and rebuild the branch IN PLACE, then capture branch and compare main branch. Scratch branch source files and binary SHA match the validated worktree. The manifest rev is the calling worktree's babd15f; source verification identifies the executable. No canonical build or binary was used. - Each capture: 521 programs, 5 denied. Compare exits 0: provenance: base=305b8c73a74e(force=0) ref=305b8c73a74e(force=0) gated=11310dc5efba(force=0) corpus entries with captures: 520 nondeterministic under a FIXED build (excluded): 23 compared: 497 (informative: 401, silent: 96) mismatches: 0 RESULT: PASS — 497 programs byte-identical Known measurement limit, unchanged from round 2: test_terminal.eigs consumes corpus stdin, truncating the next path to ts/test_throw_unwind.eigs in all three captures. The real tests/test_throw_unwind.eigs is outside the 497 compared programs. No differential normalization, exclusions or floors were changed. No new measurement contradicted the round-3 brief. Final artifact: git diff --check clean; 13 modified files on fix-1038. Git metadata is read-only in this session; no commit or push attempted. References #1038 and #1028; closes round-3 F4, F5 and F6. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Kpzyjv1SaLaqBf45FSFDhB
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1038. Closes #1028.
#1038 — a linked-runtime embedder no longer gets a silently-closed observer
The #915 gate was closed-by-default and armed only by
compile_ast's scan, so an embedder that drives observer slots without ever compiling (ouroboros's AOT runtime; any native/assembled-code host) got a gate that never opened — every predicate answered from an empty window (VMimproving=1, embedder all-zeros, rc 0), the silent-wrong the gate's own header forbids. Now a state starts with recording OPEN, and only an initial compile verdict may CLOSE it. A read-free compiled program still gates closed (the #915 8.5x is preserved — verifiedunobservedviaEIGS_OBS_GATE_STATS=1and the 497-program observer differential byte-identical).#1028 — a host opt-in for per-unit eval gating
eigs_set_eval_observer_isolated(1)lets a host promise that noeigs_eval_stringunit interrogates a binding an earlier unit assigned; under it each unit gates on its own compile verdict, and a broken promise raises the stickyobs_history_gaperror namingEIGS_OBS_FORCE=1before any later read can answer from incomplete history — never a wrong verdict, never silent. Without the opt-in, behavior is exactly today's force-armed accumulation. EigenOS measures and adopts on its own schedule.New
tests/test_embed_observer.c(+.sh): the embedder arm is red on main (all-zeros) and green here; the isolation arm's missing-history raise is the planted fault.docs/EMBED_OBSERVER_VALIDATION.mdrecords the method and the 9.42x measurement.Filed while validating: a pre-existing SIGSEGV in
replay_diffreproduced 20/20 on both main and this branch (#1112), unrelated to this change.🤖 Generated with Claude Code
https://claude.ai/code/session_01Kpzyjv1SaLaqBf45FSFDhB