Embed observer contract: gate open by default; eval-isolation opt-in (#1038, #1028)

CHANGELOG.md

@@ -37,6 +37,20 @@ All notable changes to EigenScript are documented here.
 
 ### Added
 
+- **Embed observer contract (#1038/#1028).** States start with recording open,
+  so native/assembled hosts need no startup `eigs_obs_enable()` workaround.
+  Embed initialization keeps later module compilations from closing recording
+  behind a native caller. Compile verdicts still close the CLI gate for read-free programs. The additive
+  `eigs_set_eval_observer_isolated` host opt-in allows read-free eval units to
+  skip observation; default evals retain cross-unit history. Missing history
+  rejects later reader units with `EIGS_OBS_FORCE=1` guidance, conservatively
+  even for independent bindings. Escaped compiled functions retain the gate's
+  accumulated verdict; registered C callbacks keep eval recording open.
+  Explicit host `eigs_obs_enable()` also pins the next isolated eval open,
+  so direct host predicates can read that unit's recorded assignments; the
+  request is consumed at one eval boundary. A C regression runs under the
+  suite's build variant.
+
 - **`is_file of path` (#1058).** 1 iff the path names a REGULAR file
   (`S_ISREG`); 0 for a directory, a device/fifo/socket, a missing path, or a
   non-string. `read_file_util` admits only regular files, and a driver that

Makefile

@@ -212,6 +212,15 @@ $(NATIVEFN_TEST): $(NATIVEFN_TEST_OBJ) $(filter-out build/release/main.o build/r
 nativefn-test: $(NATIVEFN_TEST)
 	@echo "Native-fn identity test built: $(NATIVEFN_TEST)"
 
+# #1038/#1028: same runtime variant as the suite; never repoint the CLI alias.
+EMBED_OBSERVER_VARIANT ?= release
+EMBED_OBSERVER_OBJ := $(filter-out build/$(EMBED_OBSERVER_VARIANT)/main.o,$(OBJ_$(EMBED_OBSERVER_VARIANT)))
+build/$(EMBED_OBSERVER_VARIANT)/test_embed_observer: tests/test_embed_observer.c $(EMBED_OBSERVER_OBJ) $(wildcard $(SRC_DIR)/*.h) Makefile
+	$(CC) $(FLAGS_$(EMBED_OBSERVER_VARIANT)) -I$(SRC_DIR) -o $@ $< $(EMBED_OBSERVER_OBJ) $(LIBS_$(EMBED_OBSERVER_VARIANT))
+.PHONY: embed-observer-test
+embed-observer-test: build/$(EMBED_OBSERVER_VARIANT)/test_embed_observer
+	@echo "Embed observer test built: $<"
+
 # #1056: use the same variant as the CLI under test, without relinking it.
 ROAD_VARIANT ?= release
 EMBED_ROADS_OBJ := $(filter-out build/$(ROAD_VARIANT)/main.o,$(OBJ_$(ROAD_VARIANT)))

docs/COMPARISON.md

@@ -533,3 +533,10 @@ The existing function-slot exception remains: a binder with no prior binding
 inside a function retains its final value after the loop on every road. A
 pre-existing parameter or local is restored. This change preserves that
 exception; see the scope notes in LANGUAGE_CONTRACT.md.
+
+<!-- Embed contract: #1038/#1028; language-level observer semantics unchanged. -->
+The C embedding API starts observer recording open. Source evals retain
+cross-unit history by default; hosts may explicitly promise isolated observer
+use with `eigs_set_eval_observer_isolated`. Missing history then raises
+conservatively instead of answering a rest value. See the
+[embedding observer contract](EMBEDDING.md#observer-contract-1038--1028).

docs/EMBEDDING.md

@@ -137,6 +137,99 @@ if (!r) {
 }
 ```
 
+## Observer contract (#1038 / #1028)
+
+A newly opened state records observer updates by default. This also applies to
+`eigs_state_new` plus attach/init: a native or assembled-code host that never
+calls `compile_ast` has a live observer. Calling `eigs_obs_enable()` at startup
+is **no longer required**. The call remains idempotent and can pin the current
+unit open before compilation; an explicit host call also pins the next embed
+eval that reaches compilation, including an isolated eval. Embed initialization
+pins recording open because
+a subsequently compiled module cannot classify its surrounding native caller.
+The explicit eval opt-in below renews permission to choose a compile verdict.
+Only a compile verdict proving no observer reads
+may close the gate; recording stays monotonic within that execution unit.
+The CLI's read-free programs still compile as `unobserved`.
+
+A raw host using `compile_ast` before runtime initialization must serialize
+the first compilation against worker arming and execution: no worker may call
+`eigs_obs_enable()` until that compile returns. Atomic flag accesses do not
+make the compiler's check-then-clear decision a transaction. Subsequent arming
+and flag reads may be concurrent within the established execution unit;
+isolated eval boundaries still require exclusive state access.
+
+By default, `eigs_eval_string` and `eigs_eval_file` keep recording across calls.
+A later unit may interrogate a binding an earlier unit assigned, with its full
+history available. Hosts can explicitly opt in to compile-time gating:
+
+```c
+EigsState *st = eigs_open();
+eigs_set_eval_observer_isolated(1);
+/* Each eval unit obeys the promise below. */
+```
+
+**The host promises: no eval unit interrogates a binding assigned by an earlier
+unit, directly or through called code.** This covers observer predicates,
+reports and temporal queries. The setting belongs to the attached `EigsState`,
+applies to both string and file evals, and defaults off on every new state.
+Call the setter only between evals while the host has exclusive use of the
+state, as required for eval itself. Globals still accumulate normally.
+This promise covers eval-unit cross-reads; a host calling predicates directly
+between isolated units must arrange recording before the relevant assignments
+(`eigs_obs_enable()` before the eval that makes those assignments, or
+`EIGS_OBS_FORCE=1` before the first eval), because direct predicate calls bypass
+the eval guard and arming after
+the assignments cannot recover their missing history.
+An explicit arm is consumed at one eval compilation boundary; call it again
+before each unit whose assignments the host will interrogate directly. Arming
+by a source scan or by internal runtime code does not create that host request.
+
+```c
+eigs_set_eval_observer_isolated(1);
+eigs_obs_enable();
+EigsValue *r = eigs_eval_string(assignments); /* this unit stays observed */
+eigs_value_release(r);
+/* Direct observer_predicate_at calls can now read this unit's history. */
+```
+
+| API | Contract |
+|---|---|
+| `eigs_set_eval_observer_isolated(int enabled)` | Nonzero opts in to the promise above; zero restores recording for subsequent work. |
+| `eigs_obs_enable(void)` | Idempotently arms the current unit and pins the next eval compilation boundary open; does not reconstruct missing history. |
+
+An opted-in eval starts a new unit and may choose a new verdict unless the
+host explicitly armed it. Use default eval mode or `EIGS_OBS_FORCE=1` to keep
+all subsequent units observed.
+
+With the opt-in, a read-free unit can run `unobserved`, visible through
+`EIGS_OBS_GATE_STATS=1`. The existing source scan supplies each verdict. Functions
+can escape through globals, module exports, containers and callbacks, so once
+any functions have been compiled the state conservatively retains its accumulated
+gate verdict across later evals; a later call site cannot rescan their source.
+Registered C callbacks have no scannable source at all, so
+`eigs_register_function` pins subsequent evals open, even if the host enables
+the opt-in after registration. Registering a callback after a history gap causes
+subsequent evals to raise conservatively before entering opaque host code.
+
+**Missing history is a sticky error, never a rest-value answer.** After an
+unobserved unit executes, a subsequent unit whose scan reads observer state is
+rejected before execution, with an error naming `EIGS_OBS_FORCE=1`. This is
+conservative, like `load_file`: it can reject a unit even if that unit would
+observe only independent bindings. Catching/clearing the error, calling
+`eigs_obs_enable`, or disabling the opt-in does not restore the missing history.
+Restart the state with `EIGS_OBS_FORCE=1` set **before the first eval**, or restart
+without the opt-in, when cross-unit observation is needed. Force-on preserves
+correct cross-unit queries even with the opt-in enabled.
+
+The regression instrument is `bash tests/test_embed_observer.sh`: native slot
+updates and assembled bytecode without compilation, default cross-unit history,
+isolated read-free units, a rejected cross-unit read, retained functions and the
+force-on recovery path, plus C callback observation and late registration. It uses the same build variant as `src/eigenscript`,
+including ASan, and is enrolled in the full suite.
+The [validation record](EMBED_OBSERVER_VALIDATION.md) contains the baseline
+reproducer, planted-fault output and measurement setup.
+
 ## Error retrieval
 
 ```c