From 10018344ea43e0341f282ea63136cb4f2110c97c Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 6 Sep 2026 16:59:13 +0000 Subject: [PATCH 1/4] probes: canonical witness identity survives the cycle store on both tails Two pre-registered probes for the temporal-architecture recovery, plus their STATUS_BOARD rows. D-TEMPORAL-1 measures whether canonical address identity survives the cycle store's round trip: NodeGuid -> NodeRow -> NodeRowPacket::as_le_bytes -> cycle commit -> drop writer -> reopen -> scan_image. Green on both rails. Payload bytes are byte-exact across two cycles and a reopen, payload[0..16] is the submitted NodeGuid, the value slab travels too (so the equality is not a degenerate zero-match), each cycle mints exactly base+1, and the head is recovered after restart. Keys are minted through mint_for(classid_read_mode(c).tail_variant, ...) rather than a hardcoded constructor, so the fixture cannot silently test only the deprecated V1 tail. D-TEMPORAL-1a pins the false-green that motivates the two-rail shape: 256 V3 addresses varying only in leaf present as 256 distinct keys and 256 distinct to_hex_v2() decodes, but as ONE distinct identity() -- the V1 accessor reads bytes 13..16 of a V3 key. The same test pins the other half, which is not a defect: local_key_v2 also yields one across that set, because leaf is the 4th HHTL routing tier and belongs to the addressing prefix, not the basin-local key. Choosing the wrong V3 accessor fails the same way as choosing the V1 one. D-V3-MINT-1 checks a gate asymmetry rather than assuming it: mint_for's V2/V3 arm is gated on guid-v2-tail, which is not in default, while the registry's V3 entries are gated on guid-v3-tail, which is. The implication holds -- guid-v3-tail = ["guid-v2-tail"] -- so a V3-registered classid does mint a V3 key by default. The probe discriminates on leaf, which the V3 arm forwards to new_v2 and the V1 fallback discards, so it keeps working if the feature graph is ever flattened. Scope stated where it is recorded: these prove byte and address fidelity of the store, not production reachability. LanceCycleWriter still has zero non-test callers. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_012wrzeZAdwGYTCKoxamwQht --- .claude/board/STATUS_BOARD.md | 3 + .../tests/v3_mint_reachability_probe.rs | 62 +++++ .../tests/canonical_witness_identity_probe.rs | 255 ++++++++++++++++++ 3 files changed, 320 insertions(+) create mode 100644 crates/lance-graph-contract/tests/v3_mint_reachability_probe.rs create mode 100644 crates/lance-graph/tests/canonical_witness_identity_probe.rs diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index 0c69a1a06..2c3c3d3a5 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -17,6 +17,9 @@ identity) is real and probe-gated. | D-LNC-5 | Replace `VersionedGraph::diff`'s two-version full materialization (`read_all_batches` = `scan().try_into_stream()`, `versioned.rs:645`) with lance 11's native "row ids deleted between two versions". zero-copy-warden LENS-CLEAN; byte-identical output | Queued (blocked on D-LNC-3 — the API is lance 11) | | D-LNC-6 | Session reuse: end `Dataset::open`-per-call (17 sites in `lance-graph`); instrumented open-count strictly decreases, results identical | Queued (independent; lance 10 ships shared session) | | D-LNC-5a | **Delta version-columns probe** (pre-registered, `crates/lance-graph/tests/delta_version_columns_probe.rs`): are `_row_created_at_version` / `_row_last_updated_at_version` populated WITHOUT stable row ids? Stable-row-id arm is the CONTROL so a null is attributable to lance, not the apparatus | **In PR** #1198. **RED, both arms.** insert: control 2, physical **0**; update: control HELD (version advanced AND value changed), stable **1**, physical **0**. ⊘ the update arm's first run was INCONCLUSIVE — its control did not hold because `MergeInsertBuilder::when_matched` defaults to `DoNothing`, so no update ever committed (caught by a CodeRabbit finding on #1198). **Consequence: D-LNC-5's insert/update half is gated behind the stable-row-id decision, same as the delete arm** | +| D-TEMPORAL-1 | **Canonical witness identity probe** (`crates/lance-graph/tests/canonical_witness_identity_probe.rs`): does canonical address identity survive the A-store round trip on lance 11? `NodeGuid → NodeRow → NodeRowPacket::as_le_bytes → cycle commit → drop writer → reopen → scan_image`. Minted through `mint_for(classid_read_mode(c).tail_variant, …)`, never a hardcoded constructor | **GREEN, both rails.** V3 (`CLASSID_OSINT_V3`): payload bytes byte-exact across two cycles and a reopen; `payload[0..16] == G.as_bytes()`; value slab survived (so equality is not a zero-match); each cycle minted exactly `base+1`; head recovered at V2 after restart. V1 legacy control (`CLASSID_OSINT`) round-trips too. **Scope:** proves byte and address fidelity of the store; does NOT prove production reachability — `LanceCycleWriter` still has zero non-test callers (see D-MW-P2) | +| D-TEMPORAL-1a | **Pinned false-green: the V1 accessor on a V3 key.** 256 V3 addresses varying only in `leaf` → 256 distinct 16-byte keys, 256 distinct `to_hex_v2()`, but **1** distinct `identity()`. Byte fidelity does not imply address fidelity | **Pinned as a positive assertion.** Also pins the second half, which is not a defect: `local_key_v2` yields **1** across the same set BY CONSTRUCTION — `leaf` is the 4th HHTL routing tier (bytes 10..12), part of the addressing prefix, deliberately outside the basin-local key (bytes 12..16). So "use the V3 accessor" is two choices, not one, and the wrong V3 one fails like the V1 one. Do not update the numbers if the assertion breaks — re-audit the call sites | +| D-V3-MINT-1 | **V3 mint reachability** (`crates/lance-graph-contract/tests/v3_mint_reachability_probe.rs`): `mint_for`'s V2/V3 arm is gated on `guid-v2-tail`, which is NOT in `default`; the registry's V3 entries are gated on `guid-v3-tail`, which is. Two different features, so premise and conclusion can come apart — is a V3-registered classid silently minted into the deprecated V1 layout? | **GREEN — no defect.** `guid-v3-tail = ["guid-v2-tail"]` (`Cargo.toml:69`), so the implication holds. Measured, not inferred: `tail_variant=V3 guid-v3-tail=true guid-v2-tail=true leaf_is_live=true`, key bytes show the V2/V3 carving with `leaf` live at byte 10. Discriminator is `leaf` (the V3 arm feeds it to `new_v2`; the V1 fallback discards it), so the probe stays valid if the feature graph is ever flattened | | D-MW-P2 | **MemWAL crash-atomicity probe** (`crates/lance-graph/tests/memwal_atomicity_probe.rs`): the one measurement `.claude/temporal/08-memwal-vs-batchwriter.md` named as overturning its FOLD verdict — SIGKILL mid-flush on a 5 000-row put | **In PR** #1198. **GREEN.** A1 clean-drop control 5000; A2 SIGKILL sweep 5–500 ms → `0,0,0,5000,5000,5000,5000`. Boundary bracketed on both sides, no partial batch. FOLD stands. Scope recorded as NOT settled: one put, local store, no concurrent writers, no fencing path, and NOT the landing-rows-plus-frame-row composite (that is `LanceCycleWriter`'s, which has zero callers) | | D-GTM-0n / P3 | **Ternlog chaining amortization** (`ndarray/examples/ternlog_amortization_probe.rs`, re-run `--release` 2026-09-06): does chaining pay on THIS mask shape? Pre-registered pass criterion = per-constraint cost flat in K, with a **K=1 control** (nothing to chain ⇒ no win) | **GREEN, bounded.** `T3/T1` bottoms at **0.50** by K=8; T3 total flat in K (57.6 ns @ K=4 → 54.1 ns @ K=32, 16× more constraints). K=1 control reads **1.03** as it must. Two bounds ride with the number and may not be dropped when it is cited: the win is **L2-residency-contingent** (bandwidth `138→15 GB/s` past L2, ratio back to 1.03 at a 512 KiB mask), and mask loses to sparse below **0.1%** active. **Does NOT retire the standing caveat** — 2× on one of three passes still cannot account for 5×. Gates Stage-3 SPEED claims only; the shape/correctness case never depended on it | | D-ALPHA-1b | **`AlphaMask::zip` release-mode length guard** (`lance-graph-contract/src/alpha.rs`): the only guard was `debug_assert_eq!`, compiled out in release, so mismatched-length operands produced a mask claiming `self.len` addresses over the SHORTER operand's word count — `count`/`is_empty` under-report silently, `contains`/`materialize_ordinals` panic at a distance | **Fixed** — promoted to `assert_eq!`. Defect reproduced in release BEFORE the fix (the `should_panic` falsifier reported "test did not panic as expected"); two-sided falsifiers both disable-verified. No API change, no external callers. Stage 1b of `.claude/temporal/09-plan.md`, now DONE | diff --git a/crates/lance-graph-contract/tests/v3_mint_reachability_probe.rs b/crates/lance-graph-contract/tests/v3_mint_reachability_probe.rs new file mode 100644 index 000000000..f9ed1e657 --- /dev/null +++ b/crates/lance-graph-contract/tests/v3_mint_reachability_probe.rs @@ -0,0 +1,62 @@ +// SPDX-License-Identifier: Apache-2.0 + +//! D-V3-MINT-1 — is the V3 tail actually reachable through `mint_for` in the +//! build configuration production uses? +//! +//! `mint_for`'s doc justifies its V1 fallback arm as dead code: *"With the +//! feature off no classid registers a V2/V3 `tail_variant` (`classid_read_mode` +//! returns V1), so the fallback arm is dead — it exists purely so the crate +//! compiles `--no-default-features`."* +//! +//! That justification is checkable. The registry's V3 entries are gated on +//! `guid-v3-tail` (default ON); `mint_for`'s V3 arm is gated on `guid-v2-tail` +//! (default OFF). Those are two different features, so the premise and the +//! conclusion can come apart. +//! +//! The discriminator is `leaf`: the V3 arm feeds it to `new_v2`, the V1 +//! fallback discards it (`let _ = leaf;`). Two mints differing only in `leaf` +//! are therefore byte-equal iff the fallback ran. + +use lance_graph_contract::canonical_node::{classid_read_mode, NodeGuid, TailVariant}; + +#[test] +fn a_v3_registered_classid_resolves_to_the_v3_tail_variant() { + let m = classid_read_mode(NodeGuid::CLASSID_OSINT_V3); + #[cfg(feature = "guid-v3-tail")] + assert_eq!( + m.tail_variant, + TailVariant::V3, + "CLASSID_OSINT_V3 must register the V3 tail when guid-v3-tail is on" + ); + #[cfg(not(feature = "guid-v3-tail"))] + assert_eq!(m.tail_variant, TailVariant::V1); +} + +#[test] +fn minting_through_the_registered_tail_variant_honours_leaf() { + let c = NodeGuid::CLASSID_OSINT_V3; + let tv = classid_read_mode(c).tail_variant; + + // The canonical consumer incantation, verbatim from mint_for's doc. + let a = NodeGuid::mint_for(tv, c, 0x1111, 0x2222, 0x3333, 0x00AA, 0x0001, 0x0002); + let b = NodeGuid::mint_for(tv, c, 0x1111, 0x2222, 0x3333, 0x00BB, 0x0001, 0x0002); + + let leaf_is_live = a.as_bytes() != b.as_bytes(); + + eprintln!( + "tail_variant={tv:?} guid-v3-tail={} guid-v2-tail={} leaf_is_live={leaf_is_live}\n a={:02x?}\n b={:02x?}", + cfg!(feature = "guid-v3-tail"), + cfg!(feature = "guid-v2-tail"), + a.as_bytes(), + b.as_bytes(), + ); + + if tv == TailVariant::V3 { + assert!( + leaf_is_live, + "classid {c:#010x} registers TailVariant::V3, but mint_for produced a \ + leaf-insensitive key — the V1 fallback arm ran. A V3 class is being \ + minted into the deprecated V1 layout, silently." + ); + } +} diff --git a/crates/lance-graph/tests/canonical_witness_identity_probe.rs b/crates/lance-graph/tests/canonical_witness_identity_probe.rs new file mode 100644 index 000000000..0acdc8a12 --- /dev/null +++ b/crates/lance-graph/tests/canonical_witness_identity_probe.rs @@ -0,0 +1,255 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright The Lance Authors + +//! D-TEMPORAL-1 — does canonical address identity survive the cycle store? +//! +//! The claim under test, stated as bytes rather than as prose: +//! +//! ```text +//! NodeGuid G -> NodeRow -> 512-byte LE packet -> cycle Vn commit +//! -> drop writer -> reopen -> scan_image(cycle) +//! -> payload == submitted bytes AND payload[0..16] == G.as_bytes() +//! ``` +//! +//! Three assertions are kept apart, because they fail independently: +//! +//! 1. the raw key bytes survive byte-exact (a storage claim); +//! 2. the V3 address decodes as V3 (a reader claim); +//! 3. reading a V3 key through the V1 accessors is WRONG (a selection claim) — +//! pinned here as a positive assertion so that a future reader who reaches +//! for `identity()` on a V3 key breaks this test instead of shipping. +//! +//! Assertion 3 exists because that exact false-green has already happened: a +//! byte-exact V3 round-trip read back through the V1 `identity()` accessor +//! collapsed a five-figure row count to a three-figure one, and the round-trip +//! stayed green throughout. Byte fidelity does not imply address fidelity. +//! +//! Keys are minted through `mint_for(classid_read_mode(c).tail_variant, …)` — +//! never by hardcoding `new` (V1, deprecated) or `new_v2`. The V1 rail is +//! retained only as a read-side legacy control (RESERVE, DON'T RECLAIM); no new +//! unit is minted V1. +//! +//! Two Arrow claims are also kept apart: the schema's column 0 is `kind`, not +//! an address. The address claim is about the first 16 bytes of the +//! `FixedSizeBinary(512)` payload blob. + +use lance_graph::graph::cycle_sink::{LanceCycleWriter, EPISODIC_WITNESS_BYTES}; +use lance_graph_contract::canonical_node::{ + classid_read_mode, EdgeBlock, NodeGuid, NodeRow, NodeRowPacket, TailVariant, +}; +use lance_graph_contract::collapse_gate::MailboxId; +use lance_graph_contract::kanban::{ExecTarget, KanbanColumn, KanbanMove}; +use lance_graph_contract::scheduler::DatasetVersion; +use lance_graph_planner::persist_sink::{ + persist_cycle, CommitOutcome, CycleFrame, CycleId, SweepSlot, +}; + +/// The canonical consumer incantation — the registry picks the tail, not us. +fn mint(classid: u32, leaf: u16, family: u32, identity: u32) -> NodeGuid { + let tv = classid_read_mode(classid).tail_variant; + NodeGuid::mint_for(tv, classid, 0x1111, 0x2222, 0x3333, leaf, family, identity) +} + +fn row_of(guid: NodeGuid, fill: u8) -> NodeRow { + NodeRow { + key: guid, + edges: EdgeBlock::default(), + value: [fill; 480], + } +} + +fn packet_bytes(rows: &[NodeRow], cycle: u32) -> Vec { + use lance_graph_contract::soa_envelope::SoaEnvelope; + NodeRowPacket::new(rows, cycle).as_le_bytes().to_vec() +} + +fn mv(owner: MailboxId) -> KanbanMove { + KanbanMove { + mailbox: owner, + from: KanbanColumn::Planning, + to: KanbanColumn::CognitiveWork, + witness_chain_position: 1, + exec: ExecTarget::Elixir, + } +} + +fn cast(cycle: u64, sp: u64, owner: MailboxId, row: u64, payload: Vec) -> SweepSlot { + SweepSlot { + cycle: CycleId(cycle), + stream_position: sp, + owner, + row, + paired_move: Some(mv(owner)), + payload, + } +} + +/// Write one cycle per (row, guid, fill) and return the writer's head. +async fn write_cycles( + w: &mut LanceCycleWriter, + units: &[(u64, NodeGuid, u8)], +) -> Vec<(u64, Vec, u64)> { + let mut out = Vec::new(); + let mut base = w.head(); + for (i, (row, g, fill)) in units.iter().enumerate() { + let cycle = (i + 1) as u64; + let bytes = packet_bytes(&[row_of(*g, *fill)], cycle as u32); + assert_eq!(bytes.len(), EPISODIC_WITNESS_BYTES); + let outcome = persist_cycle( + w, + CycleFrame::new(CycleId(cycle), base), + vec![cast(cycle, 0, 7, *row, bytes.clone())], + ) + .await + .unwrap(); + match outcome { + CommitOutcome::Committed { version, .. } => { + assert_eq!( + version, + DatasetVersion(base.0 + 1), + "cycle {cycle} minted more than one version" + ); + base = version; + out.push((cycle, bytes, *row)); + } + other => panic!("cycle {cycle} did not commit: {other:?}"), + } + } + out +} + +/// RAIL V3 (the live rail) — bytes survive, and the V3 address decodes as V3. +#[tokio::test] +async fn a_v3_address_survives_two_cycles_and_a_reopen_byte_for_byte() { + let c = NodeGuid::CLASSID_OSINT_V3; + assert_eq!( + classid_read_mode(c).tail_variant, + TailVariant::V3, + "this rail is only meaningful if the classid actually registers V3" + ); + + let g1 = mint(c, 0x00AA, 0x1234, 0x5678); + let g2 = mint(c, 0x00BB, 0x4321, 0x8765); + assert_ne!( + g1.as_bytes(), + g2.as_bytes(), + "leaf must discriminate on the V3 rail — if this fails the V1 fallback ran" + ); + + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("cycles.lance"); + let p = path.to_str().unwrap(); + + let mut w = LanceCycleWriter::open(p).await.unwrap(); + let written = write_cycles(&mut w, &[(100, g1, 0xA1), (200, g2, 0xB2)]).await; + assert_eq!(w.head(), DatasetVersion(2)); + + drop(w); + let w2 = LanceCycleWriter::open(p).await.unwrap(); + assert_eq!(w2.head(), DatasetVersion(2), "head did not survive restart"); + + let expect = [g1, g2]; + for (i, (cycle, submitted, row)) in written.iter().enumerate() { + let img = w2.scan_image(CycleId(*cycle)).await.unwrap(); + let back = img + .get(row) + .unwrap_or_else(|| panic!("cycle {cycle} row {row} missing after reopen")); + + // (1) raw bytes + assert_eq!(back, submitted, "cycle {cycle} payload bytes changed"); + assert_eq!( + &back[0..16], + expect[i].as_bytes(), + "cycle {cycle} payload[0..16] is not the submitted NodeGuid" + ); + // the value slab travelled too, so (1) is not a degenerate zero-match + assert_eq!(back[32], if i == 0 { 0xA1 } else { 0xB2 }); + } + + // (2) V3 semantic decode — the V3-aware accessors, never the V1 ones. + assert_eq!(g1.leaf(), 0x00AA); + assert_eq!(g1.family_v2(), 0x1234); + assert_eq!(g1.identity_v2(), 0x5678); + assert_eq!(g2.leaf(), 0x00BB); + assert_eq!(g2.family_v2(), 0x4321); + assert_eq!(g2.identity_v2(), 0x8765); + assert_ne!(g1.to_hex_v2(), g2.to_hex_v2(), "V3 rail addresses differ"); +} + +/// (3) The selection claim: the V1 accessors are WRONG on a V3 key, and wrong +/// in the specific way that looks like success — many distinct addresses +/// collapsing onto one apparent identity while every byte round-trips. +#[test] +fn reading_a_v3_key_through_the_v1_accessor_collapses_distinct_addresses() { + let c = NodeGuid::CLASSID_OSINT_V3; + assert_eq!(classid_read_mode(c).tail_variant, TailVariant::V3); + + // 256 genuinely distinct V3 addresses, varying only the leaf tier. + let keys: Vec = (0..256u16).map(|l| mint(c, l, 0x1234, 0x5678)).collect(); + + let distinct_bytes: std::collections::HashSet<[u8; 16]> = + keys.iter().map(|k| *k.as_bytes()).collect(); + assert_eq!(distinct_bytes.len(), 256, "the 256 addresses are distinct"); + + // The V3-aware full-tier decode keeps all 256 apart. `local_key_v2` alone + // does NOT and must not: leaf is an HHTL routing tier (bytes 10..12), part + // of the addressing PREFIX, deliberately outside the basin-local key + // (bytes 12..16). Both facts are pinned, because picking the wrong one of + // the two V3 accessors is the same class of error as picking the V1 one. + let v3_view: std::collections::HashSet = keys.iter().map(NodeGuid::to_hex_v2).collect(); + assert_eq!( + v3_view.len(), + 256, + "the V3-aware full-tier decode must preserve all 256" + ); + let v3_basin: std::collections::HashSet = + keys.iter().map(NodeGuid::local_key_v2).collect(); + assert_eq!( + v3_basin.len(), + 1, + "leaf is a routing tier, not part of the basin-local key — these 256 \ + addresses share one basin by construction" + ); + + let v1_view: std::collections::HashSet = keys.iter().map(NodeGuid::identity).collect(); + assert_eq!( + v1_view.len(), + 1, + "PINNED FALSE-GREEN: the V1 identity() accessor reads bytes 13..16 of a \ + V3 key, so 256 distinct addresses present as ONE. If this assertion \ + ever fails, the tail layout moved and every V1-accessor call site on \ + V3 data must be re-audited — do not simply update the number." + ); +} + +/// RAIL V1 (legacy control) — a pre-flip class still decodes as V1. No new unit +/// is minted this way; this only proves old rows stay readable. +#[tokio::test] +async fn a_legacy_v1_address_still_round_trips_and_decodes_as_v1() { + let c = NodeGuid::CLASSID_OSINT; // pre-V3 exemplar + assert_eq!( + classid_read_mode(c).tail_variant, + TailVariant::V1, + "this control is only meaningful if the classid still registers V1" + ); + + let g = mint(c, 0, 0x00AB_CDEF, 0x0012_3456); + assert_eq!(g.family(), 0x00AB_CDEF); + assert_eq!(g.identity(), 0x0012_3456); + + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("cycles.lance"); + let p = path.to_str().unwrap(); + + let mut w = LanceCycleWriter::open(p).await.unwrap(); + let written = write_cycles(&mut w, &[(300, g, 0xC3)]).await; + drop(w); + + let w2 = LanceCycleWriter::open(p).await.unwrap(); + let (cycle, submitted, row) = &written[0]; + let img = w2.scan_image(CycleId(*cycle)).await.unwrap(); + let back = img.get(row).expect("legacy row missing after reopen"); + + assert_eq!(back, submitted, "V1 payload bytes changed"); + assert_eq!(&back[0..16], g.as_bytes(), "V1 payload[0..16] != G"); +} From 01829f6a786b6be5a7753ea4e74ece9b788399b9 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 6 Sep 2026 17:02:58 +0000 Subject: [PATCH 2/4] board: regenerate the supersession index for the new persist_cycle reference MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The `Supersession index is current` gate went red on this branch. The cause is this PR's own: the index counts references per ruled symbol, and the new canonical-witness probe calls `persist_cycle`, moving its plan-reference count from 10 to 11. Regenerated with the documented command and verified idempotent (a second run diffs clean). The prediction in the PR body that this gate would not fire was wrong. It read the index's board inputs as `entries/*.md` + `EPIPHANIES.md` and concluded a STATUS_BOARD-only change could not move it — but `crates/` is also an input, and adding a test that calls a ruled symbol is a `crates/` change. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_012wrzeZAdwGYTCKoxamwQht --- .claude/board/SUPERSESSION-INDEX.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.claude/board/SUPERSESSION-INDEX.md b/.claude/board/SUPERSESSION-INDEX.md index c2921a54a..2fdd26b94 100644 --- a/.claude/board/SUPERSESSION-INDEX.md +++ b/.claude/board/SUPERSESSION-INDEX.md @@ -53,7 +53,7 @@ coverage instead. | `StepMask` | BLOCKED | — | 3 | 9 | 4 | | `commit_to_l4` | BLOCKED | — | 2 | 2 | 0 | | `dispatch_busdto` | BLOCKED | — | 3 | 8 | 5 | -| `persist_cycle` | BLOCKED | — | 10 | 8 | 5 | +| `persist_cycle` | BLOCKED | — | 11 | 8 | 5 | | `CognitiveMarkers` | REPURPOSE | `Commit` | 1 | 0 | 0 | | `DominoCascade` | REPURPOSE | `Commit` | 7 | 1 | 0 | | `GateDecision` | REPURPOSE | — | 25 | 27 | 24 | From fc26869720e2243003c97851cd3797869d7c43d1 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 6 Sep 2026 17:07:36 +0000 Subject: [PATCH 3/4] probes: the old snapshot path yields the full tripartite change set MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit TEST E asked whether store B's version-to-version comparison can produce inserted / updated / removed without any lance delta API. It can. With V1={A,B,D} and V2={A,B',C}, GraphDiff reports inserted={C} and updated={B}. It has no removed-nodes field, and D appears nowhere in it — but the probe recomputes removed={D} from exactly the two seal maps diff() already materialises before throwing their difference away. So the gap is an API omission, not information the store lost. The seal gate is exercised on both sides: a removal reads Staunen, a version compared against itself reads Wisdom. An unchanged node A is asserted to appear in neither change set, so "updated" cannot degenerate into "present". This makes historical snapshot comparison the correctness oracle for the changed set. Native lance delta, wherever it works, is an accelerator measured against this, not a replacement for it. The companion probe test_c_cycle_payload_roundtrip.rs was written alongside this one and is deliberately NOT landed: D-TEMPORAL-1 already proves the stronger V1/V3 byte, address, restart and version result, and a second green test of the same claim adds review cost without adding a falsifier. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_012wrzeZAdwGYTCKoxamwQht --- .claude/board/STATUS_BOARD.md | 1 + .../tests/test_e_old_diff_tripartite.rs | 232 ++++++++++++++++++ 2 files changed, 233 insertions(+) create mode 100644 crates/lance-graph/tests/test_e_old_diff_tripartite.rs diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index 2c3c3d3a5..98f5c4891 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -20,6 +20,7 @@ identity) is real and probe-gated. | D-TEMPORAL-1 | **Canonical witness identity probe** (`crates/lance-graph/tests/canonical_witness_identity_probe.rs`): does canonical address identity survive the A-store round trip on lance 11? `NodeGuid → NodeRow → NodeRowPacket::as_le_bytes → cycle commit → drop writer → reopen → scan_image`. Minted through `mint_for(classid_read_mode(c).tail_variant, …)`, never a hardcoded constructor | **GREEN, both rails.** V3 (`CLASSID_OSINT_V3`): payload bytes byte-exact across two cycles and a reopen; `payload[0..16] == G.as_bytes()`; value slab survived (so equality is not a zero-match); each cycle minted exactly `base+1`; head recovered at V2 after restart. V1 legacy control (`CLASSID_OSINT`) round-trips too. **Scope:** proves byte and address fidelity of the store; does NOT prove production reachability — `LanceCycleWriter` still has zero non-test callers (see D-MW-P2) | | D-TEMPORAL-1a | **Pinned false-green: the V1 accessor on a V3 key.** 256 V3 addresses varying only in `leaf` → 256 distinct 16-byte keys, 256 distinct `to_hex_v2()`, but **1** distinct `identity()`. Byte fidelity does not imply address fidelity | **Pinned as a positive assertion.** Also pins the second half, which is not a defect: `local_key_v2` yields **1** across the same set BY CONSTRUCTION — `leaf` is the 4th HHTL routing tier (bytes 10..12), part of the addressing prefix, deliberately outside the basin-local key (bytes 12..16). So "use the V3 accessor" is two choices, not one, and the wrong V3 one fails like the V1 one. Do not update the numbers if the assertion breaks — re-audit the call sites | | D-V3-MINT-1 | **V3 mint reachability** (`crates/lance-graph-contract/tests/v3_mint_reachability_probe.rs`): `mint_for`'s V2/V3 arm is gated on `guid-v2-tail`, which is NOT in `default`; the registry's V3 entries are gated on `guid-v3-tail`, which is. Two different features, so premise and conclusion can come apart — is a V3-registered classid silently minted into the deprecated V1 layout? | **GREEN — no defect.** `guid-v3-tail = ["guid-v2-tail"]` (`Cargo.toml:69`), so the implication holds. Measured, not inferred: `tail_variant=V3 guid-v3-tail=true guid-v2-tail=true leaf_is_live=true`, key bytes show the V2/V3 carving with `leaf` live at byte 10. Discriminator is `leaf` (the V3 arm feeds it to `new_v2`; the V1 fallback discards it), so the probe stays valid if the feature graph is ever flattened | +| D-TEMPORAL-2 | **Old-path tripartite change set** (`crates/lance-graph/tests/test_e_old_diff_tripartite.rs`): can store B's snapshot comparison produce `inserted / updated / removed` without any lance delta API? `V1={A,B,D}`, `V2={A,B',C}` | **GREEN.** `inserted={C}` and `updated={B}` from `GraphDiff`; `removed={D}` DERIVED from the same two seal maps `diff()` already materialises (`versioned.rs:550-554`) before discarding the difference — so the missing `removed_nodes` field is an **API omission, not information loss**. Seal gate two-sided: removal ⇒ `Staunen`, self-compare ⇒ `Wisdom`. Anti-vacuity: unchanged `A` appears in neither set. Blind spot pinned — if a `removed_nodes` field lands, the assertion must be re-pinned deliberately. **Consequence: historical snapshot comparison is the correctness ORACLE; native lance delta is an optional accelerator, never the architecture** | | D-MW-P2 | **MemWAL crash-atomicity probe** (`crates/lance-graph/tests/memwal_atomicity_probe.rs`): the one measurement `.claude/temporal/08-memwal-vs-batchwriter.md` named as overturning its FOLD verdict — SIGKILL mid-flush on a 5 000-row put | **In PR** #1198. **GREEN.** A1 clean-drop control 5000; A2 SIGKILL sweep 5–500 ms → `0,0,0,5000,5000,5000,5000`. Boundary bracketed on both sides, no partial batch. FOLD stands. Scope recorded as NOT settled: one put, local store, no concurrent writers, no fencing path, and NOT the landing-rows-plus-frame-row composite (that is `LanceCycleWriter`'s, which has zero callers) | | D-GTM-0n / P3 | **Ternlog chaining amortization** (`ndarray/examples/ternlog_amortization_probe.rs`, re-run `--release` 2026-09-06): does chaining pay on THIS mask shape? Pre-registered pass criterion = per-constraint cost flat in K, with a **K=1 control** (nothing to chain ⇒ no win) | **GREEN, bounded.** `T3/T1` bottoms at **0.50** by K=8; T3 total flat in K (57.6 ns @ K=4 → 54.1 ns @ K=32, 16× more constraints). K=1 control reads **1.03** as it must. Two bounds ride with the number and may not be dropped when it is cited: the win is **L2-residency-contingent** (bandwidth `138→15 GB/s` past L2, ratio back to 1.03 at a 512 KiB mask), and mask loses to sparse below **0.1%** active. **Does NOT retire the standing caveat** — 2× on one of three passes still cannot account for 5×. Gates Stage-3 SPEED claims only; the shape/correctness case never depended on it | | D-ALPHA-1b | **`AlphaMask::zip` release-mode length guard** (`lance-graph-contract/src/alpha.rs`): the only guard was `debug_assert_eq!`, compiled out in release, so mismatched-length operands produced a mask claiming `self.len` addresses over the SHORTER operand's word count — `count`/`is_empty` under-report silently, `contains`/`materialize_ordinals` panic at a distance | **Fixed** — promoted to `assert_eq!`. Defect reproduced in release BEFORE the fix (the `should_panic` falsifier reported "test did not panic as expected"); two-sided falsifiers both disable-verified. No API change, no external callers. Stage 1b of `.claude/temporal/09-plan.md`, now DONE | diff --git a/crates/lance-graph/tests/test_e_old_diff_tripartite.rs b/crates/lance-graph/tests/test_e_old_diff_tripartite.rs new file mode 100644 index 000000000..bd0d3d47e --- /dev/null +++ b/crates/lance-graph/tests/test_e_old_diff_tripartite.rs @@ -0,0 +1,232 @@ +//! TEST E — can the OLD path (store B, `VersionedGraph`) produce the tripartite +//! change set `inserted / updated / removed`? +//! +//! Scenario: `V1 = {A, B, D}`, `V2 = {A, B', C}`. Desired: +//! +//! - `inserted = to - from = {C}` +//! - `updated = seals differ on the intersection = {B}` +//! - `removed = from - to = {D}` +//! +//! `GraphDiff` (`graph/versioned.rs:70`) carries `new_nodes` and +//! `modified_nodes` and NO removed-nodes field; `diff()` computes +//! `to_ids.difference(&from_ids)` (`:556`) and the seal-differing intersection +//! (`:560`) but never `from_ids.difference(&to_ids)` — even though both sets +//! are materialized side by side at `:553-554`. +//! +//! This probe asserts the two sets the API does expose, then DERIVES the third +//! from exactly the material `diff()` already reads (`extract_node_seals` over +//! both checked-out versions), showing the omission is API-only rather than an +//! information loss in the store. +//! +//! Modelled on `tests/lance_row_identity_probe.rs` (D-LNC-2) for the write +//! idioms; no `_rowaddr` here — this probe is about the change SETS. + +use std::collections::{BTreeMap, BTreeSet}; +use std::sync::Arc; + +use arrow_array::builder::FixedSizeBinaryBuilder; +use arrow_array::{Array, FixedSizeBinaryArray, RecordBatch, UInt32Array}; +use arrow_schema::{DataType, SchemaRef}; +use futures::TryStreamExt; +use lance::dataset::Dataset; +use lance_graph::graph::blasgraph::columnar::{EdgeSchema, FingerprintSchema, NodeSchema}; +use lance_graph::graph::versioned::{GraphSealStatus, VersionedGraph}; + +/// (node_id, seal tag): a changed tag on the same id is an UPDATE, not an insert. +type Round = Vec<(u32, u8)>; + +const A: u32 = 1; +const B: u32 = 2; +const C: u32 = 3; +const D: u32 = 4; + +const V1: &[(u32, u8)] = &[(A, 0x10), (B, 0x10), (D, 0x10)]; +const V2: &[(u32, u8)] = &[(A, 0x10), (B, 0x20), (C, 0x10)]; + +fn fsb_width(schema: &SchemaRef, name: &str) -> i32 { + match schema.field_with_name(name).expect(name).data_type() { + DataType::FixedSizeBinary(w) => *w, + other => panic!("{name}: expected FixedSizeBinary, got {other:?}"), + } +} + +fn seal_bytes(node_id: u32, tag: u8, width: usize) -> Vec { + let mut v = vec![tag; width]; + v[..4].copy_from_slice(&node_id.to_le_bytes()); + v +} + +fn node_batch(round: &Round) -> RecordBatch { + let schema = NodeSchema::arrow_schema_ref(); + let plane_w = fsb_width(&schema, "plane_s"); + let seal_w = fsb_width(&schema, "seal_s"); + let n = round.len(); + let mut planes = (0..3) + .map(|_| FixedSizeBinaryBuilder::with_capacity(n, plane_w)) + .collect::>(); + let mut seals = (0..3) + .map(|_| FixedSizeBinaryBuilder::with_capacity(n, seal_w)) + .collect::>(); + let plane = vec![0u8; plane_w as usize]; + for &(id, tag) in round { + for p in planes.iter_mut() { + p.append_value(&plane).unwrap(); + } + for (k, s) in seals.iter_mut().enumerate() { + s.append_value(seal_bytes(id, tag.wrapping_add(k as u8), seal_w as usize)) + .unwrap(); + } + } + let ids: Vec = round.iter().map(|r| r.0).collect(); + let enc: Vec = round.iter().map(|_| 1).collect(); + let mut cols: Vec> = vec![Arc::new(UInt32Array::from(ids))]; + for p in planes.iter_mut() { + cols.push(Arc::new(p.finish())); + } + for s in seals.iter_mut() { + cols.push(Arc::new(s.finish())); + } + cols.push(Arc::new(UInt32Array::from(enc))); + RecordBatch::try_new(schema, cols).unwrap() +} + +fn empty_batch(schema: SchemaRef) -> RecordBatch { + let cols = schema + .fields() + .iter() + .map(|f| arrow_array::new_empty_array(f.data_type())) + .collect(); + RecordBatch::try_new(schema, cols).unwrap() +} + +/// `node_id -> seal_s ++ seal_p ++ seal_o` — the same material +/// `VersionedGraph::extract_node_seals` builds (`graph/versioned.rs:660`), +/// reproduced here because that helper is private. +async fn seals_at(ds: &Dataset) -> BTreeMap> { + let batches: Vec = ds + .scan() + .try_into_stream() + .await + .expect("scan") + .try_collect() + .await + .expect("collect"); + let mut out = BTreeMap::new(); + for b in &batches { + let ids = b + .column_by_name("node_id") + .and_then(|c| c.as_any().downcast_ref::()) + .expect("node_id"); + let seals: Vec<&FixedSizeBinaryArray> = ["seal_s", "seal_p", "seal_o"] + .iter() + .map(|c| { + b.column_by_name(c) + .and_then(|x| x.as_any().downcast_ref::()) + .expect("seal column") + }) + .collect(); + for i in 0..b.num_rows() { + let mut all = Vec::new(); + for s in &seals { + all.extend_from_slice(s.value(i)); + } + out.insert(ids.value(i), all); + } + } + out +} + +#[tokio::test(flavor = "multi_thread")] +async fn old_diff_gives_inserted_and_updated_and_removed_is_derivable() { + let tmp = tempfile::tempdir().expect("tempdir"); + let graph = VersionedGraph::local(tmp.path().to_str().unwrap()); + + let mut versions = Vec::new(); + for round in [V1.to_vec(), V2.to_vec()] { + let v = graph + .commit_encounter_round( + node_batch(&round), + empty_batch(EdgeSchema::arrow_schema_ref()), + empty_batch(FingerprintSchema::arrow_schema_ref()), + ) + .await + .expect("commit"); + versions.push(v); + } + let (v1, v2) = (versions[0], versions[1]); + + // ---- what the API DOES report ---------------------------------------- + let diff = graph.diff(v1, v2).await.expect("diff V1->V2"); + + let inserted: BTreeSet = diff.new_nodes.iter().copied().collect(); + assert_eq!( + inserted, + BTreeSet::from([C]), + "inserted = to - from = {{C}} (versioned.rs:556)" + ); + + let updated: BTreeSet = diff.modified_nodes.iter().copied().collect(); + assert_eq!( + updated, + BTreeSet::from([B]), + "updated = seal-differing intersection = {{B}} (versioned.rs:560)" + ); + + // Anti-vacuity: A is in both versions with an UNCHANGED seal and must + // appear in neither set — otherwise "updated" would just mean "present". + assert!( + !updated.contains(&A) && !inserted.contains(&A), + "A is quiet" + ); + + // ---- the omission: GraphDiff has no removed-nodes field --------------- + // D is gone in V2 and appears NOWHERE in the diff. Pinned; if a + // `removed_nodes` field lands, this assertion must be re-pinned deliberately. + assert!( + !inserted.contains(&D) && !updated.contains(&D), + "PINNED BLIND SPOT: removed node D is absent from GraphDiff: {diff:?}" + ); + + // ---- removals ARE detected, just not reported ------------------------- + // graph_seal_check walks `from_seals.keys()` and returns Staunen on a + // removal (versioned.rs:632-637) — so the store distinguishes the states. + assert_eq!( + graph.graph_seal_check(v1, v2).await.expect("seal check"), + GraphSealStatus::Staunen, + "a removal is Staunen" + ); + // The silence half: same version against itself is Wisdom. + assert_eq!( + graph.graph_seal_check(v2, v2).await.expect("seal check"), + GraphSealStatus::Wisdom, + "no change is Wisdom" + ); + + // ---- DERIVATION: removed = from - to, from the same material ---------- + // `diff()` materializes exactly these two maps (`from_nodes` / `to_nodes`, + // versioned.rs:550-551) and their key sets (`from_ids` / `to_ids`, :553-554) + // before throwing the difference away. Recomputing it here proves the + // information is present in the store, not lost. + let from_seals = seals_at(&graph.at_version(v1).await.expect("checkout V1")).await; + let to_seals = seals_at(&graph.at_version(v2).await.expect("checkout V2")).await; + let from_ids: BTreeSet = from_seals.keys().copied().collect(); + let to_ids: BTreeSet = to_seals.keys().copied().collect(); + + let removed: BTreeSet = from_ids.difference(&to_ids).copied().collect(); + assert_eq!( + removed, + BTreeSet::from([D]), + "removed = from - to = {{D}} — derivable from the SAME read diff() does" + ); + + // And the derivation reproduces the two sets the API does expose, so the + // three are one pass over one pair of maps. + let derived_inserted: BTreeSet = to_ids.difference(&from_ids).copied().collect(); + assert_eq!(derived_inserted, inserted); + let derived_updated: BTreeSet = to_ids + .intersection(&from_ids) + .copied() + .filter(|id| from_seals.get(id) != to_seals.get(id)) + .collect(); + assert_eq!(derived_updated, updated); +} From 91cfe1c12a6b9b35371ebcf42fe10c6d98120251 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 6 Sep 2026 18:02:46 +0000 Subject: [PATCH 4/4] probes: isolate the removal seal-check, which was confounded MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CodeRabbit flagged that the removal probe could pass without validating removal handling. It was right. graph_seal_check returns on its FIRST divergence while walking to_seals, so on V1={A,B,D} -> V2={A,B',C} the update to B or the insert of C tripped it long before the removal branch ran. The assertion "a removal is Staunen" would have held identically with that branch deleted outright, which makes it a guard that cannot distinguish what it claims to test. Adds V3={A,B'} — V2 minus C, nothing else touched — so the Staunen arm now runs on a transition whose only change is a removal. That reaches the removal branch or nothing does. The isolation also sharpens the blind spot rather than merely documenting it: for a pure removal GraphDiff is entirely empty AND its own seal_status reads Wisdom, while graph_seal_check calls the same version pair Staunen. The two surfaces disagree, which is a stronger statement than "the removed set is missing". Also moves the four new STATUS_BOARD rows to the section's append point; they were inserted mid-table because the patch anchored on a convenient unique string rather than on the ordering convention. Doc comments added to the nine undocumented test helpers, each stating a contract rather than narrating. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_012wrzeZAdwGYTCKoxamwQht --- .claude/board/STATUS_BOARD.md | 8 +-- .../tests/v3_mint_reachability_probe.rs | 8 +++ .../tests/canonical_witness_identity_probe.rs | 14 ++++ .../tests/test_e_old_diff_tripartite.rs | 69 +++++++++++++++++-- 4 files changed, 89 insertions(+), 10 deletions(-) diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index 98f5c4891..923c0781b 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -17,14 +17,14 @@ identity) is real and probe-gated. | D-LNC-5 | Replace `VersionedGraph::diff`'s two-version full materialization (`read_all_batches` = `scan().try_into_stream()`, `versioned.rs:645`) with lance 11's native "row ids deleted between two versions". zero-copy-warden LENS-CLEAN; byte-identical output | Queued (blocked on D-LNC-3 — the API is lance 11) | | D-LNC-6 | Session reuse: end `Dataset::open`-per-call (17 sites in `lance-graph`); instrumented open-count strictly decreases, results identical | Queued (independent; lance 10 ships shared session) | | D-LNC-5a | **Delta version-columns probe** (pre-registered, `crates/lance-graph/tests/delta_version_columns_probe.rs`): are `_row_created_at_version` / `_row_last_updated_at_version` populated WITHOUT stable row ids? Stable-row-id arm is the CONTROL so a null is attributable to lance, not the apparatus | **In PR** #1198. **RED, both arms.** insert: control 2, physical **0**; update: control HELD (version advanced AND value changed), stable **1**, physical **0**. ⊘ the update arm's first run was INCONCLUSIVE — its control did not hold because `MergeInsertBuilder::when_matched` defaults to `DoNothing`, so no update ever committed (caught by a CodeRabbit finding on #1198). **Consequence: D-LNC-5's insert/update half is gated behind the stable-row-id decision, same as the delete arm** | -| D-TEMPORAL-1 | **Canonical witness identity probe** (`crates/lance-graph/tests/canonical_witness_identity_probe.rs`): does canonical address identity survive the A-store round trip on lance 11? `NodeGuid → NodeRow → NodeRowPacket::as_le_bytes → cycle commit → drop writer → reopen → scan_image`. Minted through `mint_for(classid_read_mode(c).tail_variant, …)`, never a hardcoded constructor | **GREEN, both rails.** V3 (`CLASSID_OSINT_V3`): payload bytes byte-exact across two cycles and a reopen; `payload[0..16] == G.as_bytes()`; value slab survived (so equality is not a zero-match); each cycle minted exactly `base+1`; head recovered at V2 after restart. V1 legacy control (`CLASSID_OSINT`) round-trips too. **Scope:** proves byte and address fidelity of the store; does NOT prove production reachability — `LanceCycleWriter` still has zero non-test callers (see D-MW-P2) | -| D-TEMPORAL-1a | **Pinned false-green: the V1 accessor on a V3 key.** 256 V3 addresses varying only in `leaf` → 256 distinct 16-byte keys, 256 distinct `to_hex_v2()`, but **1** distinct `identity()`. Byte fidelity does not imply address fidelity | **Pinned as a positive assertion.** Also pins the second half, which is not a defect: `local_key_v2` yields **1** across the same set BY CONSTRUCTION — `leaf` is the 4th HHTL routing tier (bytes 10..12), part of the addressing prefix, deliberately outside the basin-local key (bytes 12..16). So "use the V3 accessor" is two choices, not one, and the wrong V3 one fails like the V1 one. Do not update the numbers if the assertion breaks — re-audit the call sites | -| D-V3-MINT-1 | **V3 mint reachability** (`crates/lance-graph-contract/tests/v3_mint_reachability_probe.rs`): `mint_for`'s V2/V3 arm is gated on `guid-v2-tail`, which is NOT in `default`; the registry's V3 entries are gated on `guid-v3-tail`, which is. Two different features, so premise and conclusion can come apart — is a V3-registered classid silently minted into the deprecated V1 layout? | **GREEN — no defect.** `guid-v3-tail = ["guid-v2-tail"]` (`Cargo.toml:69`), so the implication holds. Measured, not inferred: `tail_variant=V3 guid-v3-tail=true guid-v2-tail=true leaf_is_live=true`, key bytes show the V2/V3 carving with `leaf` live at byte 10. Discriminator is `leaf` (the V3 arm feeds it to `new_v2`; the V1 fallback discards it), so the probe stays valid if the feature graph is ever flattened | -| D-TEMPORAL-2 | **Old-path tripartite change set** (`crates/lance-graph/tests/test_e_old_diff_tripartite.rs`): can store B's snapshot comparison produce `inserted / updated / removed` without any lance delta API? `V1={A,B,D}`, `V2={A,B',C}` | **GREEN.** `inserted={C}` and `updated={B}` from `GraphDiff`; `removed={D}` DERIVED from the same two seal maps `diff()` already materialises (`versioned.rs:550-554`) before discarding the difference — so the missing `removed_nodes` field is an **API omission, not information loss**. Seal gate two-sided: removal ⇒ `Staunen`, self-compare ⇒ `Wisdom`. Anti-vacuity: unchanged `A` appears in neither set. Blind spot pinned — if a `removed_nodes` field lands, the assertion must be re-pinned deliberately. **Consequence: historical snapshot comparison is the correctness ORACLE; native lance delta is an optional accelerator, never the architecture** | | D-MW-P2 | **MemWAL crash-atomicity probe** (`crates/lance-graph/tests/memwal_atomicity_probe.rs`): the one measurement `.claude/temporal/08-memwal-vs-batchwriter.md` named as overturning its FOLD verdict — SIGKILL mid-flush on a 5 000-row put | **In PR** #1198. **GREEN.** A1 clean-drop control 5000; A2 SIGKILL sweep 5–500 ms → `0,0,0,5000,5000,5000,5000`. Boundary bracketed on both sides, no partial batch. FOLD stands. Scope recorded as NOT settled: one put, local store, no concurrent writers, no fencing path, and NOT the landing-rows-plus-frame-row composite (that is `LanceCycleWriter`'s, which has zero callers) | | D-GTM-0n / P3 | **Ternlog chaining amortization** (`ndarray/examples/ternlog_amortization_probe.rs`, re-run `--release` 2026-09-06): does chaining pay on THIS mask shape? Pre-registered pass criterion = per-constraint cost flat in K, with a **K=1 control** (nothing to chain ⇒ no win) | **GREEN, bounded.** `T3/T1` bottoms at **0.50** by K=8; T3 total flat in K (57.6 ns @ K=4 → 54.1 ns @ K=32, 16× more constraints). K=1 control reads **1.03** as it must. Two bounds ride with the number and may not be dropped when it is cited: the win is **L2-residency-contingent** (bandwidth `138→15 GB/s` past L2, ratio back to 1.03 at a 512 KiB mask), and mask loses to sparse below **0.1%** active. **Does NOT retire the standing caveat** — 2× on one of three passes still cannot account for 5×. Gates Stage-3 SPEED claims only; the shape/correctness case never depended on it | | D-ALPHA-1b | **`AlphaMask::zip` release-mode length guard** (`lance-graph-contract/src/alpha.rs`): the only guard was `debug_assert_eq!`, compiled out in release, so mismatched-length operands produced a mask claiming `self.len` addresses over the SHORTER operand's word count — `count`/`is_empty` under-report silently, `contains`/`materialize_ordinals` panic at a distance | **Fixed** — promoted to `assert_eq!`. Defect reproduced in release BEFORE the fix (the `should_panic` falsifier reported "test did not panic as expected"); two-sided falsifiers both disable-verified. No API change, no external callers. Stage 1b of `.claude/temporal/09-plan.md`, now DONE | | D-LNC-7 | **One-WAL ruling**: `LanceCycleWriter`'s watermark WAL vs lance MemWAL — `applied_through` ≡ index catch-up position; two under one commit = two truths. Plan recommends keep-ours until D-LNC-3, then re-decide | **Operator decision** | +| D-TEMPORAL-1 | **Canonical witness identity probe** (`crates/lance-graph/tests/canonical_witness_identity_probe.rs`): does canonical address identity survive the A-store round trip on lance 11? `NodeGuid → NodeRow → NodeRowPacket::as_le_bytes → cycle commit → drop writer → reopen → scan_image`. Minted through `mint_for(classid_read_mode(c).tail_variant, …)`, never a hardcoded constructor | **GREEN, both rails.** V3 (`CLASSID_OSINT_V3`): payload bytes byte-exact across two cycles and a reopen; `payload[0..16] == G.as_bytes()`; value slab survived (so equality is not a zero-match); each cycle minted exactly `base+1`; head recovered at V2 after restart. V1 legacy control (`CLASSID_OSINT`) round-trips too. **Scope:** proves byte and address fidelity of the store; does NOT prove production reachability — `LanceCycleWriter` still has zero non-test callers (see D-MW-P2) | +| D-TEMPORAL-1a | **Pinned false-green: the V1 accessor on a V3 key.** 256 V3 addresses varying only in `leaf` → 256 distinct 16-byte keys, 256 distinct `to_hex_v2()`, but **1** distinct `identity()`. Byte fidelity does not imply address fidelity | **Pinned as a positive assertion.** Also pins the second half, which is not a defect: `local_key_v2` yields **1** across the same set BY CONSTRUCTION — `leaf` is the 4th HHTL routing tier (bytes 10..12), part of the addressing prefix, deliberately outside the basin-local key (bytes 12..16). So "use the V3 accessor" is two choices, not one, and the wrong V3 one fails like the V1 one. Do not update the numbers if the assertion breaks — re-audit the call sites | +| D-V3-MINT-1 | **V3 mint reachability** (`crates/lance-graph-contract/tests/v3_mint_reachability_probe.rs`): `mint_for`'s V2/V3 arm is gated on `guid-v2-tail`, which is NOT in `default`; the registry's V3 entries are gated on `guid-v3-tail`, which is. Two different features, so premise and conclusion can come apart — is a V3-registered classid silently minted into the deprecated V1 layout? | **GREEN — no defect.** `guid-v3-tail = ["guid-v2-tail"]` (`Cargo.toml:69`), so the implication holds. Measured, not inferred: `tail_variant=V3 guid-v3-tail=true guid-v2-tail=true leaf_is_live=true`, key bytes show the V2/V3 carving with `leaf` live at byte 10. Discriminator is `leaf` (the V3 arm feeds it to `new_v2`; the V1 fallback discards it), so the probe stays valid if the feature graph is ever flattened | +| D-TEMPORAL-2 | **Old-path tripartite change set** (`crates/lance-graph/tests/test_e_old_diff_tripartite.rs`): can store B's snapshot comparison produce `inserted / updated / removed` without any lance delta API? `V1={A,B,D}`, `V2={A,B',C}` | **GREEN.** `inserted={C}` and `updated={B}` from `GraphDiff`; `removed={D}` DERIVED from the same two seal maps `diff()` already materialises (`versioned.rs:550-554`) before discarding the difference — so the missing `removed_nodes` field is an **API omission, not information loss**. Seal gate two-sided AND isolated: the `Staunen` arm runs on a THIRD version `V3={A,B'}` whose only change is dropping `C` — on `V1→V2` that assertion was CONFOUNDED, because `graph_seal_check` returns on its FIRST divergence while walking `to_seals`, so `B`'s update or `C`'s insert trips it before the removal branch (`versioned.rs:632-637`) runs at all, and it would have passed with removal detection deleted (caught by a CodeRabbit finding on #1209). Self-compare ⇒ `Wisdom`. Sharper blind spot now pinned: for a PURE removal `GraphDiff` is entirely empty and its own `seal_status` reads `Wisdom`, contradicting `graph_seal_check`'s `Staunen` on the same version pair. Anti-vacuity: unchanged `A` appears in neither set. Blind spot pinned — if a `removed_nodes` field lands, the assertion must be re-pinned deliberately. **Consequence: historical snapshot comparison is the correctness ORACLE; native lance delta is an optional accelerator, never the architecture** | ## bindspace-mailbox-soa-wiring-v1 (D-ids minted 2026-09-05 with the plan) diff --git a/crates/lance-graph-contract/tests/v3_mint_reachability_probe.rs b/crates/lance-graph-contract/tests/v3_mint_reachability_probe.rs index f9ed1e657..a1b28170f 100644 --- a/crates/lance-graph-contract/tests/v3_mint_reachability_probe.rs +++ b/crates/lance-graph-contract/tests/v3_mint_reachability_probe.rs @@ -19,6 +19,8 @@ use lance_graph_contract::canonical_node::{classid_read_mode, NodeGuid, TailVariant}; +/// The PREMISE half of `mint_for`'s dead-code claim: does a classid actually +/// register a V2/V3 tail variant in this build? #[test] fn a_v3_registered_classid_resolves_to_the_v3_tail_variant() { let m = classid_read_mode(NodeGuid::CLASSID_OSINT_V3); @@ -32,6 +34,12 @@ fn a_v3_registered_classid_resolves_to_the_v3_tail_variant() { assert_eq!(m.tail_variant, TailVariant::V1); } +/// The CONCLUSION half: given a classid that registers V3, does `mint_for` +/// reach the V3 arm — or silently fall back to the deprecated V1 layout? +/// +/// `leaf` is the discriminator, because the V3 arm forwards it to `new_v2` +/// while the V1 fallback discards it. That keeps this probe meaningful even if +/// the feature graph is later flattened. #[test] fn minting_through_the_registered_tail_variant_honours_leaf() { let c = NodeGuid::CLASSID_OSINT_V3; diff --git a/crates/lance-graph/tests/canonical_witness_identity_probe.rs b/crates/lance-graph/tests/canonical_witness_identity_probe.rs index 0acdc8a12..7bb0aac62 100644 --- a/crates/lance-graph/tests/canonical_witness_identity_probe.rs +++ b/crates/lance-graph/tests/canonical_witness_identity_probe.rs @@ -50,6 +50,10 @@ fn mint(classid: u32, leaf: u16, family: u32, identity: u32) -> NodeGuid { NodeGuid::mint_for(tv, classid, 0x1111, 0x2222, 0x3333, leaf, family, identity) } +/// A canonical row at `guid` whose value slab is filled with `fill`. +/// +/// The fill matters: without it two rows sharing a key are all-zero past byte +/// 32, and a byte-equality assertion would pass on a slab that never travelled. fn row_of(guid: NodeGuid, fill: u8) -> NodeRow { NodeRow { key: guid, @@ -58,11 +62,17 @@ fn row_of(guid: NodeGuid, fill: u8) -> NodeRow { } } +/// The rows' 512-byte LE image, produced by the contract's own packer rather +/// than assembled here — so a change to the 16/16/480 carving falsifies this +/// probe instead of silently agreeing with it. fn packet_bytes(rows: &[NodeRow], cycle: u32) -> Vec { use lance_graph_contract::soa_envelope::SoaEnvelope; NodeRowPacket::new(rows, cycle).as_le_bytes().to_vec() } +/// A paired kanban move. Its content is irrelevant to identity; only its +/// PRESENCE makes the cast artifact-backed rather than intent-only, and only +/// artifact-backed casts reach the store at all. fn mv(owner: MailboxId) -> KanbanMove { KanbanMove { mailbox: owner, @@ -73,6 +83,10 @@ fn mv(owner: MailboxId) -> KanbanMove { } } +/// An artifact-backed cast placing `payload` at `row` in `cycle`. +/// +/// `payload` must be exactly [`EPISODIC_WITNESS_BYTES`]; the writer refuses any +/// other width permanently rather than as retryable IO. fn cast(cycle: u64, sp: u64, owner: MailboxId, row: u64, payload: Vec) -> SweepSlot { SweepSlot { cycle: CycleId(cycle), diff --git a/crates/lance-graph/tests/test_e_old_diff_tripartite.rs b/crates/lance-graph/tests/test_e_old_diff_tripartite.rs index bd0d3d47e..2462b0565 100644 --- a/crates/lance-graph/tests/test_e_old_diff_tripartite.rs +++ b/crates/lance-graph/tests/test_e_old_diff_tripartite.rs @@ -42,6 +42,13 @@ const D: u32 = 4; const V1: &[(u32, u8)] = &[(A, 0x10), (B, 0x10), (D, 0x10)]; const V2: &[(u32, u8)] = &[(A, 0x10), (B, 0x20), (C, 0x10)]; +/// V2 minus `C`, nothing else touched — a transition whose ONLY change is a +/// removal. Needed because `graph_seal_check` returns on its FIRST divergence +/// while walking `to_seals`, so on `V1 → V2` the update to `B` or the insert of +/// `C` trips it long before the removal branch (`versioned.rs:632-637`) is +/// reached: a `Staunen` there would pass identically with removal detection +/// deleted outright. +const V3: &[(u32, u8)] = &[(A, 0x10), (B, 0x20)]; fn fsb_width(schema: &SchemaRef, name: &str) -> i32 { match schema.field_with_name(name).expect(name).data_type() { @@ -50,12 +57,22 @@ fn fsb_width(schema: &SchemaRef, name: &str) -> i32 { } } +/// A deterministic seal for `(node_id, tag)`, zero-padded to the schema width. +/// +/// The `tag` is the whole mechanism of the fixture: the same `node_id` carrying +/// a different tag across two versions is what makes a node UPDATED rather than +/// inserted, and it is the only thing `diff()` compares. fn seal_bytes(node_id: u32, tag: u8, width: usize) -> Vec { let mut v = vec![tag; width]; v[..4].copy_from_slice(&node_id.to_le_bytes()); v } +/// One `NodeSchema` batch for a round. +/// +/// `node_id` is supplied here because the schema has no producer in-repo — +/// `commit_encounter_round` takes a caller-built batch, so a fixture must mint +/// the ids the store will key on. fn node_batch(round: &Round) -> RecordBatch { let schema = NodeSchema::arrow_schema_ref(); let plane_w = fsb_width(&schema, "plane_s"); @@ -90,6 +107,11 @@ fn node_batch(round: &Round) -> RecordBatch { RecordBatch::try_new(schema, cols).unwrap() } +/// A zero-row batch for the edge and fingerprint datasets. +/// +/// `commit_encounter_round` writes all three tables per round, but this probe +/// is about NODE change sets, so the other two are committed empty rather than +/// left out. fn empty_batch(schema: SchemaRef) -> RecordBatch { let cols = schema .fields() @@ -136,13 +158,15 @@ async fn seals_at(ds: &Dataset) -> BTreeMap> { out } +/// `V1 = {A, B, D}` then `V2 = {A, B', C}`: the two sets the API exposes, plus +/// the third derived from the same material, plus both halves of the seal gate. #[tokio::test(flavor = "multi_thread")] async fn old_diff_gives_inserted_and_updated_and_removed_is_derivable() { let tmp = tempfile::tempdir().expect("tempdir"); let graph = VersionedGraph::local(tmp.path().to_str().unwrap()); let mut versions = Vec::new(); - for round in [V1.to_vec(), V2.to_vec()] { + for round in [V1.to_vec(), V2.to_vec(), V3.to_vec()] { let v = graph .commit_encounter_round( node_batch(&round), @@ -153,7 +177,7 @@ async fn old_diff_gives_inserted_and_updated_and_removed_is_derivable() { .expect("commit"); versions.push(v); } - let (v1, v2) = (versions[0], versions[1]); + let (v1, v2, v3) = (versions[0], versions[1], versions[2]); // ---- what the API DOES report ---------------------------------------- let diff = graph.diff(v1, v2).await.expect("diff V1->V2"); @@ -188,12 +212,15 @@ async fn old_diff_gives_inserted_and_updated_and_removed_is_derivable() { ); // ---- removals ARE detected, just not reported ------------------------- - // graph_seal_check walks `from_seals.keys()` and returns Staunen on a - // removal (versioned.rs:632-637) — so the store distinguishes the states. + // ISOLATED on V2 -> V3, whose only change is dropping C. On V1 -> V2 this + // assertion would be confounded: graph_seal_check returns on the FIRST + // divergence while walking `to_seals`, so B's update or C's insert trips it + // before the removal branch (versioned.rs:632-637) runs at all, and the + // assertion would hold with that branch deleted. assert_eq!( - graph.graph_seal_check(v1, v2).await.expect("seal check"), + graph.graph_seal_check(v2, v3).await.expect("seal check"), GraphSealStatus::Staunen, - "a removal is Staunen" + "a removal ALONE is Staunen — this is the removal branch or nothing" ); // The silence half: same version against itself is Wisdom. assert_eq!( @@ -202,6 +229,21 @@ async fn old_diff_gives_inserted_and_updated_and_removed_is_derivable() { "no change is Wisdom" ); + // And the sharp shape of the blind spot: for a pure removal `GraphDiff` + // reports NOTHING AT ALL, while the seal gate above says Staunen. The two + // surfaces disagree about the same pair of versions. + let pure_removal = graph.diff(v2, v3).await.expect("diff V2->V3"); + assert!( + pure_removal.new_nodes.is_empty() && pure_removal.modified_nodes.is_empty(), + "a pure removal produces an empty GraphDiff: {pure_removal:?}" + ); + assert_eq!( + pure_removal.seal_status, + GraphSealStatus::Wisdom, + "GraphDiff's own seal_status is derived from its two (empty) sets, so it \ + reads Wisdom for a transition graph_seal_check calls Staunen" + ); + // ---- DERIVATION: removed = from - to, from the same material ---------- // `diff()` materializes exactly these two maps (`from_nodes` / `to_nodes`, // versioned.rs:550-551) and their key sets (`from_ids` / `to_ids`, :553-554) @@ -219,6 +261,21 @@ async fn old_diff_gives_inserted_and_updated_and_removed_is_derivable() { "removed = from - to = {{D}} — derivable from the SAME read diff() does" ); + // Same derivation on the ISOLATED pair recovers the only thing that changed. + let v3_ids: BTreeSet = seals_at(&graph.at_version(v3).await.expect("checkout V3")) + .await + .keys() + .copied() + .collect(); + assert_eq!( + to_ids + .difference(&v3_ids) + .copied() + .collect::>(), + BTreeSet::from([C]), + "the pure-removal pair yields exactly {{C}}" + ); + // And the derivation reproduces the two sets the API does expose, so the // three are one pass over one pair of maps. let derived_inserted: BTreeSet = to_ids.difference(&from_ids).copied().collect();