diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index 0c69a1a06..923c0781b 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -21,6 +21,10 @@ identity) is real and probe-gated. | D-GTM-0n / P3 | **Ternlog chaining amortization** (`ndarray/examples/ternlog_amortization_probe.rs`, re-run `--release` 2026-09-06): does chaining pay on THIS mask shape? Pre-registered pass criterion = per-constraint cost flat in K, with a **K=1 control** (nothing to chain ⇒ no win) | **GREEN, bounded.** `T3/T1` bottoms at **0.50** by K=8; T3 total flat in K (57.6 ns @ K=4 → 54.1 ns @ K=32, 16× more constraints). K=1 control reads **1.03** as it must. Two bounds ride with the number and may not be dropped when it is cited: the win is **L2-residency-contingent** (bandwidth `138→15 GB/s` past L2, ratio back to 1.03 at a 512 KiB mask), and mask loses to sparse below **0.1%** active. **Does NOT retire the standing caveat** — 2× on one of three passes still cannot account for 5×. Gates Stage-3 SPEED claims only; the shape/correctness case never depended on it | | D-ALPHA-1b | **`AlphaMask::zip` release-mode length guard** (`lance-graph-contract/src/alpha.rs`): the only guard was `debug_assert_eq!`, compiled out in release, so mismatched-length operands produced a mask claiming `self.len` addresses over the SHORTER operand's word count — `count`/`is_empty` under-report silently, `contains`/`materialize_ordinals` panic at a distance | **Fixed** — promoted to `assert_eq!`. Defect reproduced in release BEFORE the fix (the `should_panic` falsifier reported "test did not panic as expected"); two-sided falsifiers both disable-verified. No API change, no external callers. Stage 1b of `.claude/temporal/09-plan.md`, now DONE | | D-LNC-7 | **One-WAL ruling**: `LanceCycleWriter`'s watermark WAL vs lance MemWAL — `applied_through` ≡ index catch-up position; two under one commit = two truths. Plan recommends keep-ours until D-LNC-3, then re-decide | **Operator decision** | +| D-TEMPORAL-1 | **Canonical witness identity probe** (`crates/lance-graph/tests/canonical_witness_identity_probe.rs`): does canonical address identity survive the A-store round trip on lance 11? `NodeGuid → NodeRow → NodeRowPacket::as_le_bytes → cycle commit → drop writer → reopen → scan_image`. Minted through `mint_for(classid_read_mode(c).tail_variant, …)`, never a hardcoded constructor | **GREEN, both rails.** V3 (`CLASSID_OSINT_V3`): payload bytes byte-exact across two cycles and a reopen; `payload[0..16] == G.as_bytes()`; value slab survived (so equality is not a zero-match); each cycle minted exactly `base+1`; head recovered at V2 after restart. V1 legacy control (`CLASSID_OSINT`) round-trips too. **Scope:** proves byte and address fidelity of the store; does NOT prove production reachability — `LanceCycleWriter` still has zero non-test callers (see D-MW-P2) | +| D-TEMPORAL-1a | **Pinned false-green: the V1 accessor on a V3 key.** 256 V3 addresses varying only in `leaf` → 256 distinct 16-byte keys, 256 distinct `to_hex_v2()`, but **1** distinct `identity()`. Byte fidelity does not imply address fidelity | **Pinned as a positive assertion.** Also pins the second half, which is not a defect: `local_key_v2` yields **1** across the same set BY CONSTRUCTION — `leaf` is the 4th HHTL routing tier (bytes 10..12), part of the addressing prefix, deliberately outside the basin-local key (bytes 12..16). So "use the V3 accessor" is two choices, not one, and the wrong V3 one fails like the V1 one. Do not update the numbers if the assertion breaks — re-audit the call sites | +| D-V3-MINT-1 | **V3 mint reachability** (`crates/lance-graph-contract/tests/v3_mint_reachability_probe.rs`): `mint_for`'s V2/V3 arm is gated on `guid-v2-tail`, which is NOT in `default`; the registry's V3 entries are gated on `guid-v3-tail`, which is. Two different features, so premise and conclusion can come apart — is a V3-registered classid silently minted into the deprecated V1 layout? | **GREEN — no defect.** `guid-v3-tail = ["guid-v2-tail"]` (`Cargo.toml:69`), so the implication holds. Measured, not inferred: `tail_variant=V3 guid-v3-tail=true guid-v2-tail=true leaf_is_live=true`, key bytes show the V2/V3 carving with `leaf` live at byte 10. Discriminator is `leaf` (the V3 arm feeds it to `new_v2`; the V1 fallback discards it), so the probe stays valid if the feature graph is ever flattened | +| D-TEMPORAL-2 | **Old-path tripartite change set** (`crates/lance-graph/tests/test_e_old_diff_tripartite.rs`): can store B's snapshot comparison produce `inserted / updated / removed` without any lance delta API? `V1={A,B,D}`, `V2={A,B',C}` | **GREEN.** `inserted={C}` and `updated={B}` from `GraphDiff`; `removed={D}` DERIVED from the same two seal maps `diff()` already materialises (`versioned.rs:550-554`) before discarding the difference — so the missing `removed_nodes` field is an **API omission, not information loss**. Seal gate two-sided AND isolated: the `Staunen` arm runs on a THIRD version `V3={A,B'}` whose only change is dropping `C` — on `V1→V2` that assertion was CONFOUNDED, because `graph_seal_check` returns on its FIRST divergence while walking `to_seals`, so `B`'s update or `C`'s insert trips it before the removal branch (`versioned.rs:632-637`) runs at all, and it would have passed with removal detection deleted (caught by a CodeRabbit finding on #1209). Self-compare ⇒ `Wisdom`. Sharper blind spot now pinned: for a PURE removal `GraphDiff` is entirely empty and its own `seal_status` reads `Wisdom`, contradicting `graph_seal_check`'s `Staunen` on the same version pair. Anti-vacuity: unchanged `A` appears in neither set. Blind spot pinned — if a `removed_nodes` field lands, the assertion must be re-pinned deliberately. **Consequence: historical snapshot comparison is the correctness ORACLE; native lance delta is an optional accelerator, never the architecture** | ## bindspace-mailbox-soa-wiring-v1 (D-ids minted 2026-09-05 with the plan) diff --git a/.claude/board/SUPERSESSION-INDEX.md b/.claude/board/SUPERSESSION-INDEX.md index c2921a54a..2fdd26b94 100644 --- a/.claude/board/SUPERSESSION-INDEX.md +++ b/.claude/board/SUPERSESSION-INDEX.md @@ -53,7 +53,7 @@ coverage instead. | `StepMask` | BLOCKED | — | 3 | 9 | 4 | | `commit_to_l4` | BLOCKED | — | 2 | 2 | 0 | | `dispatch_busdto` | BLOCKED | — | 3 | 8 | 5 | -| `persist_cycle` | BLOCKED | — | 10 | 8 | 5 | +| `persist_cycle` | BLOCKED | — | 11 | 8 | 5 | | `CognitiveMarkers` | REPURPOSE | `Commit` | 1 | 0 | 0 | | `DominoCascade` | REPURPOSE | `Commit` | 7 | 1 | 0 | | `GateDecision` | REPURPOSE | — | 25 | 27 | 24 | diff --git a/crates/lance-graph-contract/tests/v3_mint_reachability_probe.rs b/crates/lance-graph-contract/tests/v3_mint_reachability_probe.rs new file mode 100644 index 000000000..a1b28170f --- /dev/null +++ b/crates/lance-graph-contract/tests/v3_mint_reachability_probe.rs @@ -0,0 +1,70 @@ +// SPDX-License-Identifier: Apache-2.0 + +//! D-V3-MINT-1 — is the V3 tail actually reachable through `mint_for` in the +//! build configuration production uses? +//! +//! `mint_for`'s doc justifies its V1 fallback arm as dead code: *"With the +//! feature off no classid registers a V2/V3 `tail_variant` (`classid_read_mode` +//! returns V1), so the fallback arm is dead — it exists purely so the crate +//! compiles `--no-default-features`."* +//! +//! That justification is checkable. The registry's V3 entries are gated on +//! `guid-v3-tail` (default ON); `mint_for`'s V3 arm is gated on `guid-v2-tail` +//! (default OFF). Those are two different features, so the premise and the +//! conclusion can come apart. +//! +//! The discriminator is `leaf`: the V3 arm feeds it to `new_v2`, the V1 +//! fallback discards it (`let _ = leaf;`). Two mints differing only in `leaf` +//! are therefore byte-equal iff the fallback ran. + +use lance_graph_contract::canonical_node::{classid_read_mode, NodeGuid, TailVariant}; + +/// The PREMISE half of `mint_for`'s dead-code claim: does a classid actually +/// register a V2/V3 tail variant in this build? +#[test] +fn a_v3_registered_classid_resolves_to_the_v3_tail_variant() { + let m = classid_read_mode(NodeGuid::CLASSID_OSINT_V3); + #[cfg(feature = "guid-v3-tail")] + assert_eq!( + m.tail_variant, + TailVariant::V3, + "CLASSID_OSINT_V3 must register the V3 tail when guid-v3-tail is on" + ); + #[cfg(not(feature = "guid-v3-tail"))] + assert_eq!(m.tail_variant, TailVariant::V1); +} + +/// The CONCLUSION half: given a classid that registers V3, does `mint_for` +/// reach the V3 arm — or silently fall back to the deprecated V1 layout? +/// +/// `leaf` is the discriminator, because the V3 arm forwards it to `new_v2` +/// while the V1 fallback discards it. That keeps this probe meaningful even if +/// the feature graph is later flattened. +#[test] +fn minting_through_the_registered_tail_variant_honours_leaf() { + let c = NodeGuid::CLASSID_OSINT_V3; + let tv = classid_read_mode(c).tail_variant; + + // The canonical consumer incantation, verbatim from mint_for's doc. + let a = NodeGuid::mint_for(tv, c, 0x1111, 0x2222, 0x3333, 0x00AA, 0x0001, 0x0002); + let b = NodeGuid::mint_for(tv, c, 0x1111, 0x2222, 0x3333, 0x00BB, 0x0001, 0x0002); + + let leaf_is_live = a.as_bytes() != b.as_bytes(); + + eprintln!( + "tail_variant={tv:?} guid-v3-tail={} guid-v2-tail={} leaf_is_live={leaf_is_live}\n a={:02x?}\n b={:02x?}", + cfg!(feature = "guid-v3-tail"), + cfg!(feature = "guid-v2-tail"), + a.as_bytes(), + b.as_bytes(), + ); + + if tv == TailVariant::V3 { + assert!( + leaf_is_live, + "classid {c:#010x} registers TailVariant::V3, but mint_for produced a \ + leaf-insensitive key — the V1 fallback arm ran. A V3 class is being \ + minted into the deprecated V1 layout, silently." + ); + } +} diff --git a/crates/lance-graph/tests/canonical_witness_identity_probe.rs b/crates/lance-graph/tests/canonical_witness_identity_probe.rs new file mode 100644 index 000000000..7bb0aac62 --- /dev/null +++ b/crates/lance-graph/tests/canonical_witness_identity_probe.rs @@ -0,0 +1,269 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright The Lance Authors + +//! D-TEMPORAL-1 — does canonical address identity survive the cycle store? +//! +//! The claim under test, stated as bytes rather than as prose: +//! +//! ```text +//! NodeGuid G -> NodeRow -> 512-byte LE packet -> cycle Vn commit +//! -> drop writer -> reopen -> scan_image(cycle) +//! -> payload == submitted bytes AND payload[0..16] == G.as_bytes() +//! ``` +//! +//! Three assertions are kept apart, because they fail independently: +//! +//! 1. the raw key bytes survive byte-exact (a storage claim); +//! 2. the V3 address decodes as V3 (a reader claim); +//! 3. reading a V3 key through the V1 accessors is WRONG (a selection claim) — +//! pinned here as a positive assertion so that a future reader who reaches +//! for `identity()` on a V3 key breaks this test instead of shipping. +//! +//! Assertion 3 exists because that exact false-green has already happened: a +//! byte-exact V3 round-trip read back through the V1 `identity()` accessor +//! collapsed a five-figure row count to a three-figure one, and the round-trip +//! stayed green throughout. Byte fidelity does not imply address fidelity. +//! +//! Keys are minted through `mint_for(classid_read_mode(c).tail_variant, …)` — +//! never by hardcoding `new` (V1, deprecated) or `new_v2`. The V1 rail is +//! retained only as a read-side legacy control (RESERVE, DON'T RECLAIM); no new +//! unit is minted V1. +//! +//! Two Arrow claims are also kept apart: the schema's column 0 is `kind`, not +//! an address. The address claim is about the first 16 bytes of the +//! `FixedSizeBinary(512)` payload blob. + +use lance_graph::graph::cycle_sink::{LanceCycleWriter, EPISODIC_WITNESS_BYTES}; +use lance_graph_contract::canonical_node::{ + classid_read_mode, EdgeBlock, NodeGuid, NodeRow, NodeRowPacket, TailVariant, +}; +use lance_graph_contract::collapse_gate::MailboxId; +use lance_graph_contract::kanban::{ExecTarget, KanbanColumn, KanbanMove}; +use lance_graph_contract::scheduler::DatasetVersion; +use lance_graph_planner::persist_sink::{ + persist_cycle, CommitOutcome, CycleFrame, CycleId, SweepSlot, +}; + +/// The canonical consumer incantation — the registry picks the tail, not us. +fn mint(classid: u32, leaf: u16, family: u32, identity: u32) -> NodeGuid { + let tv = classid_read_mode(classid).tail_variant; + NodeGuid::mint_for(tv, classid, 0x1111, 0x2222, 0x3333, leaf, family, identity) +} + +/// A canonical row at `guid` whose value slab is filled with `fill`. +/// +/// The fill matters: without it two rows sharing a key are all-zero past byte +/// 32, and a byte-equality assertion would pass on a slab that never travelled. +fn row_of(guid: NodeGuid, fill: u8) -> NodeRow { + NodeRow { + key: guid, + edges: EdgeBlock::default(), + value: [fill; 480], + } +} + +/// The rows' 512-byte LE image, produced by the contract's own packer rather +/// than assembled here — so a change to the 16/16/480 carving falsifies this +/// probe instead of silently agreeing with it. +fn packet_bytes(rows: &[NodeRow], cycle: u32) -> Vec { + use lance_graph_contract::soa_envelope::SoaEnvelope; + NodeRowPacket::new(rows, cycle).as_le_bytes().to_vec() +} + +/// A paired kanban move. Its content is irrelevant to identity; only its +/// PRESENCE makes the cast artifact-backed rather than intent-only, and only +/// artifact-backed casts reach the store at all. +fn mv(owner: MailboxId) -> KanbanMove { + KanbanMove { + mailbox: owner, + from: KanbanColumn::Planning, + to: KanbanColumn::CognitiveWork, + witness_chain_position: 1, + exec: ExecTarget::Elixir, + } +} + +/// An artifact-backed cast placing `payload` at `row` in `cycle`. +/// +/// `payload` must be exactly [`EPISODIC_WITNESS_BYTES`]; the writer refuses any +/// other width permanently rather than as retryable IO. +fn cast(cycle: u64, sp: u64, owner: MailboxId, row: u64, payload: Vec) -> SweepSlot { + SweepSlot { + cycle: CycleId(cycle), + stream_position: sp, + owner, + row, + paired_move: Some(mv(owner)), + payload, + } +} + +/// Write one cycle per (row, guid, fill) and return the writer's head. +async fn write_cycles( + w: &mut LanceCycleWriter, + units: &[(u64, NodeGuid, u8)], +) -> Vec<(u64, Vec, u64)> { + let mut out = Vec::new(); + let mut base = w.head(); + for (i, (row, g, fill)) in units.iter().enumerate() { + let cycle = (i + 1) as u64; + let bytes = packet_bytes(&[row_of(*g, *fill)], cycle as u32); + assert_eq!(bytes.len(), EPISODIC_WITNESS_BYTES); + let outcome = persist_cycle( + w, + CycleFrame::new(CycleId(cycle), base), + vec![cast(cycle, 0, 7, *row, bytes.clone())], + ) + .await + .unwrap(); + match outcome { + CommitOutcome::Committed { version, .. } => { + assert_eq!( + version, + DatasetVersion(base.0 + 1), + "cycle {cycle} minted more than one version" + ); + base = version; + out.push((cycle, bytes, *row)); + } + other => panic!("cycle {cycle} did not commit: {other:?}"), + } + } + out +} + +/// RAIL V3 (the live rail) — bytes survive, and the V3 address decodes as V3. +#[tokio::test] +async fn a_v3_address_survives_two_cycles_and_a_reopen_byte_for_byte() { + let c = NodeGuid::CLASSID_OSINT_V3; + assert_eq!( + classid_read_mode(c).tail_variant, + TailVariant::V3, + "this rail is only meaningful if the classid actually registers V3" + ); + + let g1 = mint(c, 0x00AA, 0x1234, 0x5678); + let g2 = mint(c, 0x00BB, 0x4321, 0x8765); + assert_ne!( + g1.as_bytes(), + g2.as_bytes(), + "leaf must discriminate on the V3 rail — if this fails the V1 fallback ran" + ); + + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("cycles.lance"); + let p = path.to_str().unwrap(); + + let mut w = LanceCycleWriter::open(p).await.unwrap(); + let written = write_cycles(&mut w, &[(100, g1, 0xA1), (200, g2, 0xB2)]).await; + assert_eq!(w.head(), DatasetVersion(2)); + + drop(w); + let w2 = LanceCycleWriter::open(p).await.unwrap(); + assert_eq!(w2.head(), DatasetVersion(2), "head did not survive restart"); + + let expect = [g1, g2]; + for (i, (cycle, submitted, row)) in written.iter().enumerate() { + let img = w2.scan_image(CycleId(*cycle)).await.unwrap(); + let back = img + .get(row) + .unwrap_or_else(|| panic!("cycle {cycle} row {row} missing after reopen")); + + // (1) raw bytes + assert_eq!(back, submitted, "cycle {cycle} payload bytes changed"); + assert_eq!( + &back[0..16], + expect[i].as_bytes(), + "cycle {cycle} payload[0..16] is not the submitted NodeGuid" + ); + // the value slab travelled too, so (1) is not a degenerate zero-match + assert_eq!(back[32], if i == 0 { 0xA1 } else { 0xB2 }); + } + + // (2) V3 semantic decode — the V3-aware accessors, never the V1 ones. + assert_eq!(g1.leaf(), 0x00AA); + assert_eq!(g1.family_v2(), 0x1234); + assert_eq!(g1.identity_v2(), 0x5678); + assert_eq!(g2.leaf(), 0x00BB); + assert_eq!(g2.family_v2(), 0x4321); + assert_eq!(g2.identity_v2(), 0x8765); + assert_ne!(g1.to_hex_v2(), g2.to_hex_v2(), "V3 rail addresses differ"); +} + +/// (3) The selection claim: the V1 accessors are WRONG on a V3 key, and wrong +/// in the specific way that looks like success — many distinct addresses +/// collapsing onto one apparent identity while every byte round-trips. +#[test] +fn reading_a_v3_key_through_the_v1_accessor_collapses_distinct_addresses() { + let c = NodeGuid::CLASSID_OSINT_V3; + assert_eq!(classid_read_mode(c).tail_variant, TailVariant::V3); + + // 256 genuinely distinct V3 addresses, varying only the leaf tier. + let keys: Vec = (0..256u16).map(|l| mint(c, l, 0x1234, 0x5678)).collect(); + + let distinct_bytes: std::collections::HashSet<[u8; 16]> = + keys.iter().map(|k| *k.as_bytes()).collect(); + assert_eq!(distinct_bytes.len(), 256, "the 256 addresses are distinct"); + + // The V3-aware full-tier decode keeps all 256 apart. `local_key_v2` alone + // does NOT and must not: leaf is an HHTL routing tier (bytes 10..12), part + // of the addressing PREFIX, deliberately outside the basin-local key + // (bytes 12..16). Both facts are pinned, because picking the wrong one of + // the two V3 accessors is the same class of error as picking the V1 one. + let v3_view: std::collections::HashSet = keys.iter().map(NodeGuid::to_hex_v2).collect(); + assert_eq!( + v3_view.len(), + 256, + "the V3-aware full-tier decode must preserve all 256" + ); + let v3_basin: std::collections::HashSet = + keys.iter().map(NodeGuid::local_key_v2).collect(); + assert_eq!( + v3_basin.len(), + 1, + "leaf is a routing tier, not part of the basin-local key — these 256 \ + addresses share one basin by construction" + ); + + let v1_view: std::collections::HashSet = keys.iter().map(NodeGuid::identity).collect(); + assert_eq!( + v1_view.len(), + 1, + "PINNED FALSE-GREEN: the V1 identity() accessor reads bytes 13..16 of a \ + V3 key, so 256 distinct addresses present as ONE. If this assertion \ + ever fails, the tail layout moved and every V1-accessor call site on \ + V3 data must be re-audited — do not simply update the number." + ); +} + +/// RAIL V1 (legacy control) — a pre-flip class still decodes as V1. No new unit +/// is minted this way; this only proves old rows stay readable. +#[tokio::test] +async fn a_legacy_v1_address_still_round_trips_and_decodes_as_v1() { + let c = NodeGuid::CLASSID_OSINT; // pre-V3 exemplar + assert_eq!( + classid_read_mode(c).tail_variant, + TailVariant::V1, + "this control is only meaningful if the classid still registers V1" + ); + + let g = mint(c, 0, 0x00AB_CDEF, 0x0012_3456); + assert_eq!(g.family(), 0x00AB_CDEF); + assert_eq!(g.identity(), 0x0012_3456); + + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("cycles.lance"); + let p = path.to_str().unwrap(); + + let mut w = LanceCycleWriter::open(p).await.unwrap(); + let written = write_cycles(&mut w, &[(300, g, 0xC3)]).await; + drop(w); + + let w2 = LanceCycleWriter::open(p).await.unwrap(); + let (cycle, submitted, row) = &written[0]; + let img = w2.scan_image(CycleId(*cycle)).await.unwrap(); + let back = img.get(row).expect("legacy row missing after reopen"); + + assert_eq!(back, submitted, "V1 payload bytes changed"); + assert_eq!(&back[0..16], g.as_bytes(), "V1 payload[0..16] != G"); +} diff --git a/crates/lance-graph/tests/test_e_old_diff_tripartite.rs b/crates/lance-graph/tests/test_e_old_diff_tripartite.rs new file mode 100644 index 000000000..2462b0565 --- /dev/null +++ b/crates/lance-graph/tests/test_e_old_diff_tripartite.rs @@ -0,0 +1,289 @@ +//! TEST E — can the OLD path (store B, `VersionedGraph`) produce the tripartite +//! change set `inserted / updated / removed`? +//! +//! Scenario: `V1 = {A, B, D}`, `V2 = {A, B', C}`. Desired: +//! +//! - `inserted = to - from = {C}` +//! - `updated = seals differ on the intersection = {B}` +//! - `removed = from - to = {D}` +//! +//! `GraphDiff` (`graph/versioned.rs:70`) carries `new_nodes` and +//! `modified_nodes` and NO removed-nodes field; `diff()` computes +//! `to_ids.difference(&from_ids)` (`:556`) and the seal-differing intersection +//! (`:560`) but never `from_ids.difference(&to_ids)` — even though both sets +//! are materialized side by side at `:553-554`. +//! +//! This probe asserts the two sets the API does expose, then DERIVES the third +//! from exactly the material `diff()` already reads (`extract_node_seals` over +//! both checked-out versions), showing the omission is API-only rather than an +//! information loss in the store. +//! +//! Modelled on `tests/lance_row_identity_probe.rs` (D-LNC-2) for the write +//! idioms; no `_rowaddr` here — this probe is about the change SETS. + +use std::collections::{BTreeMap, BTreeSet}; +use std::sync::Arc; + +use arrow_array::builder::FixedSizeBinaryBuilder; +use arrow_array::{Array, FixedSizeBinaryArray, RecordBatch, UInt32Array}; +use arrow_schema::{DataType, SchemaRef}; +use futures::TryStreamExt; +use lance::dataset::Dataset; +use lance_graph::graph::blasgraph::columnar::{EdgeSchema, FingerprintSchema, NodeSchema}; +use lance_graph::graph::versioned::{GraphSealStatus, VersionedGraph}; + +/// (node_id, seal tag): a changed tag on the same id is an UPDATE, not an insert. +type Round = Vec<(u32, u8)>; + +const A: u32 = 1; +const B: u32 = 2; +const C: u32 = 3; +const D: u32 = 4; + +const V1: &[(u32, u8)] = &[(A, 0x10), (B, 0x10), (D, 0x10)]; +const V2: &[(u32, u8)] = &[(A, 0x10), (B, 0x20), (C, 0x10)]; +/// V2 minus `C`, nothing else touched — a transition whose ONLY change is a +/// removal. Needed because `graph_seal_check` returns on its FIRST divergence +/// while walking `to_seals`, so on `V1 → V2` the update to `B` or the insert of +/// `C` trips it long before the removal branch (`versioned.rs:632-637`) is +/// reached: a `Staunen` there would pass identically with removal detection +/// deleted outright. +const V3: &[(u32, u8)] = &[(A, 0x10), (B, 0x20)]; + +fn fsb_width(schema: &SchemaRef, name: &str) -> i32 { + match schema.field_with_name(name).expect(name).data_type() { + DataType::FixedSizeBinary(w) => *w, + other => panic!("{name}: expected FixedSizeBinary, got {other:?}"), + } +} + +/// A deterministic seal for `(node_id, tag)`, zero-padded to the schema width. +/// +/// The `tag` is the whole mechanism of the fixture: the same `node_id` carrying +/// a different tag across two versions is what makes a node UPDATED rather than +/// inserted, and it is the only thing `diff()` compares. +fn seal_bytes(node_id: u32, tag: u8, width: usize) -> Vec { + let mut v = vec![tag; width]; + v[..4].copy_from_slice(&node_id.to_le_bytes()); + v +} + +/// One `NodeSchema` batch for a round. +/// +/// `node_id` is supplied here because the schema has no producer in-repo — +/// `commit_encounter_round` takes a caller-built batch, so a fixture must mint +/// the ids the store will key on. +fn node_batch(round: &Round) -> RecordBatch { + let schema = NodeSchema::arrow_schema_ref(); + let plane_w = fsb_width(&schema, "plane_s"); + let seal_w = fsb_width(&schema, "seal_s"); + let n = round.len(); + let mut planes = (0..3) + .map(|_| FixedSizeBinaryBuilder::with_capacity(n, plane_w)) + .collect::>(); + let mut seals = (0..3) + .map(|_| FixedSizeBinaryBuilder::with_capacity(n, seal_w)) + .collect::>(); + let plane = vec![0u8; plane_w as usize]; + for &(id, tag) in round { + for p in planes.iter_mut() { + p.append_value(&plane).unwrap(); + } + for (k, s) in seals.iter_mut().enumerate() { + s.append_value(seal_bytes(id, tag.wrapping_add(k as u8), seal_w as usize)) + .unwrap(); + } + } + let ids: Vec = round.iter().map(|r| r.0).collect(); + let enc: Vec = round.iter().map(|_| 1).collect(); + let mut cols: Vec> = vec![Arc::new(UInt32Array::from(ids))]; + for p in planes.iter_mut() { + cols.push(Arc::new(p.finish())); + } + for s in seals.iter_mut() { + cols.push(Arc::new(s.finish())); + } + cols.push(Arc::new(UInt32Array::from(enc))); + RecordBatch::try_new(schema, cols).unwrap() +} + +/// A zero-row batch for the edge and fingerprint datasets. +/// +/// `commit_encounter_round` writes all three tables per round, but this probe +/// is about NODE change sets, so the other two are committed empty rather than +/// left out. +fn empty_batch(schema: SchemaRef) -> RecordBatch { + let cols = schema + .fields() + .iter() + .map(|f| arrow_array::new_empty_array(f.data_type())) + .collect(); + RecordBatch::try_new(schema, cols).unwrap() +} + +/// `node_id -> seal_s ++ seal_p ++ seal_o` — the same material +/// `VersionedGraph::extract_node_seals` builds (`graph/versioned.rs:660`), +/// reproduced here because that helper is private. +async fn seals_at(ds: &Dataset) -> BTreeMap> { + let batches: Vec = ds + .scan() + .try_into_stream() + .await + .expect("scan") + .try_collect() + .await + .expect("collect"); + let mut out = BTreeMap::new(); + for b in &batches { + let ids = b + .column_by_name("node_id") + .and_then(|c| c.as_any().downcast_ref::()) + .expect("node_id"); + let seals: Vec<&FixedSizeBinaryArray> = ["seal_s", "seal_p", "seal_o"] + .iter() + .map(|c| { + b.column_by_name(c) + .and_then(|x| x.as_any().downcast_ref::()) + .expect("seal column") + }) + .collect(); + for i in 0..b.num_rows() { + let mut all = Vec::new(); + for s in &seals { + all.extend_from_slice(s.value(i)); + } + out.insert(ids.value(i), all); + } + } + out +} + +/// `V1 = {A, B, D}` then `V2 = {A, B', C}`: the two sets the API exposes, plus +/// the third derived from the same material, plus both halves of the seal gate. +#[tokio::test(flavor = "multi_thread")] +async fn old_diff_gives_inserted_and_updated_and_removed_is_derivable() { + let tmp = tempfile::tempdir().expect("tempdir"); + let graph = VersionedGraph::local(tmp.path().to_str().unwrap()); + + let mut versions = Vec::new(); + for round in [V1.to_vec(), V2.to_vec(), V3.to_vec()] { + let v = graph + .commit_encounter_round( + node_batch(&round), + empty_batch(EdgeSchema::arrow_schema_ref()), + empty_batch(FingerprintSchema::arrow_schema_ref()), + ) + .await + .expect("commit"); + versions.push(v); + } + let (v1, v2, v3) = (versions[0], versions[1], versions[2]); + + // ---- what the API DOES report ---------------------------------------- + let diff = graph.diff(v1, v2).await.expect("diff V1->V2"); + + let inserted: BTreeSet = diff.new_nodes.iter().copied().collect(); + assert_eq!( + inserted, + BTreeSet::from([C]), + "inserted = to - from = {{C}} (versioned.rs:556)" + ); + + let updated: BTreeSet = diff.modified_nodes.iter().copied().collect(); + assert_eq!( + updated, + BTreeSet::from([B]), + "updated = seal-differing intersection = {{B}} (versioned.rs:560)" + ); + + // Anti-vacuity: A is in both versions with an UNCHANGED seal and must + // appear in neither set — otherwise "updated" would just mean "present". + assert!( + !updated.contains(&A) && !inserted.contains(&A), + "A is quiet" + ); + + // ---- the omission: GraphDiff has no removed-nodes field --------------- + // D is gone in V2 and appears NOWHERE in the diff. Pinned; if a + // `removed_nodes` field lands, this assertion must be re-pinned deliberately. + assert!( + !inserted.contains(&D) && !updated.contains(&D), + "PINNED BLIND SPOT: removed node D is absent from GraphDiff: {diff:?}" + ); + + // ---- removals ARE detected, just not reported ------------------------- + // ISOLATED on V2 -> V3, whose only change is dropping C. On V1 -> V2 this + // assertion would be confounded: graph_seal_check returns on the FIRST + // divergence while walking `to_seals`, so B's update or C's insert trips it + // before the removal branch (versioned.rs:632-637) runs at all, and the + // assertion would hold with that branch deleted. + assert_eq!( + graph.graph_seal_check(v2, v3).await.expect("seal check"), + GraphSealStatus::Staunen, + "a removal ALONE is Staunen — this is the removal branch or nothing" + ); + // The silence half: same version against itself is Wisdom. + assert_eq!( + graph.graph_seal_check(v2, v2).await.expect("seal check"), + GraphSealStatus::Wisdom, + "no change is Wisdom" + ); + + // And the sharp shape of the blind spot: for a pure removal `GraphDiff` + // reports NOTHING AT ALL, while the seal gate above says Staunen. The two + // surfaces disagree about the same pair of versions. + let pure_removal = graph.diff(v2, v3).await.expect("diff V2->V3"); + assert!( + pure_removal.new_nodes.is_empty() && pure_removal.modified_nodes.is_empty(), + "a pure removal produces an empty GraphDiff: {pure_removal:?}" + ); + assert_eq!( + pure_removal.seal_status, + GraphSealStatus::Wisdom, + "GraphDiff's own seal_status is derived from its two (empty) sets, so it \ + reads Wisdom for a transition graph_seal_check calls Staunen" + ); + + // ---- DERIVATION: removed = from - to, from the same material ---------- + // `diff()` materializes exactly these two maps (`from_nodes` / `to_nodes`, + // versioned.rs:550-551) and their key sets (`from_ids` / `to_ids`, :553-554) + // before throwing the difference away. Recomputing it here proves the + // information is present in the store, not lost. + let from_seals = seals_at(&graph.at_version(v1).await.expect("checkout V1")).await; + let to_seals = seals_at(&graph.at_version(v2).await.expect("checkout V2")).await; + let from_ids: BTreeSet = from_seals.keys().copied().collect(); + let to_ids: BTreeSet = to_seals.keys().copied().collect(); + + let removed: BTreeSet = from_ids.difference(&to_ids).copied().collect(); + assert_eq!( + removed, + BTreeSet::from([D]), + "removed = from - to = {{D}} — derivable from the SAME read diff() does" + ); + + // Same derivation on the ISOLATED pair recovers the only thing that changed. + let v3_ids: BTreeSet = seals_at(&graph.at_version(v3).await.expect("checkout V3")) + .await + .keys() + .copied() + .collect(); + assert_eq!( + to_ids + .difference(&v3_ids) + .copied() + .collect::>(), + BTreeSet::from([C]), + "the pure-removal pair yields exactly {{C}}" + ); + + // And the derivation reproduces the two sets the API does expose, so the + // three are one pass over one pair of maps. + let derived_inserted: BTreeSet = to_ids.difference(&from_ids).copied().collect(); + assert_eq!(derived_inserted, inserted); + let derived_updated: BTreeSet = to_ids + .intersection(&from_ids) + .copied() + .filter(|id| from_seals.get(id) != to_seals.get(id)) + .collect(); + assert_eq!(derived_updated, updated); +}