PiPNN 6/6: add HashPrune candidate merging

[weiyaoluo (SeliMeli)]

Purpose

This PR adds optional HashPrune candidate merging to PiPNN.

Direct candidate merging remains the default.

Main changes

• lsh.rs creates deterministic random-hyperplane sketches.
• Each directed edge receives a relative-direction hash.
• Each source owns one bounded reservoir and one lock.
• A hash bucket keeps its nearest candidate.
• Full reservoirs reject or evict candidates by total order.
• final_prune=true sends reservoir candidates to shared RobustPrune.
• final_prune=false returns the nearest graph-degree candidates.
• Disk and benchmark configuration accept the optional HashPrune policy.

Required invariants

• Effective hash capacity covers graph degree.
• l_max does not exceed the structural reservoir limit.
• Padded cells never become candidates.
• One source lock protects each reservoir mutation.
• Leaf-local IDs become global IDs before insertion.
• Tie order uses distance, relative hash, and neighbor ID.

Review order

1. Review HashPruneConfig and with_hash_prune in mod.rs.
2. Review sketch construction in lsh.rs.
3. Review slab ownership and locking in hash_prune.rs.
4. Review insertion and extraction in hash_prune.rs.
5. Review leaf CSR conversion in leaf_build.rs.
6. Review disk and benchmark configuration.

Validation

• All 452 diskann library tests pass.
• Seven disk-adapter tests pass.
• The benchmark lifecycle test passes.
• Targeted strict-provenance Miri tests pass.
• Linux, Windows GNU, and AArch64 Clippy pass with -Dwarnings.

Stack

Stack 6/6. Depends on #1294 and completes the PiPNN stack.

[weiyaoluo (SeliMeli)]

Azure BigANN10M validation (Standard_D16as_v5, AVX2, 16 threads, 4× interleaved per binary) used R=64, c512/c64, fanout=[10,3], k=2, l_max=72, hp=12, final_prune=true.

• Build: 76.777s clean → 79.601s stack (+3.68%)
• Peak RSS: 12.224 → 11.709 GiB (-4.22%)
• Recall@10 L=50: 0.957825 → 0.957790 (-0.0035pp)
• Mean comparisons/hops L=50: +0.023% / +0.006%

Raw runs: ~/pipnn-pr5-ab/results/final-head-k2-20260729T171245Z on the benchmark VM. The residual build gap is being investigated only on local, unpushed pipnn-stack/07-perf-investigation; current attribution points to repeated per-row FP16 conversion dispatch and CSR bookkeeping, not graph quality or HashPrune semantics.

[weiyaoluo (SeliMeli)]

Direct final-stack QC

Reviewed directly against current origin/main; no subagents were used.

Fixed during QC

• Removed the proposed PiPNN-owned unsafe FP16 gather because its measured ~0.48% gain did not meet the unsafe RFC's safety/CI-evidence bar.
• Added PR6's missing direct bytemuck dependency (caught by a clean coverage build).
• Updated shared RobustPrune for the latest Neighbor accessor API.
• Rebuilt the stack from exact per-layer commit boundaries and rebased it onto current main.
• Preserved current-main nightly feature coverage while retaining strict HashPrune Miri jobs.

Remaining findings

1. Evidence gap: performance-motivated unsafe kernels have Criterion/Azure/Miri evidence, but no CI-integrated performance regression threshold as required by rfcs/00109-unsafe-rust.md.
2. Performance objective not met: allocator-matched BigANN10M 4× interleaved A/B reports build median 75.899s clean vs 77.473s final (+2.07%); wall +1.62%; RSS -3.60%; recall effectively unchanged (0.957805 vs 0.957795). No parity/win claim is made.

Validation

• Formatting and CI-style workspace Clippy: pass.
• PiPNN all targets: pass (81 unit tests plus integration/config/kernel suites and benchmark smoke executables).
• RobustPrune/Vamana tests: 22 pass.
• Focused disk PiPNN tests: 5 pass.
• pipnn and disk-index,pipnn feature compilation/Clippy: pass.
• x86-64 baseline and AArch64 all-target checks: pass.
• Strict-provenance Miri HashPrune kernel checks: pass.
• PiPNN ISA/TLS/RefCell/broadcast-cleanup scans: zero hits.
• Stable production-source coverage: 94.19% lines / 91.88% regions; experimental branch coverage 91.32%.
• Complexity/duplication audit: no maintainability blocker.
• Final Azure BigANN10M smoke: Recall@10 L=50 95.774%, build 77.918s, peak RSS 11.68 GiB.

Local runtime benchmark integration remains blocked by unavailable Git LFS (the fixture is a 123-byte pointer); CI's LFS checkout is the remaining runtime oracle.

Full report: /tmp/pipnn-qc-direct.md in the authoring environment.

[weiyaoluo (SeliMeli)]

QC follow-up

The first post-rebase CI run exposed a real AArch64-only failure in cosine_zero_norm_masks_nan_norm_at_simd_boundaries: NEON and x86 have different max(NaN) behavior, so a NaN pair could become distance zero on ARM. PR1 now preserves NaNs explicitly with a validity mask before selecting the non-negative clamp.

The replacement run is green on:

• Ubuntu AArch64 runtime tests
• x86-64 Nehalem/SDE baseline
• AVX-512/SDE
• Ubuntu and Windows default/all-feature tests
• coverage, CodeQL, formatting, and all Clippy matrices

Remote stack metadata was also rebuilt as one stack, #1301:
#1287 → #1288 → #1290 → #1291 → #1294 → #1295.
The obsolete closed HashPrune PR is no longer a stack member.

[Copilot]

Pull request overview

Adds an optional HashPrune/LSH-based candidate-merging path to PiPNN builds, wiring it through disk build configuration and benchmarks, and extending SIMD/mask utilities needed by the new kernels.

Changes:

• Introduces HashPrune reservoirs plus random-hyperplane LSH sketch computation, and integrates them into PiPNN leaf building / extraction (optionally followed by RobustPrune).
• Extends disk-build and benchmark pipelines to accept and validate HashPrune parameters for PiPNN.
• Adds supporting utilities (trusted adjacency-list constructor, mask helpers, SIMD eq optimization) and CI Miri coverage for the raw-pointer kernels.

Reviewed changes

Copilot reviewed 25 out of 26 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
diskann/src/graph/adjacencylist.rs	Adds from_vec_trusted for zero-copy construction when uniqueness is guaranteed.
diskann-wide/src/doubled.rs	Adds first() support for doubled masks.
diskann-wide/src/arch/x86_64/v3/i16x16_.rs	Optimizes SIMD equality mask generation using movemask+pext.
diskann-vector/src/lib.rs	Makes x86_64 prefetch helpers available without requiring AVX2.
diskann-pipnn/tests/config.rs	Adds validation tests for HashPruneConfig parameters.
diskann-pipnn/tests/build_graph.rs	Adds parallel HashPrune build invariant test.
diskann-pipnn/src/lsh.rs	New LSH sketch computation (seeded random hyperplanes) and errors.
diskann-pipnn/src/lib.rs	Adds HashPrune config/types and integrates candidate-merge selection into build_graph.
diskann-pipnn/src/leaf_build/tests.rs	Adds CSR construction tests used by HashPrune leaf streaming.
diskann-pipnn/src/leaf_build.rs	Refactors leaf computation; adds CSR edge streaming into HashPrune reservoirs.
diskann-pipnn/src/hash_prune.rs	New HashPrune implementation (hot/cold slabs, per-row locking, SIMD hash ops, extraction).
diskann-pipnn/src/hash_prune/tests.rs	Adds unit and concurrency tests for HashPrune kernels and reservoir behavior.
diskann-pipnn/src/bf16.rs	Adds bf16 packing helpers for compact distance keys.
diskann-pipnn/Cargo.toml	Adds new dependencies and a HashPrune benchmark target.
diskann-pipnn/benches/hash_prune.rs	Adds criterion benchmark comparing direct vs HashPrune merge paths.
diskann-disk/src/lib.rs	Re-exports HashPruneParameters when pipnn feature is enabled.
diskann-disk/src/build/mod.rs	Re-exports HashPruneParameters from configuration.
diskann-disk/src/build/configuration/mod.rs	Exposes HashPrune parameters in configuration module exports.
diskann-disk/src/build/configuration/build_algorithm.rs	Extends PiPNNParameters with hash_prune and serde defaults.
diskann-disk/src/build/configuration/disk_index_build_parameter.rs	Switches to returning borrowed PiPNN parameters for build selection.
diskann-disk/src/build/builder/build/pipnn.rs	Wires optional HashPrune parameters into PiPNNBuildContext.
diskann-disk/src/build/builder/build/pipnn/tests.rs	Updates PiPNN disk builder tests for new parameter passing.
diskann-disk/src/build/builder/build.rs	Validates PiPNN + HashPrune config when pipnn is selected.
diskann-benchmark/src/index/build.rs	Wires optional HashPrune parameters into benchmark PiPNN builds.
Cargo.lock	Adds new transitive dependencies for diskann-pipnn changes.
.github/workflows/nightly.yml	Improves feature quoting/formatting and adds Miri strict-provenance coverage for HashPrune kernels.

Comments suppressed due to low confidence (1)

diskann-disk/src/build/configuration/build_algorithm.rs:157

• If PiPNNParameters::default() is changed to keep hash_prune opt-in, this serde-defaults test should be updated to expect None instead of Some(HashPruneParameters::default()).

        assert_eq!(config.hash_prune, Some(HashPruneParameters::default()));

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 25 out of 26 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 25 out of 26 changed files in this pull request and generated no new comments.

Suppressed comments (1)

diskann-disk/src/build/configuration/build_algorithm.rs:82

• PiPNNParameters is #[serde(default)], so deserializing PiPNN configs that omit the new hash_prune field will inherit this Default value. Setting hash_prune: Some(HashPruneParameters::default()) therefore enables HashPrune by default and can change behavior for existing JSON configs that previously used direct candidate merging. If HashPrune is meant to be opt-in (as described), make the default None and require explicit configuration to enable it.

impl Default for PiPNNParameters {
    fn default() -> Self {
        Self {
            c_max: 256,
            c_min: 16,
            p_samp: 0.005,
            fanout: vec![8, 3],
            k: 2,
            replicas: 1,
            hash_prune: Some(HashPruneParameters::default()),
        }