feat(rust)!: decompress and validate the prebuilt fullmap in Rust, matching a force build

[SkyeAv]

Summary

The prebuilt fullmap.tar.zst that build-fullmap downloads is now decompressed, extracted, validated, and installed entirely in the Rust extension — streaming zstd→tar with the GIL released — and the installed bundle is checked against the exact contract a --force build produces before anything is renamed into place.

Changes

• What was changed

  Rust extraction — new rs.extract_prebuilt_fullmap(archive, output, progress=None) in rust/src/fullmap.rs streams File → 8 MiB BufReader → zstd::Decoder → tar::Archive (the multi-GB decompressed tar never touches disk), extracts into a dot-prefixed temp dir on the output's filesystem, validates the bundle, then atomically renames the primary → output and shard i → <output-stem>.s<i>.redb. The whole extract→validate→rename runs inside py.detach, mirroring build_fullmap_db; the optional progress callback re-acquires the GIL briefly per member ("opening archive", "extracting <entry>", "validating"). Adds zstd (libzstd, vendored) + tar deps.

  Force-build validation — before any rename, the extracted bundle must satisfy the same contract the read path enforces: the primary opens read-only, META schema is exactly tablassert.fullmap.v5 (older tags are rejected loudly, not warned), build_id is present, the shard set is exactly s0..s{shards-1} with no gaps or extras, and every shard's build_id equals the primary's. It reuses the read path's own helpers (validate_schema, read_build_id, shard_count_of, open_read_only), so the contract cannot drift.

  Hardening — symlink/device entries, PAX GNU.sparse.* records (tar-rs ignores them, which would silently corrupt a bsdtar-packed archive), .. path traversal, multiple unnamed primaries, and non-zstd garbage (a 4-byte frame-magic pre-check; zstd::Decoder::new is lazy) are all rejected loudly. Every failure leaves nothing beside output and removes the temp dir.

  CLI wiring — fetch_prebuilt_fullmap keeps download + sha256 orchestration and delegates extraction to the new _extract_prebuilt_fullmap seam (Rust RuntimeError → PrebuiltFullmapUnavailable, chained; KeyboardInterrupt passes through). _extract_zst_tar/_stream_tar and their shutil/tarfile/tempfile imports are deleted. Fallback semantics are unchanged: any failure falls back to the from-scratch BABEL build, and the archive is kept on failure so the retry reuses the cached download.

• Why this change is needed

  Extraction previously ran in Python — native tarfile zstd on 3.14+ or a zstd subprocess pipe on older interpreters — holding the GIL for the multi-GB decode and adding a process boundary in the IO path. Moving it into the extension makes the decompression a single streaming Rust pipeline with the GIL released. And nothing verified that what RENCI published actually matched a local build: a torn or wrong-version archive would install silently and only fail later at lookup time. Validation now happens before install, so a bad prebuilt falls back to the build path instead of landing a broken database.

• How it affects the project

  User-visible: faster extraction, progress rendering preserved, and an invalid prebuilt now surfaces as the existing "fall back to BABEL build" path instead of a silently broken DB. The zstd system binary is no longer consulted at all — Python 3.11–3.13 no longer need it installed.

Breaking change

Internals-only; no user-facing migration. The private cli._extract_zst_tar/cli._stream_tar helpers are gone (the monkeypatch seam is now cli._extract_prebuilt_fullmap). Two environment deltas: building from source now needs a C compiler for the vendored libzstd (cc — already present in CI and any Rust toolchain), and the runtime zstd binary requirement on Python < 3.14 is dropped, not added.

Docs

• docs/fullmap.md and docs/cli.md — the prebuilt path now describes the Rust-side streaming extraction and the validate-before-install contract (no stale tarfile/zstd-binary wording remains).
• CHANGELOG.md — ## Unreleased → ### Changed entry covering the move, the validation contract, and the loud rejections.

Testing

• Commands run and results
  • make check — exit 0: ruff + ruff format clean, pyright 0 errors, uv run pytest 883 passed, 35 skipped, cargo test 69 + 10 + 14 passed (1 ignored), cargo clippy --all-targets -- -D warnings clean, cargo fmt --check clean.
  • cargo test --manifest-path rust/Cargo.toml --test extract_prebuilt — 14 passed: the positive round-trip extracts a real force-built fixture (custom stem, nested members) and pins the canonical term → CURIEs map identical to the source DB, plus the progress-callback contract (opening archive → per-member extracting → validating); 12 negatives cover corrupt/garbage, missing primary, shard gap, extra shard, outdated schema (v1 rewrite via writable redb copy), non-redb primary, traversal, symlink, missing archive, PAX-sparse, duplicate/stray primaries — each asserting actionable error text, nothing landed beside output, and no temp-dir residue.
  • uv run pytest tests/test_cover_cli.py -q — 49 passed, including the real end-to-end test: a force-built DB packed into a real .tar.zst, downloaded through a faked download_babel_file, extracted by the real Rust path with zero extract monkeypatching, and pinned to answer rs.lookup_fullmap_terms identically to the force-built DB.
  • rust/tests/build_golden.rs stays byte-exact after its helpers moved to rust/tests/common/mod.rs (shared with the new test): 10 passed including golden_output_is_pinned.

Deferred

• A tar --sparse-packed fixture test for the GNUSparse expansion branch, and negatives for duplicate shard indices across subdirectories and a .redb-suffixed directory — behavior is correct (verified live during audit) but unpinned.
• Concurrent extractions to the same output share the fixed temp-dir name; the failure is loud and cannot land a corrupt bundle (documented in-source), but no test pins it.

[coderabbitai]

Important

Review available on request

• 🔍 Trigger review

Reviews should be triggered manually for repositories with fewer than 10 stars. Select Trigger review above or comment @coderabbitai review to review the latest changes. For a full review, comment @coderabbitai full review.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: dbdff87e-51a3-434f-a553-5260ef036de1

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}