feat: add build-kg --threads to control parallel fullmap reads

[SkyeAv]

build-kg gains a --threads flag for the fullmap reads behind entity resolution — and those reads can finally scale past the 16 RECORDS shard files, because the fan-out no longer stops at one reader per shard.

CLI plumbing

• Flag: build-kg accepts --threads/-t (src/tablassert/cli.py), mirroring build-fullmap --threads; a non-positive value exits 2 before any build work starts, matching the --gepa-threads gate.
• Forwarding: threads flows build_kg → build_pipeline → build_graph_pipeline → the stage-3 Tcode.model_validate dict; Tcode gains a threads: int | None field (src/tablassert/lib.py).
• Op wiring: the resolve_batch op in _node_ops spells the "_two" tag explicitly so self.threads can follow positionally (compile_subgraph applies op args positionally); from there the existing resolve_batch → lookup_rows → rs.lookup_fullmap_terms plumbing carries it into Rust unchanged.

Rust fan-out beyond 16 shards

• Why the cap wasn't redb's: redb readers hold a SHARED lock and each opens its own read transaction — any number of readers may read one shard file concurrently; only a writer conflicts. The ≤16 ceiling was an implementation choice: lookup_pair_terms_db spawned exactly one reader per non-empty shard bucket.
• Change: new split_counts / split_bucket / plan_shard_jobs (rust/src/fullmap.rs): when workers exceeds the non-empty shard count, the busiest buckets are split across additional readers of the SAME shard file (greedy most-terms-per-reader, deterministic, never more readers than a bucket has terms). workers ≤ shard count is byte-identical to the previous behavior.
• Determinism: hits stay tagged by input index and are re-merged by sort, so results are content- and order-identical to the serial path at any worker count — asserted by the new Rust test at 2 × SHARD_COUNT_SHARDS.

Design

• Default unchanged in intent: unset keeps the auto behavior (large batches ≥ 1024 terms fan out, small ones stay serial); default_lookup_workers and LOOKUP_PARALLEL_MIN untouched.
• Accepted caveat: gains are workload-dependent — largest when lookups are I/O-bound (fullmap larger than page cache on multi-hour builds), near-neutral when fully cached.
• Accepted caveat: -tc (the long-removed --table-config alias) now parses as the cluster -t c and is rejected with a coercion error instead of unknown-option; still unusable — the pinning test was updated to the new rejection mode.
• Deferred: the agent's measurement builds (build_graph_pipeline calls in src/tablassert/agent.py) stay on auto behavior.

Docs

• docs/cli.md — build-kg --threads row (enforced by the live-surface guardrail tests/test_docs_cli_coverage.py).
• CHANGELOG.md — Unreleased → Added entry.

Testing

• make check → exit 0: ruff check all passed, ruff format --check + cargo fmt --check clean, pyright 0 errors.
• uv run pytest -q → 898 passed, 35 skipped.
• cargo test → 72 lib + 10 + 14 integration tests pass, incl. the new split_counts/split_bucket unit tests and workers_above_shard_count_still_match_serial.
• cargo clippy --all-targets -- -D warnings → clean.
• Manual: tablassert build-kg --help shows the flag; tablassert build-kg x.yaml --threads 0 → exit 2 with --threads must be a positive integer before any build work.

Questions for the reviewer

• Default fan-out on big hosts. threads=None still resolves to available_parallelism, which is no longer silently truncated at 16 — default builds on >16-core hosts now spawn more shard readers than before. Intended, or should the default stay capped at the shard count?

[coderabbitai]

Important

Review available on request

• 🔍 Trigger review

Reviews should be triggered manually for repositories with fewer than 10 stars. Select Trigger review above or comment @coderabbitai review to review the latest changes. For a full review, comment @coderabbitai full review.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 52e35e7d-5bee-4c68-be99-a500d669310a

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}