Skip to content

fix(family,ci): index the phased VCF without pysam and rebuild the fixture without a deleted script - #455

Open
geiresx wants to merge 8 commits into
GeiserX:mainfrom
geiresx:pgp/port-step-oneliners
Open

geiresx wants to merge 8 commits into
GeiserX:mainfrom
geiresx:pgp/port-step-oneliners

Conversation

@geiresx

@geiresx geiresx commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

The problem

Four Python one-liners were still in what the pipeline runs.

  • Step 38 and the Nextflow WHATSHAP_PED process indexed the phased family VCF with python -c "import pysam; pysam.tabix_index(...)". That was our own Python inside the WhatsHap image, and 1.0.0 ships no owned Python.
  • Step 02b and scripts/benchmark-variants.sh resolved paths with a python3 -c os.path.realpath fallback on the host.
  • scripts/ci/build-fixture.sh still called bin/cyp2d6_depth_check.py bed, which refactor(pgx,str): run the CYP2D6 depth check and RFC1 motif in og, and drop their Python #372 deleted. The next fixture rebuild would stop there.

The fix

  • og vcf index [-f] VCF.gz is ported (rust/crates/og-io/src/tabix_write.rs). It writes VCF.gz.tbi the way htslib 1.24 builds one. htslib 1.24 is the htslib of the pysam in the WhatsHap image (pysam 0.24.1) and of BCFTOOLS_IMAGE. The inflated .tbi has the same bins in the same order as htslib's (its hash table order is emulated), the same chunks and the same linear index. Only the deflate stream can differ, because htslib and og use different deflate libraries. Every reader inflates both to the same bytes.

  • It reads the VCF once, one BGZF block at a time, and allocates nothing per line. It writes atomically and refuses to write over an index without -f, as tabix does. A usage error (for example --og-scaffold-probe) exits 2 before any file is read.

  • Step 38 runs /pgp-bin/og vcf index -f in PYTHON_IMAGE, like its other og calls. WHATSHAP_PED runs og vcf index -f from the task's bin/.

  • Steps 02b and the benchmark use realpath, then readlink -f, then the path as given.

  • The fixture builder prints the flanks with og pgx cyp2d6-bed. It builds og first when rust/og.sh check --required fails, as scripts/alt-depth-ab.sh does. python3 leaves its tool list.

  • Docs that described the old state are corrected, in files outside this lane's first list: docs/rust.md no longer lists og vcf index as a stub, the og-io crate doc (rust/crates/og-io/src/lib.rs) drops "not ported yet", docs/testing.md drops python3 from the fixture build's tools, and the WhatsHap image smoke in tests/smoke/commands.tsv stops saying the module and step 38 index with pysam (the smoke itself still checks the image's pysam).

The .tbi keeps its name and format. No option, config key or output file name changes.

Deliberate behaviour, named

These follow htslib 1.24, so they match what the pysam call did:

  • A record ends at the largest of INFO/END (when after POS), the REF length, the |SVLEN| of a <DEL>, <DUP>, <INV> or <CNV> ALT, and the largest sample LEN on a <*> or <NON_REF> block.
  • These stop the index with a message, and nothing is written: a line with no position to read, a sequence that starts again after another one, positions that go back, a record past 2^29, and a file that is not BGZF.

Decisions

  • The reference is htslib 1.24, not the tabix binary in BCFTOOLS_IMAGE. That binary is htslib 1.19. Its rules predate SVLEN and sample LEN, and it skips unreadable lines where 1.24 stops. The CI comparison uses bcftools index -t (htslib 1.24) from the same image.
  • The CI comparison against htslib runs inside cargo test, with Docker on the Guard runner, so no workflow file changes. With CI set and no Docker, the test fails. Without CI and without Docker, it says so and passes.

How it was checked

  • cargo test -p og-io: 27 passed (10 new in tabix_write):
    • The 11 committed .vcf.gz.tbi files in the repo that htslib wrote are rebuilt byte for byte (inflated).
    • In the pipeline's bcftools image, bcftools index -t writes the same index as og on 6 inputs: a synthetic VCF that crosses every bin level and odd BGZF block edges, with empty blocks, CRLF lines, SVLEN and gVCF LEN records; a quirks file; and 4 public HG002 VCFs. bcftools view -r returns the same records over og's index as over htslib's, for whole sequences and 2 kb regions.
    • Region queries through og's index equal a full scan over more than 10,000 hits.
    • Planted control inside the test: moving one chunk start to its chunk end makes the same region check find wrong regions.
    • Unit tests for strtoll and atoll as C reads them, the interval rules, the bins, and bgzf_tell offsets across empty blocks and at the end of the file.
  • Red first (run on a build host):
    • With the verb body put back to the stub, the_verb_writes_the_index_beside_the_vcf fails: left: 2, right: 0.
    • With SVLEN dropped from the end, intervals_follow_tbx_parse1 fails, and so does htslib_writes_the_same_index_and_reads_it_the_same ("synthetic.vcf.gz: bcftools index -t writes another index").
    • With bins written in sorted order instead of htslib's hash order, rebuilds_the_committed_htslib_indexes_byte_for_byte fails on abo.vcf.gz.
  • Against the pysam call it replaces, in the WhatsHap image: the inflated index is identical on a WhatsHap-phased trio VCF (whatshap phase --ped, 804 records), two public HG002 VCFs and a 5-million-record three-sample synthetic VCF.
  • Speed and memory, on that 5-million-record VCF (23 MB BGZF), 3 runs each on an Apple M-series host: og 0.75 to 0.91 s, 31 MB peak; htslib 1.24 tabix -p vcf, built natively, 0.81 to 0.93 s, 19 MB peak. pysam runs the same htslib C code, plus a Python start in its image.
  • Fixture builder: og pgx cyp2d6-bed | awk '$4 == "flank"' prints the same two rows as the deleted cyp2d6_depth_check.py bed did (chr22 42050000 42100000 flank, chr22 42200000 42250000 flank).
  • cargo fmt --check, cargo clippy --workspace --all-targets --all-features -D warnings and cargo doc -D warnings (og-io) are clean. og/tests/port_scaffold.rs passes.
  • e2e: family-joint-1 exercises both indexing paths. It needs the full fixture and GIAB streaming, so it runs in the landing train, not on the build host.

Left for the landing train

  • scripts/ci/build-fixture.sh changed, so tests/fixtures/VERSION must move to fixture-v8 and a maintainer must build that release (docs/testing.md, "Edit build-fixture.sh"). Without the bump, the e2e build-fixture job fails on purpose. That file is outside this lane.
  • The build-fixture job of this PR stops with "build-fixture.sh changed but tests/fixtures/VERSION is still fixture-v7". That is the guard working as designed, not a bug in this PR.

Part of #411. Bead: D1 (owner decision: no owned Python in 1.0.0).

@coderabbitai

coderabbitai Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 31 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 9078a1d and 2115d9f.

Files ignored due to path filters: 1

  • tests/smoke/commands.tsv is excluded by !**/*.tsv
Files selected for processing: 10

  • CHANGELOG.md
  • docs/rust.md
  • docs/testing.md
  • modules/local/whatshap_ped/main.nf
  • rust/crates/og-io/src/lib.rs
  • rust/crates/og-io/src/tabix_write.rs
  • scripts/02b-alignment-longread.sh
  • scripts/38-family.sh
  • scripts/benchmark-variants.sh
  • scripts/ci/build-fixture.sh
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

docs/rust.md no longer lists og vcf index as a stub, og-io's crate doc
drops 'not ported yet', docs/testing.md drops python3 from the fixture
build's tools, and the WhatsHap image smoke stops saying the module and
step 38 index with pysam.

Bead: D1

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants