Repository navigation
fix(family,ci): index the phased VCF without pysam and rebuild the fixture without a deleted script - #455
Open
geiresx wants to merge 8 commits into
Open
fix(family,ci): index the phased VCF without pysam and rebuild the fixture without a deleted script#455geiresx wants to merge 8 commits into
geiresx wants to merge 8 commits into
Conversation
…unreadable lines) Bead: D1
…uilder The fixture builder called bin/cyp2d6_depth_check.py, which GeiserX#372 deleted; it now prints the flanks with og pgx cyp2d6-bed. Bead: D1
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 31 minutes. View limit details
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
docs/rust.md no longer lists og vcf index as a stub, og-io's crate doc drops 'not ported yet', docs/testing.md drops python3 from the fixture build's tools, and the WhatsHap image smoke stops saying the module and step 38 index with pysam. Bead: D1
# Conflicts: # CHANGELOG.md
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
Four Python one-liners were still in what the pipeline runs.
WHATSHAP_PEDprocess indexed the phased family VCF withpython -c "import pysam; pysam.tabix_index(...)". That was our own Python inside the WhatsHap image, and 1.0.0 ships no owned Python.scripts/benchmark-variants.shresolved paths with apython3 -c os.path.realpathfallback on the host.scripts/ci/build-fixture.shstill calledbin/cyp2d6_depth_check.py bed, which refactor(pgx,str): run the CYP2D6 depth check and RFC1 motif in og, and drop their Python #372 deleted. The next fixture rebuild would stop there.The fix
og vcf index [-f] VCF.gzis ported (rust/crates/og-io/src/tabix_write.rs). It writesVCF.gz.tbithe way htslib 1.24 builds one. htslib 1.24 is the htslib of the pysam in the WhatsHap image (pysam 0.24.1) and ofBCFTOOLS_IMAGE. The inflated.tbihas the same bins in the same order as htslib's (its hash table order is emulated), the same chunks and the same linear index. Only the deflate stream can differ, because htslib and og use different deflate libraries. Every reader inflates both to the same bytes.It reads the VCF once, one BGZF block at a time, and allocates nothing per line. It writes atomically and refuses to write over an index without
-f, astabixdoes. A usage error (for example--og-scaffold-probe) exits 2 before any file is read.Step 38 runs
/pgp-bin/og vcf index -finPYTHON_IMAGE, like its otherogcalls.WHATSHAP_PEDrunsog vcf index -ffrom the task'sbin/.Steps 02b and the benchmark use
realpath, thenreadlink -f, then the path as given.The fixture builder prints the flanks with
og pgx cyp2d6-bed. It buildsogfirst whenrust/og.sh check --requiredfails, asscripts/alt-depth-ab.shdoes.python3leaves its tool list.Docs that described the old state are corrected, in files outside this lane's first list:
docs/rust.mdno longer listsog vcf indexas a stub, theog-iocrate doc (rust/crates/og-io/src/lib.rs) drops "not ported yet",docs/testing.mddropspython3from the fixture build's tools, and the WhatsHap image smoke intests/smoke/commands.tsvstops saying the module and step 38 index with pysam (the smoke itself still checks the image's pysam).The
.tbikeeps its name and format. No option, config key or output file name changes.Deliberate behaviour, named
These follow htslib 1.24, so they match what the pysam call did:
|SVLEN|of a<DEL>,<DUP>,<INV>or<CNV>ALT, and the largest sampleLENon a<*>or<NON_REF>block.Decisions
tabixbinary inBCFTOOLS_IMAGE. That binary is htslib 1.19. Its rules predate SVLEN and sample LEN, and it skips unreadable lines where 1.24 stops. The CI comparison usesbcftools index -t(htslib 1.24) from the same image.cargo test, with Docker on the Guard runner, so no workflow file changes. WithCIset and no Docker, the test fails. WithoutCIand without Docker, it says so and passes.How it was checked
cargo test -p og-io: 27 passed (10 new intabix_write):.vcf.gz.tbifiles in the repo that htslib wrote are rebuilt byte for byte (inflated).bcftools index -twrites the same index as og on 6 inputs: a synthetic VCF that crosses every bin level and odd BGZF block edges, with empty blocks, CRLF lines, SVLEN and gVCF LEN records; a quirks file; and 4 public HG002 VCFs.bcftools view -rreturns the same records over og's index as over htslib's, for whole sequences and 2 kb regions.strtollandatollas C reads them, the interval rules, the bins, andbgzf_telloffsets across empty blocks and at the end of the file.the_verb_writes_the_index_beside_the_vcffails:left: 2, right: 0.intervals_follow_tbx_parse1fails, and so doeshtslib_writes_the_same_index_and_reads_it_the_same("synthetic.vcf.gz: bcftools index -t writes another index").rebuilds_the_committed_htslib_indexes_byte_for_bytefails onabo.vcf.gz.whatshap phase --ped, 804 records), two public HG002 VCFs and a 5-million-record three-sample synthetic VCF.tabix -p vcf, built natively, 0.81 to 0.93 s, 19 MB peak. pysam runs the same htslib C code, plus a Python start in its image.og pgx cyp2d6-bed | awk '$4 == "flank"'prints the same two rows as the deletedcyp2d6_depth_check.py beddid (chr22 42050000 42100000 flank,chr22 42200000 42250000 flank).cargo fmt --check,cargo clippy --workspace --all-targets --all-features -D warningsandcargo doc -D warnings(og-io) are clean.og/tests/port_scaffold.rspasses.family-joint-1exercises both indexing paths. It needs the full fixture and GIAB streaming, so it runs in the landing train, not on the build host.Left for the landing train
scripts/ci/build-fixture.shchanged, sotests/fixtures/VERSIONmust move tofixture-v8and a maintainer must build that release (docs/testing.md, "Edit build-fixture.sh"). Without the bump, the e2ebuild-fixturejob fails on purpose. That file is outside this lane.build-fixturejob of this PR stops with "build-fixture.sh changed but tests/fixtures/VERSION is still fixture-v7". That is the guard working as designed, not a bug in this PR.Part of #411. Bead: D1 (owner decision: no owned Python in 1.0.0).