Repository navigation
fix(roh): measure F_ROH over the chromosomes the run covered - #415
Merged
Merged
Conversation
…iftover og archaic table|strict|genotypes|report and og ancient select|lift|person-geno|align|report port bin/archaic_report.py and bin/ancient_report.py. og-intervals gains chain, the UCSC chain reader with a single-position lift. Bead: pgp-bs9u.21
…d og ancient The parity cases compare og with bin/archaic_report.py and bin/ancient_report.py on synthetic inputs. setup.sh stamps the archaic table with a format id instead of the Python's hash, and takes a table the last Python built as current. Bead: pgp-bs9u.21
…ests, demo and evidence run og The step 43 and 44 tests drive og archaic and og ancient as the pipeline does and join Guard's OG_REQUIRED list; their unit checks of internals moved to the Rust tests. The docs demo writes step 44 with og ancient report, archaic_compare.py keeps its own copy of the IBDmix reader, and the archaic-share evidence job builds og. The table and VCF readers no longer build a path string per line. Bead: pgp-bs9u.21
Bead: pgp-bs9u.21
…hat holds its end Reverted in the next commit. Guard must go red: cargo test in og-ancestry and og-intervals, the archaic-table and ancient-lift parity cases, and tests/test_archaic_report.py. Bead: pgp-bs9u.21
# Conflicts: # CHANGELOG.md
Bead: pgp-bs9u.21
# Conflicts: # .github/workflows/guard.yml # CHANGELOG.md # docs/rust.md # tests/parity/cases.tsv # tests/parity/oracle.ref
…lect Without --only, og summary collect reads all three groups over one walk of the folder and runs the post pass: the readings that join sections, the cyp2d6 block, not_run, not_assessed, the database re-check and new_since. A part of the post pass that fails makes only its own section unreadable; the summary is still written. Bead: pgp-bs9u.44
Nine cases run og summary collect without --only against bin/collect_summary.py on folders that join the three groups' fixtures, with failed steps, an empty folder, kept runs in history/ and a database re-check. A telomere table with an infinite read length stopped the Python with an uncaught OverflowError; og keeps the summary with that section unreadable (the all-inf golden). Bead: pgp-bs9u.44
Bead: pgp-bs9u.44
not_run keeps the sections that are no step, a stale VNtyper reading counts as assessed, new_since takes the current run's date, and a panic in a post-pass part is no longer contained. The next commit removes them. Bead: pgp-bs9u.44
# Conflicts: # .github/workflows/guard.yml # CHANGELOG.md # tests/parity/cases.tsv # tests/parity/oracle.ref
# Conflicts: # CHANGELOG.md
Reverts 888cd36, the red control of this PR. Bead: pgp-bs9u.44
The images check reads any image name in a tracked file as one that bypasses versions.env; the fixture's DEEPVARIANT_IMAGE rows now hold plain version text. Bead: pgp-bs9u.44
Chromosomes 1 to 6 and 21 of the installed table, rebuilt by og from the same public inputs, equal the Python-built table after decompression. The synthetic table figure understated mask memory, so the changelog now gives the measured chromosome 1 cost.
# Conflicts: # CHANGELOG.md # tests/parity/cases.tsv # tests/parity/oracle.ref
# Conflicts: # CHANGELOG.md
The release profile aborts on a panic, so catch_unwind in contain() caught nothing in the shipped og. contain() now maps the error a part returns, and the docs say the parts read with defaults instead of promising to catch a panic.
…d refuse bad report inputs og archaic genotypes now stages samples.txt, every genotype file and every mask, and publishes them only when the whole run has passed. A chromosome with no row deletes its old genotype file only on success. The VCF order check compares each record with the last record handed to the join, so a skipped record of another chromosome cannot hide an unsorted return. og ancient report stops on a non-finite eigenvalue or person PC, and on an ancient row with fewer PCs than the distance uses. The parity goldens of the four failing genotypes cases now expect the previous outputs to stay (no new file).
…aic output is in place spans.tsv is now staged with the other outputs, and a chromosome's old genotype file is removed last. A failed publication keeps it.
render_report.py (step 24, HTML_REPORT and generate-report.sh) now runs og summary collect instead of the Python collector. generate-report.sh renders the text report from the summary step 24 wrote when nothing it reads has changed since, so a run collects each sample once. The folder walk leaves out the Nextflow launch folder run-all.sh keeps in the sample folder. collect_summary.py keeps a collect() that runs og, and only the names other Python still imports. Bead: pgp-bs9u.45
# Conflicts: # bin/collect_summary.py
The tests that called the Python section readers now read the section og summary collect writes for the same folder. Checks of a Python internal (the G2 reader, the probable CYP2D6 rule, the poly-C tracts, the imprinted regions, the repeat ranges table) move to og-summary's own cargo tests. The demo maker writes step 26b, 49 and 05 tables without the Python remnants. collect_summary.py imports re for half_call. Bead: pgp-bs9u.45
The supported builder stages rust at /src, so the table needs a read-only mount at /assets. Include its blob in the source stamp so an asset edit makes old binaries stale. Keep the stamp format and release-install checks. The new build checks failed five assertions against the previous builder and all pass with this change. The original missing-mount failure is upstream run 43658012836. Bead: lic-41 Bead: lin-17
The built site cannot link outside its own root. The strict local build passes, and site-check reports 97 pages with zero errors. Bead: lic-41
Restore and CRAM_TO_BAM turned MD/NM generation off for every CRAM, so a CRAM written without store_md/store_nm (an older step 34, or sarek) lost the tags samtools drops where they match the reference. Turn generation off only when the CRAM's last writer stored both tags, and decode the CRAM the same way on both sides of the content check. The e2e case now restores a plain CRAM in bash and Nextflow and checks NM, and proves the module content checks fail on a changed CRAM-side checksum.
An item also fails when the run's manifest lacks the value to compare, which no later lookup fixes, so the closing line counts looked-up items that failed and points at each item's own line for why. Bead: plat-26
Bead: pgp-iqon.11
…the landing train # Conflicts: # CHANGELOG.md
Replace iframe fragment navigation, keep the requested locus on resize, and add only one camera history entry from the Reads gauge. Bead: pgp-iqon.11
Keep the Reads navigation guard distinct from the two drawing guards the existing failed-reload control replaces. Bead: pgp-iqon.11
The dispatcher already routes freqs, project, paint and simulate, but the usage line named only the six ported verbs. A test now checks that the help names every verb.
The root .gitignore drops every results/ folder (Nextflow run output), so a new file under rust/crates/og-live/src/results/ was silently ignored. A crate-level .gitignore re-includes that one source folder.
…e landing train # Conflicts: # CHANGELOG.md
…nding train # Conflicts: # CHANGELOG.md
… the reads level A resize applied the URL's locus again even after the reader moved inside igv, so any resize threw the reader back. The viewer now follows the URL only until the reader pans, zooms or types a locus, and again after the next locus from the map. docs/genome-map.md now names the Reads stop, the R key, the Esc order and the card's See the reads button.
Bead: trait-40 Red first: all three new report tests fail on the pre-fix accumulator.
…e landing train # Conflicts: # CHANGELOG.md
An empty f64 sum in Rust is -0.0, so every sample without a run of 5 Mb or more read F_ROH -0.0000 in both reports. Fold from +0.0, check the sign in the scope test, and keep the ROH file's open and read errors named the way every other section names them.
Take GeiserX#413's version of the two e2e callers, so this PR no longer carries its own copy of those lines.
Keep called ALT identities, deduplicate repeated dosages and refuse conflicting or impossible totals. Both VCF readers now preserve split/unsplit tables and evidence. Bead: trait-40
… a ~ bash would expand An edited logs/run_status.tsv could name BASH_ENV, PATH or LD_PRELOAD in both 'meta switches' and 'meta command', and the replay then set it for the bash it starts. A switch is now a name in run-all.sh's SWITCHES list too (a unit test keeps the Rust copy equal to it). A ~ after an unquoted = or : is refused, since bash expands it there and printf %q never writes it so.
… train # Conflicts: # CHANGELOG.md
# Conflicts: # CHANGELOG.md
…or the landing train # Conflicts: # CHANGELOG.md
…anding train # Conflicts: # CHANGELOG.md
Collaborator
Author
|
@coderabbitai review |
Action not completed
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A chromosome-subset ROH run used all 22 autosomes as its denominator. It could show a reassuring relationship label or an unsupported UPD flag. The flag also listed syndrome names from a region lookup.
The Rust collector now streams the site rows and uses the assessed chromosomes for F_ROH. It embeds one attributed imprinting table and emits neutral chromosome and region hints.
Part of #411: lin-10, lin-13, lin-15, lin-16, lin-17, lic-41. Stacked on #413.
Output changes and decisions
roh.data.scopeadds the chromosomes, unmasked denominator, source, partial flag and scope note. Existing segment counts and the step 11 summary table keep their format."-"for reader compatibility. The note names deletion, acquired LOH and shared ancestry as alternatives. Copy neutrality and parental origin are not established.The retained report test follows #413's collector switch; only its ROH expectations change. The two e2e callers that still ran the old collector are #413's: this branch merged #413's head (cd1d2d8) and took its version of both. The only e2e change of its own is
bin/ogin thealign-read-count-gate-1dependency row, because that case now callsbin/og. Guard gains a planted control. The supported builder mounts the embedded table read-only. Its source fingerprint includes the table blob, so table edits make an old binary stale. The stamp format stays unchanged.tests/test_og_check.shcovers the mount, asset edits and release installation. The build notes inrust/README.mdanddocs/rust.mddescribe that input. Report and live wording for partial scope belong to the display lanes.Files outside the first plan list, each needed by the change:
.github/workflows/guard.yml(the planted control),rust/og.shandtests/test_og_check.sh(the builder mount and fingerprint for the embedded table),docs/rust.mdandrust/README.md(they describerust/og.sh),tests/test_collect_summary.py(its fixtures need site rows) andscripts/ci/e2e-case-paths.tsv(one dependency row).How it was checked
cargo test --no-fail-fast, exactly five tests fail. Restoring the source gives 116 passed, zero failed.long_mbas 0.0. An empty floating-point sum in Rust is -0.0, which the reports would print as-0.0000. The scope test now checks the sign, because -0.0 and 0.0 compare equal. Against the unfixed line it failed withf_roh must be +0.0 without long runs, got Float(-0.0)(cargo test -p og-summary roh_scope_countson the build host); with the fold from +0.0 all 116 og-summary tests pass. The two goldens that recorded -0.0 now read 0.0. The ROH file's open and read errors keep the error names every other section uses.cargo clippy --locked -p og-summary --all-targets -- -D warningspassed. Documentation built withRUSTDOCFLAGS=-Dwarnings.OG_REQUIRED=1 python3 tests/test_collect_summary.pyended withRESULT: ALL PASSon synthetic folders.summary-collectoracle covered all 71 cases: all 284 items agree, with seven ROH outputs held to goldens on purpose. Those goldens change only the ROH section. The parity self-test passed 15 harness checks and 44 comparator checks. The Linux shim shadow check remains CI's check.All upstream fast checks are green on
9b70dc299d78a8832b3a749d73e9eab24e06e14b: Guard with ci-ok, Rust, Lint, Nextflow, Docs, Container Test and Personal data. Guard includes the planted ROH control, normal Rust tests, full oracle parity and all three unit-test shards. E2E and landing are owned by the landing stage; the fork E2E run on this head is queued and was not waited on here. No new e2e case was added, and the two existing e2e callers are #413's. The Rust and report checks above exercise the collector on synthetic input. No real-genome run is required for this lane.Beads: lin-10, lin-13, lin-15, lin-16, lin-17, lic-41.