Skip to content

fix(roh): measure F_ROH over the chromosomes the run covered - #415

Merged
GeiserX merged 104 commits into
GeiserX:mainfrom
geiresx:pgp/i411-roh-scope
Nov 15, 2026
Merged

GeiserX merged 104 commits into
GeiserX:mainfrom
geiresx:pgp/i411-roh-scope

Conversation

@geiresx

@geiresx geiresx commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

A chromosome-subset ROH run used all 22 autosomes as its denominator. It could show a reassuring relationship label or an unsupported UPD flag. The flag also listed syndrome names from a region lookup.

The Rust collector now streams the site rows and uses the assessed chromosomes for F_ROH. It embeds one attributed imprinting table and emits neutral chromosome and region hints.

Part of #411: lin-10, lin-13, lin-15, lin-16, lin-17, lic-41. Stacked on #413.

Output changes and decisions

  • roh.data.scope adds the chromosomes, unmasked denominator, source, partial flag and scope note. Existing segment counts and the step 11 summary table keep their format.
  • A relationship label or UPD flag requires sites on all 22 autosomes and the full 2,658,001,522 bp unmasked length. No marker-count floor excludes sparse exome or chip input. Chromosome presence does not prove callable breadth within a chromosome.
  • Segment-only files use observed chromosomes as an unverified lower bound. They get neither label. Empty input has null F_ROH. This avoids treating an old run list as evidence of whole-genome scope.
  • UPD hints name regions and citations. The existing maternal and paternal keys remain as "-" for reader compatibility. The note names deletion, acquired LOH and shared ancestry as alternatives. Copy neutrality and parental origin are not established.
  • All imprinting rows use the same chromosome-wide lookup. Mixed iso/heterodisomy can leave an imprinted locus outside the ROH. Broad band names remain reference labels. This change does not invent diagnostic locus boundaries. The table credits its overview and per-row sources.

The retained report test follows #413's collector switch; only its ROH expectations change. The two e2e callers that still ran the old collector are #413's: this branch merged #413's head (cd1d2d8) and took its version of both. The only e2e change of its own is bin/og in the align-read-count-gate-1 dependency row, because that case now calls bin/og. Guard gains a planted control. The supported builder mounts the embedded table read-only. Its source fingerprint includes the table blob, so table edits make an old binary stale. The stamp format stays unchanged. tests/test_og_check.sh covers the mount, asset edits and release installation. The build notes in rust/README.md and docs/rust.md describe that input. Report and live wording for partial scope belong to the display lanes.

Files outside the first plan list, each needed by the change: .github/workflows/guard.yml (the planted control), rust/og.sh and tests/test_og_check.sh (the builder mount and fingerprint for the embedded table), docs/rust.md and rust/README.md (they describe rust/og.sh), tests/test_collect_summary.py (its fixtures need site rows) and scripts/ci/e2e-case-paths.tsv (one dependency row).

How it was checked

  • Red first: the test commit failed all five new scope tests against the unchanged reader. The Guard ROH control passed on the current head. It restores the wrong denominator, inserts syndrome text and corrupts a citation. With cargo test --no-fail-fast, exactly five tests fail. Restoring the source gives 116 passed, zero failed.
  • The container build exposed a missing asset mount in this red run. New builder tests failed five assertions against the prior script. After the mount and fingerprint fix, every build-check case passes, including the release-download cases. The packaged-binary CI build also passed on the current head.
  • The first docs-site check caught the table link leaving the built site in this red run. The repository URL fixes that link. The local strict Zensical build passes, and the site checker reports 97 pages with zero errors.
  • A sample with no run of 5 Mb or more must write F_ROH and long_mb as 0.0. An empty floating-point sum in Rust is -0.0, which the reports would print as -0.0000. The scope test now checks the sign, because -0.0 and 0.0 compare equal. Against the unfixed line it failed with f_roh must be +0.0 without long runs, got Float(-0.0) (cargo test -p og-summary roh_scope_counts on the build host); with the fold from +0.0 all 116 og-summary tests pass. The two goldens that recorded -0.0 now read 0.0. The ROH file's open and read errors keep the error names every other section uses.
  • cargo clippy --locked -p og-summary --all-targets -- -D warnings passed. Documentation built with RUSTDOCFLAGS=-Dwarnings.
  • OG_REQUIRED=1 python3 tests/test_collect_summary.py ended with RESULT: ALL PASS on synthetic folders.
  • The local summary-collect oracle covered all 71 cases: all 284 items agree, with seven ROH outputs held to goldens on purpose. Those goldens change only the ROH section. The parity self-test passed 15 harness checks and 44 comparator checks. The Linux shim shadow check remains CI's check.
  • Three timed repeats over 1.1 million synthetic site rows used 65.7-67.0 MB peak RSS before and 3.0-3.5 MB after. CPU time stayed near 0.2 seconds. Wall time under shared host load ranged from 1.40-21.46 seconds before and 7.89-38.16 seconds after. These measurements establish the memory reduction, not a speedup.

All upstream fast checks are green on 9b70dc299d78a8832b3a749d73e9eab24e06e14b: Guard with ci-ok, Rust, Lint, Nextflow, Docs, Container Test and Personal data. Guard includes the planted ROH control, normal Rust tests, full oracle parity and all three unit-test shards. E2E and landing are owned by the landing stage; the fork E2E run on this head is queued and was not waited on here. No new e2e case was added, and the two existing e2e callers are #413's. The Rust and report checks above exercise the collector on synthetic input. No real-genome run is required for this lane.

Beads: lin-10, lin-13, lin-15, lin-16, lin-17, lic-41.

…iftover

og archaic table|strict|genotypes|report and og ancient
select|lift|person-geno|align|report port bin/archaic_report.py and
bin/ancient_report.py. og-intervals gains chain, the UCSC chain reader
with a single-position lift.

Bead: pgp-bs9u.21
…d og ancient

The parity cases compare og with bin/archaic_report.py and
bin/ancient_report.py on synthetic inputs. setup.sh stamps the archaic
table with a format id instead of the Python's hash, and takes a table
the last Python built as current.

Bead: pgp-bs9u.21
…ests, demo and evidence run og

The step 43 and 44 tests drive og archaic and og ancient as the pipeline
does and join Guard's OG_REQUIRED list; their unit checks of internals
moved to the Rust tests. The docs demo writes step 44 with og ancient
report, archaic_compare.py keeps its own copy of the IBDmix reader, and
the archaic-share evidence job builds og. The table and VCF readers no
longer build a path string per line.

Bead: pgp-bs9u.21
…hat holds its end

Reverted in the next commit. Guard must go red: cargo test in
og-ancestry and og-intervals, the archaic-table and ancient-lift parity
cases, and tests/test_archaic_report.py.

Bead: pgp-bs9u.21
# Conflicts:
#	.github/workflows/guard.yml
#	CHANGELOG.md
#	docs/rust.md
#	tests/parity/cases.tsv
#	tests/parity/oracle.ref
…lect

Without --only, og summary collect reads all three groups over one walk
of the folder and runs the post pass: the readings that join sections,
the cyp2d6 block, not_run, not_assessed, the database re-check and
new_since. A part of the post pass that fails makes only its own section
unreadable; the summary is still written.

Bead: pgp-bs9u.44
Nine cases run og summary collect without --only against
bin/collect_summary.py on folders that join the three groups' fixtures,
with failed steps, an empty folder, kept runs in history/ and a database
re-check. A telomere table with an infinite read length stopped the
Python with an uncaught OverflowError; og keeps the summary with that
section unreadable (the all-inf golden).

Bead: pgp-bs9u.44
not_run keeps the sections that are no step, a stale VNtyper reading
counts as assessed, new_since takes the current run's date, and a panic
in a post-pass part is no longer contained. The next commit removes them.

Bead: pgp-bs9u.44
# Conflicts:
#	.github/workflows/guard.yml
#	CHANGELOG.md
#	tests/parity/cases.tsv
#	tests/parity/oracle.ref
Reverts 888cd36, the red control of this PR.

Bead: pgp-bs9u.44
The images check reads any image name in a tracked file as one that
bypasses versions.env; the fixture's DEEPVARIANT_IMAGE rows now hold
plain version text.

Bead: pgp-bs9u.44
Chromosomes 1 to 6 and 21 of the installed table, rebuilt by og from the
same public inputs, equal the Python-built table after decompression. The
synthetic table figure understated mask memory, so the changelog now gives
the measured chromosome 1 cost.
# Conflicts:
#	CHANGELOG.md
#	tests/parity/cases.tsv
#	tests/parity/oracle.ref
The release profile aborts on a panic, so catch_unwind in contain() caught
nothing in the shipped og. contain() now maps the error a part returns, and
the docs say the parts read with defaults instead of promising to catch a
panic.
…d refuse bad report inputs

og archaic genotypes now stages samples.txt, every genotype file and every
mask, and publishes them only when the whole run has passed. A chromosome
with no row deletes its old genotype file only on success. The VCF order
check compares each record with the last record handed to the join, so a
skipped record of another chromosome cannot hide an unsorted return.

og ancient report stops on a non-finite eigenvalue or person PC, and on
an ancient row with fewer PCs than the distance uses.

The parity goldens of the four failing genotypes cases now expect the
previous outputs to stay (no new file).
…aic output is in place

spans.tsv is now staged with the other outputs, and a chromosome's old
genotype file is removed last. A failed publication keeps it.
render_report.py (step 24, HTML_REPORT and generate-report.sh) now runs
og summary collect instead of the Python collector. generate-report.sh
renders the text report from the summary step 24 wrote when nothing it
reads has changed since, so a run collects each sample once. The folder
walk leaves out the Nextflow launch folder run-all.sh keeps in the sample
folder. collect_summary.py keeps a collect() that runs og, and only the
names other Python still imports.

Bead: pgp-bs9u.45
The tests that called the Python section readers now read the section og
summary collect writes for the same folder. Checks of a Python internal
(the G2 reader, the probable CYP2D6 rule, the poly-C tracts, the
imprinted regions, the repeat ranges table) move to og-summary's own
cargo tests. The demo maker writes step 26b, 49 and 05 tables without the
Python remnants. collect_summary.py imports re for half_call.

Bead: pgp-bs9u.45
The supported builder stages rust at /src, so the table needs a read-only mount at /assets. Include its blob in the source stamp so an asset edit makes old binaries stale. Keep the stamp format and release-install checks.

The new build checks failed five assertions against the previous builder and all pass with this change. The original missing-mount failure is upstream run 43658012836.

Bead: lic-41
Bead: lin-17
The built site cannot link outside its own root. The strict local build passes, and site-check reports 97 pages with zero errors.

Bead: lic-41
Restore and CRAM_TO_BAM turned MD/NM generation off for every CRAM, so a
CRAM written without store_md/store_nm (an older step 34, or sarek) lost
the tags samtools drops where they match the reference. Turn generation
off only when the CRAM's last writer stored both tags, and decode the
CRAM the same way on both sides of the content check. The e2e case now
restores a plain CRAM in bash and Nextflow and checks NM, and proves the
module content checks fail on a changed CRAM-side checksum.
An item also fails when the run's manifest lacks the value to compare,
which no later lookup fixes, so the closing line counts looked-up items
that failed and points at each item's own line for why.

Bead: plat-26
…the landing train

# Conflicts:
#	CHANGELOG.md
Replace iframe fragment navigation, keep the requested locus on resize, and add only one camera history entry from the Reads gauge.

Bead: pgp-iqon.11
Keep the Reads navigation guard distinct from the two drawing guards the existing failed-reload control replaces.

Bead: pgp-iqon.11
The dispatcher already routes freqs, project, paint and simulate, but the
usage line named only the six ported verbs. A test now checks that the help
names every verb.
The root .gitignore drops every results/ folder (Nextflow run output), so a
new file under rust/crates/og-live/src/results/ was silently ignored. A
crate-level .gitignore re-includes that one source folder.
… the reads level

A resize applied the URL's locus again even after the reader moved inside
igv, so any resize threw the reader back. The viewer now follows the URL
only until the reader pans, zooms or types a locus, and again after the
next locus from the map. docs/genome-map.md now names the Reads stop, the
R key, the Esc order and the card's See the reads button.
Bead: trait-40

Red first: all three new report tests fail on the pre-fix accumulator.
An empty f64 sum in Rust is -0.0, so every sample without a run of
5 Mb or more read F_ROH -0.0000 in both reports. Fold from +0.0, check
the sign in the scope test, and keep the ROH file's open and read errors
named the way every other section names them.
Take GeiserX#413's version of the two e2e callers, so this PR no longer
carries its own copy of those lines.
Keep called ALT identities, deduplicate repeated dosages and refuse conflicting or impossible totals. Both VCF readers now preserve split/unsplit tables and evidence.

Bead: trait-40
… a ~ bash would expand

An edited logs/run_status.tsv could name BASH_ENV, PATH or LD_PRELOAD in
both 'meta switches' and 'meta command', and the replay then set it for
the bash it starts. A switch is now a name in run-all.sh's SWITCHES list
too (a unit test keeps the Rust copy equal to it). A ~ after an unquoted
= or : is refused, since bash expands it there and printf %q never
writes it so.
…or the landing train

# Conflicts:
#	CHANGELOG.md
@geiresx

geiresx commented Nov 15, 2026

Copy link
Copy Markdown
Collaborator Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@GeiserX
GeiserX merged commit 9078a1d into GeiserX:main Nov 15, 2026
41 of 51 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants