Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,10 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n

## Unreleased

- The tests of `og sv postfilter`, `og sv inv-filter`, `og sv svlen-fix`, `og sv add-callers`, `og clinvar join-adjacent`, `og clinvar hits`, `og clinvar deep-intronic` and `og str catalog` now run in cargo test, in the crates that own the tools, with the same inputs and the same red controls. Four Python test files are gone: `tests/test_annotsv_postfilter.py`, `tests/test_annotsv_sv_inputs.py`, `tests/test_clinvar_join_adjacent.py` and `tests/test_str_catalog.py`. The checks of what `og str motif`'s RFC1 goldens must say run in og-bam's tests; `tests/test_rfc1_motif.py` stays for now, because three e2e cases and the og-bam parity generator import its read sets. The STRchive tests download the pinned files and run Stranger in docker, so they run in CI, or locally with `OG_NET_TESTS=1`. No output changes.
- ROH summaries now divide F_ROH by the GRCh38 length outside the bands of the assessed autosomes. Per-site rows identify those chromosomes, including chromosomes with no ROH. A new `roh.data.scope` records the chromosomes, denominator and scope source. Relationship labels and UPD flags require sites on all 22 autosomes and the full unmasked autosome length. Segment-only files use their observed chromosomes as an unverified lower bound and get neither label. Empty input has null F_ROH. Segment counts and the step 11 summary table keep their format.
- A one-chromosome ROH flag now names regions and citations without syndrome names. Its note includes deletion, acquired loss of heterozygosity and shared ancestry as alternatives. The maternal and paternal JSON keys remain as `"-"` for reader compatibility. The attributed `assets/imprinted_regions.tsv` is embedded at build time, replacing the duplicated Rust table. The container builder mounts it, and the source stamp includes its blob hash, so a table edit makes an old binary stale. The stamp format and `tree-hash` command name stay the same. Every listed chromosome uses the same lookup, including chr11 and chr15. A region hint does not establish overlap with its imprinted locus or parental origin. Broad cytoband names remain reference labels, not diagnostic intervals.
- The ROH collector now streams per-site rows instead of keeping the whole file in memory. On 1.1 million synthetic site rows, peak RSS fell from 65.7-67.0 MB to 3.0-3.5 MB. CPU time stayed near 0.2 seconds. Under shared host load, wall time ranged from 1.40-21.46 seconds before and 7.89-38.16 seconds after; these timings do not establish a speedup.

- The reports now get summary.json from the Rust `og` binary. Step 24, the Nextflow HTML_REPORT process and `generate-report.sh` collect it with `og summary collect` (through `bin/render_report.py`), and a run collects each sample once: `generate-report.sh` renders the text report from the summary.json step 24 wrote, and collects again when a file it reads is newer than that summary.json, or a file was added to or removed from a step folder since, or a file a section names as its source, the run status or a table in `bin/` or `assets/` is gone. When it renders from step 24's summary.json, the text report's generated time is step 24's, the same as the HTML report's; it was the minute of its own collect. When it collects again, the text report carries the new collect's values and time, and the HTML report keeps step 24's until step 24 runs again, as before. The deliberate fixes the `og summary collect` entries below list now reach the reports; apart from them, nothing in summary.json or either report changes. `bin/collect_summary.py` keeps only what the report renderer (`bin/render_report.py`), the couple report, the results index and the live view import from it. The Python modules that only the old collector imported are gone: `bin/pgx_outside_calls.py`, `ancestry_fine_report.py`, `clip_check.py`, `annotsv_postfilter.py`, `chip_report.py`, `traits_report.py`, `run_delta.py` and `genotype_rules_indels.py`. On the synthetic demo sample (4.7 million VCF records) the collect takes 1.1 s of CPU and 12-15 MB, where the Python took 7.8-8.0 s and 34-37 MB. What changes on purpose:
- The walk of the sample folder leaves out `<sample>/nextflow`. `run-all.sh` launches Nextflow there, and its `work/` holds every task's files and staged links. No section reads a file there. With 200,000 files under it, the collect took 67 MB walked and 14 MB skipped.
- The sections other than the sample VCF are read one at a time, while the sample VCF is read on its own thread, so the peak memory is near the biggest section's and not the sum of all of them. With every side table 50 times its demo size, the peak footprint fell from 119-130 MB to 105-112 MB (the Python: 100-102 MB), with the same CPU time.
Expand Down
2 changes: 1 addition & 1 deletion docs/06-clinvar-screen.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ In the Nextflow pipeline the same floor is `ext.min_gq` and `ext.min_dp` of `CLI
10. Runs the cDNA read check on each hit and the cDNA scan of every exon (below).
11. Prints the hits grouped by ClinVar's review stars (`og clinvar hits`): 4 practice guideline, 3 expert panel, 2 multiple submitters with no conflict, 1 a single submitter or conflicting classifications, 0 no assertion criteria, each with its combined-allele note when it has one and "LOW-CONFIDENCE CALL (GQ .., DP ..): check the reads" when the call is below the floor, and the count of low-confidence calls. The screening file keeps every Pathogenic/Likely_pathogenic submission whatever its review status, so a zero-star hit counts as a hit; the stars say how much weight it deserves. The Nextflow module runs the same steps and prints the same counts; its `CLINVAR_CDNA_CHECK` process runs the cDNA read check and the cDNA scan.

A delins that the caller writes as two records, while ClinVar holds it as one, is matched exactly once, as the combined record of item 5: `tests/test_clinvar_join_adjacent.py` check 8 runs the real step on such a fixture and counts the hits.
A delins that the caller writes as two records, while ClinVar holds it as one, is matched exactly once, as the combined record of item 5: the e2e case `clinvar-combined-allele-1` runs the real step on such a fixture and counts the hits, and `rust/crates/og-clinvar/tests/port_tests_sv_clinvar_str_join.rs` checks og's part on the same fixture.

### The always-report list

Expand Down
2 changes: 1 addition & 1 deletion docs/09-str-expansions.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ The reports read the counts (`og summary collect`, under the repeat expansions s
- **UGT1A1**: the TA count of each copy as its allele (`*36`, `*1`, `*28`, `*37`), and the number of decreased-function copies (`*28`, `*37`) compared with PharmCAT's UGT1A1 call of step 27, which reads `*28` from the small-variant calls with the reference-block fill of `og pgx refblock-fill`. The line says whether the two agree; when they differ, read the reads before using either. A TA length outside TA5 to TA8 has no named allele, so the line makes no function claim and does not compare it. A step 27 table from an earlier run is compared and said to be so; one of a failed or unreadable step is not compared. PharmCAT's call is still the one step 27's drug guidance uses.
- Without the opt-in, both lines say the repeat was not sized and give the command that adds it: `STR_EXTRA_LOCI=yes ./scripts/setup.sh --strchive-data <genome_dir>`.

`tests/test_str_catalog.py` checks each region against the GRCh38 sequence, `tests/test_collect_summary.py` checks the readings on made-up counts (a TG11-7T/TG11-7T sample is the red control of the 5T line), the e2e case `genotype-rule-rows-1-str-extra` sizes both tracts on made-up reads with ExpansionHunter 5.0.0, and the e2e case `str-extra-repeats-1-readings` runs step 09 and both reports on made-up reads and checks the lines.
`rust/crates/og-setup/tests/strchive_catalog.rs` checks each region against the GRCh38 sequence, `tests/test_collect_summary.py` checks the readings on made-up counts (a TG11-7T/TG11-7T sample is the red control of the 5T line), the e2e case `genotype-rule-rows-1-str-extra` sizes both tracts on made-up reads with ExpansionHunter 5.0.0, and the e2e case `str-extra-repeats-1-readings` runs step 09 and both reports on made-up reads and checks the lines.

The opt-in stays off by default: turning it on changes the catalog every run sizes, and the CFTR and UGT1A1 lines are readings of a modifier and of a PGx allele PharmCAT already calls, not expansions.

Expand Down
2 changes: 1 addition & 1 deletion docs/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -257,7 +257,7 @@ The first port is the reference mask of `setup.sh --masked-reference`. On a 254

On the whole GRCh38 analysis set (3.1 GB) `og` writes the file `GRC_MASKED_REF_SHA256` pins in 29 s on the same machine, 1.2 s of it CPU: writing the copy is what takes the time.

The next four ports are the VCF filters of steps 04, 05 and 07: `og sv inv-filter`, `og sv svlen-fix`, `og pgx refblock-fill` and `og pgx mito-merge`. Their cases run each oracle in the image the step used (`inv-filter` on the Manta image's Python 2.7, and on Python 3 too). `refblock-fill` reads a stream on stdin, so its cases stage a file named `stdin`. e2e does not shadow these four: the step scripts call them from the image's own working directory, `refblock-fill` reads stdin, which the shadow cannot replay, and `mito-merge` writes `--out` outside the call's directory. Their unit tests (`tests/test_annotsv_sv_inputs.py`, `tests/test_pgx_refblock_fill.py`, `tests/test_pgx_mito_merge.py`) and the three step 05 tests run `og`. The unit-tests job has no `og`, so there they skip with a note. The Guard job `rust` runs them with `OG_REQUIRED=1` after its musl builds, and there a skip fails.
The next four ports are the VCF filters of steps 04, 05 and 07: `og sv inv-filter`, `og sv svlen-fix`, `og pgx refblock-fill` and `og pgx mito-merge`. Their cases run each oracle in the image the step used (`inv-filter` on the Manta image's Python 2.7, and on Python 3 too). `refblock-fill` reads a stream on stdin, so its cases stage a file named `stdin`. e2e does not shadow these four: the step scripts call them from the image's own working directory, `refblock-fill` reads stdin, which the shadow cannot replay, and `mito-merge` writes `--out` outside the call's directory. The two SV filters are checked in cargo test (`rust/crates/og-steps/tests/sv_inputs.rs`). The unit tests of the other two (`tests/test_pgx_refblock_fill.py`, `tests/test_pgx_mito_merge.py`) and the three step 05 tests run `og`. The unit-tests job has no `og`, so there they skip with a note. The Guard job `rust` runs them with `OG_REQUIRED=1` after its musl builds, and there a skip fails.

The checks CI runs on the pipeline itself are ports too: `og dev contract` (the output contract), `og dev schema` (a `summary.json` against `tests/schema/summary.schema.json`) and `og dev report-only` (the report-only error strategy). Their cargo tests in `rust/crates/og-dev/tests/repo.rs` run them on this checkout and on copies with one planted break each, in the Guard job `rust`. `og dev schema` and the run check of `og dev contract` have parity cases; `--static` and `og dev report-only` read a whole checkout, which a case cannot stage, so their parity was shown on the checkout and the planted copies when they were ported. Each fixes a check that passed on what it should refuse: a schema default of another JSON type (`false` for `0`), `additionalProperties: false`, `true` against `enum: [1]`, NaN against `minimum`, a `$ref` to nothing (now exit 2, a broken schema), a `//` inside a string and a nested brace in a strategy closure. `tests/schema/validate.py` stays only for three Python test files that validate the summaries they build in-process, in the unit-tests job, which has no `og`; it follows the same rules. The fake-docker case `reports-stale-step` checks its `summary.json` with `og dev schema` in the job `rust`; the job `fake-docker` has no `og` and says it skipped that check.

Expand Down
153 changes: 153 additions & 0 deletions rust/crates/og-bam/tests/motif_meaning.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,153 @@
//! What `og str motif`'s goldens must mean, by meaning: the checks of the
//! old `tests/test_rfc1_motif.py`, in cargo test.
//!
//! `tests/parity.rs` holds og to its goldens byte for byte
//! (`tests/data/parity/want/<case>`, or `og/<case>` for a deliberate fix).
//! This test says what those goldens must say about the made-up reads at
//! the GRCh38 RFC1 repeat, so a golden rewritten with another verdict fails
//! here even when og agrees with it:
//!
//! - benign: AAAAG reads across the repeat and soft-clipped, benign_motif;
//! - pathogenic: AAGGG reads soft-clipped, unmapped and placed at their
//! mate, rescued from another contig and on the reverse strand, plus three
//! expanded AAAAG reads; the eight reads across an unexpanded (AAAAG)11
//! allele are not counted: pathogenic_motif with both shares;
//! - mixed (AAAAG and AAAGG): benign_motif with both shares; acagg:
//! pathogenic_motif; unknown (AAGAG): uncertain_motif; few (two reads):
//! not_resolved;
//! - controls, each failing when the rescue it names is off: `--no-softclip`
//! counts the 6 clipped reads fewer, `--no-mates` the 3 rescued mates;
//! - `--skip` and a VCF without an RFC1 record: not_checked.

use std::collections::HashMap;
use std::path::{Path, PathBuf};

fn parity() -> PathBuf {
Path::new(env!("CARGO_MANIFEST_DIR")).join("tests/data/parity")
}

/// og's golden of a parity case when it has one (a deliberate fix), else
/// the oracle's.
fn golden(case: &str) -> PathBuf {
let og = parity().join("og").join(case);
if og.is_dir() {
og
} else {
parity().join("want").join(case)
}
}

/// The one row of the case's motif.tsv, by column.
fn table(case: &str) -> HashMap<String, String> {
let path = golden(case).join("files/motif.tsv");
let text = std::fs::read_to_string(&path).unwrap_or_else(|e| panic!("{}: {e}", path.display()));
let mut lines = text.lines();
let head = lines.next().unwrap_or_default().split('\t');
let row = lines.next().unwrap_or_default().split('\t');
head.zip(row)
.map(|(k, v)| (k.to_owned(), v.to_owned()))
.collect()
}

fn exit_code(case: &str) -> i32 {
let path = golden(case).join("exit");
std::fs::read_to_string(&path)
.unwrap_or_else(|e| panic!("{}: {e}", path.display()))
.trim()
.parse()
.unwrap()
}

fn get<'a>(t: &'a HashMap<String, String>, k: &str) -> &'a str {
t.get(k).map_or("<missing>", String::as_str)
}

#[test]
fn each_read_set_gets_its_verdict() {
for (kind, verdict, reads) in [
("benign", "benign_motif", "10"),
("pathogenic", "pathogenic_motif", "16"),
("mixed", "benign_motif", "10"),
("acagg", "pathogenic_motif", "5"),
("unknown", "uncertain_motif", "5"),
("few", "not_resolved", "2"),
] {
let case = format!("motif-{kind}");
let t = table(&case);
assert_eq!(
(
exit_code(&case),
get(&t, "verdict"),
get(&t, "repeat_reads")
),
(0, verdict, reads),
"{kind}: exit 0, {verdict}, {reads} repeat reads"
);
}
}

#[test]
fn the_pathogenic_reads_and_their_shares() {
let t = table("motif-pathogenic");
assert_eq!(
["chrom", "start", "end", "dominant_motif", "motif_shares"].map(|k| get(&t, k)),
[
"chr4",
"39348425",
"39348479",
"AAGGG",
"AAGGG=0.81,AAAAG=0.19"
],
"pathogenic: the locus from the VCF, AAGGG dominant, both shares; the 8 reference-allele reads not counted"
);
assert!(
get(&t, "note").contains("repeat-primed PCR"),
"pathogenic: the note asks for repeat-primed PCR: {}",
get(&t, "note")
);
let benign = table("motif-benign");
assert!(
get(&benign, "note").contains("the flag is not supported"),
"benign: the note says the flag is not supported: {}",
get(&benign, "note")
);
assert_eq!(
get(&table("motif-mixed"), "motif_shares"),
"AAAAG=0.60,AAAGG=0.40",
"mixed: AAAAG and AAAGG counted apart"
);
let unknown = table("motif-unknown");
assert_eq!(
(
get(&unknown, "dominant_motif"),
get(&unknown, "dominant_class")
),
("AAGAG", "unknown"),
"unknown: AAGAG named by its smallest form, class unknown"
);
}

#[test]
fn the_rescue_controls_and_the_unchecked_cases() {
assert_eq!(
get(&table("motif-pathogenic-no-softclip"), "repeat_reads"),
"10",
"control: --no-softclip loses the 6 clipped reads"
);
assert_eq!(
get(&table("motif-pathogenic-no-mates"), "repeat_reads"),
"13",
"control: --no-mates loses the 3 rescued mates"
);
let skip = table("motif-skip");
assert_eq!(
(get(&skip, "verdict"), get(&skip, "note")),
("not_checked", "no BAM"),
"--skip: not_checked with the reason"
);
assert_eq!(
get(&table("motif-no-rfc1-record"), "verdict"),
"not_checked",
"a VCF without an RFC1 record: not_checked"
);
}
Loading
Loading