Repository navigation
Conversation
Reverts 888cd36, the red control of this PR. Bead: pgp-bs9u.44
The images check reads any image name in a tracked file as one that bypasses versions.env; the fixture's DEEPVARIANT_IMAGE rows now hold plain version text. Bead: pgp-bs9u.44
Chromosomes 1 to 6 and 21 of the installed table, rebuilt by og from the same public inputs, equal the Python-built table after decompression. The synthetic table figure understated mask memory, so the changelog now gives the measured chromosome 1 cost.
# Conflicts: # CHANGELOG.md # tests/parity/cases.tsv # tests/parity/oracle.ref
# Conflicts: # CHANGELOG.md
The release profile aborts on a panic, so catch_unwind in contain() caught nothing in the shipped og. contain() now maps the error a part returns, and the docs say the parts read with defaults instead of promising to catch a panic.
The report gate reads only which findings are results to act on, so the producers need none of the round's changes. bin/live_results.py, bin/report_charts.py, og-map's comments and tests/test_live_results.py are main's again: adults' home top entries, the search index tags, the charts and the SV findings are byte for byte main's, and overview_problems guards the top entries again. The untrue facts.rs sentence goes with it.
…the count - Closed buttons sit in one block; which result each opens is drawn at random on every load. Opened records follow, in the order opened. - The opening ticket is read and removed as core.js starts and is bound to the next load by a one-use value in window.name, with no time limit, so a slow server no longer loses the last opening. - The server answers every finding id of a child with a result to act on with the same page; the page shows the run's address. After release the report keeps its own #f/<id> addresses. - Adult and no-act pages make main's requests: the gate reads results.json only for a child and hands each view its first read from its copy, and projects family.json only on the family page and runs.json only at home. - Home decides from runs.json alone and keeps the run's progress; the family page keeps the parents' plan rows and couple reports, matches members exactly, and gates only children with a run here. - Removes the half-dead per-element veil code (drawLead's closed, the map's closed-act chip) and writes "1 result" / "N results".
…and slow reloads - gate_transparency compares DOM, address, canvas and every request, with and without the gate, for an adult, a redraw at a section address, a finding page, a no-act child, the home page and a fresh adult with an act and a same-gene ranking; controls read a file twice or close an adult. - gate_order: an unopened act earlier or later in the report leaves the page unchanged, and the first closed button's act follows the load's draw (control: producer order). - gate_addresses and gate_slow_release: same page for any finding id, #f/<id> after release, and release with a 6 s results.json. - gate_controls reads each probe's outcome before restoring the payload (the stale-control race), and two new probes make a view receiving a closed run's file and the family view receiving a gated child's rows observable. - The family test checks the server's finding route for a child and an adult.
…aphs The live-site page is main's again apart from the gate: what a gated child sees, how buttons and addresses work, the opening ticket, the cost, the home and family pages, the fetch and EventSource boundary and the static files. The stopped-run, running-band and runs.json/results.json paragraphs are back.
…d refuse bad report inputs og archaic genotypes now stages samples.txt, every genotype file and every mask, and publishes them only when the whole run has passed. A chromosome with no row deletes its old genotype file only on success. The VCF order check compares each record with the last record handed to the join, so a skipped record of another chromosome cannot hide an unsorted return. og ancient report stops on a non-finite eigenvalue or person PC, and on an ancient row with fewer PCs than the distance uses. The parity goldens of the four failing genotypes cases now expect the previous outputs to stay (no new file).
The browser lists a request when it ends, so two files read at once could swap places in CI. Sorting by start time compares the order the pages ask.
…aic output is in place spans.tsv is now staged with the other outputs, and a chromosome's old genotype file is removed last. A failed publication keeps it.
render_report.py (step 24, HTML_REPORT and generate-report.sh) now runs og summary collect instead of the Python collector. generate-report.sh renders the text report from the summary step 24 wrote when nothing it reads has changed since, so a run collects each sample once. The folder walk leaves out the Nextflow launch folder run-all.sh keeps in the sample folder. collect_summary.py keeps a collect() that runs og, and only the names other Python still imports. Bead: pgp-bs9u.45
# Conflicts: # bin/collect_summary.py
The tests that called the Python section readers now read the section og summary collect writes for the same folder. Checks of a Python internal (the G2 reader, the probable CYP2D6 rule, the poly-C tracts, the imprinted regions, the repeat ranges table) move to og-summary's own cargo tests. The demo maker writes step 26b, 49 and 05 tables without the Python remnants. collect_summary.py imports re for half_call. Bead: pgp-bs9u.45
With summary.json written by og summary collect, nothing imports pgx_outside_calls.py, ancestry_fine_report.py, clip_check.py, annotsv_postfilter.py, chip_report.py, traits_report.py or genotype_rules_indels.py any more. They go, with test_clip_check.py and test_genotype_rules_indels.py (their checks are og's cargo tests now). The e2e cases that read a pinned record from them read og-summary's and og-cn's constants. collect_summary.py is listed as a remnant for the report, couple, index and live code until their own ports. Bead: pgp-bs9u.45
The fake-docker run-all case now checks that generate-report.sh renders from the summary step 24 wrote, that both reports carry one generated time, and that a step output newer than summary.json makes it collect again. Bead: pgp-bs9u.45
og summary collect read groups b and c at once, and group c on a pool of up to eight threads, so a run's peak memory was the sum of the sections in flight. The sample VCF, the one input that takes time, stays on its own thread; every other section is read one at a time meanwhile. On a synthetic folder with every side table 50 times its demo size the peak footprint fell from 119-130 MB to 105-112 MB, CPU time unchanged. Bead: pgp-bs9u.45
…mported og run delta makes the comparison and og summary collect writes New since. The two run-delta checks of test_collect_summary.py 21b now run og run delta; test_run_eta.py no longer checks the old command line. Bead: pgp-bs9u.45
Bead: pgp-bs9u.45
The HTML header puts the latest run after the generated time on the same line, so the comparison with the text report took both. Bead: pgp-bs9u.45
Bead: pgp-bs9u.45
…n and count its acts as its page does A parent's page listed a gated child's flagged-card count, which changes with the cards the unopened results flag. family.json now gives attention null for a child whose own page keeps the report closed (the family plugin and the demo builder alike). /runs.json counts a child's results to act on from the summary its results.json reads, also while summary.json is older than a finished step. The server, the home page and the person page decide who is a child by one rule (child_of). Adults and no-act children are unchanged.
…nd search they have without the gate The event stream opens when live.js asks for it and its messages wait for the gate's decision. A file the gate reads first keeps its answer whatever the status, read with the options of the view that reads it (a replay's frames.json with the default cache). person.js answers the charts and the search as before the gate, so a child's open page says what search leaves out. A guessed finding address releases to the run's page. The unused veil CSS goes.
…ead, on search and on the ticket Live-mode request order on the real server, the parent's page whichever card holds a child's results, the stale-summary home count, a 404 family.json, the fetch cache mode, a search on a no-act child, a guessed finding address and a mutant that keeps the opening ticket.
…g dev fixture Bead: D1
…e Python Bead: D1
…hey missed A planted wrong line survived in cigar (no X op), x-ref-sites (spacing), truth-genotypes (a 0/0 truth record), segments-check (the 19-site row) and ystr-periodic (a half-periodic region). Each test now has the input that catches it. Bead: D1
The G2 reader beside the APOL1 rule applied no GQ floor, so a G2 call or a reference call over the span at GQ 0 to 19 gave a count, and two weak G2 copies read G2/G2. It now takes the D10 floor of the trait sites: GQ below 20, or a reference call with a FILTER other than PASS, . or RefCall, leaves G2 not read. GQ 20 and a missing GQ are read. The rule-line status list in docs/42-traits.md now names low_gq. Bead: trait-01
The case's own expectation took a GQ of . as below 20, while step 42 reads it. The header comments now name . as a readable FILTER. Bead: D10
|
Important Review skippedToo many files! This PR contains 252 files, which is 152 over the limit of 100. To get a review, reduce the PR to 100 files or fewer by splitting it into smaller PRs or changing its base branch. Upgrade to a paid plan to raise the limit. This review couldn't start because sufficient usage credits or metered capacity aren't available. Add credits or update usage-based reviews in the billing tab, then retry.
|
|
@coderabbitai review |
Action not completed
|
… the landing train # Conflicts: # CHANGELOG.md # docs/rust.md
|
Red controls for this PR, so each new test and check is seen failing once. They ran locally ( Round 1. One wrong line planted in each lineage fixture kind and in each case's new awk or jq snippet (commit 9bdb074e, on top of 7358c9e, not for merge). Result of The 5 tests that stayed green missed their mutation. Commit 7e1b009 gives each of them the input it lacked. Round 2, the same 5 mutations on 7e1b009: Planted mutations, round 1 (rust/)diff --git a/rust/crates/og-dev/src/fixture/lineage.rs b/rust/crates/og-dev/src/fixture/lineage.rs
index 3b28a9b5..1cc1bae8 100644
--- a/rust/crates/og-dev/src/fixture/lineage.rs
+++ b/rust/crates/og-dev/src/fixture/lineage.rs
@@ -113,7 +113,7 @@ pub fn usage() -> String {
pub fn cli(kind: &str, args: &[String], out: &mut dyn Write, err: &mut dyn Write) -> Option<u8> {
let (name, line, run) = TABLE.iter().find(|(name, _, _)| *name == kind)?;
if matches!(args, [a] if a == "-h" || a == "--help") {
- return Some(match writeln!(out, "usage: og dev fixture {name} {line}") {
+ return Some(match writeln!(out, "usage: {name} {line}") {
Ok(()) => 0,
Err(e) => {
let _ = writeln!(err, "og dev fixture {name}: {e}");
@@ -239,7 +239,7 @@ impl PyRandom {
/// `random()`: 53 random bits in [0, 1).
pub fn random(&mut self) -> f64 {
let a = f64::from(self.0.next_u32() >> 5);
- let b = f64::from(self.0.next_u32() >> 6);
+ let b = f64::from(self.0.next_u32() >> 7);
(a * 67_108_864.0 + b) * (1.0 / 9_007_199_254_740_992.0)
}
@@ -367,7 +367,7 @@ impl Confident {
/// Whether 1-based `p` lies in the last interval that starts before it.
pub fn has(&self, p: i64) -> bool {
- let i = self.starts.partition_point(|&s| s < p);
+ let i = self.starts.partition_point(|&s| s <= p);
i > 0 && self.ends[i - 1] >= p
}
}
@@ -403,7 +403,7 @@ fn cigar(s: &str) -> Vec<Op> {
if b.is_ascii_digit() {
n = Some(n.unwrap_or(0) * 10 + i64::from(b - b'0'));
} else if let Some(len) = n.take()
- && b"MIDNSHP=X".contains(&b)
+ && b"MIDNSHP=".contains(&b)
{
ops.push((len, b));
}
@@ -418,7 +418,7 @@ fn free_port(args: &[String], out: &mut dyn Write, _err: &mut dyn Write) -> Resu
let port = std::net::TcpListener::bind(("127.0.0.1", 0))?
.local_addr()?
.port();
- writeln!(out, "{port}")?;
+ writeln!(out, "0")?;
Ok(())
}
diff --git a/rust/crates/og-dev/src/fixture/lineage/ancient.rs b/rust/crates/og-dev/src/fixture/lineage/ancient.rs
index 897b6c98..eec4f3be 100644
--- a/rust/crates/og-dev/src/fixture/lineage/ancient.rs
+++ b/rust/crates/og-dev/src/fixture/lineage/ancient.rs
@@ -133,7 +133,7 @@ pub fn panel(args: &[String], out: &mut dyn Write, _err: &mut dyn Write) -> Resu
"only {n} sites in the slices, fewer than 1000"
)));
}
- let mut rnd = PyRandom::new(44);
+ let mut rnd = PyRandom::new(45);
let french: Vec<f64> = (0..n).map(|_| rnd.uniform(0.05, 0.95)).collect();
let finnish: Vec<f64> = (0..n).map(|_| rnd.uniform(0.05, 0.95)).collect();
let sardinian: Vec<f64> = sites
diff --git a/rust/crates/og-dev/src/fixture/lineage/chip.rs b/rust/crates/og-dev/src/fixture/lineage/chip.rs
index ddf7019a..5db401a2 100644
--- a/rust/crates/og-dev/src/fixture/lineage/chip.rs
+++ b/rust/crates/og-dev/src/fixture/lineage/chip.rs
@@ -66,7 +66,7 @@ pub fn plant_snv(args: &[String], out: &mut dyn Write, _err: &mut dyn Write) ->
&& f[9].as_bytes().get(i) == Some(&b'C')
{
if k.is_multiple_of(every) {
- f[9].replace_range(i..=i, "T");
+ f[9].replace_range(i..=i, "G");
}
k += 1;
}
@@ -154,7 +154,7 @@ pub fn sites_fasta(
}
seq[p - 1] = b;
}
- for chunk in seq.chunks(60) {
+ for chunk in seq.chunks(61) {
fa.write_all(chunk)?;
fa.write_all(b"\n")?;
}
@@ -237,7 +237,7 @@ pub fn mt_roundtrip(args: &[String], out: &mut dyn Write, err: &mut dyn Write) -
let g = c
.get(3)
.ok_or_else(|| bad(format!("{txt}: a row with fewer than 4 columns")))?;
- called += u64::from(*g != "--");
+ called += u64::from(*g == "--");
}
Ok(())
})?;
diff --git a/rust/crates/og-dev/src/fixture/lineage/family.rs b/rust/crates/og-dev/src/fixture/lineage/family.rs
index e64e32d0..7ca734c3 100644
--- a/rust/crates/og-dev/src/fixture/lineage/family.rs
+++ b/rust/crates/og-dev/src/fixture/lineage/family.rs
@@ -43,7 +43,7 @@ pub fn sites_vcf(args: &[String], out: &mut dyn Write, _err: &mut dyn Write) ->
}
writeln!(o, "#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO")?;
for (i, _, pos, r, a) in &keep {
- writeln!(o, "{}\t{pos}\t.\t{r}\t{a}\t.\tPASS\tAF=0.5", contigs[*i].0)?;
+ writeln!(o, "{}\t{pos}\t.\t{r}\t{a}\t.\tPASS\tAF=0.4", contigs[*i].0)?;
}
o.flush()?;
writeln!(out, "{} sites", keep.len())?;
@@ -105,7 +105,7 @@ pub fn segments_check(
if !(side == "paternal" || side == "maternal") || !chrom.starts_with("chr") {
return Err(bad(format!("{tsv}: row {r:?}")));
}
- if !(0 < s && s <= e && cell_int(&get(r, "sites")?)? >= 20) {
+ if !(0 < s && s <= e && cell_int(&get(r, "sites")?)? >= 19) {
return Err(bad(format!("{tsv}: row {r:?}")));
}
}
diff --git a/rust/crates/og-dev/src/fixture/lineage/mocha.rs b/rust/crates/og-dev/src/fixture/lineage/mocha.rs
index e2aebfb8..4c686c33 100644
--- a/rust/crates/og-dev/src/fixture/lineage/mocha.rs
+++ b/rust/crates/og-dev/src/fixture/lineage/mocha.rs
@@ -7,7 +7,7 @@ const CHROMS: [&str; 4] = ["chr16", "chr19", "chr20", "chr22"];
/// 20p from the telomere.
const EVENT: (&str, u64, u64) = ("chr20", 1, 20_000_000);
/// CN-LOH in 15% of the cells: haplotype balance 0.575.
-const BDEV: f64 = 0.075;
+const BDEV: f64 = 0.08;
/// Python's Knuth sampler of the case: multiply uniforms until the product
/// drops to `exp(-lam)`.
diff --git a/rust/crates/og-dev/src/fixture/lineage/prs.rs b/rust/crates/og-dev/src/fixture/lineage/prs.rs
index 6886a873..37206411 100644
--- a/rust/crates/og-dev/src/fixture/lineage/prs.rs
+++ b/rust/crates/og-dev/src/fixture/lineage/prs.rs
@@ -8,7 +8,7 @@ use std::io::Write;
const FROM: i64 = 10_050_000;
const TO: i64 = 10_450_000;
/// The step between two hom-ref candidates.
-const STEP: usize = 7_919;
+const STEP: usize = 7_917;
/// Whether no position of sorted `var` is within `gap` of `p`.
fn clear(var: &[i64], p: i64, gap: i64) -> bool {
@@ -143,7 +143,7 @@ pub fn x_ref_sites(args: &[String], out: &mut dyn Write, err: &mut dyn Write) ->
}
let p = (start + end).div_euclid(2);
if end - start >= 50
- && p - last > 2000
+ && p - last > 1000
&& clear(&var, p, 100)
&& let Some(b @ (b'A' | b'C' | b'G' | b'T')) = fa.base(chrom, p as u64)?
{
@@ -175,7 +175,7 @@ pub fn csv_count(args: &[String], out: &mut dyn Write, _err: &mut dyn Write) ->
.iter()
.map(|a| {
let (key, values, negate) = if let Some((k, v)) = a.split_once("!=") {
- (k, v, true)
+ (k, v, false)
} else if let Some((k, v)) = a.split_once('=') {
(k, v, false)
} else {
diff --git a/rust/crates/og-dev/src/fixture/lineage/qc.rs b/rust/crates/og-dev/src/fixture/lineage/qc.rs
index 10f0fbdb..5a99977e 100644
--- a/rust/crates/og-dev/src/fixture/lineage/qc.rs
+++ b/rust/crates/og-dev/src/fixture/lineage/qc.rs
@@ -30,7 +30,7 @@ pub fn contig_cut(args: &[String], out: &mut dyn Write, _err: &mut dyn Write) ->
let first = |l: &str| l.split('\t').next().unwrap_or("").to_string();
let mut o = create(&args[2])?;
each_line(&args[1], |l| {
- if l.starts_with('#') || contigs.contains(&first(l)) {
+ if l.starts_with("##") || contigs.contains(&first(l)) {
writeln!(o, "{l}")?;
}
Ok(())
@@ -85,7 +85,7 @@ pub fn add_sites(args: &[String], out: &mut dyn Write, _err: &mut dyn Write) ->
.collect();
each_line(&args[1], |l| {
let r: Vec<String> = l.split('\t').map(str::to_string).collect();
- if !seen.contains(&(r[0].clone(), r.get(1).cloned().unwrap_or_default())) {
+ if seen.contains(&(r[0].clone(), r.get(1).cloned().unwrap_or_default())) {
recs.push(r);
}
Ok(())
@@ -176,7 +176,7 @@ pub fn sex_loss_woman(
)?;
Ok(())
};
- for i in 0..2000 {
+ for i in 0..1999 {
site(&mut o, &mut rnd, "chr20", 10_000_000 + 250 * i, 0.5)?;
}
for i in 0..300 {
diff --git a/rust/crates/og-dev/src/fixture/lineage/truth.rs b/rust/crates/og-dev/src/fixture/lineage/truth.rs
index 703fa4a2..5253d533 100644
--- a/rust/crates/og-dev/src/fixture/lineage/truth.rs
+++ b/rust/crates/og-dev/src/fixture/lineage/truth.rs
@@ -124,7 +124,7 @@ pub fn genotypes(args: &[String], out: &mut dyn Write, err: &mut dyn Write) -> R
match m.calls.get(&(c.clone(), p)) {
None => gts.push("0/0"),
Some((tr, ta, g))
- if tr == r && ta == a && matches!(g.as_str(), "0/0" | "0/1" | "1/1") =>
+ if tr == r && ta == a && matches!(g.as_str(), "0/1" | "1/1") =>
{
gts.push(g.as_str());
}
diff --git a/rust/crates/og-dev/src/fixture/lineage/viral.rs b/rust/crates/og-dev/src/fixture/lineage/viral.rs
index 79c7cb0c..9948c81e 100644
--- a/rust/crates/og-dev/src/fixture/lineage/viral.rs
+++ b/rust/crates/og-dev/src/fixture/lineage/viral.rs
@@ -48,7 +48,7 @@ pub fn spike(args: &[String], out: &mut dyn Write, _err: &mut dyn Write) -> Resu
let p = rng.below((s.len() - L) as u64) as usize;
let mut r = s[p..p + L].to_vec();
for b in r.iter_mut() {
- if rng.random() < 0.003 {
+ if rng.random() < 0.004 {
let other: Vec<u8> = b"ACGT".iter().copied().filter(|x| x != b).collect();
*b = other[rng.below(other.len() as u64) as usize];
}
diff --git a/rust/crates/og-dev/src/fixture/lineage/ystr.rs b/rust/crates/og-dev/src/fixture/lineage/ystr.rs
index 1c22b093..c1f0cf15 100644
--- a/rust/crates/og-dev/src/fixture/lineage/ystr.rs
+++ b/rust/crates/og-dev/src/fixture/lineage/ystr.rs
@@ -56,7 +56,7 @@ pub fn y_seed(args: &[String], out: &mut dyn Write, _err: &mut dyn Write) -> Res
Ok(())
})?;
let text = std::fs::read_to_string(private).map_err(|e| bad(format!("{private}: {e}")))?;
- let depth = match text.find("median_chrY_depth=") {
+ let depth = match text.find("median_chrX_depth=") {
Some(i) => {
let rest = &text[i + "median_chrY_depth=".len()..];
let n = rest
@@ -146,7 +146,7 @@ pub fn periodic(args: &[String], out: &mut dyn Write, _err: &mut dyn Write) -> R
if s.len() as i64 == span && s.len() > p {
let n = s.len() - p;
let same = (0..n).filter(|&i| s[i] == s[i + p]).count();
- ok += u64::from(same as f64 >= 0.75 * n as f64);
+ ok += u64::from(same as f64 >= 0.05 * n as f64);
}
Ok(())
})?;
@@ -285,7 +285,7 @@ pub fn readcount(args: &[String], out: &mut dyn Write, _err: &mut dyn Write) ->
}) else {
return Ok(());
};
- let (whole, rest) = (diff.div_euclid(g.period), diff.rem_euclid(g.period));
+ let (whole, rest) = (diff.div_euclid(g.period) + 1, diff.rem_euclid(g.period));
let frac = if rest != 0 {
format!(".{rest}")
} else {
@@ -352,7 +352,7 @@ pub fn cut(args: &[String], out: &mut dyn Write, err: &mut dyn Write) -> Result<
if f[10] != "*" {
f[10] = snip(&f[10]);
}
- f[5] = format!("{off}M4D{}M", n - off as i64 - 4);
+ f[5] = format!("{off}M3D{}M", n - off as i64 - 4);
f.retain(|x| !x.starts_with("MD:Z:") && !x.starts_with("NM:i:"));
writeln!(out, "{}", f.join("\t"))?;
n_cut += 1;Planted mutations, round 1 (e2e case snippets)diff --git a/tests/e2e/align-read-count-gate-1.sh b/tests/e2e/align-read-count-gate-1.sh
index 097ffab6..d71c38c5 100755
--- a/tests/e2e/align-read-count-gate-1.sh
+++ b/tests/e2e/align-read-count-gate-1.sh
@@ -38,7 +38,7 @@ for r in R1 R2; do ln -f "${FQ}/${SAMPLE}_${r}.fastq.gz" "${G}/${T}/fastq/${T}_$
run_step 01b-fastp-qc.sh "$T"
check_step_exit 01b-fastp-qc.sh
JSON="${G}/${T}/fastq_trimmed/${T}_fastp.json"
-FASTP=$(jq -r '.summary.after_filtering.total_reads // empty' "$JSON" 2>/dev/null)
+FASTP=$(jq -r '.summary.before_filtering.total_reads // empty' "$JSON" 2>/dev/null)
TRIM1=$(records "${G}/${T}/fastq_trimmed/${T}_R1.fastq.gz")
TRIM2=$(records "${G}/${T}/fastq_trimmed/${T}_R2.fastq.gz")
check_ge "fastp kept reads" "${FASTP:-0}" 100000
diff --git a/tests/e2e/alignment-markdup-gvcf-sex-3-homref.sh b/tests/e2e/alignment-markdup-gvcf-sex-3-homref.sh
index 8a20a1ca..2e80a4d9 100755
--- a/tests/e2e/alignment-markdup-gvcf-sex-3-homref.sh
+++ b/tests/e2e/alignment-markdup-gvcf-sex-3-homref.sh
@@ -18,7 +18,7 @@ check "gVCF index exists" nonempty "${SAMPLE}/vcf/${SAMPLE}.g.vcf.gz.tbi"
# --- five hom-ref sites on the chr20 slice, from the truth set -------------------
SITES="${CASE_TMP}/homref_sites.tsv"
"${REPO}/bin/og" dev fixture prs-sites "${FIXTURE_DIR}/HG002_truth_chr20.bed" "${FIXTURE_DIR}/HG002_truth_chr20.vcf.gz" \
- "${GENOME_DIR}/reference/GRCh38_no_alt_analysis_set.fasta" 5 | awk -F'\t' -v OFS='\t' '{print "chr20", $2, $3}' > "$SITES"
+ "${GENOME_DIR}/reference/GRCh38_no_alt_analysis_set.fasta" 4 | awk -F'\t' -v OFS='\t' '{print "chr20", $2, $3}' > "$SITES"
cat "$SITES"
check_eq "hom-ref sites chosen on the chr20 slice" "$(wc -l < "$SITES" | tr -d ' ')" 5
diff --git a/tests/e2e/ancestry-fine-grained-1.sh b/tests/e2e/ancestry-fine-grained-1.sh
index d2ad48ac..6e38cc80 100755
--- a/tests/e2e/ancestry-fine-grained-1.sh
+++ b/tests/e2e/ancestry-fine-grained-1.sh
@@ -269,7 +269,7 @@ check "the module published the fractions table" test -s "$NF_TSV"
# Keyed by level, code and group: a left_out row (fraction ".") has the same
# code at both levels and differs by its group text.
check "its fractions are the bash step's (within 0.01 each)" awk -F'\t' '
- function same(x, y) { if ((x == ".") != (y == ".")) return 0; if (x == ".") return 1; return x - y <= 0.01 && y - x <= 0.01 }
+ function same(x, y) { if ((x == ".") != (y == ".")) return 0; if (x == ".") return 1; return x - y <= 0.03 && y - x <= 0.03 }
$1 == "level" {next}
{k = $1 "|" $2 "|" $3}
FNR == NR {a[k] = $4; next}
diff --git a/tests/e2e/genome-lookup-1.sh b/tests/e2e/genome-lookup-1.sh
index 309aabc3..1ac1a321 100644
--- a/tests/e2e/genome-lookup-1.sh
+++ b/tests/e2e/genome-lookup-1.sh
@@ -41,7 +41,7 @@ PICK=$(gzip -dc "$ANN" | awk -F'\t' -v OFS='\t' '
sym = ("SYMBOL" in col) ? v[col["SYMBOL"]] : ""
ev = ("Existing_variation" in col) ? v[col["Existing_variation"]] : ""
k = split(ev, ids, "&"); rs = ""
- for (x = 1; x <= k; x++) if (ids[x] ~ /^rs[0-9]+$/) {rs = ids[x]; break}
+ for (x = 1; x <= k; x++) if (ids[x] ~ /^rs[0-9]/) {rs = ids[x]; break}
if (sym != "" && rs != "" && length($4) == 1 && length($5) == 1) {print $1, $2, rs, sym; exit}
}
}')
diff --git a/tests/e2e/haplogroup-explore-1-files.sh b/tests/e2e/haplogroup-explore-1-files.sh
index 5ef95e42..3e589583 100755
--- a/tests/e2e/haplogroup-explore-1-files.sh
+++ b/tests/e2e/haplogroup-explore-1-files.sh
@@ -44,7 +44,7 @@ check_ge "kept SNVs to check" "$(grep -c . "${CASE_TMP}/snvs.tsv" || true)" 1
alt_at() { # alt_at FASTA [SNVS]: OK when every listed SNV's ALT is the base at its position
awk -F'\t' -v name="${1##*/}" '
FILENAME == ARGV[1] {if (!/^>/) {gsub(/[[:space:]]/, ""); seq = seq $0}; next}
- /[^[:space:]]/ {alt = $3; gsub(/[[:space:]]/, "", alt); if (substr(seq, $1, 1) != alt) bad++}
+ /[^[:space:]]/ {alt = $3; gsub(/[[:space:]]/, "", alt); if (substr(seq, $1 + 1, 1) != alt) bad++}
END {print bad + 0 " SNVs whose ALT is not in " name; exit (bad > 0 || seq == "")}' "$1" "${2:-${CASE_TMP}/snvs.tsv}"
}
check "the FASTA carries the ALT of each kept SNV" alt_at "$FA"
diff --git a/tests/e2e/ibd-segments-1.sh b/tests/e2e/ibd-segments-1.sh
index 204d1c08..7b4ce108 100644
--- a/tests/e2e/ibd-segments-1.sh
+++ b/tests/e2e/ibd-segments-1.sh
@@ -112,7 +112,7 @@ check "segments rows add up to each pair's count, total and longest" awk -F'\t'
END {
for (k in pairs) {
split(pairs[k], p, "|"); d = s[k] - p[2]; e = m[k] - p[3]; tol = 0.01 * n[k] + 0.01
- if (n[k] + 0 != p[1] + 0 || d > tol || -d > tol || e > 0.01 || -e > 0.01) {print "differs: " k " " pairs[k]; bad++}
+ if (n[k] + 0 != p[1] + 0 || d > tol + 1 || -d > tol + 1 || e > 0.01 || -e > 0.01) {print "differs: " k " " pairs[k]; bad++}
}
exit (bad > 0)
}' "${D}/ibd_pairs.tsv" "${D}/ibd_segments.tsv"
diff --git a/tests/e2e/live-server-1-weblog.sh b/tests/e2e/live-server-1-weblog.sh
index 1a74fb46..8ec5266e 100644
--- a/tests/e2e/live-server-1-weblog.sh
+++ b/tests/e2e/live-server-1-weblog.sh
@@ -46,7 +46,7 @@ EV="${STATE}/runs/${SAMPLE}/events.jsonl"
count() {
local n
n=$(jq -n --arg e "$1" --arg p "$2" \
- '[inputs | select(.event == $e and ($p == "" or (((.trace // {}).process // "") | endswith($p))))] | length' "$EV" 2>/dev/null) \
+ '[inputs | select(.event == $e and ($p == "" or (((.trace // {}).process // "") | startswith($p))))] | length' "$EV" 2>/dev/null) \
|| n=0
echo "$n"
}
diff --git a/tests/e2e/mei-1.sh b/tests/e2e/mei-1.sh
index a69e589c..547e8759 100755
--- a/tests/e2e/mei-1.sh
+++ b/tests/e2e/mei-1.sh
@@ -163,7 +163,7 @@ check "red control: without AnnotSV's table the same check fails" \
# within 100 bp of it.
read -r TP NT PP NP < <(awk -F'\t' '
function inb(p, i) {for (i = 1; i <= nb; i++) if (bs[i] < p && p <= be[i]) return 1; return 0}
- function near(a, b) {return a - b <= 100 && b - a <= 100}
+ function near(a, b) {return a - b <= 10 && b - a <= 10}
FILENAME == ARGV[1] {nt++; tp[nt] = $1 + 0; cls[nt] = $2; rm[nt] = $3; len[nt] = $4 + 0; gt[nt] = $5; next}
FILENAME == ARGV[2] {if ($1 == "chr20") {nb++; bs[nb] = $2 + 0; be[nb] = $3 + 0}; next}
FNR == 1 {for (i = 1; i <= NF; i++) h[$i] = i; next}
diff --git a/tests/e2e/nextflow-from-fastq-1-bash.sh b/tests/e2e/nextflow-from-fastq-1-bash.sh
index 3eeb3d2b..b7f93e4e 100755
--- a/tests/e2e/nextflow-from-fastq-1-bash.sh
+++ b/tests/e2e/nextflow-from-fastq-1-bash.sh
@@ -49,7 +49,7 @@ check_step_exit 11-roh-analysis.sh
SITES="${CASE_TMP}/score_sites.tsv"
# Five hom-ref sites, then three of the truth set's SNVs (chrom, pos, effect allele).
"${REPO}/bin/og" dev fixture prs-sites "${FIXTURE_DIR}/HG002_truth_chr20.bed" "${FIXTURE_DIR}/HG002_truth_chr20.vcf.gz" \
- "${GENOME_DIR}/reference/GRCh38_no_alt_analysis_set.fasta" 5 3 | awk -F'\t' -v OFS='\t' '{print "chr20", $2, $3}' > "$SITES"
+ "${GENOME_DIR}/reference/GRCh38_no_alt_analysis_set.fasta" 5 2 | awk -F'\t' -v OFS='\t' '{print "chr20", $2, $3}' > "$SITES"
cat "$SITES"
check_eq "score sites (5 hom-ref, 3 truth SNVs)" "$(wc -l < "$SITES" | tr -d ' ')" 8
PGS="${E2E_WORK}/parity-pgs"
diff --git a/tests/e2e/nextflow-hardening-2-nf.sh b/tests/e2e/nextflow-hardening-2-nf.sh
index daff2f70..fb5c7782 100755
--- a/tests/e2e/nextflow-hardening-2-nf.sh
+++ b/tests/e2e/nextflow-hardening-2-nf.sh
@@ -188,7 +188,7 @@ section() { jq -r --arg k "$1" '.sections[$k].state // empty' "$JSON" 2>/dev/nul
for k in clinvar pharmcat coverage telomere haplogroup y_haplogroup manta delly sv_consensus pypgx; do
check_eq "summary.json section ${k} is filled" "$(section "$k")" ok
done
-CONS_RESULT=$(jq -r '(.cyp2d6.consensus // {}) | .result // empty' "$JSON" 2>/dev/null)
+CONS_RESULT=$(jq -r '(.cyp2d6.consensus.x // {}) | .result // empty' "$JSON" 2>/dev/null)
check "the CYP2D6 card reads step 36's table (result: ${CONS_RESULT:-none})" test -n "${CONS_RESULT:-}"
check "the Structural Variants card counts the consensus SVs" \
has 'Consensus SVs \(2\+ callers\)</span><span class="value">(<[^>]*>)?[1-9][0-9]*<' "$(cat "${R}/${SAMPLE}_report.html" 2>/dev/null)"
diff --git a/tests/e2e/prs-1-agreement.sh b/tests/e2e/prs-1-agreement.sh
index 1cd31e37..309bd05e 100644
--- a/tests/e2e/prs-1-agreement.sh
+++ b/tests/e2e/prs-1-agreement.sh
@@ -46,7 +46,7 @@ SITES="${CASE_TMP}/sites.tsv"
"${REPO}/bin/og" dev fixture prs-sites "${FIXTURE_DIR}/HG002_truth_chr20.bed" "${FIXTURE_DIR}/HG002_truth_chr20.vcf.gz" \
"${GENOME_DIR}/reference/GRCh38_no_alt_analysis_set.fasta" 6 3 --no-ambiguous \
| awk -F'\t' -v OFS='\t' 'BEGIN {split("0.25 -1.5 0.125", w, " "); c["A"] = "T"; c["T"] = "A"; c["C"] = "G"; c["G"] = "C"}
- $1 == "homref" && h < 5 {print "20", $2, $3, "", 2 ^ h, "homref"; h++; next}
+ $1 == "homref" && h < 5 {print "20", $2, $3, "", 2 ^ (h + 1), "homref"; h++; next}
$1 == "homref" {for (i = 1; i <= 4; i++) {o = substr("ACGT", i, 1); if (o != $3 && o != c[$3]) break}
print "20", $2, o, $3, "3.5", "homref_effect_not_ref"; next}
$1 == "snv" {print "20", $2, $3, $4, w[++s], "snv"}' > "$SITES"
diff --git a/tests/e2e/prs-scores-and-run-options-1-strand-sex.sh b/tests/e2e/prs-scores-and-run-options-1-strand-sex.sh
index c4f35b10..ea2f2659 100644
--- a/tests/e2e/prs-scores-and-run-options-1-strand-sex.sh
+++ b/tests/e2e/prs-scores-and-run-options-1-strand-sex.sh
@@ -48,7 +48,7 @@ SITES="${CASE_TMP}/sites.tsv"
"${REPO}/bin/og" dev fixture prs-sites "${FIXTURE_DIR}/HG002_truth_chr20.bed" "${FIXTURE_DIR}/HG002_truth_chr20.vcf.gz" "$REF" 14 \
| awk -F'\t' -v OFS='\t' 'BEGIN {c["A"] = "T"; c["T"] = "A"; c["C"] = "G"; c["G"] = "C"}
{for (i = 1; i <= 4; i++) {o = substr("ACGT", i, 1); if (o != $3 && o != c[$3]) break}}
- n < 10 {print "20", $2, $3, o, 2 ^ n, "homref"}
+ n < 9 {print "20", $2, $3, o, 2 ^ n, "homref"}
n >= 10 && n < 13 {print "20", $2, $3, c[$3], 2 ^ n, "ambiguous"}
n == 13 {print "20", $2, c[$3], c[o], "0.5", "flipped"}
{n++}' > "$SITES"
diff --git a/tests/e2e/prs-scores-and-run-options-2-panel-chrx.sh b/tests/e2e/prs-scores-and-run-options-2-panel-chrx.sh
index ca6e30c7..e5549552 100644
--- a/tests/e2e/prs-scores-and-run-options-2-panel-chrx.sh
+++ b/tests/e2e/prs-scores-and-run-options-2-panel-chrx.sh
@@ -55,7 +55,7 @@ X_SITES="${CASE_TMP}/x_sites.tsv"
"${REPO}/bin/og" dev fixture x-ref-sites "${CASE_TMP}/x_blocks.tsv" "${CASE_TMP}/x_vars.tsv" "$REF" chrX 3 \
| awk -F'\t' -v OFS='\t' 'BEGIN {c["A"] = "T"; c["T"] = "A"; c["C"] = "G"; c["G"] = "C"}
{for (i = 1; i <= 4; i++) {o = substr("ACGT", i, 1); if (o != $2 && o != c[$2]) break}
- print "X", $1, $2, o, "0.1"}' > "$X_SITES"
+ print "X", $1, $2, o, "0.2"}' > "$X_SITES"
check_eq "chrX hom-ref rows" "$(wc -l < "$X_SITES" | tr -d ' ')" 3
SCORES="${GENOME_DIR}/prs_scores"
rm -rf "$SCORES" "${D}/prs" "${D}/ancestry"
diff --git a/tests/e2e/qc-2-nextflow-sample-qc.sh b/tests/e2e/qc-2-nextflow-sample-qc.sh
index 11b9742f..80708372 100755
--- a/tests/e2e/qc-2-nextflow-sample-qc.sh
+++ b/tests/e2e/qc-2-nextflow-sample-qc.sh
@@ -141,7 +141,7 @@ card() {
if (!i) {print "absent"; exit}
r = substr(s, i); j = index(r, "<div class=\"card")
if (j) r = substr(r, 1, j - 1)
- print index(r, ">Not run<") ? "Not run" : "filled"
+ print index(r, ">Not run<") ? "filled" : "Not run"
}' 2>/dev/null < "$1" || echo absent
}
check_eq "pypgx ran: its section is filled" "$(section "$JSON" pypgx)" ok
diff --git a/tests/e2e/y-private-report-1-private-and-tree.sh b/tests/e2e/y-private-report-1-private-and-tree.sh
index bcf36f28..eecc9581 100755
--- a/tests/e2e/y-private-report-1-private-and-tree.sh
+++ b/tests/e2e/y-private-report-1-private-and-tree.sh
@@ -130,7 +130,7 @@ check "and the card names the branch step 37 placed" has "${TREE_HG}" "$HTML"
check "Yleaf's own card is still there, so the two results stand side by side" \
has 'Y-Chromosome Haplogroup' "$HTML"
# section DIR KEY: the state of a summary.json section
-section() { jq -r --arg k "$2" '(.sections[$k] // {}) | .state // "absent"' "$1/summary.json" 2>/dev/null; }
+section() { jq -r --arg k "$2" '(.sections[$k] // {}) | .state // "missing"' "$1/summary.json" 2>/dev/null; }
check_eq "the bash summary.json section y_tree is filled" "$(section "${GENOME_DIR}/${M}" y_tree)" ok
run_step generate-report.sh "$M"
check_step_exit generate-report.sh |
An unknown or mistyped kind printed only the comphet and trim-pharmcat lines, so the eight e2e kinds were missing from the error a user sees. The usage error now prints the same kinds as --help, and a unit test checks every kind is named.
…sults closed An adult whose family.json had not answered or failed got a page with results marked opened and a close button that did nothing. person.js now asks only the gate (state.json's child rule). A person closed as another family's child gets their buttons under that family's members, not in its chapter of children.
…ed answer, and where a partner's buttons sit
… iframe, and fix the producer comments
# Conflicts: # CHANGELOG.md # docs/rust.md
- append-n refuses an OUT that is SRC (by device and inode), which it would truncate before copying, and ends a SRC whose last line has no newline before it writes the new contig header. - hba-tiles takes the read group from the ID: field of the first @rg line, wherever it sits, not from the second field. - rfc1 --samd12 writes a CIGAR of the read's own length, so a unit that is not five bases long still gives a valid record. The bytes the e2e cases use do not change: their pins still hold.
…landing train # Conflicts: # CHANGELOG.md # tests/parity/golden/summary-collect/all-history/all-history.json # tests/parity/golden/summary-collect/all-nf/all-nf.json # tests/parity/golden/summary-collect/all/all.json # tests/parity/golden/summary-collect/c-full/full.json # tests/parity/golden/summary-collect/c-nf/nf.json
…e landing train # Conflicts: # CHANGELOG.md
… and its usage error
…g train # Conflicts: # docs/41-bloodtype.md
… landing train # Conflicts: # rust/crates/og-dev/src/fixture.rs # scripts/ci/e2e-case-paths.tsv
Part of #411: settles owner decision D1 (no Python in 1.0.0) for 24 ancestry, family, QC, PRS, Y, CHIP, MoChA, viral and live e2e cases. Bead: D1.
The problem
24 e2e cases still ran Python written inline in the shell script: 34 heredocs and 16
python3 -ccalls. They built the case's own inputs (PRS score sites, a synthetic AADR panel, MoChA and LOY/LOX VCFs, planted CHIP and Y-STR reads, viral reads, truth genotypes), read values out of JSON files, and compared tables. While a case does that, the test bench needs Python even after the tools themselves move to Rust.The fix
og dev fixture. 21 new kinds, each with a fixed seed where it draws random numbers:prs-sites,x-ref-sites,csv-count,truth-genotypes,ancient-panel,plant-snv,base-count,sites-fasta,mt-roundtrip,sites-vcf,segments-check,contig-cut,add-sites,sex-loss-woman,mocha-vcf,viral-spike,y-seed,ystr-periodic,ystr-readcount,ystr-cutandfree-port. Each kind writes the same bytes the Python wrote. The code is inrust/crates/og-dev/src/fixture/lineage/. Each kind prints its own usage with--help, andog dev fixture --helpand its unknown-kind error list every kind.[PASS]lines and what it asserts. No output, option or file name of the pipeline changes.Decisions
og dev fixture, beside the kinds of the sister lane. The e2e legs already get the og binary. The kinds are in their own module,fixture/lineage/, so this branch and the genotype lane touch different lines offixture.rs.random.Randomexactly.random(),uniform(),randrange()andrandint()follow CPython's algorithms on the Mersenne Twister thatfixture.rsalready has. So every input keeps its bytes, every threshold a case counts stays true, and no case needed recalibration. Unit tests pin FNV-1a hashes of what the Python wrote, and CPython's own values for seed 44.prs-sitesandx-ref-sitesprint the picked positions, and a short awk in each case adds the weights and kinds. So one kind serves five cases.truth-genotypesalways writes the##FILTER=<ID=PASS>header line. The IBD case's Python did not write it. That is one extra header line in the case's family VCF. The records are the same bytes.0before it fails on a missing file, so the live case'scountkeeps jq's output only when jq exits 0. A missing key reads as empty, as the Python's caught exception did.Files outside the package's list
rust/crates/og-dev/src/fixture.rs: 15 lines. It declarespub mod lineage, passes a kind name tolineage::clibefore the existing match, and adds the lineage usage lines to--helpand to the unknown-kind error. Without the hook the kinds cannot be called. The genotype lane (test(e2e): build genotype and repeat e2e inputs without inline Python #442) edits the same function with its own hook. The kind names do not collide, so whichever PR lands second keeps both hooks when it merges main.Left as is, owned by other packages
bin/render_report.pyby file, andlive-server-1-weblog.shcallsbin/owngenome_live.pyby file. These are not inline Python.og report rendersays "not ported yet". They go when the report port and the live port land.bin/ogandrust/crates/og-dev/src/fixture.rs. A change underfixture/lineage/alone does not select those cases, becausescripts/ci/e2e-scope.shand the e2e path filter list onlyfixture.rs. The unit test pins catch a change to the bytes. Follow-up: addrust/crates/og-dev/src/fixture/**toSCOPE_FILESand to the e2e path filter, then point the 18 rows atrust/crates/og-dev/src/fixture*. The rows cannot change first:e2e-scope.sh checkrefuses a glob that reaches a file outside the scope.jq1.7 or newer (for@uri) andunzipon the runner. Both are onubuntu-latest, andscripts/ci/parity-diff.shalready uses jq in the e2e legs.How it was checked
Byte parity with the Python. I ran each removed heredoc from the base commit and the matching og kind on the same synthetic inputs.
cmpfound 43 of 43 outputs identical: every fileancient-panelwrites plus its stdout, both MoChA VCFs,sex-loss-womanat two loss fractions,viral-spikeat two depths,plant-snvat three spacings,sites-fasta,mt-roundtrip(stdout and stderr),sites-vcf,segments-check(output and exit),contig-cut(all four files and stdout),add-sites,y-seed,ystr-periodic,ystr-readcountandystr-cut.truth-genotypesmatched both the ancestry heredoc (994 records) and the IBD heredoc (205 records, the same stderr), apart from the FILTER header line named above.Glue parity. A harness takes each new og, awk and jq snippet from the case file and runs it next to the Python it replaced, taken from git, on the same synthetic inputs. It covers all 24 cases:
The one difference is expected: on an empty FASTA the old Python crashed with an IndexError and printed nothing, and the awk prints its count. Both exit 1. The awk and jq results are the same on macOS BWK awk and in a Debian container with mawk 1.3.4, gawk 5.2.1 and jq 1.7.
Red controls, one per case. The planted diffs and the failing test lists: test(e2e): build ancestry, family and QC e2e inputs without inline Python #456 (comment)
2^(h+1), a 10 bp Alu window, a tolerance oftol + 1,startswithforendswith, or a swapped card state. Every case then showed at least one difference: 29 differences in total.cargo test -p og-dev --no-fail-fast. 18 of the 23 lineage tests failed.cigarhad noXop,x-ref-siteshad no block between 1,000 and 2,000 bases from the last pick,truth-genotypeshad no 0/0 truth record,segments-checkfailed its 19-site row on the CSV rather than on the site count, andystr-periodichad no half-periodic region. Each test now has that input, and with the same 5 mutations all 5 fail.Usage listing. A new test checks that
og dev fixture --helpand the error for an unknown kind name every lineage kind. With thefixture.rswiring taken out, it fails (fixture_help_and_an_unknown_kind_list_every_lineage_kind ... FAILED, run locally).Unit tests. The 24 lineage tests pass.
cargo clippy -p og-dev --all-targets -D warnings,cargo fmt --checkandcargo docare clean.No inline Python is left.
grep -nE "python3? +(-I +)?-( |$)|python3? +-c|<<'?PY"over the 24 edited cases finds nothing. The same grep finds 57 lines in them at the base commit. The onlypython3lines left are the by-file calls listed above.shellcheck -S warningis clean on the edited cases.scripts/ci/e2e-scope.sh checkpasses: every one of the 163 cases has one paths row.tests/test_e2e_shards.sh,tests/test_e2e_scope.shand the personal-data scan pass.The red controls above are local. They ran on synthetic inputs, through the harness and
cargo test, not as a planted-wrong case on the CI runner. They show that each new snippet and kind can fail. They do not show a case's[PASS]line going red on CI.The 24 cases have not yet run end to end. They need the HG002 fixture and amd64 tool images, which this box cannot run. The e2e runs on earlier heads were cancelled before any case ran. The e2e run on the final head is the end-to-end check, and this PR lands only when every one of the 24 cases is green there. The paths that only real data reaches (the GIAB truth calls, the pgsc_calc logs, the fastp JSON, jq
@urion the runner,unzipon the 23andMe zip) are proven only by that run.Time and memory
Synthetic inputs. Wall time and peak RSS from
/usr/bin/time -l, on an Apple M-series Mac mini:mocha-vcf(21 Mb over four contigs, both VCFs)viral-spike(17,386 reads)sex-loss-woman(525,000 windows)ancient-panel(3,091 sites, 109 people)prs-sites(chr20 slice)