Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n

## Unreleased

- Step 51 (MoChA) now reads "not assessed" when too few heterozygous sites are phased to see a mosaic event. It read "not assessed" only when no site at all was phased, so a sample with one phased site in the whole genome, or with one autosome left mostly unphased by a panel cut to regions, read "assessed" with no call, and a real 6% CN-LOH above the floor went unreported. The sample is now not assessed when an autosome has under 0.80 of its heterozygous sites phased, and the message names each such autosome. An autosome the panel has no file for is still left out of the step before this check, so it is not assessed and the report does not say so. The minimum comes from two public genomes, HG002 and HG005, where every autosome keeps 0.933 to 0.975 of its heterozygous sites phased by the full panel; `docs/51-mocha.md` has the table. The page also says that MoChA's call rate is always 1 on a WGS VCF, so the call-rate half of its sample filter never fires there. The summary file keeps its rows; only the status and message of such a sample change. Part of #411.
- ROH summaries now divide F_ROH by the GRCh38 length outside the bands of the assessed autosomes. Per-site rows identify those chromosomes, including chromosomes with no ROH. A new `roh.data.scope` records the chromosomes, denominator and scope source. Relationship labels and UPD flags require sites on all 22 autosomes and the full unmasked autosome length. Segment-only files use their observed chromosomes as an unverified lower bound and get neither label. Empty input has null F_ROH. Segment counts and the step 11 summary table keep their format.
- A one-chromosome ROH flag now names regions and citations without syndrome names. Its note includes deletion, acquired loss of heterozygosity and shared ancestry as alternatives. The maternal and paternal JSON keys remain as `"-"` for reader compatibility. The attributed `assets/imprinted_regions.tsv` is embedded at build time, replacing the duplicated Rust table. The container builder mounts it, and the source stamp includes its blob hash, so a table edit makes an old binary stale. The stamp format and `tree-hash` command name stay the same. Every listed chromosome uses the same lookup, including chr11 and chr15. A region hint does not establish overlap with its imprinted locus or parental origin. Broad cytoband names remain reference labels, not diagnostic intervals.
- The ROH collector now streams per-site rows instead of keeping the whole file in memory. On 1.1 million synthetic site rows, peak RSS fell from 65.7-67.0 MB to 3.0-3.5 MB. CPU time stayed near 0.2 seconds. Under shared host load, wall time ranged from 1.40-21.46 seconds before and 7.89-38.16 seconds after; these timings do not establish a speedup.
Expand Down
40 changes: 39 additions & 1 deletion docs/51-mocha.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,45 @@ A call is reported when all of these hold:

At 30x with about 3,000 heterozygous sites per 10 Mb the read-sampling floor is about 0.02, so the floor is 0.05. At 5x with 300 sites per 10 Mb it is about 0.15. The floor is a lower bound: phasing errors and read bias are not in it, so an event above it is not always found.

The sample is **not assessed** when MoChA's sample filter of the same README fails (a call rate under 0.97, or a phased BAF autocorrelation over 0.03), when MoChA gives no value for one of the two, when MoChA gives no median depth (so no floor can be set), or when no heterozygous site on chr1 to chr22 reached MoChA or none is phased. Then no call is reported, and the reports say why.
The sample is **not assessed** when MoChA's sample filter of the same README fails (a call rate under 0.97, or a phased BAF autocorrelation over 0.03), when MoChA gives no value for one of the two, when MoChA gives no median depth (so no floor can be set), when no heterozygous site on chr1 to chr22 reached MoChA, or when an autosome has under 0.80 of its heterozygous sites phased. Then no call is reported, and the reports say why. The message names each autosome under the minimum, with its phased and heterozygous sites.

**The call rate on WGS.** MoChA reads a WGS VCF by its allele depths (AD). In that mode it drops a site with a missing genotype before it counts the sites, so the call rate in its stats is 1 on every WGS VCF, whatever step 1's depth and GQ filter set missing. The call-rate half of the sample filter therefore never fires on this step's input. It applies only to a stats table that carries a real call rate. On WGS the phased BAF autocorrelation, the phased share below and the floor are what gate the sample.

**The phased share.** MoChA follows a haplotype only across phased heterozygous sites. An unphased site adds nothing to its phased model and adds 0 to its phased BAF autocorrelation, so a chromosome that is mostly unphased passes that filter and shows no event, even one above the floor. Beagle phases only the sites the panel holds, so the share of an autosome's heterozygous sites that end up phased is the share the panel holds. A panel cut with `MOCHA_REGIONS`, or a `MOCHA_PHASED_VCF` that is only partly phased, leaves whole stretches unphased. One gap stays: with Beagle, an autosome the panel has no file for (one a `MOCHA_REGIONS` file does not name) is left out of the step before MoChA runs. That autosome has no row in the sites table, so this check cannot see it, and the step does not assess it and does not say so.

We measured that share on two public genomes, with step 1's filters (biallelic PASS SNVs, at least 10 reads and GQ 20) and the panel's sites (biallelic SNVs seen twice or more, the same position and alleles):

- HG002, of Ashkenazi Jewish descent: the DeepVariant 1.0 call set of a 30x Illumina whole genome (Google's DeepVariant WGS case study), every region.
- HG005, of Han Chinese descent: the Genome in a Bottle v4.2.1 benchmark call set, which holds only its benchmark regions.

| Autosome | HG002 heterozygous sites | HG002 share in the panel | HG005 heterozygous sites | HG005 share in the panel |
|---|---|---|---|---|
| chr1 | 167816 | 0.9705 | 149656 | 0.9614 |
| chr2 | 168167 | 0.9698 | 162468 | 0.9684 |
| chr3 | 157028 | 0.9732 | 138142 | 0.9752 |
| chr4 | 158076 | 0.9729 | 145939 | 0.9717 |
| chr5 | 148594 | 0.9733 | 124577 | 0.9709 |
| chr6 | 145168 | 0.9598 | 131235 | 0.9680 |
| chr7 | 131005 | 0.9692 | 115320 | 0.9676 |
| chr8 | 124187 | 0.9712 | 106123 | 0.9685 |
| chr9 | 95946 | 0.9614 | 81566 | 0.9570 |
| chr10 | 113721 | 0.9706 | 95840 | 0.9641 |
| chr11 | 103275 | 0.9723 | 89849 | 0.9730 |
| chr12 | 107852 | 0.9720 | 92700 | 0.9739 |
| chr13 | 81021 | 0.9747 | 68967 | 0.9679 |
| chr14 | 73041 | 0.9679 | 63691 | 0.9733 |
| chr15 | 64200 | 0.9555 | 55996 | 0.9566 |
| chr16 | 66355 | 0.9683 | 61311 | 0.9561 |
| chr17 | 58942 | 0.9554 | 48337 | 0.9647 |
| chr18 | 59704 | 0.9747 | 56497 | 0.9742 |
| chr19 | 53980 | 0.9517 | 44114 | 0.9695 |
| chr20 | 51614 | 0.9471 | 40867 | 0.9751 |
| chr21 | 29933 | 0.9335 | 24548 | 0.9733 |
| chr22 | 31687 | 0.9410 | 24923 | 0.9651 |

Over the whole genome the share is 0.967 (HG002) and 0.968 (HG005). The lowest share on one autosome is 0.933 (HG002, chr21). The minimum is that lowest share less 0.10, rounded down to a multiple of 0.05: 0.80. The 0.10 margin is for a genome whose ancestry the panel holds fewer people of, which has more variants the panel lacks, and for a caller with more false heterozygous calls. A whole genome phased against the full panel stays well above it; a chromosome that the phasing left mostly unphased falls far below it.

Below the minimum, MoChA's unphased model can still see a large event, but the step reports no call, the same as when nothing is phased. The check counts genotypes written with `|`. A VCF phased only in short read-backed blocks (WhatsHap, for example) passes it, but MoChA needs phase along the whole chromosome: give `MOCHA_PHASED_VCF` phase from a population panel or a trio.

## Output

Expand Down
44 changes: 40 additions & 4 deletions rust/crates/og-reports/src/mocha.rs
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,16 @@
//! 6 / sqrt(D x H), D MoChA's median depth and H the heterozygous sites per
//! 10 Mb. The sample is not assessed when MoChA's sample filter fails
//! (call rate under 0.97, phased BAF autocorrelation over 0.03) or cannot
//! be applied. Exit 0 whatever the result; 2 on unreadable input.
//! be applied, or when an autosome has under 80 percent of its
//! heterozygous sites phased. Exit 0 whatever the result; 2 on unreadable
//! input.
//!
//! The call-rate half of the sample filter never fires on a WGS VCF:
//! MoChA's WGS mode (allele depths from AD) drops a site with a missing
//! genotype before it counts it, so its call rate is 1 whatever the step's
//! depth and GQ filter set missing. The gate stays for an input whose stats
//! carry a real call rate; on WGS the phased BAF autocorrelation, the
//! phased share and the floor are what gate the sample.
//!
//! NaN is kept as Python has it: every comparison with NaN is false, so a
//! call with no BAF deviation, no N50 and no concordance LOD is dropped as
Expand All @@ -29,9 +38,10 @@
//! What changes on purpose: `calls_mocha` counts the calls of this sample
//! (the Python counted every row of a multi-sample table, other samples'
//! included; a pipeline run gives MoChA one sample, so its value does not
//! change), and a calls row that ends before a needed column, or a sites
//! change), a calls row that ends before a needed column, or a sites
//! value that is not ASCII digits, is an input error (exit 2), not a
//! traceback.
//! traceback, and a sample with an autosome under the phased minimum is not
//! assessed (the Python assessed it once one site was phased).

use crate::{Error, Outcome};
use og_filters::args::Spec;
Expand All @@ -44,6 +54,14 @@ const FLOOR_Z: f64 = 3.0;
const FLOOR_MB: f64 = 10.0;
const CALL_RATE_MIN: f64 = 0.97;
const BAF_AUTO_MAX: f64 = 0.03;
/// The least share of an autosome's heterozygous sites that must be phased,
/// in percent. Beagle phases only the sites its panel holds, so a whole
/// genome phased against the 1000 Genomes panel keeps most of its
/// heterozygous sites phased on every autosome (docs/51-mocha.md gives the
/// measured shares); a panel cut to regions or a partly phased input VCF
/// leaves whole stretches unphased, where MoChA's phased model is blind and
/// its phased BAF autocorrelation reads 0.
const PHASED_MIN_PERCENT: i128 = 80;

pub const CALL_COLUMNS: [&str; 14] = [
"chrom",
Expand Down Expand Up @@ -198,6 +216,19 @@ fn read_sites(path: &str) -> Result<Vec<Sites>, Error> {
Ok(out)
}

/// The autosomes whose phased share of heterozygous sites is under
/// [`PHASED_MIN_PERCENT`], as `chr7 (12 of 95000)`; `None` when every one
/// with a heterozygous site meets it. Integer arithmetic, so a share of
/// exactly the minimum passes.
fn few_phased(auto: &[&Sites]) -> Option<String> {
let few: Vec<String> = auto
.iter()
.filter(|s| s.hets > 0 && s.phased * 100 < s.hets * PHASED_MIN_PERCENT)
.map(|s| format!("{} ({} of {})", s.chrom, s.phased, s.hets))
.collect();
(!few.is_empty()).then(|| few.join(", "))
}
Comment thread
geiresx marked this conversation as resolved.
Resolved

/// The read-sampling floor: 6 / sqrt(D x H); NaN without data.
fn floor_read(depth: f64, het_per_10mb: f64) -> f64 {
if !(depth > 0.0 && het_per_10mb > 0.0) {
Expand Down Expand Up @@ -318,6 +349,11 @@ pub fn report(
Some("no heterozygous site on chr1 to chr22 reached MoChA".to_owned())
} else if phased == 0 {
Some("no heterozygous site is phased, so MoChA cannot follow a haplotype".into())
} else if let Some(few) = few_phased(&auto) {
Some(format!(
"under {} of the heterozygous sites are phased on {few}, so MoChA cannot follow a haplotype there",
fmt(PHASED_MIN_PERCENT as f64 / 100.0, 2)
))
} else if floor.is_nan() {
Some("MoChA gave no median depth, so no floor can be set".into())
} else if call_rate.is_nan() || baf_auto.is_nan() {
Expand Down Expand Up @@ -568,7 +604,7 @@ mod tests {
first: 10001,
last: 100_010_001,
hets: 120_000,
phased: 94_000,
phased: 114_000,
})
.collect();
report("S", &c, &s, &sites, "beagle").unwrap()
Expand Down
187 changes: 187 additions & 0 deletions rust/crates/og-reports/tests/i411_mocha_phase_gate.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,187 @@
//! Issue #411 (trait-65): `og mocha report` marks a sample "not assessed"
//! when an autosome has too few of its heterozygous sites phased for MoChA
//! to follow a haplotype, not only when no site at all is phased.
//!
//! Before the fix one phased site in the whole genome was enough for
//! "assessed", and a real 6% CN-LOH above the floor then went unreported
//! under a summary that read as a negative result. The minimum is a share
//! per autosome; `docs/51-mocha.md` gives the shares measured on two public
//! genomes and how the minimum follows from them.

use std::path::{Path, PathBuf};

/// The minimum, in percent: `PHASED_MIN_PERCENT` in `src/mocha.rs`.
const MIN_PERCENT: u64 = 80;

const STATS: &str =
"sample_id\tcall_rate\tcov_median\tbaf_auto\tn_hets\nS\t1.0000\t31\t0.0124\t2090000\n";
const CALLS: &str = "sample_id\tchrom\tbeg_GRCh38\tend_GRCh38\tlength\tp_arm\tq_arm\tn_hets\tn50_hets\tbdev\tbdev_se\trel_cov\tlod_baf_phase\tlod_lrr_baf\tlod_baf_conc\ttype\tcf\n\
S\tchr20\t30100000\t42300000\t12200000\tN\tY\t4210\t350000\t0.031\t0.004\t1.86\t48.2\t35.1\tnan\tLoss\t0.120\n";

struct Run {
dir: PathBuf,
}

impl Run {
fn new(name: &str) -> Run {
let dir = std::env::temp_dir().join(format!("og-i411-mocha-{name}-{}", std::process::id()));
let _ = std::fs::remove_dir_all(&dir);
std::fs::create_dir_all(&dir).unwrap();
Run { dir }
}

fn path(&self, f: &str) -> String {
self.dir.join(f).to_string_lossy().into_owned()
}

/// Runs `og mocha report` on these sites rows (chrom, hets, phased);
/// the summary as (metric, value) pairs and the one call row.
fn report(&self, sites: &[(&str, u64, u64)]) -> (Vec<(String, String)>, Vec<String>) {
let mut text = String::new();
for (c, hets, phased) in sites {
text.push_str(&format!(
"{c}\t10001\t100010001\t{}\t{hets}\t{phased}\n",
hets * 3
));
}
std::fs::write(self.path("sites.tsv"), text).unwrap();
std::fs::write(self.path("stats.tsv"), STATS).unwrap();
std::fs::write(self.path("calls.tsv"), CALLS).unwrap();
let args: Vec<String> = [
"--sample",
"S",
"--calls",
&self.path("calls.tsv"),
"--stats",
&self.path("stats.tsv"),
"--sites",
&self.path("sites.tsv"),
"--phasing",
"beagle",
"--out-calls",
&self.path("o.tsv"),
"--out-summary",
&self.path("s.tsv"),
]
.iter()
.map(|s| s.to_string())
.collect();
let (mut out, mut err) = (Vec::new(), Vec::new());
let code = og_reports::mocha::cli(Some("report"), &args, &mut out, &mut err);
assert_eq!(code, 0, "{}", String::from_utf8_lossy(&err));
let summary = read(&self.dir.join("s.tsv"))
.lines()
.skip(1)
.map(|l| {
let (k, v) = l.split_once('\t').unwrap();
(k.to_owned(), v.to_owned())
})
.collect();
let call = read(&self.dir.join("o.tsv"))
.lines()
.nth(1)
.unwrap()
.split('\t')
.map(str::to_owned)
.collect();
(summary, call)
}
}

impl Drop for Run {
fn drop(&mut self) {
let _ = std::fs::remove_dir_all(&self.dir);
}
}

fn read(p: &Path) -> String {
std::fs::read_to_string(p).unwrap()
}

fn get<'a>(summary: &'a [(String, String)], k: &str) -> &'a str {
summary
.iter()
.find(|(n, _)| n == k)
.map(|(_, v)| v.as_str())
.unwrap()
}

fn autosomes(hets: u64, phased: impl Fn(usize) -> u64) -> Vec<(String, u64, u64)> {
(1..=22)
.map(|i| (format!("chr{i}"), hets, phased(i)))
.collect()
}

fn rows(v: &[(String, u64, u64)]) -> Vec<(&str, u64, u64)> {
v.iter().map(|(c, h, p)| (c.as_str(), *h, *p)).collect()
}

/// Red first: the issue's case, 22 autosomes of 95,000 heterozygous sites
/// with one site phased in the whole genome. Before the fix: "assessed".
#[test]
fn red_first_one_phased_site_in_the_genome_is_not_assessed() {
let run = Run::new("one-site");
let sites = autosomes(95_000, |i| u64::from(i == 1));
let (summary, call) = run.report(&rows(&sites));
assert_eq!(get(&summary, "status"), "not assessed", "{summary:?}");
let msg = get(&summary, "message");
assert!(msg.starts_with("under 0.80 of the heterozygous sites are phased on chr1 (1 of 95000), chr2 (0 of 95000)"), "{msg}");
assert!(
msg.ends_with("chr22 (0 of 95000), so MoChA cannot follow a haplotype there"),
"{msg}"
);
assert_eq!(get(&summary, "phased_het_sites"), "1");
assert_eq!(call[12], "no");
assert_eq!(call[13], "sample not assessed");
}

/// Red first: one autosome left unphased (a panel cut to regions) among 21
/// phased ones. The genome-wide share is still 0.95, so only a check per
/// autosome sees it. Before the fix: "assessed".
#[test]
fn red_first_one_unphased_autosome_is_not_assessed() {
let run = Run::new("one-chrom");
let sites = autosomes(95_000, |i| if i == 7 { 12 } else { 93_000 });
let (summary, call) = run.report(&rows(&sites));
assert_eq!(get(&summary, "status"), "not assessed", "{summary:?}");
assert_eq!(
get(&summary, "message"),
"under 0.80 of the heterozygous sites are phased on chr7 (12 of 95000), so MoChA cannot follow a haplotype there"
);
assert_eq!(call[13], "sample not assessed");
}

/// Red first at the minimum minus one site: 95,000 heterozygous sites on
/// chr5 need 76,000 phased; 75,999 is not assessed, 76,000 is.
#[test]
fn red_first_the_minimum_minus_one_site_is_not_assessed() {
let need = 95_000 * MIN_PERCENT / 100;
assert_eq!(need, 76_000);
for (phased5, status) in [(need - 1, "not assessed"), (need, "assessed")] {
let run = Run::new(&format!("min-{phased5}"));
let sites = autosomes(95_000, |i| if i == 5 { phased5 } else { 93_000 });
let (summary, call) = run.report(&rows(&sites));
assert_eq!(
get(&summary, "status"),
status,
"chr5 {phased5}: {summary:?}"
);
let reported = if status == "assessed" { "yes" } else { "no" };
assert_eq!(call[12], reported, "chr5 {phased5}: {call:?}");
}
}

/// Control: a whole genome with 0.92 of every autosome phased, under the
/// lowest share measured on one autosome of the two public genomes (0.933,
/// docs/51-mocha.md), is assessed and its call reported; chrX, which the
/// report never reads, is not gated even when none of it is phased.
#[test]
fn the_measured_shares_are_assessed() {
let run = Run::new("measured");
let mut sites = autosomes(100_000, |_| 92_000);
sites.push(("chrX".into(), 40_000, 0));
let (summary, call) = run.report(&rows(&sites));
assert_eq!(get(&summary, "status"), "assessed", "{summary:?}");
assert_eq!(get(&summary, "message"), ".");
assert_eq!(call[12], "yes", "{call:?}");
}
Loading