diff --git a/CHANGELOG.md b/CHANGELOG.md index b3e8e4021..f4ff31091 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,10 +4,10 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n ## Unreleased +- The tests of `og sv postfilter`, `og sv inv-filter`, `og sv svlen-fix`, `og sv add-callers`, `og clinvar join-adjacent`, `og clinvar hits`, `og clinvar deep-intronic` and `og str catalog` now run in cargo test, in the crates that own the tools, with the same inputs and the same red controls. Four Python test files are gone: `tests/test_annotsv_postfilter.py`, `tests/test_annotsv_sv_inputs.py`, `tests/test_clinvar_join_adjacent.py` and `tests/test_str_catalog.py`. The checks of what `og str motif`'s RFC1 goldens must say run in og-bam's tests; `tests/test_rfc1_motif.py` stays for now, because three e2e cases and the og-bam parity generator import its read sets. The STRchive tests download the pinned files and run Stranger in docker, so they run in CI, or locally with `OG_NET_TESTS=1`. No output changes. - ROH summaries now divide F_ROH by the GRCh38 length outside the bands of the assessed autosomes. Per-site rows identify those chromosomes, including chromosomes with no ROH. A new `roh.data.scope` records the chromosomes, denominator and scope source. Relationship labels and UPD flags require sites on all 22 autosomes and the full unmasked autosome length. Segment-only files use their observed chromosomes as an unverified lower bound and get neither label. Empty input has null F_ROH. Segment counts and the step 11 summary table keep their format. - A one-chromosome ROH flag now names regions and citations without syndrome names. Its note includes deletion, acquired loss of heterozygosity and shared ancestry as alternatives. The maternal and paternal JSON keys remain as `"-"` for reader compatibility. The attributed `assets/imprinted_regions.tsv` is embedded at build time, replacing the duplicated Rust table. The container builder mounts it, and the source stamp includes its blob hash, so a table edit makes an old binary stale. The stamp format and `tree-hash` command name stay the same. Every listed chromosome uses the same lookup, including chr11 and chr15. A region hint does not establish overlap with its imprinted locus or parental origin. Broad cytoband names remain reference labels, not diagnostic intervals. - The ROH collector now streams per-site rows instead of keeping the whole file in memory. On 1.1 million synthetic site rows, peak RSS fell from 65.7-67.0 MB to 3.0-3.5 MB. CPU time stayed near 0.2 seconds. Under shared host load, wall time ranged from 1.40-21.46 seconds before and 7.89-38.16 seconds after; these timings do not establish a speedup. - - The reports now get summary.json from the Rust `og` binary. Step 24, the Nextflow HTML_REPORT process and `generate-report.sh` collect it with `og summary collect` (through `bin/render_report.py`), and a run collects each sample once: `generate-report.sh` renders the text report from the summary.json step 24 wrote, and collects again when a file it reads is newer than that summary.json, or a file was added to or removed from a step folder since, or a file a section names as its source, the run status or a table in `bin/` or `assets/` is gone. When it renders from step 24's summary.json, the text report's generated time is step 24's, the same as the HTML report's; it was the minute of its own collect. When it collects again, the text report carries the new collect's values and time, and the HTML report keeps step 24's until step 24 runs again, as before. The deliberate fixes the `og summary collect` entries below list now reach the reports; apart from them, nothing in summary.json or either report changes. `bin/collect_summary.py` keeps only what the report renderer (`bin/render_report.py`), the couple report, the results index and the live view import from it. The Python modules that only the old collector imported are gone: `bin/pgx_outside_calls.py`, `ancestry_fine_report.py`, `clip_check.py`, `annotsv_postfilter.py`, `chip_report.py`, `traits_report.py`, `run_delta.py` and `genotype_rules_indels.py`. On the synthetic demo sample (4.7 million VCF records) the collect takes 1.1 s of CPU and 12-15 MB, where the Python took 7.8-8.0 s and 34-37 MB. What changes on purpose: - The walk of the sample folder leaves out `/nextflow`. `run-all.sh` launches Nextflow there, and its `work/` holds every task's files and staged links. No section reads a file there. With 200,000 files under it, the collect took 67 MB walked and 14 MB skipped. - The sections other than the sample VCF are read one at a time, while the sample VCF is read on its own thread, so the peak memory is near the biggest section's and not the sum of all of them. With every side table 50 times its demo size, the peak footprint fell from 119-130 MB to 105-112 MB (the Python: 100-102 MB), with the same CPU time. diff --git a/docs/06-clinvar-screen.md b/docs/06-clinvar-screen.md index 3a8e5873a..16c3a4a29 100644 --- a/docs/06-clinvar-screen.md +++ b/docs/06-clinvar-screen.md @@ -60,7 +60,7 @@ In the Nextflow pipeline the same floor is `ext.min_gq` and `ext.min_dp` of `CLI 10. Runs the cDNA read check on each hit and the cDNA scan of every exon (below). 11. Prints the hits grouped by ClinVar's review stars (`og clinvar hits`): 4 practice guideline, 3 expert panel, 2 multiple submitters with no conflict, 1 a single submitter or conflicting classifications, 0 no assertion criteria, each with its combined-allele note when it has one and "LOW-CONFIDENCE CALL (GQ .., DP ..): check the reads" when the call is below the floor, and the count of low-confidence calls. The screening file keeps every Pathogenic/Likely_pathogenic submission whatever its review status, so a zero-star hit counts as a hit; the stars say how much weight it deserves. The Nextflow module runs the same steps and prints the same counts; its `CLINVAR_CDNA_CHECK` process runs the cDNA read check and the cDNA scan. -A delins that the caller writes as two records, while ClinVar holds it as one, is matched exactly once, as the combined record of item 5: `tests/test_clinvar_join_adjacent.py` check 8 runs the real step on such a fixture and counts the hits. +A delins that the caller writes as two records, while ClinVar holds it as one, is matched exactly once, as the combined record of item 5: the e2e case `clinvar-combined-allele-1` runs the real step on such a fixture and counts the hits, and `rust/crates/og-clinvar/tests/port_tests_sv_clinvar_str_join.rs` checks og's part on the same fixture. ### The always-report list diff --git a/docs/09-str-expansions.md b/docs/09-str-expansions.md index c46b8a99e..795c99b44 100644 --- a/docs/09-str-expansions.md +++ b/docs/09-str-expansions.md @@ -53,7 +53,7 @@ The reports read the counts (`og summary collect`, under the repeat expansions s - **UGT1A1**: the TA count of each copy as its allele (`*36`, `*1`, `*28`, `*37`), and the number of decreased-function copies (`*28`, `*37`) compared with PharmCAT's UGT1A1 call of step 27, which reads `*28` from the small-variant calls with the reference-block fill of `og pgx refblock-fill`. The line says whether the two agree; when they differ, read the reads before using either. A TA length outside TA5 to TA8 has no named allele, so the line makes no function claim and does not compare it. A step 27 table from an earlier run is compared and said to be so; one of a failed or unreadable step is not compared. PharmCAT's call is still the one step 27's drug guidance uses. - Without the opt-in, both lines say the repeat was not sized and give the command that adds it: `STR_EXTRA_LOCI=yes ./scripts/setup.sh --strchive-data `. -`tests/test_str_catalog.py` checks each region against the GRCh38 sequence, `tests/test_collect_summary.py` checks the readings on made-up counts (a TG11-7T/TG11-7T sample is the red control of the 5T line), the e2e case `genotype-rule-rows-1-str-extra` sizes both tracts on made-up reads with ExpansionHunter 5.0.0, and the e2e case `str-extra-repeats-1-readings` runs step 09 and both reports on made-up reads and checks the lines. +`rust/crates/og-setup/tests/strchive_catalog.rs` checks each region against the GRCh38 sequence, `tests/test_collect_summary.py` checks the readings on made-up counts (a TG11-7T/TG11-7T sample is the red control of the 5T line), the e2e case `genotype-rule-rows-1-str-extra` sizes both tracts on made-up reads with ExpansionHunter 5.0.0, and the e2e case `str-extra-repeats-1-readings` runs step 09 and both reports on made-up reads and checks the lines. The opt-in stays off by default: turning it on changes the catalog every run sizes, and the CFTR and UGT1A1 lines are readings of a modifier and of a PGx allele PharmCAT already calls, not expansions. diff --git a/docs/testing.md b/docs/testing.md index 7302f594c..3911bbd64 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -257,7 +257,7 @@ The first port is the reference mask of `setup.sh --masked-reference`. On a 254 On the whole GRCh38 analysis set (3.1 GB) `og` writes the file `GRC_MASKED_REF_SHA256` pins in 29 s on the same machine, 1.2 s of it CPU: writing the copy is what takes the time. -The next four ports are the VCF filters of steps 04, 05 and 07: `og sv inv-filter`, `og sv svlen-fix`, `og pgx refblock-fill` and `og pgx mito-merge`. Their cases run each oracle in the image the step used (`inv-filter` on the Manta image's Python 2.7, and on Python 3 too). `refblock-fill` reads a stream on stdin, so its cases stage a file named `stdin`. e2e does not shadow these four: the step scripts call them from the image's own working directory, `refblock-fill` reads stdin, which the shadow cannot replay, and `mito-merge` writes `--out` outside the call's directory. Their unit tests (`tests/test_annotsv_sv_inputs.py`, `tests/test_pgx_refblock_fill.py`, `tests/test_pgx_mito_merge.py`) and the three step 05 tests run `og`. The unit-tests job has no `og`, so there they skip with a note. The Guard job `rust` runs them with `OG_REQUIRED=1` after its musl builds, and there a skip fails. +The next four ports are the VCF filters of steps 04, 05 and 07: `og sv inv-filter`, `og sv svlen-fix`, `og pgx refblock-fill` and `og pgx mito-merge`. Their cases run each oracle in the image the step used (`inv-filter` on the Manta image's Python 2.7, and on Python 3 too). `refblock-fill` reads a stream on stdin, so its cases stage a file named `stdin`. e2e does not shadow these four: the step scripts call them from the image's own working directory, `refblock-fill` reads stdin, which the shadow cannot replay, and `mito-merge` writes `--out` outside the call's directory. The two SV filters are checked in cargo test (`rust/crates/og-steps/tests/sv_inputs.rs`). The unit tests of the other two (`tests/test_pgx_refblock_fill.py`, `tests/test_pgx_mito_merge.py`) and the three step 05 tests run `og`. The unit-tests job has no `og`, so there they skip with a note. The Guard job `rust` runs them with `OG_REQUIRED=1` after its musl builds, and there a skip fails. The checks CI runs on the pipeline itself are ports too: `og dev contract` (the output contract), `og dev schema` (a `summary.json` against `tests/schema/summary.schema.json`) and `og dev report-only` (the report-only error strategy). Their cargo tests in `rust/crates/og-dev/tests/repo.rs` run them on this checkout and on copies with one planted break each, in the Guard job `rust`. `og dev schema` and the run check of `og dev contract` have parity cases; `--static` and `og dev report-only` read a whole checkout, which a case cannot stage, so their parity was shown on the checkout and the planted copies when they were ported. Each fixes a check that passed on what it should refuse: a schema default of another JSON type (`false` for `0`), `additionalProperties: false`, `true` against `enum: [1]`, NaN against `minimum`, a `$ref` to nothing (now exit 2, a broken schema), a `//` inside a string and a nested brace in a strategy closure. `tests/schema/validate.py` stays only for three Python test files that validate the summaries they build in-process, in the unit-tests job, which has no `og`; it follows the same rules. The fake-docker case `reports-stale-step` checks its `summary.json` with `og dev schema` in the job `rust`; the job `fake-docker` has no `og` and says it skipped that check. diff --git a/rust/crates/og-bam/tests/motif_meaning.rs b/rust/crates/og-bam/tests/motif_meaning.rs new file mode 100644 index 000000000..b1bb9d280 --- /dev/null +++ b/rust/crates/og-bam/tests/motif_meaning.rs @@ -0,0 +1,153 @@ +//! What `og str motif`'s goldens must mean, by meaning: the checks of the +//! old `tests/test_rfc1_motif.py`, in cargo test. +//! +//! `tests/parity.rs` holds og to its goldens byte for byte +//! (`tests/data/parity/want/`, or `og/` for a deliberate fix). +//! This test says what those goldens must say about the made-up reads at +//! the GRCh38 RFC1 repeat, so a golden rewritten with another verdict fails +//! here even when og agrees with it: +//! +//! - benign: AAAAG reads across the repeat and soft-clipped, benign_motif; +//! - pathogenic: AAGGG reads soft-clipped, unmapped and placed at their +//! mate, rescued from another contig and on the reverse strand, plus three +//! expanded AAAAG reads; the eight reads across an unexpanded (AAAAG)11 +//! allele are not counted: pathogenic_motif with both shares; +//! - mixed (AAAAG and AAAGG): benign_motif with both shares; acagg: +//! pathogenic_motif; unknown (AAGAG): uncertain_motif; few (two reads): +//! not_resolved; +//! - controls, each failing when the rescue it names is off: `--no-softclip` +//! counts the 6 clipped reads fewer, `--no-mates` the 3 rescued mates; +//! - `--skip` and a VCF without an RFC1 record: not_checked. + +use std::collections::HashMap; +use std::path::{Path, PathBuf}; + +fn parity() -> PathBuf { + Path::new(env!("CARGO_MANIFEST_DIR")).join("tests/data/parity") +} + +/// og's golden of a parity case when it has one (a deliberate fix), else +/// the oracle's. +fn golden(case: &str) -> PathBuf { + let og = parity().join("og").join(case); + if og.is_dir() { + og + } else { + parity().join("want").join(case) + } +} + +/// The one row of the case's motif.tsv, by column. +fn table(case: &str) -> HashMap { + let path = golden(case).join("files/motif.tsv"); + let text = std::fs::read_to_string(&path).unwrap_or_else(|e| panic!("{}: {e}", path.display())); + let mut lines = text.lines(); + let head = lines.next().unwrap_or_default().split('\t'); + let row = lines.next().unwrap_or_default().split('\t'); + head.zip(row) + .map(|(k, v)| (k.to_owned(), v.to_owned())) + .collect() +} + +fn exit_code(case: &str) -> i32 { + let path = golden(case).join("exit"); + std::fs::read_to_string(&path) + .unwrap_or_else(|e| panic!("{}: {e}", path.display())) + .trim() + .parse() + .unwrap() +} + +fn get<'a>(t: &'a HashMap, k: &str) -> &'a str { + t.get(k).map_or("", String::as_str) +} + +#[test] +fn each_read_set_gets_its_verdict() { + for (kind, verdict, reads) in [ + ("benign", "benign_motif", "10"), + ("pathogenic", "pathogenic_motif", "16"), + ("mixed", "benign_motif", "10"), + ("acagg", "pathogenic_motif", "5"), + ("unknown", "uncertain_motif", "5"), + ("few", "not_resolved", "2"), + ] { + let case = format!("motif-{kind}"); + let t = table(&case); + assert_eq!( + ( + exit_code(&case), + get(&t, "verdict"), + get(&t, "repeat_reads") + ), + (0, verdict, reads), + "{kind}: exit 0, {verdict}, {reads} repeat reads" + ); + } +} + +#[test] +fn the_pathogenic_reads_and_their_shares() { + let t = table("motif-pathogenic"); + assert_eq!( + ["chrom", "start", "end", "dominant_motif", "motif_shares"].map(|k| get(&t, k)), + [ + "chr4", + "39348425", + "39348479", + "AAGGG", + "AAGGG=0.81,AAAAG=0.19" + ], + "pathogenic: the locus from the VCF, AAGGG dominant, both shares; the 8 reference-allele reads not counted" + ); + assert!( + get(&t, "note").contains("repeat-primed PCR"), + "pathogenic: the note asks for repeat-primed PCR: {}", + get(&t, "note") + ); + let benign = table("motif-benign"); + assert!( + get(&benign, "note").contains("the flag is not supported"), + "benign: the note says the flag is not supported: {}", + get(&benign, "note") + ); + assert_eq!( + get(&table("motif-mixed"), "motif_shares"), + "AAAAG=0.60,AAAGG=0.40", + "mixed: AAAAG and AAAGG counted apart" + ); + let unknown = table("motif-unknown"); + assert_eq!( + ( + get(&unknown, "dominant_motif"), + get(&unknown, "dominant_class") + ), + ("AAGAG", "unknown"), + "unknown: AAGAG named by its smallest form, class unknown" + ); +} + +#[test] +fn the_rescue_controls_and_the_unchecked_cases() { + assert_eq!( + get(&table("motif-pathogenic-no-softclip"), "repeat_reads"), + "10", + "control: --no-softclip loses the 6 clipped reads" + ); + assert_eq!( + get(&table("motif-pathogenic-no-mates"), "repeat_reads"), + "13", + "control: --no-mates loses the 3 rescued mates" + ); + let skip = table("motif-skip"); + assert_eq!( + (get(&skip, "verdict"), get(&skip, "note")), + ("not_checked", "no BAM"), + "--skip: not_checked with the reason" + ); + assert_eq!( + get(&table("motif-no-rfc1-record"), "verdict"), + "not_checked", + "a VCF without an RFC1 record: not_checked" + ); +} diff --git a/rust/crates/og-clinvar/tests/port_tests_sv_clinvar_str_join.rs b/rust/crates/og-clinvar/tests/port_tests_sv_clinvar_str_join.rs new file mode 100644 index 000000000..ec99f0a80 --- /dev/null +++ b/rust/crates/og-clinvar/tests/port_tests_sv_clinvar_str_join.rs @@ -0,0 +1,707 @@ +//! `og clinvar join-adjacent`, `og clinvar hits` and +//! `og clinvar deep-intronic` on hand-made records at real GRCh38 chr22 +//! positions: what the old `tests/test_clinvar_join_adjacent.py` checked, +//! by meaning, in cargo test. +//! +//! 1. Two records within 10 bp, both het with the same unphased GT, become +//! one REF/ALT with the reference bases between them; 0/1 and 1/0 are +//! the same het call. +//! 2. Phased records join only in the same phase set with the same GT. +//! 3. A het next to a hom, an unphased half call, overlapping records and +//! records 11 bp apart are not joined; 10 bp apart are; a phased half +//! call joins a record with its ALT on the same haplotype. +//! 4. A single SNV gives no combined record. +//! 5. finish drops a combined record whose part disagrees with the +//! reference. +//! 6. The combined records come out in POS order. +//! 7. The tag pass marks the parts of a matched combined allele, flags two +//! hits in one gene that can be one allele, and leaves the rest alone. +//! 8. The fixture records of `tests/fixtures/vcf/combined_allele_*.vcf`, +//! read in place: the one combined record, left-trimmed as step 06's +//! `bcftools norm` writes it, is ClinVar's 2-star record; its parts are +//! ClinVar's two 1-star records; the tag pass marks both and leaves the +//! single SNV hit unchanged, four hits in all. The real step 06 with +//! bcftools on the same files is the e2e case `clinvar-combined-allele-1`. +//! 9. A 1/2 record at a poly-A tract split as step 06 splits it (1/. for +//! ClinVar's allele): the hit is worded "one copy, other copy not called +//! ref", never het; a plain 0/1 hit stays het; and the always-report +//! list gives every site a call. `tests/test_isec_columns.sh` runs the +//! real split through step 06. +//! +//! Prepare, finish and build must agree: every join check runs prepare, +//! a fill of `PGP_JOIN_SEQ` as `bcftools +fill-from-fasta` does it, then +//! finish, and build on a FASTA, and both must write the same records. +//! Set `WINDOW` to 0 in `src/join.rs` and checks 1, 2, 3, 6, 7 and 8 fail. + +use std::io::Cursor; +use std::path::{Path, PathBuf}; + +use og_clinvar::join::{self, Mode, WINDOW}; +use og_clinvar::{hits, sites}; +use og_intervals::fasta::Indexed; + +/// GRCh38 chr22 (the same in the NCBI no-ALT analysis set), two short +/// stretches. +const REF: [(u64, &str); 2] = [ + ( + 30000000, + "AGACTCTGTCGCAAAAAAAAAATTAATGTAACCATTTAGCCATTACCCAGTTCCTTAAAAT", + ), + (30000200, "GCCTGTAATCCCAGCACTTTGGGAGGCTGAG"), +]; + +const HEAD: &str = "##fileformat=VCFv4.2\n##contig=\n\ +##FORMAT=\n\ +##FORMAT=\n\ +#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT\tS\n"; + +fn repo() -> PathBuf { + Path::new(env!("CARGO_MANIFEST_DIR")).join("../../..") +} + +/// A scratch folder of its own for one test, removed when dropped. +struct Scratch(PathBuf); + +impl Scratch { + fn new(name: &str) -> Scratch { + let d = std::env::temp_dir().join(format!("og-clinvar-join-{name}-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&d); + std::fs::create_dir_all(&d).unwrap(); + Scratch(d) + } +} + +impl Drop for Scratch { + fn drop(&mut self) { + let _ = std::fs::remove_dir_all(&self.0); + } +} + +fn ref_at(pos: u64, n: usize) -> String { + for (start, seq) in REF { + if start <= pos && pos as usize + n <= start as usize + seq.len() { + let i = (pos - start) as usize; + return seq[i..i + n].to_owned(); + } + } + panic!("chr22:{pos} (+{n}) is outside the test's reference stretches"); +} + +/// chr22 up to the last stretch, N everywhere else, 60 bases a line, with +/// its .fai. +fn write_reference(dir: &Path) -> String { + let length = REF + .iter() + .map(|(s, q)| *s as usize + q.len()) + .max() + .unwrap() + + 50; + let mut seq = vec![b'N'; length]; + for (start, q) in REF { + let s = start as usize - 1; + seq[s..s + q.len()].copy_from_slice(q.as_bytes()); + } + let mut fa = b">chr22\n".to_vec(); + for chunk in seq.chunks(60) { + fa.extend_from_slice(chunk); + fa.push(b'\n'); + } + let path = dir.join("ref.fa"); + std::fs::write(&path, fa).unwrap(); + std::fs::write( + dir.join("ref.fa.fai"), + format!("chr22\t{length}\t7\t60\t61\n"), + ) + .unwrap(); + path.to_string_lossy().into_owned() +} + +fn rec(pos: u64, r: &str, alt: &str, gt: &str, ps: Option) -> String { + let (fmt, smp) = match ps { + None => ("GT".to_owned(), gt.to_owned()), + Some(p) => ("GT:PS".to_owned(), format!("{gt}:{p}")), + }; + format!("chr22\t{pos}\t.\t{r}\t{alt}\t50\tPASS\t.\t{fmt}\t{smp}\n") +} + +/// What `bcftools +fill-from-fasta -- -c INFO/PGP_JOIN_SEQ` does: the +/// reference bases from POS for the length of REF. +fn fill(vcf: &str) -> String { + vcf.lines() + .map(|line| { + if line.starts_with('#') { + return line.to_owned() + "\n"; + } + let mut f: Vec = line.split('\t').map(str::to_owned).collect(); + let seq = ref_at(f[1].parse().unwrap(), f[3].len()); + f[7] = f[7].replacen("PGP_JOIN_SEQ=.", &format!("PGP_JOIN_SEQ={seq}"), 1); + f.join("\t") + "\n" + }) + .collect() +} + +fn body(text: &str) -> Vec { + text.lines() + .filter(|l| !l.is_empty() && !l.starts_with('#')) + .map(str::to_owned) + .collect() +} + +/// prepare | fill | finish, and build, which must write the same records: +/// the records as fields, and the whole output of finish. +fn joined(fasta: &str, records: &[String]) -> (Vec>, String) { + let input = HEAD.to_owned() + &records.concat(); + let mut prep = Vec::new(); + join::join( + Mode::Prepare, + WINDOW, + Cursor::new(input.as_bytes()), + &mut prep, + ) + .expect("prepare"); + let mut done = Vec::new(); + join::finish( + Cursor::new(fill(&String::from_utf8(prep).unwrap()).into_bytes()), + &mut done, + ) + .expect("finish"); + let done = String::from_utf8(done).unwrap(); + let mut ix = Indexed::open(fasta).expect("the test reference"); + let mut built = Vec::new(); + join::join( + Mode::Build(&mut ix), + WINDOW, + Cursor::new(input.as_bytes()), + &mut built, + ) + .expect("build"); + let built = String::from_utf8(built).unwrap(); + assert_eq!( + body(&built), + body(&done), + "build must write the records of prepare and finish" + ); + let recs = body(&done) + .iter() + .map(|l| l.split('\t').map(str::to_owned).collect()) + .collect(); + (recs, done) +} + +/// POS, REF, ALT, FORMAT and the sample of each record. +fn core(recs: &[Vec]) -> Vec<[&str; 5]> { + recs.iter() + .map(|r| [&*r[1], &*r[3], &*r[4], &*r[8], &*r[9]]) + .collect() +} + +#[test] +fn neighbouring_records_join_into_one_allele() { + let w = Scratch::new("join"); + let fasta = write_reference(&w.0); + let j = |records: &[String]| joined(&fasta, records); + + // 1. unphased het pair, 3 bp apart: AGC>A (a 2 bp deletion) and T>TGGG + let (out, raw) = j(&[ + rec(30000037, "AGC", "A", "0/1", None), + rec(30000042, "T", "TGGG", "0/1", None), + ]); + assert_eq!( + core(&out), + [["30000037", "AGCCAT", "ACATGGG", "GT", "0/1"]], + "two het records 3 bp apart become one REF/ALT with the reference bases between them" + ); + assert_eq!( + out[0][7], "PGP_JOINED_FROM=chr22:30000037:AGC:A|chr22:30000042:T:TGGG", + "the combined record names its two parts" + ); + assert!( + raw.contains("ID=PGP_JOINED_FROM") && !raw.contains("ID=PGP_JOIN_SEQ"), + "the header keeps PGP_JOINED_FROM and drops the temporary PGP_JOIN_SEQ: {raw}" + ); + let (out, _) = j(&[ + rec(30000203, "T", "A", "1/0", None), + rec(30000208, "T", "C", "0/1", None), + ]); + assert_eq!( + core(&out), + [["30000203", "TGTAAT", "AGTAAC", "GT", "0/1"]], + "1/0 and 0/1 (unphased) are the same het call and are joined" + ); + + // 2. phase sets + let (out, _) = j(&[ + rec(30000203, "T", "A", "0|1", Some(100)), + rec(30000208, "T", "C", "0|1", Some(100)), + ]); + assert_eq!( + core(&out), + [["30000203", "TGTAAT", "AGTAAC", "GT:PS", "0|1:100"]], + "phased, same phase set and GT: joined, with GT and PS" + ); + let none: [[&str; 5]; 0] = []; + for (desc, a, b) in [ + ( + "phased on different phase sets: not joined", + rec(30000203, "T", "A", "0|1", Some(100)), + rec(30000208, "T", "C", "0|1", Some(200)), + ), + ( + "phased on the two haplotypes of one phase set: not joined", + rec(30000203, "T", "A", "0|1", Some(100)), + rec(30000208, "T", "C", "1|0", Some(100)), + ), + ( + "one phased and one unphased record: not joined", + rec(30000203, "T", "A", "0|1", Some(100)), + rec(30000208, "T", "C", "0/1", None), + ), + // 3. what is not one allele, and the window's edge + ( + "a het next to a hom: not joined", + rec(30000203, "T", "A", "0/1", None), + rec(30000208, "T", "C", "1/1", None), + ), + ( + "unphased half calls (./1): not joined", + rec(30000203, "T", "A", "./1", None), + rec(30000208, "T", "C", "./1", None), + ), + ( + "a phased half call and an ALT on the other haplotype: not joined", + rec(30000203, "T", "A", "1|.", Some(100)), + rec(30000208, "T", "C", "0|1", Some(100)), + ), + ( + "overlapping records: not joined", + rec(30000203, "TGT", "T", "0/1", None), + rec(30000204, "G", "A", "0/1", None), + ), + ( + "11 bp apart: not joined", + rec(30000203, "T", "A", "0/1", None), + rec(30000214, "C", "A", "0/1", None), + ), + // 5. a part whose REF is not the reference's base (30000203 is T) + ( + "a part that disagrees with the reference: dropped by finish", + rec(30000203, "C", "A", "0/1", None), + rec(30000208, "T", "C", "0/1", None), + ), + ] { + let (out, raw) = j(&[a, b]); + assert_eq!(core(&out), none, "{desc}: {raw}"); + } + for (a, b, gt) in [ + ("1|.", "1|0", "1|."), + ("1|0", "1|.", "1|."), + (".|1", ".|1", ".|1"), + ] { + let (out, _) = j(&[ + rec(30000203, "T", "A", a, Some(100)), + rec(30000208, "T", "C", b, Some(100)), + ]); + let sample = format!("{gt}:100"); + assert_eq!( + core(&out), + [["30000203", "TGTAAT", "AGTAAC", "GT:PS", sample.as_str()]], + "phased half calls {a} and {b} in one phase set: joined, '.' kept on the other haplotype" + ); + } + let (out, _) = j(&[ + rec(30000203, "T", "A", "0/1", None), + rec(30000213, "G", "T", "0/1", None), + ]); + let three: Vec<[&str; 3]> = core(&out).iter().map(|c| [c[0], c[1], c[2]]).collect(); + assert_eq!( + three, + [["30000203", "TGTAATCCCAG", "AGTAATCCCAT"]], + "10 bp apart: joined" + ); + + // 4. a single SNV + let (out, _) = j(&[rec(30000210, "C", "T", "0/1", None)]); + assert_eq!(core(&out), none, "a single SNV gives no combined record"); + + // 6. POS order: a 12 bp deletion that ends after the two records inside it + let (out, raw) = j(&[ + rec(30000201, "CCTGTAATCCCA", "C", "0/1", None), + rec(30000205, "T", "G", "0/1", None), + rec(30000208, "T", "C", "0/1", None), + rec(30000214, "C", "A", "0/1", None), + ]); + let pos: Vec<&str> = out.iter().map(|r| r[1].as_str()).collect(); + assert_eq!( + pos, + ["30000201", "30000205", "30000205", "30000208"], + "four combined records, in POS order: {raw}" + ); + + // mode is required + let args = ["join-adjacent".to_owned()]; + assert_eq!( + og_clinvar::cli::main(&args), + 2, + "no mode: exit 2 with a message" + ); +} + +const CV: &str = "GENEINFO={g};CLNSIG=Pathogenic;CLNREVSTAT=criteria_provided,_single_submitter"; + +fn cv(g: &str) -> String { + CV.replace("{g}", g) +} + +/// The tag pass: its output and the INFO of each record by POS. +fn tag(hits: &str) -> (String, std::collections::HashMap) { + let mut out = Vec::new(); + join::tag(Cursor::new(hits.as_bytes()), WINDOW, &mut out).expect("tag"); + let out = String::from_utf8(out).unwrap(); + let info = body(&out) + .iter() + .map(|l| { + let f: Vec<&str> = l.split('\t').collect(); + (f[1].to_owned(), f[7].to_owned()) + }) + .collect(); + (out, info) +} + +#[test] +fn the_tag_pass_marks_parts_and_one_allele_pairs() { + let lines = [ + format!( + "chr22\t30000037\t9900001\tAGC\tA\t50\tPASS\t{}\tGT\t0/1", + cv("SYNTH1:1") + ), + format!( + "chr22\t30000038\t9900003\tGCCAT\tCATGGG\t.\tPASS\tPGP_JOINED_FROM=chr22:30000037:AGC:A|chr22:30000042:T:TGGG;{}\tGT\t0/1", + cv("SYNTH1:1").replace("single_submitter", "multiple_submitters,_no_conflicts") + ), + format!( + "chr22\t30000042\t9900002\tT\tTGGG\t50\tPASS\t{}\tGT\t0/1", + cv("SYNTH1:1") + ), + format!( + "chr22\t30000210\t9900004\tC\tT\t50\tPASS\t{}\tGT\t0/1", + cv("SYNTH2:2") + ), + format!( + "chr22\t30000300\t9900005\tA\tG\t50\tPASS\t{}\tGT\t0/1", + cv("SYNTH3:3") + ), + format!( + "chr22\t30000305\t9900006\tC\tT\t50\tPASS\t{}\tGT\t0/1", + cv("SYNTH3:3") + ), + format!( + "chr22\t30000400\t9900007\tA\tG\t50\tPASS\t{}\tGT\t0/1", + cv("SYNTH4:4") + ), + format!( + "chr22\t30000404\t9900008\tC\tT\t50\tPASS\t{}\tGT\t0/1", + cv("SYNTH5:5") + ), + format!( + "chr22\t30000500\t9900009\tA\tG\t50\tPASS\t{}\tGT:PS\t0|1:7", + cv("SYNTH6:6") + ), + format!( + "chr22\t30000504\t9900010\tC\tT\t50\tPASS\t{}\tGT:PS\t1|0:7", + cv("SYNTH6:6") + ), + format!( + "chr22\t30000600\t9900011\tA\tG\t50\tPASS\t{}\tGT:PS\t1|.:8", + cv("SYNTH7:7") + ), + format!( + "chr22\t30000605\t9900012\tC\tT\t50\tPASS\t{}\tGT:PS\t1|0:8", + cv("SYNTH7:7") + ), + ]; + let (out, info) = tag(&(HEAD.to_owned() + &lines.join("\n") + "\n")); + let got = body(&out); + assert!( + got.len() == lines.len() && out.starts_with("##fileformat"), + "tag: header and every record kept: {out}" + ); + let i = |p: &str| info.get(p).map_or("", String::as_str); + let part = "PGP_PART_OF=chr22:30000038:GCCAT:CATGGG"; + assert!( + i("30000037").ends_with(part) && i("30000042").ends_with(part), + "tag: both parts are marked part of the combined allele: {info:?}" + ); + assert!( + !i("30000038").contains("PGP_PART_OF") && !i("30000038").contains("PGP_ONE_ALLELE"), + "tag: the combined hit itself gets no part mark: {info:?}" + ); + assert!( + got.contains(&lines[3]), + "tag: the single SNV hit is unchanged" + ); + assert!( + i("30000300").ends_with("PGP_ONE_ALLELE=chr22:30000305:C:T") + && i("30000305").ends_with("PGP_ONE_ALLELE=chr22:30000300:A:G"), + "tag: two hits in one gene, 5 bp apart, no combined record: flagged both ways: {info:?}" + ); + assert!( + lines[6..10].iter().all(|l| got.contains(l)), + "tag: hits in two genes, or on the two haplotypes of a phase set, are unchanged" + ); + assert!( + i("30000600").ends_with("PGP_ONE_ALLELE=chr22:30000605:C:T") + && i("30000605").ends_with("PGP_ONE_ALLELE=chr22:30000600:A:G"), + "tag: a phased half call (1|.) and 1|0 in one gene and phase set: flagged both ways: {info:?}" + ); +} + +fn fixture(name: &str) -> String { + let p = repo().join("tests/fixtures/vcf").join(name); + std::fs::read_to_string(&p).unwrap_or_else(|e| panic!("{}: {e}", p.display())) +} + +/// VCF left-trimming as `bcftools norm` writes a combined record: drop a +/// first base REF and ALT share while both keep one. +fn left_trim(pos: u64, r: &str, a: &str) -> (u64, String, String) { + let (mut pos, mut r, mut a) = (pos, r.to_owned(), a.to_owned()); + while r.len() > 1 && a.len() > 1 && r.as_bytes()[0] == a.as_bytes()[0] { + r.remove(0); + a.remove(0); + pos += 1; + } + (pos, r, a) +} + +#[test] +fn the_fixture_delins_matches_clinvar_once_as_one_allele() { + let w = Scratch::new("fixture"); + let fasta = write_reference(&w.0); + let sample = fixture("combined_allele_sample.vcf"); + let clinvar = fixture("combined_allele_clinvar.vcf"); + let sample_recs: Vec> = body(&sample) + .iter() + .map(|l| l.split('\t').map(str::to_owned).collect()) + .collect(); + for r in &sample_recs { + assert_eq!( + ref_at(r[1].parse().unwrap(), r[3].len()), + r[3], + "the fixture's REF bases are the reference's at chr22:{}", + r[1] + ); + } + + let mut ix = Indexed::open(&fasta).expect("the test reference"); + let mut built = Vec::new(); + join::join( + Mode::Build(&mut ix), + WINDOW, + Cursor::new(sample.as_bytes()), + &mut built, + ) + .expect("build"); + let combined: Vec> = body(&String::from_utf8(built).unwrap()) + .iter() + .map(|l| l.split('\t').map(str::to_owned).collect()) + .collect(); + assert_eq!( + combined.len(), + 1, + "the delins written as two records gives one combined record: {combined:?}" + ); + let c = &combined[0]; + let (pos, r, a) = left_trim(c[1].parse().unwrap(), &c[3], &c[4]); + + // ClinVar's records by allele. + let cv: std::collections::HashMap<(u64, String, String), Vec> = body(&clinvar) + .iter() + .map(|l| { + let f: Vec = l.split('\t').map(str::to_owned).collect(); + ((f[1].parse().unwrap(), f[3].clone(), f[4].clone()), f) + }) + .collect(); + let two_star = cv.get(&(pos, r.clone(), a.clone())).unwrap_or_else(|| { + panic!("the combined allele chr22:{pos} {r}>{a} is no ClinVar record of the fixture") + }); + assert!( + two_star[2] == "9900003" + && two_star[7] + .contains("CLNREVSTAT=criteria_provided,_multiple_submitters,_no_conflicts"), + "the combined allele is ClinVar's 2-star record 9900003: {two_star:?}" + ); + assert_eq!( + c[7], "PGP_JOINED_FROM=chr22:30000037:AGC:A|chr22:30000042:T:TGGG", + "the combined record names its two parts" + ); + for (p, r, a, id) in [ + (30000037, "AGC", "A", "9900001"), + (30000042, "T", "TGGG", "9900002"), + ] { + assert_eq!( + cv.get(&(p, r.to_owned(), a.to_owned())) + .map(|f| f[2].as_str()), + Some(id), + "the part {p} {r}>{a} is ClinVar's 1-star record {id}" + ); + } + + // The hits as step 06 writes them before the tag pass: each candidate + // (the sample's records and the combined one) that is a ClinVar allele, + // with ClinVar's ID and INFO, in POS order. + let mut candidates = sample_recs.clone(); + let mut trimmed = c.clone(); + (trimmed[1], trimmed[3], trimmed[4]) = (pos.to_string(), r, a); + candidates.push(trimmed); + candidates.sort_by_key(|f| f[1].parse::().unwrap()); + let hits_lines: Vec = candidates + .iter() + .filter_map(|f| { + let m = cv.get(&(f[1].parse().unwrap(), f[3].clone(), f[4].clone()))?; + let info = if f[7] == "." { + m[7].clone() + } else { + format!("{};{}", f[7], m[7]) + }; + let mut h = f.clone(); + (h[2], h[7]) = (m[2].clone(), info); + Some(h.join("\t")) + }) + .collect(); + let (out, info) = tag(&(HEAD.to_owned() + &hits_lines.join("\n") + "\n")); + let got = body(&out); + assert_eq!( + got.len(), + 4, + "four hits in all: the combined allele is matched once: {out}" + ); + let i = |p: &str| info.get(p).map_or("", String::as_str); + assert!( + i("30000038").contains("PGP_JOINED_FROM=chr22:30000037:AGC:A|chr22:30000042:T:TGGG"), + "the 2-star hit is the combined allele: {info:?}" + ); + assert!( + ["30000037", "30000042"] + .iter() + .all(|p| i(p).contains("PGP_PART_OF=chr22:30000038:GCCAT:CATGGG")), + "both 1-star parts are hits, marked part of the combined allele: {info:?}" + ); + let snv = got + .iter() + .find(|l| l.split('\t').nth(1) == Some("30000210")) + .expect("the SNV hit"); + assert!( + snv.split('\t').nth(2) == Some("9900004") && !snv.contains("PGP_"), + "the single SNV hit is unchanged (its ClinVar fields, no combined-allele tag): {snv}" + ); +} + +#[test] +fn a_split_multiallelic_call_is_not_het_and_every_listed_site_gets_a_call() { + let tract = ref_at(30000011, 5); + assert_eq!( + tract, "CAAAA", + "poly-A: the reference holds C and a run of A at chr22:30000011" + ); + + // Step 06's hits after `bcftools norm -m -any --multi-overlaps .`: the + // ClinVar allele of the 1/2 record (the 4-base deletion) is 1/., the + // other copy holds CAA. + let hits_vcf = format!( + "{HEAD}chr22\t30000011\t9900005\t{tract}\tC\t50\tPASS\tGENEINFO=SYNTH4:990004;CLNSIG=Pathogenic;\ +CLNREVSTAT=criteria_provided,_single_submitter\tGT:GQ:DP\t1/.:45:30\n\ +chr22\t30000210\t9900004\tC\tT\t50\tPASS\tGENEINFO=SYNTH2:990002;CLNSIG=Pathogenic;\ +CLNREVSTAT=criteria_provided,_single_submitter\tGT:GQ:DP\t0/1:40:30\n" + ); + let mut rows = Vec::new(); + hits::hits(Cursor::new(hits_vcf.as_bytes()), &mut rows).expect("og clinvar hits"); + let rows = String::from_utf8(rows).unwrap(); + let zyg: std::collections::HashMap<&str, &str> = rows + .lines() + .map(|l| { + let f: Vec<&str> = l.split('\t').collect(); + (f[2], f[5]) + }) + .collect(); + assert_eq!( + zyg.get("30000011").copied(), + Some("one copy, other copy not called ref (1/.)"), + "poly-A: the hit is not worded het: {rows}" + ); + assert_eq!( + zyg.get("30000210").copied(), + Some("het"), + "poly-A: the plain 0/1 SNV hit beside it is still het" + ); + + // The always-report list: four sites, and the calls as step 06's + // `bcftools query '%CHROM\t%POS\t%END\t%REF\t%ALT\t[%GT]\t[%GQ]\t[%DP]\n'` + // writes them from the split PASS VCF and the gVCF. + let list = format!( + "# four sites of the test\nchrom\tpos\tref\talt\tgene\thgvs\tclinvar_id\n\ +chr1\t100\tA\tG\tSYNTHA\tc.1A>G\t1\n\ +chr22\t30000011\t{tract}\tC\tSYNTH4\tc.2del\t9900005\n\ +chr22\t30000030\tA\tG\tSYNTHB\tc.3A>G\t2\n\ +chr22\t30000210\tC\tT\tSYNTH2\tc.4C>T\t9900004\n" + ); + let vcf_calls = format!( + "chr22\t30000011\t30000015\t{tract}\tC\t1/.\t45\t30\n\ +chr22\t30000011\t30000015\t{tract}\tCAA\t./1\t45\t30\n\ +chr22\t30000210\t30000210\tC\tT\t0/1\t40\t30\n" + ); + let gvcf_calls = format!( + "chr22\t30000011\t30000015\t{tract}\tC,CAA,<*>\t1/2\t45\t30\n\ +chr22\t30000016\t30000209\tA\t<*>\t0/0\t50\t30\n\ +chr22\t30000210\t30000210\tC\tT,<*>\t0/1\t40\t30\n" + ); + let o = sites::Options { + min_gq: "30".to_owned(), + min_gq_value: 30, + gvcf: true, + }; + let mut out = Vec::new(); + sites::deep_intronic( + list.as_bytes(), + vcf_calls.as_bytes(), + gvcf_calls.as_bytes(), + &o, + &mut out, + ) + .expect("og clinvar deep-intronic"); + let out = String::from_utf8(out).unwrap(); + let got: std::collections::BTreeMap<&str, (&str, &str)> = out + .lines() + .skip(1) + .map(|l| { + let f: Vec<&str> = l.split('\t').collect(); + (f[1], (f[7], f[8])) + }) + .collect(); + assert_eq!( + got.keys().copied().collect::>(), + ["100", "30000011", "30000030", "30000210"], + "always-report: one row per site, all four: {out}" + ); + assert_eq!( + got["30000210"], + ("het", "GQ 40, DP 30"), + "always-report: the carried SNV gives het with its GQ and DP" + ); + assert_eq!( + got["30000030"], + ("ref", "GQ 50 (gVCF block)"), + "always-report: a site inside a gVCF reference block gives ref with the block's GQ" + ); + assert_eq!( + got["30000011"], + ("one copy, other copy not called ref (1/.)", "GQ 45, DP 30"), + "always-report: the poly-A site gives the 1/. call, not het" + ); + assert_eq!( + got["100"], + ( + "no call", + "no record in the VCF or the gVCF covers the site" + ), + "always-report: a site no record covers gives no call, and why" + ); +} diff --git a/rust/crates/og-setup/tests/strchive_catalog.rs b/rust/crates/og-setup/tests/strchive_catalog.rs new file mode 100644 index 000000000..4370c3d1c --- /dev/null +++ b/rust/crates/og-setup/tests/strchive_catalog.rs @@ -0,0 +1,782 @@ +//! `og str catalog` on the two STRchive files `versions.env` pins, and +//! Stranger on the catalog it builds: what the old +//! `tests/test_str_catalog.py` checked, by meaning, in cargo test. +//! +//! 1. The files: downloaded once into a cache folder (`STRCHIVE_DIR`, else +//! `/pgp-strchive-`) and checked against their sha256; a +//! copy with one byte flipped fails that check. +//! 2. The converter: one entry per STRchive ExpansionHunter locus (82), no +//! name twice, every ExpansionHunter LocusId traced to one entry by its +//! STRchive id; the report's key loci keep their names; the thresholds +//! of HTT, FMR1 and SAMD12; the motif loci; the loci without thresholds +//! and why; and a planted ExpansionHunter locus with no STRchive record +//! stops it (exit 2, nothing written). +//! 3. Stranger (`STRANGER_IMAGE`, through docker) on the catalog: HTT 26, +//! 27 and 36 give normal, pre_mutation and full_mutation; SAMD12 1 and 2 +//! normal and pre_mutation; MUC1 no status; control: Stranger's bundled +//! GRCh38 catalog does not call HTT 27 pre_mutation. +//! 4. The repeats STRchive leaves out (`assets/str_extra_loci.json`, +//! `--extra`): merged after STRchive's 82 entries, copied as they are, +//! with ranges rows and no Stranger thresholds; each region is, in +//! GRCh38 (NCBI's GRCh38.p14 sequence), a whole run of its motif that +//! the bases beside it do not continue; controls: an extra catalog +//! without UGT1A1 is caught, four planted bad entries each stop the +//! converter, and a region shifted by one base fails the sequence check. +//! +//! The tests download (GitHub and NCBI) and run docker, so they run where +//! `CI` is set (every CI job) or with `OG_NET_TESTS=1`; elsewhere each says +//! it skipped. In CI a failed download or a missing docker fails the test. +//! What `og str motif` counts on the catalog's motifs is `tests/parity.rs` +//! of og-bam and the e2e case `str-catalog-1-strchive`. + +use std::collections::{BTreeSet, HashMap}; +use std::path::{Path, PathBuf}; +use std::process::Command; +use std::sync::OnceLock; + +use indexmap::IndexMap; +use pycompat::json::{self, Dump, Value}; + +fn repo() -> PathBuf { + Path::new(env!("CARGO_MANIFEST_DIR")).join("../../..") +} + +/// Whether the tests that download and run docker run here. +fn online(test: &str) -> bool { + let on = std::env::var_os("CI").is_some_and(|v| !v.is_empty()) + || std::env::var("OG_NET_TESTS").as_deref() == Ok("1"); + if !on { + eprintln!( + "skip: {test} downloads STRchive and runs docker; set OG_NET_TESTS=1 (CI always runs it)" + ); + } + on +} + +/// A value of versions.env. +fn env(name: &str) -> String { + let text = std::fs::read_to_string(repo().join("versions.env")).unwrap(); + let prefix = format!("{name}="); + let line = text + .lines() + .find(|l| l.starts_with(&prefix)) + .unwrap_or_else(|| panic!("no {name} line in versions.env")); + let v = line[prefix.len()..].split('#').next().unwrap().trim(); + v.trim_matches('"').to_owned() +} + +fn sha256(path: &Path) -> String { + for (cmd, args) in [("sha256sum", &[][..]), ("shasum", &["-a", "256"][..])] { + if let Ok(o) = Command::new(cmd).args(args).arg(path).output() + && o.status.success() + { + let out = String::from_utf8(o.stdout).unwrap(); + return out.split_whitespace().next().unwrap_or_default().to_owned(); + } + } + panic!("neither sha256sum nor shasum could hash {}", path.display()); +} + +/// GET `url` into `dest` with curl, three tries. +fn download(url: &str, dest: &Path) -> Result<(), String> { + let o = Command::new("curl") + .args([ + "-sSLf", + "--retry", + "3", + "--retry-delay", + "5", + "--max-time", + "120", + "-o", + ]) + .arg(dest) + .arg(url) + .output() + .map_err(|e| format!("curl: {e}"))?; + if o.status.success() { + Ok(()) + } else { + Err(format!( + "curl {url}: {}", + String::from_utf8_lossy(&o.stderr) + )) + } +} + +struct Files { + release: String, + eh: String, + loci: String, +} + +/// The release and the two pinned files, downloaded once and checked. +fn files() -> &'static Files { + static FILES: OnceLock = OnceLock::new(); + FILES.get_or_init(|| { + let rel = env("STRCHIVE_VERSION"); + let dir = std::env::var_os("STRCHIVE_DIR").map_or_else( + || std::env::temp_dir().join(format!("pgp-strchive-{rel}")), + PathBuf::from, + ); + std::fs::create_dir_all(&dir).unwrap(); + let mut out = Vec::new(); + for (name, var) in [ + ( + format!("STRchive-disease-loci-{rel}.hg38.expansionhunter.json"), + "STRCHIVE_EH_SHA256", + ), + (format!("STRchive-loci-{rel}.json"), "STRCHIVE_LOCI_SHA256"), + ] { + let path = dir.join(&name); + if !(path.exists() && sha256(&path) == env(var)) { + let part = dir.join(format!("{name}.part")); + let url = format!( + "https://github.com/dashnowlab/STRchive/releases/download/{rel}/{name}" + ); + download(&url, &part).unwrap(); + std::fs::rename(&part, &path).unwrap(); + } + assert_eq!( + sha256(&path), + env(var), + "{name} does not have the sha256 {var} pins" + ); + out.push(path.to_string_lossy().into_owned()); + } + Files { + release: rel, + eh: out[0].clone(), + loci: out[1].clone(), + } + }) +} + +/// A scratch folder of its own for one test, removed when dropped. +struct Scratch(PathBuf); + +impl Scratch { + fn new(name: &str) -> Scratch { + let d = std::env::temp_dir().join(format!("og-strchive-{name}-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&d); + std::fs::create_dir_all(&d).unwrap(); + Scratch(d) + } + + fn path(&self, name: &str) -> String { + self.0.join(name).to_string_lossy().into_owned() + } +} + +impl Drop for Scratch { + fn drop(&mut self) { + let _ = std::fs::remove_dir_all(&self.0); + } +} + +fn load(path: &str) -> Value { + json::loads(&std::fs::read_to_string(path).unwrap()).unwrap() +} + +fn dump(path: &str, v: &Value) { + std::fs::write(path, json::dumps(v, &Dump::default()).unwrap()).unwrap(); +} + +fn list(v: &Value) -> &[Value] { + match v { + Value::Array(a) => a, + other => panic!("not a list: {other:?}"), + } +} + +fn obj(v: &Value) -> &IndexMap { + match v { + Value::Object(o) => o, + other => panic!("not an object: {other:?}"), + } +} + +fn s(v: &Value) -> &str { + match v { + Value::Str(s) => s, + other => panic!("not text: {other:?}"), + } +} + +fn strs(v: Option<&Value>) -> Option> { + v.map(|v| list(v).iter().map(s).collect()) +} + +fn int(v: Option<&Value>) -> Option { + match v { + Some(Value::Int(i)) => Some(*i), + _ => None, + } +} + +/// `og str catalog ARGS` on the pinned files: the exit code. +fn catalog(extra_args: &[&str]) -> u8 { + let f = files(); + let mut args: Vec = ["--eh", &f.eh, "--loci", &f.loci, "--release", &f.release] + .iter() + .map(|a| (*a).to_owned()) + .collect(); + args.extend(extra_args.iter().map(|a| (*a).to_owned())); + og_setup::strchive::cli(&args) +} + +#[test] +fn a_flipped_byte_fails_the_checksum() { + if !online("a_flipped_byte_fails_the_checksum") { + return; + } + let w = Scratch::new("flip"); + let mut data = std::fs::read(&files().eh).unwrap(); + let mid = data.len() / 2; + data[mid] ^= 1; + let bad = w.0.join("flipped.json"); + std::fs::write(&bad, data).unwrap(); + assert_ne!( + sha256(&bad), + env("STRCHIVE_EH_SHA256"), + "control: a copy with one byte flipped fails the sha256 check" + ); +} + +#[test] +fn the_converter_on_the_pinned_files() { + if !online("the_converter_on_the_pinned_files") { + return; + } + let w = Scratch::new("convert"); + let (out, rng) = (w.path("catalog.json"), w.path("str_ranges.tsv")); + assert_eq!( + catalog(&["--out", &out, "--ranges", &rng]), + 0, + "the converter runs on the pinned files" + ); + let src_v = load(&files().eh); + let src = list(&src_v); + let cat_v = load(&out); + let cat: Vec<&IndexMap> = list(&cat_v).iter().map(obj).collect(); + assert_eq!( + src.len(), + 82, + "STRchive's ExpansionHunter catalog has 82 loci" + ); + assert_eq!( + cat.len(), + src.len(), + "the catalog has one entry per ExpansionHunter locus" + ); + + let mut by_src: HashMap<&str, Vec<&str>> = HashMap::new(); + for e in &cat { + by_src + .entry(s(&e["SourceId"])) + .or_default() + .push(s(&e["LocusId"])); + } + let src_ids: BTreeSet<&str> = src.iter().map(|x| s(&obj(x)["LocusId"])).collect(); + assert!( + by_src.keys().copied().collect::>() == src_ids + && by_src.values().all(|v| v.len() == 1), + "every ExpansionHunter LocusId has exactly one entry: {:?}", + by_src + .iter() + .filter(|(_, v)| v.len() != 1) + .collect::>() + ); + let names: Vec<&str> = cat + .iter() + .flat_map(|e| match e.get("VariantId") { + Some(Value::Array(a)) => a.iter().map(s).collect(), + _ => vec![s(&e["LocusId"])], + }) + .collect(); + let unique: BTreeSet<&str> = names.iter().copied().collect(); + assert_eq!(unique.len(), names.len(), "no variant name twice"); + + let by: HashMap<&str, &IndexMap> = + cat.iter().map(|e| (s(&e["LocusId"]), *e)).collect(); + for k in ["HTT", "FMR1", "C9ORF72", "ATXN1", "DMPK", "RFC1"] { + assert!( + by.contains_key(k), + "the report's key loci and RFC1 keep their names: {k}" + ); + } + for k in ["EIEE1_ARX", "PRTS_ARX", "HFG_HOXA13-I"] { + assert!( + by.contains_key(k), + "one gene with two loci keeps STRchive's ids: {k}" + ); + } + let htt = by["HTT"]; + assert_eq!( + ( + int(htt.get("NormalMax")), + int(htt.get("PathologicMin")), + strs(htt.get("VariantId")) + ), + (Some(26), Some(36), Some(vec!["HTT", "HTT_CCG"])), + "HTT: normal to 26, pathogenic from 36 (STRchive), the CAG repeat is REPID HTT" + ); + let fmr1 = by["FMR1"]; + assert_eq!( + (int(fmr1.get("NormalMax")), int(fmr1.get("PathologicMin"))), + (Some(44), Some(201)), + "FMR1: normal to 44, pathogenic from 201" + ); + let s12 = by["SAMD12"]; + assert_eq!( + ( + int(s12.get("NormalMax")), + int(s12.get("PathologicMin")), + strs(s12.get("BenignMotifs")), + strs(s12.get("PathogenicMotifs")), + strs(s12.get("VariantId")), + ), + ( + Some(1), + Some(105), + Some(vec!["TAAAA"]), + Some(vec!["TGAAA"]), + Some(vec!["SAMD12_TAAAA", "SAMD12"]) + ), + "SAMD12 (FAME1): no benign maximum in STRchive, so normal is the reference count (1 TGAAA), pathogenic \ + from 105; TTTTA benign, TTTCA pathogenic" + ); + let rfc1 = by["RFC1"]; + assert_eq!( + ( + strs(rfc1.get("BenignMotifs")), + strs(rfc1.get("PathogenicMotifs")) + ), + ( + Some(vec!["AAAAG", "AAAGG"]), + Some(vec!["AAGGG", "ACAGG", "CAGGG"]) + ), + "RFC1: AAAGG is counted benign, CAGGG pathogenic" + ); + let motif_loci: BTreeSet<&str> = cat + .iter() + .filter(|e| e.contains_key("PathogenicMotifs")) + .map(|e| s(&e["LocusId"])) + .collect(); + for k in [ + "RFC1", "SAMD12", "STARD7", "YEATS2", "MARCHF6", "RAPGEF2", "TNRC6A", "RAI1", "BEAN1", + "DAB1", + ] { + assert!( + motif_loci.contains(k), + "motif loci: RFC1, the FAME loci, SCA31 and SCA37: {k}" + ); + } + let none: BTreeSet<&str> = cat + .iter() + .filter(|e| e.contains_key("NoThresholds")) + .map(|e| s(&e["LocusId"])) + .collect(); + assert_eq!( + none.iter().copied().collect::>(), + ["CEL", "MIR7-2", "MUC1", "PLIN4", "POLG", "TYMS", "VWA1"], + "no thresholds: the four repeats longer than two reads, two without a pathogenic size, one contraction" + ); + assert!( + none.iter() + .all(|k| !by[k].contains_key("NormalMax") && !by[k].contains_key("PathologicMin")), + "a locus without thresholds has no NormalMax, so Stranger gives it no status" + ); + assert!( + cat.iter() + .filter(|e| e.contains_key("NormalMax")) + .all(|e| int(e.get("NormalMax")) < int(e.get("PathologicMin"))), + "every threshold pair is normal below pathogenic" + ); + assert_eq!( + htt.get("SourceDisplay").map(s), + Some(format!("STRchive-{}", files().release).as_str()), + "the release is in SourceDisplay" + ); + + // Control: a planted ExpansionHunter locus with no STRchive record. + let mut planted = src.to_vec(); + planted.push(json::loads(r#"{"LocusId": "PLANTED_GENE", "LocusStructure": "(CAG)*", "ReferenceRegion": "chr1:1000-1030", "VariantType": "Repeat"}"#).unwrap()); + let eh = w.path("planted_eh.json"); + dump(&eh, &Value::Array(planted)); + let (o2, r2) = (w.path("out.json"), w.path("ranges.tsv")); + let f = files(); + let rc = og_setup::strchive::cli( + &[ + "--eh", + &eh, + "--loci", + &f.loci, + "--release", + &f.release, + "--out", + &o2, + "--ranges", + &r2, + ] + .map(str::to_owned), + ); + assert!( + rc == 2 && !Path::new(&o2).exists() && !Path::new(&r2).exists(), + "control: a planted ExpansionHunter locus with no STRchive record stops the converter (exit 2, no catalog \ + and no ranges table): exit {rc}" + ); +} + +// --- Stranger on the catalog ------------------------------------------------------------- + +const FMT: &str = "GT:SO:REPCN:REPCI:ADSP:ADFL:ADIR:LC"; + +fn id(flag: &str) -> String { + let o = Command::new("id").arg(flag).output().unwrap(); + String::from_utf8(o.stdout).unwrap().trim().to_owned() +} + +/// STR_STATUS per record id from STRANGER_IMAGE on a VCF of (id, chrom, +/// pos, end, count) records; `catalog` None is the GRCh38 catalog Stranger +/// bundles, as step 9b passes it. +fn stranger( + dir: &Path, + catalog: Option<&str>, + records: &[(&str, &str, u64, u64, u32)], +) -> HashMap { + let smoke = std::fs::read_to_string(repo().join("tests/smoke/stranger_input.vcf.in")).unwrap(); + let mut vcf: String = smoke + .lines() + .filter(|l| l.starts_with("##")) + .map(|l| l.to_owned() + "\n") + .collect(); + let alts: BTreeSet = records.iter().map(|r| r.4).collect(); + for a in alts { + vcf += &format!("##ALT=\n"); + } + let chroms: BTreeSet<&str> = records.iter().map(|r| r.1).collect(); + for c in chroms { + vcf += &format!("##contig=\n"); + } + vcf += "#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT\tS\n"; + for (i, (rid, chrom, pos, end, n)) in records.iter().enumerate() { + vcf += &format!( + "{chrom}\t{pos}\t{rid}.{i}\tA\t\t.\tPASS\tSVTYPE=STR;END={end};REF=1;RL=3;RU=CAG;VARID={rid};REPID={rid}\t\ +{FMT}\t1/1:SPANNING/SPANNING:{n}/{n}:{n}-{n}/{n}-{n}:20/20:0/0:0/0:30.0\n" + ); + } + std::fs::write(dir.join("in.vcf"), vcf).unwrap(); + let mut cmd = Command::new("docker"); + cmd.args([ + "run", + "--rm", + "--network", + "none", + "-u", + &format!("{}:{}", id("-u"), id("-g")), + "-v", + ]) + .arg(format!("{}:/w", dir.display())) + .arg(env("STRANGER_IMAGE")); + match catalog { + Some(c) => cmd.args(["stranger", "--repeats-file", &format!("/w/{c}"), "/w/in.vcf"]), + None => cmd.args([ + "sh", + "-c", + "exec stranger --repeats-file \"$(python -c 'import os, stranger; print(os.path.join(\ + os.path.dirname(stranger.__file__), \"resources\", \"variant_catalog_grch38.json\"))')\" /w/in.vcf", + ]), + }; + let o = cmd.output().unwrap_or_else(|e| panic!("docker: {e}")); + assert!( + o.status.success(), + "Stranger runs: {}", + String::from_utf8_lossy(&o.stderr) + ); + String::from_utf8(o.stdout) + .unwrap() + .lines() + .filter(|l| !l.starts_with('#')) + .map(|l| { + let f: Vec<&str> = l.split('\t').collect(); + let status = f[7] + .split(';') + .find_map(|kv| kv.strip_prefix("STR_STATUS=")) + .unwrap_or_default(); + (f[2].to_owned(), status.to_owned()) + }) + .collect() +} + +#[test] +fn stranger_grades_with_the_catalog() { + if !online("stranger_grades_with_the_catalog") { + return; + } + let w = Scratch::new("stranger"); + assert_eq!( + catalog(&["--out", &w.path("catalog.json")]), + 0, + "the converter runs on the pinned files" + ); + // The container runs as the caller and reads the folder. + assert!( + Command::new("chmod") + .args(["755"]) + .arg(&w.0) + .status() + .unwrap() + .success() + ); + let recs = [ + ("HTT", "chr4", 3074877, 3074933, 26), + ("HTT", "chr4", 3074877, 3074933, 27), + ("HTT", "chr4", 3074877, 3074933, 36), + ("SAMD12", "chr8", 118366913, 118366918, 1), + ("SAMD12", "chr8", 118366913, 118366918, 2), + ("MUC1", "chr1", 155188506, 155192239, 60), + ]; + let got = stranger(&w.0, Some("catalog.json"), &recs); + let want: HashMap = [ + ("HTT.0", "normal"), + ("HTT.1", "pre_mutation"), + ("HTT.2", "full_mutation"), + ("SAMD12.3", "normal"), + ("SAMD12.4", "pre_mutation"), + ("MUC1.5", ""), + ] + .iter() + .map(|(k, v)| ((*k).to_owned(), (*v).to_owned())) + .collect(); + assert_eq!( + got, want, + "Stranger on the catalog: HTT 26 normal, 27 pre_mutation, 36 full_mutation; SAMD12 1 normal, 2 \ + pre_mutation; MUC1 no status" + ); + let bundled = stranger(&w.0, None, &recs[..3]); + assert_ne!( + bundled.get("HTT.1").map(String::as_str), + Some("pre_mutation"), + "control: Stranger's bundled GRCh38 catalog grades HTT 27 otherwise: {bundled:?}" + ); +} + +// --- the extra repeats ------------------------------------------------------------------- + +/// GRCh38 bases [start, end) (0-based) of chrom, from NCBI (GRCh38.p14, +/// as the no-ALT set's chr2 and chr7). +fn grch38(chrom: &str, start: u64, end: u64) -> String { + let acc = match chrom { + "chr2" => "NC_000002.12", + "chr7" => "NC_000007.14", + other => panic!("no accession for {other}"), + }; + let url = format!( + "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?db=nuccore&rettype=fasta&id={acc}&seq_start={}&seq_stop={end}", + start + 1 + ); + let w = Scratch::new(&format!("ncbi-{chrom}-{start}")); + let dest = w.0.join("seq.fa"); + download(&url, &dest).unwrap(); + std::fs::read_to_string(&dest) + .unwrap() + .lines() + .skip(1) + .collect::() + .to_uppercase() +} + +fn region(text: &str) -> (&str, u64, u64) { + let (chrom, span) = text.split_once(':').unwrap(); + let (a, b) = span.split_once('-').unwrap(); + (chrom, a.parse().unwrap(), b.parse().unwrap()) +} + +/// Why each region of an extra entry is not a whole run of its motif in +/// GRCh38 (empty when it is). +fn region_errors(entry: &IndexMap, shift: u64) -> Vec { + let regions: Vec<&str> = match &entry["ReferenceRegion"] { + Value::Array(a) => a.iter().map(s).collect(), + v => vec![s(v)], + }; + let motifs = og_setup::strchive::motifs_of(s(&entry["LocusStructure"])); + let mut errs = Vec::new(); + for (m, r) in motifs.iter().zip(regions) { + let (chrom, a, b) = region(r); + let (a, b) = (a + shift, b + shift); + let k = m.len(); + let seq = grch38(chrom, a - k as u64, b + k as u64); + let (left, inner, right) = (&seq[..k], &seq[k..seq.len() - k], &seq[seq.len() - k..]); + if inner != m.repeat(((b - a) as usize) / k) { + errs.push(format!("{r}: GRCh38 has {inner}, not {m} repeated")); + } else if left == *m || right == *m { + errs.push(format!( + "{r}: the {m} run goes on past the region ({left}|...|{right})" + )); + } + } + errs +} + +#[test] +fn the_extra_repeats() { + if !online("the_extra_repeats") { + return; + } + let w = Scratch::new("extra"); + let extra_path = repo() + .join("assets/str_extra_loci.json") + .to_string_lossy() + .into_owned(); + let extra_v = load(&extra_path); + let extra = list(&extra_v); + let shape: Vec<(&str, &str)> = extra + .iter() + .map(|e| (s(&obj(e)["LocusId"]), s(&obj(e)["LocusStructure"]))) + .collect(); + assert_eq!( + shape, + [("CFTR", "(TG)*(T)*"), ("UGT1A1", "(TA)*")], + "the extra catalog names CFTR (TG and T) and UGT1A1 (TA)" + ); + let missing = |cat: &Value| -> Vec { + let have: BTreeSet<&str> = list(cat).iter().map(|e| s(&obj(e)["LocusId"])).collect(); + extra + .iter() + .map(|e| s(&obj(e)["LocusId"])) + .filter(|i| !have.contains(i)) + .map(str::to_owned) + .collect() + }; + + let (out, rng) = (w.path("catalog.json"), w.path("ranges.tsv")); + let rc = catalog(&["--out", &out, "--ranges", &rng, "--extra", &extra_path]); + assert_eq!(rc, 0, "--extra: the converter runs"); + let cat_v = load(&out); + let cat = list(&cat_v); + assert_eq!( + cat.len(), + 82 + extra.len(), + "--extra: STRchive's 82 entries and the extra loci" + ); + assert!( + missing(&cat_v).is_empty(), + "every extra locus is in the merged catalog" + ); + let tail = &cat[cat.len() - extra.len()..]; + assert_eq!(tail, extra, "the extra loci are copied as they are, last"); + assert!( + tail.iter().map(obj).all(|e| !e.contains_key("NormalMax") + && !e.contains_key("PathologicMin") + && e.get("NoThresholds").is_some_and(|v| !s(v).is_empty())), + "no extra locus has Stranger thresholds, each says why" + ); + let ranges = std::fs::read_to_string(&rng).unwrap(); + let rows: Vec> = ranges + .lines() + .skip(2) + .map(|l| l.split('\t').collect()) + .collect(); + let want_tail: Vec> = extra + .iter() + .map(|e| { + vec![ + s(&obj(e)["LocusId"]), + s(&obj(e)["SourceId"]), + "", + "", + "", + "", + ] + }) + .collect(); + let got_tail: Vec> = rows[rows.len() - extra.len()..] + .iter() + .map(|r| r[..6].to_vec()) + .collect(); + assert!( + rows.len() == cat.len() && got_tail == want_tail, + "the ranges table has a row per entry, the extra ones without STRchive ranges: {got_tail:?}" + ); + + // Control: an extra catalog without UGT1A1 gives a merged catalog the + // missing-locus check flags. + let short = w.path("short.json"); + dump( + &short, + &Value::Array( + extra + .iter() + .filter(|e| s(&obj(e)["LocusId"]) != "UGT1A1") + .cloned() + .collect(), + ), + ); + catalog(&["--out", &out, "--extra", &short]); + assert_eq!( + missing(&load(&out)), + ["UGT1A1"], + "control: an extra catalog without UGT1A1 gives a merged catalog the check flags" + ); + + // Controls: four planted bad entries each stop the converter. + let cftr = obj(&extra[0]).clone(); + let with = |k: &str, v: Value| { + let mut e = cftr.clone(); + e.insert(k.to_owned(), v); + Value::Object(e) + }; + let planted = [ + ( + "no NoThresholds", + with("NoThresholds", Value::Str(String::new())), + ), + ( + "a region that is not a whole number of motifs", + with( + "ReferenceRegion", + Value::Array(vec![ + Value::Str("chr7:117548606-117548627".into()), + Value::Str("chr7:117548627-117548635".into()), + ]), + ), + ), + ( + "a STRchive locus name", + with("LocusId", Value::Str("HTT".into())), + ), + ("Stranger thresholds", { + let mut e = cftr.clone(); + e.insert("NormalMax".into(), Value::Int(11)); + e.insert("PathologicMin".into(), Value::Int(13)); + Value::Object(e) + }), + ]; + for (desc, bad) in planted { + let (p, o, g) = ( + w.path("bad.json"), + w.path("bad_out.json"), + w.path("bad_ranges.tsv"), + ); + dump(&p, &Value::Array(vec![bad])); + let rc = catalog(&["--out", &o, "--ranges", &g, "--extra", &p]); + assert!( + rc == 2 && !Path::new(&o).exists() && !Path::new(&g).exists(), + "control: an extra entry with {desc} stops the converter (exit 2, nothing written): exit {rc}" + ); + } + + for e in extra.iter().map(obj) { + let errs = region_errors(e, 0); + assert!( + errs.is_empty(), + "{}: every region is a whole run of its motif in GRCh38: {errs:?}", + s(&e["LocusId"]) + ); + } + assert!( + !region_errors(obj(&extra[1]), 1).is_empty(), + "control: the UGT1A1 region shifted by one base fails the sequence check" + ); +} diff --git a/rust/crates/og-steps/tests/sv_inputs.rs b/rust/crates/og-steps/tests/sv_inputs.rs new file mode 100644 index 000000000..0333a2bf0 --- /dev/null +++ b/rust/crates/og-steps/tests/sv_inputs.rs @@ -0,0 +1,840 @@ +//! `og sv inv-filter`, `og sv svlen-fix` and `og sv add-callers` on made-up +//! VCFs: what the old `tests/test_annotsv_sv_inputs.py` checked, by +//! meaning, in cargo test. +//! +//! 1. inv-filter, on Manta's file and a `convertInversion.py`-style +//! converted file: a reciprocal pair and a pair that shares one Manta +//! EVENT keep their INV records (INV_JUNCTIONS=2); a single 50 Mb INV3 +//! junction gives back its two breakends (INV_JUNCTIONS=1); a single +//! 100 kb junction stays an INV, a single 400 bp one also gets +//! INVDUP_CANDIDATE; the deletion passes unchanged; the counts. Control: +//! the converted file holds that 50 Mb INV. +//! 2. svlen-fix: REF 137 bp / ALT 2 bp gets SVLEN -135 and END POS+134 with +//! the old values kept (control: the input says -136); a pure deletion +//! keeps SVLEN and gets END POS+135; a net insertion gets its net SVLEN +//! and keeps END; symbolic, breakend and multi-ALT records pass +//! unchanged; a second run changes nothing; two sample columns stop it. +//! 3. add-callers: which Delly and CNVpytor calls are added beside Manta's, +//! no-call records that must not hide another caller's call, and the +//! control that Manta's file alone lacks the CNVpytor deletion. +//! 4. add-callers and Delly's read depth (FORMAT RDCN), with and without +//! the sex and the PAR file; control: every planted call is PASS DEL or +//! DUP, so without the depth check each would be added. + +use std::collections::{BTreeSet, HashMap}; +use std::io::Write; +use std::path::{Path, PathBuf}; + +use og_filters::TextLines; +use og_filters::inv::{self, Bounds}; +use og_filters::svlen::{self, SvlenError}; +use og_steps::sv::Sex; +use og_steps::sv::add_callers::{self, Opts}; + +fn repo() -> PathBuf { + Path::new(env!("CARGO_MANIFEST_DIR")).join("../../..") +} + +/// A scratch folder of its own for one test, removed when dropped. +struct Scratch(PathBuf); + +impl Scratch { + fn new(name: &str) -> Scratch { + let d = std::env::temp_dir().join(format!("og-sv-inputs-{name}-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&d); + std::fs::create_dir_all(&d).unwrap(); + Scratch(d) + } + + fn put(&self, name: &str, text: &str) -> String { + let p = self.0.join(name); + std::fs::write(&p, text).unwrap(); + p.to_string_lossy().into_owned() + } + + fn put_gz(&self, name: &str, text: &str) -> String { + let p = self.0.join(name); + let mut gz = flate2::write::GzEncoder::new( + std::fs::File::create(&p).unwrap(), + flate2::Compression::default(), + ); + gz.write_all(text.as_bytes()).unwrap(); + gz.finish().unwrap(); + p.to_string_lossy().into_owned() + } +} + +impl Drop for Scratch { + fn drop(&mut self) { + let _ = std::fs::remove_dir_all(&self.0); + } +} + +fn records(text: &str) -> Vec> { + text.lines() + .filter(|l| !l.is_empty() && !l.starts_with('#')) + .map(|l| l.split('\t').map(str::to_owned).collect()) + .collect() +} + +/// INFO as a map; a flag maps to `None`. +fn info(rec: &[String]) -> HashMap> { + rec[7] + .split(';') + .map(|item| match item.split_once('=') { + Some((k, v)) => (k.to_owned(), Some(v.to_owned())), + None => (item.to_owned(), None), + }) + .collect() +} + +fn info_of(rec: &[String], key: &str) -> Option { + info(rec).get(key).cloned().flatten() +} + +fn by_id(recs: &[Vec]) -> HashMap> { + recs.iter().map(|r| (r[2].clone(), r.clone())).collect() +} + +// --- 1. og sv inv-filter ----------------------------------------------------------------- + +const HEAD: &str = "##fileformat=VCFv4.1\n\ +##contig=\n\ +##contig=\n\ +##INFO=\n\ +##INFO=\n\ +##FORMAT=\n\ +#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT\ts1\n"; + +fn bnd(chrom: &str, pos: u64, rid: &str, alt: &str, mate: &str) -> String { + format!("{chrom}\t{pos}\t{rid}\tN\t{alt}\t500\tPASS\tSVTYPE=BND;MATEID={mate}\tGT\t0/1\n") +} + +fn inv(chrom: &str, pos: u64, rid: &str, end: u64, side: u8, extra: &str) -> String { + format!( + "{chrom}\t{pos}\t{rid}\tN\t\t500\tPASS\tEND={end};SVTYPE=INV;SVLEN={}{extra};INV{side}\tGT\t0/1\n", + end - pos + ) +} + +const DEL: &str = "chr20\t40000000\tMantaDEL:9:0:0:0:0:0\tN\t\t500\tPASS\tEND=40010000;SVTYPE=DEL;SVLEN=-10000\tGT\t0/1\n"; + +/// Manta's own file: each junction a pair of breakends. +fn raw() -> String { + [ + // reciprocal inversion chr20:1,000,000-2,000,000: INV3 and INV5 junctions + bnd( + "chr20", + 1000000, + "MantaBND:10:0:1:0:0:0:0", + "N]chr20:2000000]", + "MantaBND:10:0:1:0:0:0:1", + ), + bnd( + "chr20", + 1000101, + "MantaBND:10:0:2:0:0:0:0", + "[chr20:2000101[N", + "MantaBND:10:0:2:0:0:0:1", + ), + bnd( + "chr20", + 2000000, + "MantaBND:10:0:1:0:0:0:1", + "N]chr20:1000000]", + "MantaBND:10:0:1:0:0:0:0", + ), + bnd( + "chr20", + 2000101, + "MantaBND:10:0:2:0:0:0:1", + "[chr20:1000101[N", + "MantaBND:10:0:2:0:0:0:0", + ), + // one INV3 junction, 50 Mb + bnd( + "chr20", + 5000000, + "MantaBND:20:0:1:0:0:0:0", + "N]chr20:55000000]", + "MantaBND:20:0:1:0:0:0:1", + ), + bnd( + "chr20", + 55000000, + "MantaBND:20:0:1:0:0:0:1", + "N]chr20:5000000]", + "MantaBND:20:0:1:0:0:0:0", + ), + // one INV5 junction, 400 bp + bnd( + "chr20", + 30000000, + "MantaBND:30:0:1:0:0:0:0", + "[chr20:30000400[N", + "MantaBND:30:0:1:0:0:0:1", + ), + bnd( + "chr20", + 30000400, + "MantaBND:30:0:1:0:0:0:1", + "[chr20:30000000[N", + "MantaBND:30:0:1:0:0:0:0", + ), + // one INV3 junction, 100 kb + bnd( + "chr1", + 3000000, + "MantaBND:40:0:1:0:0:0:0", + "N]chr1:3100000]", + "MantaBND:40:0:1:0:0:0:1", + ), + bnd( + "chr1", + 3100000, + "MantaBND:40:0:1:0:0:0:1", + "N]chr1:3000000]", + "MantaBND:40:0:1:0:0:0:0", + ), + DEL.to_owned(), + ] + .iter() + .fold(HEAD.to_owned(), |a, l| a + l) +} + +/// What `convertInversion.py` writes for it (an INV5 record moves one base +/// left). +fn converted() -> String { + let ev = ";EVENT=MantaINV:50:0:1:0:0:0"; + [ + inv("chr1", 3000000, "MantaINV:40:0:1:0:0:0", 3100000, 3, ""), + // two junctions 3 kb apart that Manta put in one EVENT + inv("chr1", 10000000, "MantaINV:50:0:1:0:0:0", 10500000, 3, ev), + inv("chr1", 10003000, "MantaINV:50:0:2:0:0:0", 10503000, 5, ev), + inv("chr20", 1000000, "MantaINV:10:0:1:0:0:0", 2000000, 3, ""), + inv("chr20", 1000100, "MantaINV:10:0:2:0:0:0", 2000100, 5, ""), + inv("chr20", 5000000, "MantaINV:20:0:1:0:0:0", 55000000, 3, ""), + inv("chr20", 29999999, "MantaINV:30:0:1:0:0:0", 30000399, 5, ""), + DEL.to_owned(), + ] + .iter() + .fold(HEAD.to_owned(), |a, l| a + l) +} + +fn inv_filter(raw: &str, conv: &str, b: Bounds) -> (String, inv::Counts) { + let mut out = Vec::new(); + let c = inv::run(raw, conv, b, &mut out).unwrap_or_else(|e| panic!("og sv inv-filter: {e}")); + (String::from_utf8(out).unwrap(), c) +} + +#[test] +fn inv_filter_keeps_paired_inversions_and_returns_long_single_junctions() { + let w = Scratch::new("inv"); + let raw_path = w.put_gz("raw.vcf.gz", &raw()); + let conv = converted(); + let conv_path = w.put("conv.vcf", &conv); + + let long_inv = records(&conv) + .into_iter() + .filter(|r| { + info_of(r, "SVTYPE").as_deref() == Some("INV") + && info_of(r, "END").unwrap().parse::().unwrap() - r[1].parse::().unwrap() + >= 1_000_000 + && r[2] == "MantaINV:20:0:1:0:0:0" + }) + .count(); + assert_eq!( + long_inv, 1, + "control: the converted file holds the 50 Mb single-junction INV" + ); + + let (out, counts) = inv_filter(&raw_path, &conv_path, Bounds::default()); + let recs = records(&out); + let by = by_id(&recs); + for rid in [ + "MantaINV:10:0:1:0:0:0", + "MantaINV:10:0:2:0:0:0", + "MantaINV:50:0:1:0:0:0", + "MantaINV:50:0:2:0:0:0", + ] { + assert_eq!( + by.get(rid) + .and_then(|r| info_of(r, "INV_JUNCTIONS")) + .as_deref(), + Some("2"), + "both junctions present: {rid} keeps its INV, INV_JUNCTIONS=2" + ); + } + assert!( + !by.contains_key("MantaINV:20:0:1:0:0:0"), + "the 50 Mb single junction is no INV record any more" + ); + let mb_invs: Vec<&String> = recs + .iter() + .filter(|r| { + info_of(r, "SVTYPE").as_deref() == Some("INV") + && info_of(r, "END").unwrap().parse::().unwrap() - r[1].parse::().unwrap() + > 1_000_000 + }) + .map(|r| &r[2]) + .collect(); + assert!( + mb_invs.is_empty(), + "no INV record longer than the largest paired inversion (1 Mb): {mb_invs:?}" + ); + for rid in ["MantaBND:20:0:1:0:0:0:0", "MantaBND:20:0:1:0:0:0:1"] { + let r = by + .get(rid) + .unwrap_or_else(|| panic!("its breakend {rid} is back")); + assert_eq!( + ( + info_of(r, "SVTYPE").as_deref(), + info_of(r, "INV_JUNCTIONS").as_deref() + ), + (Some("BND"), Some("1")), + "its breakend {rid} is back, INV_JUNCTIONS=1" + ); + } + let r = &by["MantaINV:40:0:1:0:0:0"]; + assert!( + info_of(r, "INV_JUNCTIONS").as_deref() == Some("1") + && !info(r).contains_key("INVDUP_CANDIDATE"), + "the 100 kb single junction stays an INV, INV_JUNCTIONS=1, no INVDUP_CANDIDATE: {r:?}" + ); + let r = &by["MantaINV:30:0:1:0:0:0"]; + assert!( + info_of(r, "INV_JUNCTIONS").as_deref() == Some("1") + && info(r).get("INVDUP_CANDIDATE") == Some(&None), + "the 400 bp single junction gets INVDUP_CANDIDATE: {r:?}" + ); + let del: Vec = DEL.trim_end().split('\t').map(str::to_owned).collect(); + assert!(recs.contains(&del), "the deletion passes unchanged"); + let order: Vec<(usize, u64)> = recs + .iter() + .map(|r| (usize::from(r[0] == "chr20"), r[1].parse().unwrap())) + .collect(); + let mut sorted = order.clone(); + sorted.sort_unstable(); + assert_eq!( + order, sorted, + "records sorted by the ##contig order, then POS" + ); + assert!( + out.contains("##INFO=\n\ +#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT\ts1\n"; + +fn svlen_fix(path: &str) -> Result<(String, usize), SvlenError> { + let mut out = Vec::new(); + let input = og_io::open(path).expect("open"); + let n = svlen::run(TextLines::file(input), &mut out)?; + Ok((String::from_utf8(out).unwrap(), n)) +} + +#[test] +fn svlen_fix_sets_the_net_length_change() { + let w = Scratch::new("svlen"); + let (ref137, alt2) = ("A".repeat(137), "AC".to_owned()); + let ins_alt = format!("A{}", "T".repeat(59)); + let rows = [ + ( + "1005400", + "complex", + ref137.as_str(), + alt2.as_str(), + "END=1005536;SVTYPE=DEL;SVLEN=-136", + ), + ( + "1100000", + "pure", + ref137.as_str(), + "A", + "END=1100136;SVTYPE=DEL;SVLEN=-136", + ), + ( + "1200000", + "ins", + "ACG", + ins_alt.as_str(), + "END=1200002;SVTYPE=INS;SVLEN=59", + ), + ( + "1300000", + "sym", + "N", + "", + "END=1310000;SVTYPE=DEL;SVLEN=-10000", + ), + ("1400000", "bnd", "N", "N]chr20:1500000]", "SVTYPE=BND"), + ( + "1600000", + "multi", + ref137.as_str(), + "A,AC", + "END=1600136;SVTYPE=DEL;SVLEN=-136", + ), + ]; + let vcf = rows + .iter() + .fold(SV.to_owned(), |a, (pos, id, r, alt, inf)| { + a + &format!("chr20\t{pos}\t{id}\t{r}\t{alt}\t50\tPASS\t{inf}\tGT\t0/1\n") + }); + let src = w.put("sv.vcf", &vcf); + let input = by_id(&records(&vcf)); + assert_eq!( + info_of(&input["complex"], "SVLEN").as_deref(), + Some("-136"), + "control: the input gives the REF 137 / ALT 2 record SVLEN -136" + ); + + let (out, n) = svlen_fix(&src).expect("og sv svlen-fix"); + let by = by_id(&records(&out)); + let c = info(&by["complex"]); + let get = |m: &HashMap>, k: &str| m.get(k).cloned().flatten(); + assert_eq!( + (get(&c, "SVLEN").as_deref(), get(&c, "END").as_deref()), + (Some("-135"), Some("1005534")), + "REF 137 / ALT 2: SVLEN -135, END POS+134 (an interval of 135 bp for AnnotSV)" + ); + assert_eq!( + ( + get(&c, "ORIG_SVLEN").as_deref(), + get(&c, "ORIG_END").as_deref() + ), + (Some("-136"), Some("1005536")), + "REF 137 / ALT 2: the old values kept" + ); + let p = info(&by["pure"]); + assert!( + get(&p, "SVLEN").as_deref() == Some("-136") + && get(&p, "END").as_deref() == Some("1100135") + && !p.contains_key("ORIG_SVLEN"), + "pure deletion: SVLEN stays -136, END becomes POS+135: {p:?}" + ); + let i = info(&by["ins"]); + assert_eq!( + (get(&i, "SVLEN").as_deref(), get(&i, "END").as_deref()), + (Some("57"), Some("1200002")), + "net insertion: SVLEN 57, END kept" + ); + for rid in ["sym", "bnd", "multi"] { + assert_eq!(by[rid], input[rid], "{rid} passes unchanged"); + } + assert!( + out.contains("##INFO=\t50\tPASS\tEND=1310000;SVTYPE=DEL\tGT\t0/0\t0/1\n"), + ); + match svlen_fix(&two) { + Err(e @ SvlenError::MultiSample(_)) => assert!( + e.to_string().contains("holds 2 samples (s1, s2)"), + "a VCF with two sample columns stops and names them: {e}" + ), + other => panic!("a VCF with two sample columns must stop (exit 2), not {other:?}"), + } +} + +// --- 3. og sv add-callers ---------------------------------------------------------------- + +const BASE: &str = "##fileformat=VCFv4.1\n##contig=\n##contig=\n\ +##INFO=\n\ +##INFO=\n\ +##FORMAT=\n\ +#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT\ts1\n\ +chr20\t1000000\tMantaDEL:1\tN\t\t500\tPASS\tEND=1100000;SVTYPE=DEL;SVLEN=-100000\tGT\t0/1\n\ +chr20\t3000000\tMantaINV:2\tN\t\t500\tPASS\tEND=4000000;SVTYPE=INV;SVLEN=1000000;INV3\tGT\t0/1\n"; + +const DELLY: &str = "##fileformat=VCFv4.2\n#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT\ts1\n\ +chr20\t1000500\tDEL00000001\tN\t\t60\tPASS\tIMPRECISE;SVTYPE=DEL;END=1099000;CIPOS=-50,50;CIEND=-50,50\tGT:GQ\t0/1:40\n\ +chr20\t2000000\tDEL00000002\tN\t\t60\tPASS\tPRECISE;SVTYPE=DEL;END=2050000\tGT:GQ\t1/1:40\n\ +chr20\t2500000\tDEL00000003\tN\t\t10\tLowQual\tIMPRECISE;SVTYPE=DEL;END=2600000\tGT\t0/1\n\ +chr20\t3000000\tINV00000004\tN\t\t60\tPASS\tIMPRECISE;SVTYPE=INV;END=4000000;CT=3to3\tGT\t0/1\n"; + +/// One CNVpytor record: its fields as CNVpytor writes them. +#[allow(clippy::too_many_arguments)] +fn cnv( + chrom: &str, + pos: u64, + rid: &str, + alt: &str, + end: u64, + p1: &str, + q0: &str, + gt: &str, +) -> String { + let t = &alt[1..4]; + format!( + "chr{chrom}\t{pos}\t{rid}\tN\t{alt}\t.\tPASS\tEND={end};IMPRECISE;SVLEN={};SVTYPE={t};pytorRD=0.5;pytorP1={p1};\ +pytorP2=0;pytorP3=0;pytorP4=0;pytorQ0={q0};pytorPN=0;pytorDG=1000;pytorCL=rd_mean_shift\tGT:CN\t{gt}:1\n", + end - pos + ) +} + +fn cnvpytor() -> String { + "##fileformat=VCFv4.2\n##source=CNVpytor\n#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT\ts1\n".to_owned() + + &cnv("20", 2001000, "CNVpytor_del0", "", 2049000, "1e-06", "0.01", "1/1") // Delly's added call has it + + &cnv("20", 3200000, "CNVpytor_del1", "", 3700000, "1e-08", "0.02", "0/1") // inside Manta's INV: added + + &cnv("20", 5000000, "CNVpytor_del2", "", 5100000, "0.5", "0.0", "0/1") // e-val too high + + &cnv("20", 6000000, "CNVpytor_del3", "", 6100000, "1e-08", "0.9", "0/1") // q0 too high + + &cnv("20", 7000000, "CNVpytor_loh4", "", 7100000, "1e-08", "0.0", "0/1") // LOH + + &cnv("1", 500000, "CNVpytor_dup5", "", 600000, "1e-05", "0.0", "0/1") // added +} + +fn opts(base: &str) -> Opts { + Opts { + base: base.to_owned(), + base_name: "manta".to_owned(), + delly: None, + sex: Sex::Unknown, + par_bed: None, + cnvpytor: None, + max_eval: 0.01, + max_q0: 0.5, + min_overlap: 0.5, + } +} + +/// `og sv add-callers`: the VCF it writes and the note it prints. +fn add(o: &Opts) -> (String, String) { + let mut out = Vec::new(); + let note = add_callers::run(o, &mut out).unwrap_or_else(|e| panic!("og sv add-callers: {e}")); + (String::from_utf8(out).unwrap(), note) +} + +fn ids(vcf: &str) -> Vec { + records(vcf).into_iter().map(|r| r[2].clone()).collect() +} + +#[test] +fn add_callers_adds_what_manta_lacks() { + let w = Scratch::new("callers"); + let base = w.put("base.vcf", BASE); + let delly = w.put_gz("delly.vcf.gz", DELLY); + let cnvp = w.put_gz("cnv.vcf.gz", &cnvpytor()); + assert!( + !BASE.contains("CNVpytor_del1"), + "control: Manta's file alone lacks the CNVpytor deletion" + ); + + let all = Opts { + delly: Some(delly.clone()), + cnvpytor: Some(cnvp.clone()), + ..opts(&base) + }; + let (out, note) = add(&all); + assert_eq!( + ids(&out), + [ + "CNVpytor_dup5", + "MantaDEL:1", + "DEL00000002", + "MantaINV:2", + "CNVpytor_del1" + ], + "the added records are exactly Delly's DEL00000002 and CNVpytor's del1 and dup5, sorted" + ); + let by = by_id(&records(&out)); + assert!( + ["MantaDEL:1", "MantaINV:2"] + .iter() + .all(|i| info_of(&by[*i], "SVCALLER").as_deref() == Some("manta")), + "Manta's records name their caller" + ); + let d = &by["DEL00000002"]; + assert!( + info_of(d, "SVCALLER").as_deref() == Some("delly") + && d[4] == "" + && info_of(d, "END").as_deref() == Some("2050000") + && d[8..] == ["GT", "1/1"], + "Delly's call: SVCALLER=delly, , END kept, its genotype: {d:?}" + ); + let c = &by["CNVpytor_del1"]; + assert!( + info_of(c, "SVCALLER").as_deref() == Some("cnvpytor") + && info_of(c, "END").as_deref() == Some("3700000") + && !info(c).contains_key("SVLEN") + && info(c).get("IMPRECISE") == Some(&None) + && c[8..] == ["GT", "0/1"], + "CNVpytor's deletion: SVCALLER=cnvpytor, END, no SVLEN (its positive one), IMPRECISE, its genotype: {c:?}" + ); + let c = &by["CNVpytor_dup5"]; + assert!( + c[4] == "" && info_of(c, "SVTYPE").as_deref() == Some("DUP"), + "CNVpytor's duplication is a : {c:?}" + ); + assert!( + out.contains("##INFO=\t500\t{filt}\tEND=2049900;SVTYPE=DEL;SVLEN=-49800\tGT\t{gt}\n" + ) + }; + for (label, gt, filt) in [ + ("a 0/0", "0/0", "PASS"), + ("a MinGQ", "0/1", "MinGQ"), + ("a ./.", "./.", "PASS"), + ] { + let b = w.put("base_nocall.vcf", &(BASE.to_owned() + &manta9(filt, gt))); + let got = ids(&add(&Opts { + delly: Some(delly.clone()), + ..opts(&b) + }) + .0); + assert!( + got.iter().any(|i| i == "MantaDEL:9") && got.iter().any(|i| i == "DEL00000002"), + "{label} Manta deletion stays and does not hide Delly's DEL00000002: {got:?}" + ); + } + let b = w.put( + "base_nocall.vcf", + &(BASE.to_owned() + &manta9("PASS", "0|1")), + ); + let got = ids(&add(&Opts { + delly: Some(delly.clone()), + ..opts(&b) + }) + .0); + assert!( + !got.iter().any(|i| i == "DEL00000002"), + "control: a PASS 0|1 Manta deletion still covers Delly's DEL00000002: {got:?}" + ); + let sites: String = (BASE.to_owned() + + "chr20\t2000100\tMantaDEL:9\tN\t\t500\tPASS\tEND=2049900;SVTYPE=DEL\tGT\t0/0\n") + .lines() + .map(|l| l.split('\t').take(8).collect::>().join("\t") + "\n") + .collect(); + let b = w.put("sites.vcf", &sites); + let got = ids(&add(&Opts { + delly: Some(delly.clone()), + ..opts(&b) + }) + .0); + assert!( + !got.iter().any(|i| i == "DEL00000002"), + "a sites-only base deletion covers Delly's DEL00000002 (no genotype to read): {got:?}" + ); + + // A Delly PASS deletion genotyped 0/0 or ./. is no call either: it is + // not added, and CNVpytor's deletion at the same place is added instead. + for gt in ["0/0", "./."] { + let d = w.put_gz( + "delly_nocall.vcf.gz", + &DELLY.replace("\t1/1:40\n", &format!("\t{gt}:40\n")), + ); + let got = ids(&add(&Opts { + delly: Some(d), + cnvpytor: Some(cnvp.clone()), + ..opts(&base) + }) + .0); + assert!( + !got.iter().any(|i| i == "DEL00000002") && got.iter().any(|i| i == "CNVpytor_del0"), + "a {gt} Delly deletion is not added and does not hide CNVpytor_del0: {got:?}" + ); + } + + let (out, note) = add(&Opts { + base_name: "sv_vcf".to_owned(), + ..opts(&base) + }); + let callers: Vec> = records(&out) + .iter() + .map(|r| info_of(r, "SVCALLER")) + .collect(); + assert!( + note.contains("none given") + && callers == [Some("sv_vcf".to_owned()), Some("sv_vcf".to_owned())], + "no other caller: the base out with SVCALLER=sv_vcf: {note} {callers:?}" + ); +} + +// --- 4. Delly's read depth --------------------------------------------------------------- + +const DELLY_FMT: &str = "GT:GL:GQ:FT:RCL:RC:RCR:RDCN:DR:DV:RR:RV"; + +fn delly_rdcn( + chrom: &str, + pos: u64, + rid: &str, + kind: &str, + end: u64, + rdcn: &str, + precise: bool, +) -> String { + let flag = if precise { "PRECISE" } else { "IMPRECISE" }; + format!( + "{chrom}\t{pos}\t{rid}\tN\t<{kind}>\t60\tPASS\t{flag};SVTYPE={kind};END={end}\t{DELLY_FMT}\t\ +0/1:-10,0,-10:40:PASS:100:100:100:{rdcn}:10:10:0:0\n" + ) +} + +fn depth() -> String { + "##fileformat=VCFv4.2\n#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT\ts1\n".to_owned() + + &delly_rdcn("chr20", 5000000, "DEL_rd2", "DEL", 55000000, "2", false) // 50 Mb, no loss: set aside + + &delly_rdcn("chr1", 10000000, "DEL_rd1", "DEL", 60000000, "1", false) // 50 Mb, one copy: added + + &delly_rdcn("chr1", 100000000, "DUP_rd2", "DUP", 100500000, "2", false) // no gain: set aside + + &delly_rdcn("chr1", 120000000, "DUP_rd3", "DUP", 120500000, "3", false) // a gain: added + + &delly_rdcn("chr1", 150000000, "DEL_short", "DEL", 150000500, "2", true) // 500 bp: added + + &delly_rdcn("chr1", 160000000, "DEL_rdneg", "DEL", 161000000, "-1", false) // no estimate: set aside + + &delly_rdcn("chrX", 50000000, "DEL_x1", "DEL", 51000000, "1", false) // male: normal, set aside + + &delly_rdcn("chrX", 60000000, "DUP_x2", "DUP", 60500000, "2", false) // male: a gain; female: none + + &delly_rdcn("chrX", 1000000, "DEL_par1", "DEL", 1500000, "1", false) // PAR: a loss for anyone +} + +#[test] +fn add_callers_checks_delly_read_depth() { + let w = Scratch::new("depth"); + let base = w.put("base.vcf", BASE); + let delly = w.put_gz("delly_rdcn.vcf.gz", &depth()); + let par = repo() + .join("assets/par_grch38.bed") + .to_string_lossy() + .into_owned(); + let planted = records(&depth()); + assert!( + planted.len() == 9 + && planted + .iter() + .all(|r| r[6] == "PASS" && (r[4] == "" || r[4] == "")), + "control: every planted Delly call is PASS DEL or DUP" + ); + + let added = |sex: Sex, par: Option<&str>| { + let (out, note) = add(&Opts { + delly: Some(delly.clone()), + sex, + par_bed: par.map(str::to_owned), + ..opts(&base) + }); + let ids: BTreeSet = records(&out) + .iter() + .filter(|r| info_of(r, "SVCALLER").as_deref() == Some("delly")) + .map(|r| r[2].clone()) + .collect(); + assert!( + !ids.is_empty(), + "og sv add-callers adds Delly calls: {note}" + ); + (ids, out, note) + }; + let has = |ids: &BTreeSet, i: &str| ids.contains(i); + + // No sex and no PAR file first: the arguments step 05 gave before. + let (ids, _, _) = added(Sex::Unknown, None); + assert!( + !has(&ids, "DEL_rd2"), + "no sex: the 50 Mb deletion with RDCN 2 is set aside: {ids:?}" + ); + assert!( + !has(&ids, "DUP_rd2"), + "no sex: the duplication with RDCN 2 is set aside: {ids:?}" + ); + assert!( + !has(&ids, "DEL_rdneg"), + "no sex: the long deletion with RDCN -1 is set aside: {ids:?}" + ); + assert!( + ["DEL_rd1", "DUP_rd3", "DEL_short"] + .iter() + .all(|i| has(&ids, i)), + "no sex: the deletion with RDCN 1, the duplication with RDCN 3 and the short call are added: {ids:?}" + ); + assert!( + has(&ids, "DEL_x1"), + "no sex: the chrX deletion with RDCN 1 is added: {ids:?}" + ); + assert!( + has(&ids, "DUP_x2"), + "no sex: the chrX duplication with RDCN 2 is added: {ids:?}" + ); + + let (ids, out, note) = added(Sex::Male, Some(&par)); + let want: BTreeSet = ["DEL_rd1", "DUP_rd3", "DEL_short", "DUP_x2", "DEL_par1"] + .iter() + .map(|s| (*s).to_owned()) + .collect(); + assert_eq!( + ids, want, + "male: RDCN 2 deletion and duplication, RDCN -1 and the chrX RDCN 1 deletion (his normal) set aside; \ + RDCN 1 deletion, RDCN 3 duplication, the short call, the chrX RDCN 2 duplication and the PAR RDCN 1 \ + deletion added" + ); + let line = + "delly: 5 PASS DEL/DUP calls, 5 added, and 4 PASS calls longer than 1000 bp set aside"; + assert!( + note.contains(line), + "male: the summary counts the 4 calls set aside: {note}" + ); + assert!( + out.contains(&format!("##svcallers_added=\"Callers beside manta: {line}")) + && out.find("##svcallers_added=") < out.find("#CHROM"), + "male: the input VCF's header states the same, set-aside count included" + ); + + let (ids, _, _) = added(Sex::Female, Some(&par)); + assert!( + !has(&ids, "DUP_x2"), + "female: the chrX duplication with RDCN 2 is set aside: {ids:?}" + ); + assert!( + has(&ids, "DEL_x1"), + "female: the chrX deletion with RDCN 1 is added: {ids:?}" + ); +} diff --git a/rust/crates/og-steps/tests/sv_postfilter.rs b/rust/crates/og-steps/tests/sv_postfilter.rs new file mode 100644 index 000000000..7f7d23e1e --- /dev/null +++ b/rust/crates/og-steps/tests/sv_postfilter.rs @@ -0,0 +1,902 @@ +//! `og sv postfilter` on a made-up class 4 and 5 table: what the old +//! `tests/test_annotsv_postfilter.py` checked, by meaning, in cargo test. +//! +//! 1. Exon edges, on a made-up AnnotSV genes file: a deletion from the last +//! base of exon 1 to the base before exon 3 of one transcript is flagged +//! and named; one that starts 10 bp inside the intron is not; on a minus +//! strand transcript the exons are numbered from its own start; a +//! duplication is not a deletion ("."); without the genes file the +//! column says "not checked". +//! 2. The population frequency: B_loss_AFmax for a deletion, B_gain_AFmax +//! for a duplication, AF 0.05 is common, an empty one is not. +//! 3. The caller from INFO SVCALLER. +//! 4. The support from step 22's consensus, and its red control: with an +//! empty consensus the two-caller deletion has support 1. +//! 5. The command line writes the table, and `--consensus` adds the support. +//! 6. The rule table and the depth check (`--rules` +//! assets/sv_rule_regions.tsv, `--sex`, `--par-bed`), and the red +//! controls: without `--rules` the DSPP and IGH deletions stay class 4; +//! the rule table itself has a known rule on every row, IGH and the MHC +//! where GRCh38 has them, and at least 2,000 coding tandem repeats. + +use std::collections::HashMap; +use std::path::{Path, PathBuf}; + +use og_steps::sv::Sex; +use og_steps::sv::postfilter::{self, Opts}; + +type Row = HashMap; + +fn repo() -> PathBuf { + Path::new(env!("CARGO_MANIFEST_DIR")).join("../../..") +} + +fn asset(name: &str) -> String { + repo() + .join("assets") + .join(name) + .to_string_lossy() + .into_owned() +} + +/// A scratch folder of its own for one test, removed when dropped. +struct Scratch(PathBuf); + +impl Scratch { + fn new(name: &str) -> Scratch { + let d = + std::env::temp_dir().join(format!("og-sv-postfilter-{name}-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&d); + std::fs::create_dir_all(&d).unwrap(); + Scratch(d) + } + + fn put(&self, name: &str, text: &str) -> String { + let p = self.0.join(name); + std::fs::write(&p, text).unwrap(); + p.to_string_lossy().into_owned() + } + + fn path(&self, name: &str) -> String { + self.0.join(name).to_string_lossy().into_owned() + } +} + +impl Drop for Scratch { + fn drop(&mut self) { + let _ = std::fs::remove_dir_all(&self.0); + } +} + +/// The options of `og sv postfilter --class45 C45 --out OUT` with nothing +/// else given. +fn opts(class45: &str, out: &str) -> Opts { + Opts { + class45: class45.to_owned(), + genes: None, + rules: None, + consensus: None, + sex: Sex::Unknown, + par_bed: None, + out: out.to_owned(), + } +} + +/// The written table: its header and its rows by column name. +fn table(path: &str) -> (Vec, Vec) { + let text = std::fs::read_to_string(path).unwrap_or_default(); + let mut lines = text.lines(); + let head: Vec = lines + .next() + .unwrap_or_default() + .split('\t') + .map(str::to_owned) + .collect(); + let rows = lines + .map(|l| { + head.iter() + .cloned() + .zip(l.split('\t').map(str::to_owned)) + .collect() + }) + .collect(); + (head, rows) +} + +/// Runs the step; the written rows keyed by `key`, and the summary line +/// (or the error, when it stopped). +fn run(o: &Opts, key: &str) -> (HashMap, String) { + match postfilter::run(o) { + Ok(summary) => { + let (_, rows) = table(&o.out); + let by = rows + .into_iter() + .map(|r| (r.get(key).cloned().unwrap_or_default(), r)) + .collect(); + (by, summary) + } + Err(e) => (HashMap::new(), format!("ERROR: {e}")), + } +} + +fn col<'a>(by: &'a HashMap, id: &str, c: &str) -> &'a str { + by.get(id) + .and_then(|r| r.get(c)) + .map_or("", String::as_str) +} + +// chrom txStart txEnd strand gene tx cdsStart cdsEnd exonStarts exonEnds +const GENES: &str = "20\t1000000\t1010000\t+\tFIXG1\tNM_900001\t1000100\t1009900\t1000000,1006000,1009800,\t1000200,1006100,1010000,\n\ +20\t3000000\t3005100\t-\tMING1\tNM_900005\t3000000\t3005100\t3000000,3005000,\t3000100,3005100,\n"; + +const HEAD: [&str; 16] = [ + "AnnotSV_ID", + "SV_chrom", + "SV_start", + "SV_end", + "SV_length", + "SV_type", + "ID", + "INFO", + "Annotation_mode", + "Gene_name", + "B_gain_source", + "B_gain_AFmax", + "B_loss_source", + "B_loss_AFmax", + "ACMG_class", + "AnnotSV_ranking_score", +]; + +/// One class 4 or 5 row of the first table: chr20, AnnotSV's columns. +#[allow(clippy::too_many_arguments)] +fn row( + i: u32, + start: i64, + end: i64, + t: &str, + cls: u8, + caller: &str, + loss_af: &str, + loss_src: &str, +) -> String { + let len = if t == "DEL" { start - end } else { end - start }; + [ + format!("20_{start}_{end}_{t}_1"), + "20".into(), + start.to_string(), + end.to_string(), + len.to_string(), + t.into(), + format!("SV{i}"), + format!("END={end};SVTYPE={t};SVCALLER={caller}"), + "full".into(), + format!("GENE{i}"), + String::new(), + String::new(), + loss_src.into(), + loss_af.into(), + cls.to_string(), + "0.95".into(), + ] + .join("\t") + + "\n" +} + +fn class45() -> String { + HEAD.join("\t") + + "\n" + + &row( + 1, + 1000200, + 1009800, + "DEL", + 4, + "manta", + "0.05", + "gnomAD-SV_v4.1_DEL_fixture", + ) + + &row(2, 10100999, 10102999, "DEL", 5, "manta", "", "") + + &row(3, 20000000, 20450000, "DEL", 4, "manta", "", "") + + &row(4, 30000000, 30500000, "DUP", 4, "delly", "", "") + + &row(5, 1000190, 1009800, "DEL", 4, "cnvpytor", "", "") + + &row(6, 3000100, 3005000, "DEL", 4, "manta", "", "") +} + +const CONS_HEAD: &str = "##fileformat=VCFv4.1\n##source=SURVIVOR\n#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT\tmanta\tdelly\n"; + +fn consensus() -> String { + CONS_HEAD.to_owned() + + "chr20\t10101000\tMantaDEL:1\tN\t\t.\tPASS\tSUPP=2;SUPP_VEC=11;SVLEN=-2000;SVTYPE=DEL;SVMETHOD=SURVIVOR1.0.7;\ +CHR2=chr20;END=10103000;STRANDS=+-\tGT\t0/1\t0/1\n\ +chr20\t20000010\tMantaDUP:2\tN\t\t.\tPASS\tSUPP=2;SUPP_VEC=11;SVLEN=450000;SVTYPE=DUP;END=20450010\tGT\t0/1\t0/1\n" +} + +/// The columns of the checked table: AnnotSV's kept ones, the checks', and +/// with `--consensus` the support. +const KEEP: [&str; 10] = [ + "AnnotSV_ID", + "ID", + "SV_chrom", + "SV_start", + "SV_end", + "SV_length", + "SV_type", + "Gene_name", + "ACMG_class", + "AnnotSV_ranking_score", +]; +const ADDED: [&str; 10] = [ + "Caller", + "Exon_edge", + "Exon_edge_detail", + "Population_AF", + "Population_source", + "Common", + "AnnotSV_class", + "Depth_support", + "Rule", + "Rule_detail", +]; +const SUPPORT: [&str; 2] = ["Support", "Support_callers"]; + +#[test] +fn exon_edges_population_caller_and_support() { + let w = Scratch::new("main"); + let genes = w.put("genes.RefSeq.sorted.bed", GENES); + let c45 = w.put("S_sv_class4_5.tsv", &class45()); + let dst = w.path("checked.tsv"); + let checked = |genes: Option<&str>, consensus: Option<&str>| { + let o = Opts { + genes: genes.map(str::to_owned), + consensus: consensus.map(str::to_owned), + ..opts(&c45, &dst) + }; + let (by, summary) = run(&o, "Gene_name"); + assert!( + !summary.starts_with("ERROR"), + "og sv postfilter stopped: {summary}" + ); + by + }; + + // 1. exon edges + let out = checked(Some(&genes), None); + assert_eq!( + col(&out, "GENE1", "Exon_edge"), + "yes", + "an intron-exact deletion (exon 1 end to exon 3 start) is on exon edges" + ); + assert_eq!( + col(&out, "GENE1", "Exon_edge_detail"), + "NM_900001 (FIXG1): exon 1 and exon 3 edges", + "it names the transcript and the exons" + ); + assert_eq!( + col(&out, "GENE5", "Exon_edge"), + "no", + "a deletion starting 10 bp inside the intron is not" + ); + assert_eq!( + col(&out, "GENE3", "Exon_edge"), + "no", + "a deletion in no transcript is not" + ); + assert_eq!( + col(&out, "GENE6", "Exon_edge_detail"), + "NM_900005 (MING1): exon 1 and exon 2 edges", + "minus strand: exons numbered from the transcript's start" + ); + assert_eq!( + col(&out, "GENE4", "Exon_edge"), + ".", + "a duplication is not checked as a deletion" + ); + let no_genes = checked(None, None); + assert_eq!( + col(&no_genes, "GENE1", "Exon_edge"), + "not checked", + "without the genes file: not checked" + ); + + // 2. population frequency + assert_eq!( + ( + col(&out, "GENE1", "Population_AF"), + col(&out, "GENE1", "Population_source") + ), + ("0.05", "gnomAD-SV_v4.1_DEL_fixture"), + "the deletion's B_loss_AFmax and its source" + ); + assert_eq!(col(&out, "GENE1", "Common"), "yes", "AF 0.05 is common"); + assert_eq!( + ( + col(&out, "GENE2", "Population_AF"), + col(&out, "GENE2", "Common") + ), + ("", "no"), + "no benign record: no AF, not common" + ); + assert_eq!( + col(&out, "GENE4", "Population_AF"), + "", + "a duplication reads B_gain_AFmax (empty here)" + ); + + // 3. caller + assert_eq!( + (col(&out, "GENE2", "Caller"), col(&out, "GENE4", "Caller")), + ("manta", "delly"), + "the caller from INFO SVCALLER" + ); + + // 4. consensus support + let cons = w.put("S_sv_consensus.vcf", &consensus()); + let sup = checked(Some(&genes), Some(&cons)); + assert_eq!( + ( + col(&sup, "GENE2", "Support"), + col(&sup, "GENE2", "Support_callers") + ), + ("2", "manta,delly"), + "a deletion in the consensus: support 2, manta and delly" + ); + assert_eq!( + ( + col(&sup, "GENE3", "Support"), + col(&sup, "GENE3", "Support_callers") + ), + ("1", "manta"), + "a deletion only Manta made: support 1 (the consensus DUP at its place does not count)" + ); + let gone = checked(None, Some(&w.path("none.vcf"))); + assert!( + !gone.is_empty() && gone.values().all(|r| r["Support"].is_empty()), + "without a consensus file the support is unknown, not 1: {gone:?}" + ); + + // Red control: an empty consensus (header only) gives the two-caller + // deletion support 1, so the support-2 check above can fail. + let empty = w.put("empty.vcf", CONS_HEAD); + let sup_e = checked(None, Some(&empty)); + assert_eq!( + col(&sup_e, "GENE2", "Support"), + "1", + "red control: with an empty consensus the two-caller deletion has support 1" + ); + assert!( + sup_e.values().all(|r| r["Support"] == "1"), + "red control: and no row has support 2: {sup_e:?}" + ); +} + +#[test] +fn the_command_line_writes_the_table() { + let w = Scratch::new("cli"); + let genes = w.put("genes.bed", GENES); + let c45 = w.put("S_sv_class4_5.tsv", &class45()); + let cons = w.put("S_sv_consensus.vcf", &consensus()); + let dst = w.path("checked.tsv"); + let argv = |extra: &[&str]| -> Vec { + ["--class45", &c45, "--out", &dst] + .iter() + .chain(extra) + .map(|s| (*s).to_owned()) + .collect() + }; + + assert_eq!( + postfilter::cli(&argv(&["--genes", &genes, "--consensus", &cons])), + 0, + "the command exits 0" + ); + let (head, got) = table(&dst); + assert_eq!(got.len(), 6, "one row per SV"); + let want: Vec<&str> = KEEP.iter().chain(&ADDED).chain(&SUPPORT).copied().collect(); + assert_eq!(head, want, "the columns"); + let g2 = got + .iter() + .find(|r| r["Gene_name"] == "GENE2") + .expect("GENE2"); + assert_eq!( + (g2["Support"].as_str(), g2["Support_callers"].as_str()), + ("2", "manta,delly"), + "the written support" + ); + let summary = postfilter::run(&Opts { + genes: Some(genes.clone()), + consensus: Some(cons.clone()), + ..opts(&c45, &dst) + }) + .expect("og sv postfilter"); + assert!( + summary.contains("deletions on exon edges: 2; common (benign-set AF >= 1%): 1"), + "the summary line: {summary}" + ); + + assert_eq!(postfilter::cli(&argv(&[])), 0); + let (head, _) = table(&dst); + let summary = postfilter::run(&opts(&c45, &dst)).expect("og sv postfilter"); + assert!( + !head.iter().any(|h| h == "Support") + && summary.contains("deletions on exon edges: not checked"), + "without --consensus no support columns, without --genes not checked: {head:?} {summary}" + ); + + std::fs::write(&c45, "").unwrap(); + assert_eq!(postfilter::cli(&argv(&["--genes", &genes])), 0); + let text = std::fs::read_to_string(&dst).unwrap(); + assert_eq!( + text.lines().count(), + 1, + "an empty class 4 and 5 table gives a header and no row: {text:?}" + ); +} + +// --- the rule table and the depth check -------------------------------------------------- + +const HEAD7: [&str; 16] = [ + "AnnotSV_ID", + "SV_chrom", + "SV_start", + "SV_end", + "SV_length", + "SV_type", + "Samples_ID", + "ID", + "INFO", + "FORMAT", + "S1", + "Annotation_mode", + "Gene_name", + "B_loss_AFmax", + "ACMG_class", + "AnnotSV_ranking_score", +]; + +/// One row of the rule and depth table: (ID, chrom, start, end, type, +/// class, caller, gene, FORMAT, sample). +type R7<'a> = ( + &'a str, + &'a str, + i64, + i64, + &'a str, + u8, + &'a str, + &'a str, + &'a str, + &'a str, +); + +fn row7(r: &R7) -> String { + let (id, chrom, start, end, t, cls, caller, gene, fmt, sample) = *r; + let len = if t == "DEL" { start - end } else { end - start }; + [ + format!("{chrom}_{start}_{end}_{t}_1"), + chrom.into(), + start.to_string(), + end.to_string(), + len.to_string(), + t.into(), + "S1".into(), + id.into(), + format!("END={end};SVTYPE={t};SVCALLER={caller}"), + fmt.into(), + sample.into(), + "full".into(), + gene.into(), + String::new(), + cls.to_string(), + "0.99".into(), + ] + .join("\t") + + "\n" +} + +const ROWS7: [R7; 18] = [ + ( + "TR_IN", "4", 87614600, 87615500, "DEL", 4, "manta", "DSPP", "GT:DHFFC", "0/1:0.45", + ), + ( + "TR_OUT", "4", 87614600, 87616700, "DEL", 4, "manta", "DSPP", "GT:DHFFC", "0/1:0.45", + ), + ( + "IGH_IN", "14", 106000000, 106100000, "DEL", 4, "manta", "IGHV3-23", "GT:DHFFC", "0/1:0.5", + ), + ( + "IGH_EDGE", "14", 105400000, 105700000, "DEL", 4, "manta", "TMEM121", "GT:DHFFC", "0/1:0.5", + ), + ( + "IGH_MOST", "14", 105550000, 105650000, "DEL", 4, "manta", "IGHA2", "GT:DHFFC", "0/1:0.5", + ), + ( + "MHC_IN", "6", 31000000, 31100000, "DEL", 4, "manta", "HLA-C", "GT:DHFFC", "0/1:0.5", + ), + ( + "ALIAS", "10", 46634800, 46635600, "DEL", 5, "manta", "RHEB", "GT:DHFFC", "0/1:0.3", + ), + ( + "REAL", "7", 151470000, 151480000, "DEL", 5, "manta", "RHEB", "GT:DHFFC", "0/1:0.3", + ), + ( + "D_LOW", "20", 40000000, 40300000, "DEL", 4, "manta", "G7", "GT:DHFFC", "0/1:0.95", + ), + ( + "D_OK", "20", 40000000, 40300000, "DEL", 4, "manta", "G8", "GT:DHFFC", "0/1:0.5", + ), + ( + "DUP_LOW", "20", 41000000, 41300000, "DUP", 4, "manta", "G9", "GT:DHBFC", "0/1:1.1", + ), + ( + "DELLY2", "20", 42000000, 42300000, "DEL", 4, "delly", "G10", "GT:RDCN", "0/1:2", + ), + ( + "DELLY1", "20", 43000000, 43300000, "DEL", 4, "delly", "G11", "GT:RDCN", "0/1:1", + ), + ( + "XDUP", "X", 50000000, 50300000, "DUP", 4, "delly", "G12", "GT:RDCN", "0/1:2", + ), + ( + "NODEPTH", "20", 44000000, 44300000, "DEL", 4, "manta", "G13", "GT", "0/1", + ), + ( + "CNV", "20", 45000000, 45300000, "DEL", 4, "cnvpytor", "G14", "GT", "0/1", + ), + ( + "DELLY_GT", "20", 46000000, 46300000, "DEL", 4, "delly", "G15", "GT", "0/1", + ), + ( + "DELLY_SHORT", + "20", + 47000000, + 47000800, + "DEL", + 4, + "delly", + "G16", + "GT", + "0/1", + ), +]; + +/// RHEB as AnnotSV's genes file would name it: once on the pseudogene +/// RHEBP1 (chr10, the region of the rule table) and once on RHEB itself +/// (chr7). +const GENES7: &str = "10\t46634910\t46635466\t+\tRHEB\tNM_990001\t46634910\t46635466\t46634910,\t46635466,\n\ +7\t151466000\t151520000\t-\tRHEB\tNM_005614\t151466100\t151519900\t151466000,151519000,\t151467000,151520000,\n"; + +const SEX_ROWS: [R7; 10] = [ + ( + "XDEL1", "X", 50000000, 50050000, "DEL", 4, "delly", "G20", "GT:RDCN", "1/1:1", + ), + ( + "XDEL0", "X", 51000000, 51050000, "DEL", 4, "delly", "G21", "GT:RDCN", "1/1:0", + ), + ( + "XDUP2", "X", 52000000, 52050000, "DUP", 4, "delly", "G22", "GT:RDCN", "0/1:2", + ), + ( + "PARDUP2", "X", 1000000, 1050000, "DUP", 4, "delly", "G23", "GT:RDCN", "0/1:2", + ), + ( + "PARDEL1", "X", 1000000, 1050000, "DEL", 4, "delly", "G24", "GT:RDCN", "0/1:1", + ), + ( + "YDEL1", "Y", 20000000, 20050000, "DEL", 4, "delly", "G25", "GT:RDCN", "1/1:1", + ), + ( + "YDUP2", "Y", 21000000, 21050000, "DUP", 4, "delly", "G26", "GT:RDCN", "0/1:2", + ), + // PAR1 ends at chrX:2,781,479 (assets/par_grch38.bed): the last base in + // it, then the first base past it. + ( + "PAREDGE_IN", + "X", + 2781479, + 2831479, + "DUP", + 4, + "delly", + "G27", + "GT:RDCN", + "0/1:2", + ), + ( + "PAREDGE_OUT", + "X", + 2781480, + 2831480, + "DUP", + 4, + "delly", + "G28", + "GT:RDCN", + "0/1:2", + ), + ( + "BIGDUP", + "20", + 48000000, + 48050000, + "DUP", + 4, + "delly", + "G29", + "GT:RDCN", + "0/1:1000000", + ), +]; + +fn table7(header: &[&str], rows: &[R7]) -> String { + header.join("\t") + "\n" + &rows.iter().map(row7).collect::() +} + +/// The step on ROWS7 with the genes file, and with or without the rules. +fn run7(w: &Scratch, name: &str, with_rules: bool) -> (HashMap, String) { + let c45 = w.put(&format!("{name}.tsv"), &table7(&HEAD7, &ROWS7)); + let genes = w.put(&format!("{name}_genes.bed"), GENES7); + let o = Opts { + genes: Some(genes), + rules: with_rules.then(|| asset("sv_rule_regions.tsv")), + ..opts(&c45, &w.path(&format!("{name}_checked.tsv"))) + }; + run(&o, "ID") +} + +#[test] +fn region_rules_and_the_depth_check() { + let w = Scratch::new("rules"); + let (got, summary) = run7(&w, "rules", true); + let c = |id: &str, k: &str| col(&got, id, k).to_owned(); + + // the region rules + assert_eq!( + ( + c("TR_IN", "ACMG_class"), + c("TR_IN", "AnnotSV_class"), + c("TR_IN", "Rule") + ), + ("3".into(), "4".into(), "tandem_repeat".into()), + "an in-frame deletion inside DSPP's coding tandem repeat is class 3, not likely pathogenic" + ); + assert_eq!( + (c("TR_OUT", "ACMG_class"), c("TR_OUT", "Rule")), + ("4".into(), String::new()), + "the same deletion reaching 254 bp past the repeat stays class 4" + ); + assert_eq!( + (c("IGH_IN", "ACMG_class"), c("IGH_IN", "Rule")), + ("3".into(), "ig_tr_locus".into()), + "a deletion inside IGH is class 3 (V(D)J)" + ); + assert_eq!( + c("IGH_EDGE", "ACMG_class"), + "4", + "a deletion 39% inside IGH stays class 4" + ); + assert_eq!( + c("IGH_MOST", "ACMG_class"), + "3", + "a deletion 66% inside IGH is class 3 (gnomAD-SV's rule: over half)" + ); + assert_eq!( + (c("MHC_IN", "ACMG_class"), c("MHC_IN", "Rule")), + ("3".into(), "mhc".into()), + "a deletion inside the MHC is class 3" + ); + assert_eq!( + ( + c("ALIAS", "Gene_name"), + c("ALIAS", "ACMG_class"), + c("ALIAS", "Rule") + ), + ("RHEBP1".into(), "3".into(), "gene_alias".into()), + "a class 5 deletion on the pseudogene RHEBP1 is not mapped to RHEB" + ); + assert_eq!( + ( + c("REAL", "Gene_name"), + c("REAL", "ACMG_class"), + c("REAL", "Rule") + ), + ("RHEB".into(), "5".into(), String::new()), + "a class 5 deletion on RHEB itself stays RHEB and class 5" + ); + + // the depth check + assert_eq!( + ( + c("D_LOW", "Depth_support"), + c("D_LOW", "ACMG_class"), + c("D_LOW", "Rule") + ), + ("no".into(), "3".into(), "no_depth_support".into()), + "a Manta deletion with DHFFC 0.95 has no depth support and is class 3" + ); + assert_eq!( + (c("D_OK", "Depth_support"), c("D_OK", "ACMG_class")), + ("yes".into(), "4".into()), + "the same deletion with DHFFC 0.5 stays class 4" + ); + assert_eq!( + (c("DUP_LOW", "Depth_support"), c("DUP_LOW", "ACMG_class")), + ("no".into(), "3".into()), + "a duplication with DHBFC 1.1 is class 3" + ); + assert_eq!( + (c("DELLY2", "Depth_support"), c("DELLY1", "Depth_support")), + ("no".into(), "yes".into()), + "Delly RDCN 2 on a deletion: no support; RDCN 1: support" + ); + assert_eq!( + c("XDUP", "Depth_support"), + "yes", + "a chrX duplication with RDCN 2 counts when the sex is not given" + ); + assert_eq!( + (c("NODEPTH", "Depth_support"), c("NODEPTH", "ACMG_class")), + ("not checked".into(), "4".into()), + "no depth value: not checked, class kept" + ); + assert_eq!( + c("CNV", "Depth_support"), + "yes", + "a CNVpytor call has depth support" + ); + assert_eq!( + (c("DELLY_GT", "Depth_support"), c("DELLY_GT", "ACMG_class")), + ("yes".into(), "4".into()), + "a Delly call step 05 added (FORMAT GT only) over 1,000 bp passed its RDCN gate there: support" + ); + assert_eq!( + ( + c("DELLY_SHORT", "Depth_support"), + c("DELLY_SHORT", "ACMG_class") + ), + ("not checked".into(), "4".into()), + "the same call of 800 bp had no RDCN gate: not checked, class kept" + ); + + let mut moved: Vec<&str> = got + .iter() + .filter(|(_, r)| r["ACMG_class"] != r["AnnotSV_class"]) + .map(|(i, _)| i.as_str()) + .collect(); + moved.sort_unstable(); + let mut want = vec![ + "TR_IN", "IGH_IN", "IGH_MOST", "MHC_IN", "ALIAS", "D_LOW", "DUP_LOW", "DELLY2", + ]; + want.sort_unstable(); + assert_eq!( + moved, want, + "exactly the eight rows above are moved to class 3" + ); + assert!( + summary.contains("moved to class 3: 8; depth support no: 3, not checked: 2"), + "the summary line counts them: {summary}" + ); + assert_eq!( + (c("D_OK", "FORMAT"), c("D_OK", "S1"), c("NODEPTH", "S1")), + ("GT:DHFFC".into(), "0/1:0.5".into(), "0/1".into()), + "every row keeps AnnotSV's FORMAT and sample column (GT and the depth fields)" + ); + + // A sample named like a checked column stops the step. + let clash_head: Vec<&str> = HEAD7 + .iter() + .map(|&h| if h == "S1" { "Caller" } else { h }) + .collect(); + let clash = w.put("clash.tsv", &table7(&clash_head, &ROWS7)); + let err = postfilter::run(&opts(&clash, &w.path("clash_checked.tsv"))) + .expect_err("a sample named Caller must stop the step"); + assert!( + err.to_string().contains("rename the sample"), + "a sample named Caller stops the step instead of losing its GT and PS: {err}" + ); + + // Red control: without --rules the DSPP and IGH deletions stay class 4. + let (plain, _) = run7(&w, "norules", false); + assert_eq!( + ( + col(&plain, "TR_IN", "ACMG_class"), + col(&plain, "IGH_IN", "ACMG_class") + ), + ("4", "4"), + "red control: without --rules the DSPP and IGH deletions stay class 4" + ); +} + +#[test] +fn delly_depth_by_sex_and_par() { + let w = Scratch::new("sex"); + let par = asset("par_grch38.bed"); + let c45 = w.put("sex.tsv", &table7(&HEAD7, &SEX_ROWS)); + let (yes, no) = (("yes", "4"), ("no", "3")); + let ids = [ + "XDEL1", + "XDUP2", + "PARDUP2", + "XDEL0", + "PARDEL1", + "YDEL1", + "YDUP2", + "PAREDGE_IN", + "PAREDGE_OUT", + "BIGDUP", + ]; + for (sex, want) in [ + (Sex::Male, [no, yes, no, yes, yes, no, yes, no, yes, yes]), + (Sex::Female, [yes, no, no, yes, yes, yes, no, no, no, yes]), + ( + Sex::Unknown, + [yes, yes, no, yes, yes, yes, yes, no, yes, yes], + ), + ] { + let o = Opts { + sex, + par_bed: Some(par.clone()), + ..opts(&c45, &w.path(&format!("sex_{sex:?}_checked.tsv"))) + }; + let (got, summary) = run(&o, "ID"); + assert!(!summary.starts_with("ERROR"), "{summary}"); + let got: Vec<(&str, &str)> = ids + .iter() + .map(|i| (col(&got, i, "Depth_support"), col(&got, i, "ACMG_class"))) + .collect(); + assert_eq!( + got, want, + "sex {sex:?}: chrX and chrY RDCN against 1 copy (male), 2 (female), either (not given); \ + inside the PAR, up to its last base, 2 copies; RDCN 1000000 is a gain ({ids:?})" + ); + } +} + +#[test] +fn the_rule_table() { + let text = std::fs::read_to_string(asset("sv_rule_regions.tsv")).unwrap(); + let table: Vec> = text + .lines() + .filter(|l| !l.starts_with('#')) + .map(|l| l.split('\t').collect()) + .collect(); + let rules: std::collections::BTreeSet<&str> = table + .iter() + .map(|r| r.get(3).copied().unwrap_or("")) + .collect(); + assert!( + table.iter().all(|r| r.len() == 7) + && rules + == ["gene_alias", "ig_tr_locus", "mhc", "tandem_repeat"] + .into_iter() + .collect(), + "the rule table: every row has 7 columns and a known rule: {rules:?}" + ); + let igh: Vec<&Vec<&str>> = table + .iter() + .filter(|r| r[3] == "ig_tr_locus" && r[5] == "IGH") + .collect(); + let n = |s: &str| s.parse::().unwrap(); + assert!( + igh.len() == 1 + && igh[0][0] == "chr14" + && 105_500_000 < n(igh[0][1]) + && n(igh[0][1]) < n(igh[0][2]) + && n(igh[0][2]) < 107_000_000, + "the rule table: IGH on chr14 between 105.5 and 107 Mb: {igh:?}" + ); + let mhc: Vec<&[&str]> = table + .iter() + .filter(|r| r[3] == "mhc") + .map(|r| &r[..3]) + .collect(); + assert_eq!( + mhc, + [["chr6", "28510119", "33480577"]], + "the rule table: the MHC is GRCh38's 28,510,120-33,480,577" + ); + assert!( + table.iter().filter(|r| r[3] == "tandem_repeat").count() >= 2000, + "the rule table: at least 2,000 coding tandem repeats" + ); +} diff --git a/tests/fixtures/vcf/README.md b/tests/fixtures/vcf/README.md index 16833d17b..aed7aee1d 100644 --- a/tests/fixtures/vcf/README.md +++ b/tests/fixtures/vcf/README.md @@ -18,7 +18,7 @@ combined-allele files, all described below, which are the reference genome's. SNV. Positions are on GRCh38 chr22 and the REF bases are the reference's there, so step 06 runs on them with the real fixture reference; the genes (`SYNTH1`, `SYNTH2`) and ClinVar IDs are made up. - `tests/test_clinvar_join_adjacent.py` and the e2e case + `rust/crates/og-clinvar/tests/port_tests_sv_clinvar_str_join.rs` and the e2e case `clinvar-combined-allele-1` read them. - `grch37_no_contigs.vcf`, for the e2e case `tests/e2e/vcf-precheck-ref-spotcheck-1.sh`: 60 SNVs on chr1 and chr20 diff --git a/tests/parity/gen/clinvar-cases.sh b/tests/parity/gen/clinvar-cases.sh index c49528f71..0629a190c 100755 --- a/tests/parity/gen/clinvar-cases.sh +++ b/tests/parity/gen/clinvar-cases.sh @@ -5,7 +5,7 @@ # c32), each the same two stretches of GRCh38 chr22 # (from 30000000 and 30000200, here at 100 and 300) # with N between; lc holds them in lower case -# join.vcf the scenarios of tests/test_clinvar_join_adjacent.py +# join.vcf the scenarios of og-clinvar's tests/port_tests_sv_clinvar_str_join.rs # and more, one contig each, header without a PASS # line and with old PGP_ INFO lines # join-edge.vcf lines a joiner must pass or skip: a ## line between diff --git a/tests/test_annotsv_postfilter.py b/tests/test_annotsv_postfilter.py deleted file mode 100755 index 6ec5a1cfb..000000000 --- a/tests/test_annotsv_postfilter.py +++ /dev/null @@ -1,406 +0,0 @@ -#!/usr/bin/env python3 -"""og sv postfilter on a made-up class 4 and 5 table. - -Every check runs og, the binary OG names (guard.yml sets it). Without OG -the test is skipped with a note, and with OG_REQUIRED set the skip is a -failure. - -Checks: - 1. exon edges, on a made-up AnnotSV genes file: a deletion from the last - base of exon 1 to the base before exon 3 of one transcript is flagged - and named; one that starts 10 bp inside the intron is not; on a minus - strand transcript the exons are numbered from its own start; a - duplication is not a deletion ("."); without the genes file the column - says "not checked"; - 2. the population frequency: B_loss_AFmax for a deletion, B_gain_AFmax - for a duplication, AF 0.05 is common, an empty one is not; - 3. the caller from INFO SVCALLER; - 4. the support from step 22's consensus: a deletion in it has support 2 - and its two callers by the VCF's sample names; a deletion only Manta - made has support 1; a duplication of the consensus does not count for - a deletion at the same place; without a consensus file the support is - unknown (empty), not 1; - 5. red control: with an empty consensus (header only) the two-caller - deletion has support 1, so the support-2 check above can fail; - 6. the command line writes the table, and --consensus adds the support; - 7. the rule table and the depth check, through the command line with - assets/sv_rule_regions.tsv (each check reads the written table, so on a - postfilter without them every check fails with what it got): - an in-frame deletion inside DSPP's coding tandem repeat is class 3 and - the same deletion reaching 254 bp past the repeat stays class 4; a - deletion inside IGH is class 3, one 66% inside it too, and one 39% - inside it stays; a - deletion inside the MHC is class 3; a class 5 deletion on the - pseudogene RHEBP1, which AnnotSV names RHEB (its alias), is not mapped - to RHEB and is class 3, while one on RHEB itself stays RHEB and 5; two - Manta deletions at one place, DHFFC 0.95 and 0.5: the first is class 3 - (no depth support), the second stays 4; a duplication with DHBFC 1.1 - is class 3; Delly RDCN 2 on a deletion is no support, RDCN 1 is; on - chrX a duplication with RDCN 2 counts with no sex given; with --sex - and --par-bed a male's chrX deletion needs RDCN 0, a female's chrX - duplication 3 or more, and inside the PAR a duplication needs 3 for - any sex; a record with no depth value is - "not checked" and keeps its class; a CNVpytor call has support; a - Delly call with FORMAT GT only (as step 05 adds it) has support over - 1,000 bp and is "not checked" at 800 bp; the - summary line counts the rows moved to class 3; every written row keeps - AnnotSV's FORMAT and sample columns, and a sample named like a checked - column (Caller) stops the step instead of losing its GT; - 8. red controls: without --rules the DSPP and IGH deletions stay class 4; - and the rule table itself: every row has a known rule, IGH lies on - chr14 between 105.5 and 107 Mb, the MHC is GRCh38's 28,510,120 to - 33,480,577, and there are at least 2,000 coding tandem repeats. - -Run: OG=rust/target/release/og python3 tests/test_annotsv_postfilter.py -""" -import csv -import os -import shutil -import subprocess -import sys -import tempfile - -REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) -OG = os.environ.get("OG", "") -# The columns of the checked table: AnnotSV's kept ones, the checks', and -# with --consensus the support. -KEEP = ["AnnotSV_ID", "ID", "SV_chrom", "SV_start", "SV_end", "SV_length", "SV_type", "Gene_name", "ACMG_class", - "AnnotSV_ranking_score"] -ADDED = ["Caller", "Exon_edge", "Exon_edge_detail", "Population_AF", "Population_source", "Common", "AnnotSV_class", - "Depth_support", "Rule", "Rule_detail"] -SUPPORT = ["Support", "Support_callers"] - -FAILS = 0 - - -def postfilter(*args): - """og sv postfilter ARGS: (exit code, stderr).""" - p = subprocess.run([OG, "sv", "postfilter"] + list(args), capture_output=True, text=True) - return p.returncode, p.stderr - - -def table(path): - with open(path) as f: - return list(csv.DictReader(f, delimiter="\t")) - - -def check(desc, ok, detail=""): - global FAILS - print(("ok: " if ok else "FAIL: ") + desc + ("" if ok else f": {detail}")) - if not ok: - FAILS += 1 - - -# chrom txStart txEnd strand gene tx cdsStart cdsEnd exonStarts exonEnds -GENES = ("20\t1000000\t1010000\t+\tFIXG1\tNM_900001\t1000100\t1009900\t1000000,1006000,1009800,\t1000200,1006100,1010000,\n" - "20\t3000000\t3005100\t-\tMING1\tNM_900005\t3000000\t3005100\t3000000,3005000,\t3000100,3005100,\n") - -HEAD = ["AnnotSV_ID", "SV_chrom", "SV_start", "SV_end", "SV_length", "SV_type", "ID", "INFO", "Annotation_mode", - "Gene_name", "B_gain_source", "B_gain_AFmax", "B_loss_source", "B_loss_AFmax", "ACMG_class", - "AnnotSV_ranking_score"] - - -def row(i, start, end, t, cls, caller, gain_af="", loss_af="", loss_src=""): - v = [f"20_{start}_{end}_{t}_1", "20", str(start), str(end), str(end - start if t != "DEL" else start - end), t, - f"SV{i}", f"END={end};SVTYPE={t};SVCALLER={caller}", "full", f"GENE{i}", "", gain_af, loss_src, loss_af, - str(cls), "0.95"] - return "\t".join(v) + "\n" - - -CLASS45 = "\t".join(HEAD) + "\n" + "".join([ - row(1, 1000200, 1009800, "DEL", 4, "manta", loss_af="0.05", loss_src="gnomAD-SV_v4.1_DEL_fixture"), - row(2, 10100999, 10102999, "DEL", 5, "manta"), - row(3, 20000000, 20450000, "DEL", 4, "manta"), - row(4, 30000000, 30500000, "DUP", 4, "delly", gain_af=""), - row(5, 1000190, 1009800, "DEL", 4, "cnvpytor"), - row(6, 3000100, 3005000, "DEL", 4, "manta"), -]) - -CONS_HEAD = ("##fileformat=VCFv4.1\n##source=SURVIVOR\n" - "#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT\tmanta\tdelly\n") -CONSENSUS = CONS_HEAD + ( - "chr20\t10101000\tMantaDEL:1\tN\t\t.\tPASS\tSUPP=2;SUPP_VEC=11;SVLEN=-2000;SVTYPE=DEL;SVMETHOD=SURVIVOR1.0.7;" - "CHR2=chr20;END=10103000;STRANDS=+-\tGT\t0/1\t0/1\n" - "chr20\t20000010\tMantaDUP:2\tN\t\t.\tPASS\tSUPP=2;SUPP_VEC=11;SVLEN=450000;SVTYPE=DUP;END=20450010\tGT\t0/1\t0/1\n") - - -RULES_TSV = os.path.join(REPO, "assets", "sv_rule_regions.tsv") -HEAD7 = ["AnnotSV_ID", "SV_chrom", "SV_start", "SV_end", "SV_length", "SV_type", "Samples_ID", "ID", "INFO", "FORMAT", - "S1", "Annotation_mode", "Gene_name", "B_loss_AFmax", "ACMG_class", "AnnotSV_ranking_score"] - - -def row7(i, chrom, start, end, t, cls, caller, gene, fmt="GT", sample="0/1"): - length = end - start if t != "DEL" else start - end - v = [f"{chrom}_{start}_{end}_{t}_1", chrom, str(start), str(end), str(length), t, "S1", i, - f"END={end};SVTYPE={t};SVCALLER={caller}", fmt, sample, "full", gene, "", str(cls), "0.99"] - return "\t".join(v) + "\n" - - -ROWS7 = "".join([ - row7("TR_IN", "4", 87614600, 87615500, "DEL", 4, "manta", "DSPP", "GT:DHFFC", "0/1:0.45"), - row7("TR_OUT", "4", 87614600, 87616700, "DEL", 4, "manta", "DSPP", "GT:DHFFC", "0/1:0.45"), - row7("IGH_IN", "14", 106000000, 106100000, "DEL", 4, "manta", "IGHV3-23", "GT:DHFFC", "0/1:0.5"), - row7("IGH_EDGE", "14", 105400000, 105700000, "DEL", 4, "manta", "TMEM121", "GT:DHFFC", "0/1:0.5"), - row7("IGH_MOST", "14", 105550000, 105650000, "DEL", 4, "manta", "IGHA2", "GT:DHFFC", "0/1:0.5"), - row7("MHC_IN", "6", 31000000, 31100000, "DEL", 4, "manta", "HLA-C", "GT:DHFFC", "0/1:0.5"), - row7("ALIAS", "10", 46634800, 46635600, "DEL", 5, "manta", "RHEB", "GT:DHFFC", "0/1:0.3"), - row7("REAL", "7", 151470000, 151480000, "DEL", 5, "manta", "RHEB", "GT:DHFFC", "0/1:0.3"), - row7("D_LOW", "20", 40000000, 40300000, "DEL", 4, "manta", "G7", "GT:DHFFC", "0/1:0.95"), - row7("D_OK", "20", 40000000, 40300000, "DEL", 4, "manta", "G8", "GT:DHFFC", "0/1:0.5"), - row7("DUP_LOW", "20", 41000000, 41300000, "DUP", 4, "manta", "G9", "GT:DHBFC", "0/1:1.1"), - row7("DELLY2", "20", 42000000, 42300000, "DEL", 4, "delly", "G10", "GT:RDCN", "0/1:2"), - row7("DELLY1", "20", 43000000, 43300000, "DEL", 4, "delly", "G11", "GT:RDCN", "0/1:1"), - row7("XDUP", "X", 50000000, 50300000, "DUP", 4, "delly", "G12", "GT:RDCN", "0/1:2"), - row7("NODEPTH", "20", 44000000, 44300000, "DEL", 4, "manta", "G13"), - row7("CNV", "20", 45000000, 45300000, "DEL", 4, "cnvpytor", "G14"), - row7("DELLY_GT", "20", 46000000, 46300000, "DEL", 4, "delly", "G15"), - row7("DELLY_SHORT", "20", 47000000, 47000800, "DEL", 4, "delly", "G16"), -]) -# RHEB as AnnotSV's genes file would name it: once on the pseudogene RHEBP1 -# (chr10, the region of the rule table) and once on RHEB itself (chr7). -GENES7 = ("10\t46634910\t46635466\t+\tRHEB\tNM_990001\t46634910\t46635466\t46634910,\t46635466,\n" - "7\t151466000\t151520000\t-\tRHEB\tNM_005614\t151466100\t151519900\t151466000,151519000,\t151467000,151520000,\n") - - -def run7(work, name, with_rules=True): - """{ID: written row} of the command line on ROWS7, {} when it failed.""" - c45 = os.path.join(work, name + ".tsv") - with open(c45, "w") as f: - f.write("\t".join(HEAD7) + "\n" + ROWS7) - genes = os.path.join(work, name + "_genes.bed") - with open(genes, "w") as f: - f.write(GENES7) - dst = os.path.join(work, name + "_checked.tsv") - rc, err = postfilter("--class45", c45, "--genes", genes, "--out", dst, *(["--rules", RULES_TSV] if with_rules else [])) - if rc != 0 or not os.path.isfile(dst): - print(f" (the command failed: {err.strip()[-300:]})") - return {}, err - return {r["ID"]: r for r in table(dst)}, err - - -SEX_ROWS = "".join([ - row7("XDEL1", "X", 50000000, 50050000, "DEL", 4, "delly", "G20", "GT:RDCN", "1/1:1"), - row7("XDEL0", "X", 51000000, 51050000, "DEL", 4, "delly", "G21", "GT:RDCN", "1/1:0"), - row7("XDUP2", "X", 52000000, 52050000, "DUP", 4, "delly", "G22", "GT:RDCN", "0/1:2"), - row7("PARDUP2", "X", 1000000, 1050000, "DUP", 4, "delly", "G23", "GT:RDCN", "0/1:2"), - row7("PARDEL1", "X", 1000000, 1050000, "DEL", 4, "delly", "G24", "GT:RDCN", "0/1:1"), - row7("YDEL1", "Y", 20000000, 20050000, "DEL", 4, "delly", "G25", "GT:RDCN", "1/1:1"), - row7("YDUP2", "Y", 21000000, 21050000, "DUP", 4, "delly", "G26", "GT:RDCN", "0/1:2"), - # PAR1 ends at chrX:2,781,479 (assets/par_grch38.bed): the last base in - # it, then the first base past it. - row7("PAREDGE_IN", "X", 2781479, 2831479, "DUP", 4, "delly", "G27", "GT:RDCN", "0/1:2"), - row7("PAREDGE_OUT", "X", 2781480, 2831480, "DUP", 4, "delly", "G28", "GT:RDCN", "0/1:2"), - row7("BIGDUP", "20", 48000000, 48050000, "DUP", 4, "delly", "G29", "GT:RDCN", "0/1:1000000"), -]) - - -def run_sex(work, name, sex, par): - """{ID: written row} of the command line on SEX_ROWS with --sex and --par-bed.""" - c45 = os.path.join(work, name + ".tsv") - with open(c45, "w") as f: - f.write("\t".join(HEAD7) + "\n" + SEX_ROWS) - dst = os.path.join(work, name + "_checked.tsv") - rc, err = postfilter("--class45", c45, "--sex", sex, "--par-bed", par, "--out", dst) - if rc != 0 or not os.path.isfile(dst): - print(f" (the command failed: {err.strip()[-300:]})") - return {}, err - return {r["ID"]: r for r in table(dst)}, err - - -def rules_section(work): - got, err = run7(work, "rules") - - def col(i, c): - return got.get(i, {}).get(c) - - # the region rules - check("an in-frame deletion inside DSPP's coding tandem repeat is class 3, not likely pathogenic", - (col("TR_IN", "ACMG_class"), col("TR_IN", "AnnotSV_class"), col("TR_IN", "Rule")) == ("3", "4", "tandem_repeat"), - got.get("TR_IN")) - check("the same deletion reaching 254 bp past the repeat stays class 4", - (col("TR_OUT", "ACMG_class"), col("TR_OUT", "Rule")) == ("4", ""), got.get("TR_OUT")) - check("a deletion inside IGH is class 3 (V(D)J)", - (col("IGH_IN", "ACMG_class"), col("IGH_IN", "Rule")) == ("3", "ig_tr_locus"), got.get("IGH_IN")) - check("a deletion 39% inside IGH stays class 4", col("IGH_EDGE", "ACMG_class") == "4", got.get("IGH_EDGE")) - check("a deletion 66% inside IGH is class 3 (gnomAD-SV's rule: over half)", col("IGH_MOST", "ACMG_class") == "3", - got.get("IGH_MOST")) - check("a deletion inside the MHC is class 3", - (col("MHC_IN", "ACMG_class"), col("MHC_IN", "Rule")) == ("3", "mhc"), got.get("MHC_IN")) - check("a class 5 deletion on the pseudogene RHEBP1 is not mapped to RHEB", - (col("ALIAS", "Gene_name"), col("ALIAS", "ACMG_class"), col("ALIAS", "Rule")) == ("RHEBP1", "3", "gene_alias"), - got.get("ALIAS")) - check("a class 5 deletion on RHEB itself stays RHEB and class 5", - (col("REAL", "Gene_name"), col("REAL", "ACMG_class"), col("REAL", "Rule")) == ("RHEB", "5", ""), got.get("REAL")) - - # the depth check - check("a Manta deletion with DHFFC 0.95 has no depth support and is class 3", - (col("D_LOW", "Depth_support"), col("D_LOW", "ACMG_class"), col("D_LOW", "Rule")) == ("no", "3", "no_depth_support"), - got.get("D_LOW")) - check("the same deletion with DHFFC 0.5 stays class 4", - (col("D_OK", "Depth_support"), col("D_OK", "ACMG_class")) == ("yes", "4"), got.get("D_OK")) - check("a duplication with DHBFC 1.1 is class 3", - (col("DUP_LOW", "Depth_support"), col("DUP_LOW", "ACMG_class")) == ("no", "3"), got.get("DUP_LOW")) - check("Delly RDCN 2 on a deletion: no support; RDCN 1: support", - (col("DELLY2", "Depth_support"), col("DELLY1", "Depth_support")) == ("no", "yes"), - (got.get("DELLY2"), got.get("DELLY1"))) - check("a chrX duplication with RDCN 2 counts when the sex is not given", col("XDUP", "Depth_support") == "yes", - got.get("XDUP")) - check("no depth value: not checked, class kept", - (col("NODEPTH", "Depth_support"), col("NODEPTH", "ACMG_class")) == ("not checked", "4"), got.get("NODEPTH")) - check("a CNVpytor call has depth support", col("CNV", "Depth_support") == "yes", got.get("CNV")) - check("a Delly call step 05 added (FORMAT GT only) over 1,000 bp passed its RDCN gate there: support", - (col("DELLY_GT", "Depth_support"), col("DELLY_GT", "ACMG_class")) == ("yes", "4"), got.get("DELLY_GT")) - check("the same call of 800 bp had no RDCN gate: not checked, class kept", - (col("DELLY_SHORT", "Depth_support"), col("DELLY_SHORT", "ACMG_class")) == ("not checked", "4"), - got.get("DELLY_SHORT")) - # Delly RDCN on chrX and chrY by sex and PAR (--sex, --par-bed), the rule - # og sv add-callers applies when step 05 adds a Delly call. - par = os.path.join(REPO, "assets", "par_grch38.bed") - yes, no = ("yes", "4"), ("no", "3") - for sex, want_sex in (("male", {"XDEL1": no, "XDUP2": yes, "PARDUP2": no, "XDEL0": yes, "PARDEL1": yes, - "YDEL1": no, "YDUP2": yes, "PAREDGE_IN": no, "PAREDGE_OUT": yes, - "BIGDUP": yes}), - ("female", {"XDEL1": yes, "XDUP2": no, "PARDUP2": no, "XDEL0": yes, "PARDEL1": yes, - "YDEL1": yes, "YDUP2": no, "PAREDGE_IN": no, "PAREDGE_OUT": no, - "BIGDUP": yes}), - ("", {"XDEL1": yes, "XDUP2": yes, "PARDUP2": no, "XDEL0": yes, "PARDEL1": yes, - "YDEL1": yes, "YDUP2": yes, "PAREDGE_IN": no, "PAREDGE_OUT": yes, - "BIGDUP": yes})): - sx, _ = run_sex(work, "sex_" + (sex or "none"), sex, par) - got_sex = {i: (sx.get(i, {}).get("Depth_support"), sx.get(i, {}).get("ACMG_class")) for i in want_sex} - check(f"sex {sex or 'not given'}: chrX and chrY RDCN against 1 copy (male), 2 (female), either (not " - f"given); inside the PAR, up to its last base, 2 copies; RDCN 1000000 is a gain", - got_sex == want_sex, got_sex) - - moved = sorted(i for i, r in got.items() if r["ACMG_class"] != r["AnnotSV_class"]) - want = sorted(["TR_IN", "IGH_IN", "IGH_MOST", "MHC_IN", "ALIAS", "D_LOW", "DUP_LOW", "DELLY2"]) - check("exactly the eight rows above are moved to class 3", moved == want, moved) - check("the summary line counts them", "moved to class 3: 8; depth support no: 3, not checked: 2" in err, err) - check("every row keeps AnnotSV's FORMAT and sample column (GT and the depth fields)", - (col("D_OK", "FORMAT"), col("D_OK", "S1"), col("NODEPTH", "S1")) == ("GT:DHFFC", "0/1:0.5", "0/1"), - got.get("D_OK")) - clash = os.path.join(work, "clash.tsv") - with open(clash, "w") as f: - f.write("\t".join("Caller" if h == "S1" else h for h in HEAD7) + "\n" + ROWS7) - rc, err = postfilter("--class45", clash, "--out", os.path.join(work, "clash_checked.tsv")) - check("a sample named Caller stops the step instead of losing its GT and PS", - rc != 0 and "rename the sample" in err, (rc, err[-300:])) - - # 8. red controls - plain, _ = run7(work, "norules", with_rules=False) - check("red control: without --rules the DSPP and IGH deletions stay class 4", - (plain.get("TR_IN", {}).get("ACMG_class"), plain.get("IGH_IN", {}).get("ACMG_class")) == ("4", "4"), - (plain.get("TR_IN"), plain.get("IGH_IN"))) - with open(RULES_TSV) as f: - table = [line.rstrip("\n").split("\t") for line in f if not line.startswith("#")] - rules = {r[3] for r in table} - check("the rule table: every row has 7 columns and a known rule", - all(len(r) == 7 for r in table) and rules == {"tandem_repeat", "ig_tr_locus", "mhc", "gene_alias"}, rules) - igh = [r for r in table if r[3] == "ig_tr_locus" and r[5] == "IGH"] - check("the rule table: IGH on chr14 between 105.5 and 107 Mb", - len(igh) == 1 and igh[0][0] == "chr14" and 105_500_000 < int(igh[0][1]) < int(igh[0][2]) < 107_000_000, igh) - check("the rule table: the MHC is GRCh38's 28,510,120-33,480,577", - [r[:3] for r in table if r[3] == "mhc"] == [["chr6", "28510119", "33480577"]]) - check("the rule table: at least 2,000 coding tandem repeats", sum(r[3] == "tandem_repeat" for r in table) >= 2000) - - -def main(): - if not OG: - check("OG names an og binary", not os.environ.get("OG_REQUIRED"), "OG_REQUIRED is set") - print("skip: og sv postfilter: OG is not set") - return 1 if FAILS else 0 - work = tempfile.mkdtemp() - try: - genes = os.path.join(work, "genes.RefSeq.sorted.bed") - with open(genes, "w") as f: - f.write(GENES) - c45 = os.path.join(work, "S_sv_class4_5.tsv") - with open(c45, "w") as f: - f.write(CLASS45) - dst = os.path.join(work, "checked.tsv") - - def checked(*args): - """{Gene_name: written row} of og sv postfilter on CLASS45 with ARGS.""" - rc, err = postfilter("--class45", c45, "--out", dst, *args) - check("og sv postfilter " + " ".join(a for a in args if a.startswith("--")) + " exits 0", rc == 0, err) - return {r["Gene_name"]: r for r in table(dst)} if rc == 0 else {} - - out = checked("--genes", genes) - - # 1. exon edges - check("an intron-exact deletion (exon 1 end to exon 3 start) is on exon edges", - out["GENE1"]["Exon_edge"] == "yes", out["GENE1"]) - check("it names the transcript and the exons", - out["GENE1"]["Exon_edge_detail"] == "NM_900001 (FIXG1): exon 1 and exon 3 edges", out["GENE1"]) - check("a deletion starting 10 bp inside the intron is not", out["GENE5"]["Exon_edge"] == "no", out["GENE5"]) - check("a deletion in no transcript is not", out["GENE3"]["Exon_edge"] == "no", out["GENE3"]) - check("minus strand: exons numbered from the transcript's start", - out["GENE6"]["Exon_edge_detail"] == "NM_900005 (MING1): exon 1 and exon 2 edges", out["GENE6"]) - check("a duplication is not checked as a deletion", out["GENE4"]["Exon_edge"] == ".", out["GENE4"]) - no_genes = checked() - check("without the genes file: not checked", no_genes["GENE1"]["Exon_edge"] == "not checked", no_genes["GENE1"]) - - # 2. population frequency - check("the deletion's B_loss_AFmax and its source", - (out["GENE1"]["Population_AF"], out["GENE1"]["Population_source"]) == ("0.05", "gnomAD-SV_v4.1_DEL_fixture"), - out["GENE1"]) - check("AF 0.05 is common", out["GENE1"]["Common"] == "yes") - check("no benign record: no AF, not common", - (out["GENE2"]["Population_AF"], out["GENE2"]["Common"]) == ("", "no"), out["GENE2"]) - check("a duplication reads B_gain_AFmax (empty here)", out["GENE4"]["Population_AF"] == "", out["GENE4"]) - - # 3. caller - check("the caller from INFO SVCALLER", (out["GENE2"]["Caller"], out["GENE4"]["Caller"]) == ("manta", "delly")) - - # 4. consensus support - cons = os.path.join(work, "S_sv_consensus.vcf") - with open(cons, "w") as f: - f.write(CONSENSUS) - sup = {g: {k: r[k] for k in SUPPORT} for g, r in checked("--genes", genes, "--consensus", cons).items()} - check("a deletion in the consensus: support 2, manta and delly", - sup["GENE2"] == {"Support": "2", "Support_callers": "manta,delly"}, sup["GENE2"]) - check("a deletion only Manta made: support 1", sup["GENE3"] == {"Support": "1", "Support_callers": "manta"}, - sup["GENE3"]) - gone = checked("--consensus", os.path.join(work, "none.vcf")) - check("without a consensus file the support is unknown, not 1", - all(r["Support"] == "" for r in gone.values()), gone) - - # 5. red control: an empty consensus - empty = os.path.join(work, "empty.vcf") - with open(empty, "w") as f: - f.write(CONS_HEAD) - sup_e = checked("--consensus", empty) - check("red control: with an empty consensus the two-caller deletion has support 1", - sup_e["GENE2"]["Support"] == "1", sup_e["GENE2"]) - check("red control: and no row has support 2", all(r["Support"] == "1" for r in sup_e.values()), sup_e) - - # 6. the command line - rc, err = postfilter("--class45", c45, "--genes", genes, "--consensus", cons, "--out", dst) - check("the command exits 0", rc == 0, err) - got = table(dst) - check("one row per SV", len(got) == 6, len(got)) - check("the columns", list(got[0]) == KEEP + ADDED + SUPPORT, list(got[0]) if got else None) - check("the summary line", "deletions on exon edges: 2; common (benign-set AF >= 1%): 1" in err, err) - g2 = next(r for r in got if r["Gene_name"] == "GENE2") - check("the written support", (g2["Support"], g2["Support_callers"]) == ("2", "manta,delly"), g2) - rc, err = postfilter("--class45", c45, "--out", dst) - with open(dst) as f: - head = f.readline().rstrip("\n").split("\t") - check("without --consensus no support columns, without --genes not checked", - rc == 0 and "Support" not in head and "deletions on exon edges: not checked" in err, err) - open(c45, "w").close() - rc, err = postfilter("--class45", c45, "--genes", genes, "--out", dst) - with open(dst) as f: - lines = f.read().splitlines() - check("an empty class 4 and 5 table gives a header and no row", rc == 0 and len(lines) == 1, (err, lines)) - - # 7 and 8. the rule table and the depth check - rules_section(work) - finally: - shutil.rmtree(work) - print("\nRESULT:", "ALL PASS" if FAILS == 0 else f"{FAILS} FAILED") - return 1 if FAILS else 0 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/tests/test_annotsv_sv_inputs.py b/tests/test_annotsv_sv_inputs.py deleted file mode 100755 index e79c1b481..000000000 --- a/tests/test_annotsv_sv_inputs.py +++ /dev/null @@ -1,444 +0,0 @@ -#!/usr/bin/env python3 -# og-leg: OG_REQUIRED -"""og sv inv-filter, og sv svlen-fix and og sv add-callers on made-up VCFs. - -The checks run og, the binary OG names (the rust job of guard.yml sets -it). Without OG they are skipped with a note, and with OG_REQUIRED set the -skip is a failure. - -Checks: - 1. og sv inv-filter, on Manta's file and a convertInversion.py-style - converted file: - - a reciprocal pair (INV3 and INV5 junctions 100 bp apart) and a pair - that shares one Manta EVENT keep their INV records, INV_JUNCTIONS=2; - - a single INV3 junction 50 Mb long is no longer an INV record: its two - breakends from Manta's file take its place, INV_JUNCTIONS=1, in - position order; the control: the converted file holds that 50 Mb INV; - - a single 100 kb junction stays an INV with INV_JUNCTIONS=1, a single - 400 bp one also gets INVDUP_CANDIDATE; - - the deletion passes unchanged, and the summary line counts each kind; - 2. og sv svlen-fix: REF 137 bp / ALT 2 bp gets SVLEN -135 and END - POS+134, with the old values kept (the control: the input says -136); - a pure deletion keeps SVLEN and gets END POS+135 (POS-1+len(REF)-len(ALT)); a net - insertion gets its net SVLEN and keeps END; symbolic, breakend and - multi-ALT records pass unchanged; a second run changes nothing; - 3. og sv add-callers: a Delly PASS deletion Manta has is not added, one - Manta lacks is; Delly's LowQual and INV records are left out; a CNVpytor - deletion Delly's added call covers is not added twice; CNVpytor calls - with e-val >= 0.01 or q0 >= 0.5, and LOH, are left out; the CNVpytor - deletion and duplication nobody else has are added, with their - genotype; every record names its caller, the output is sorted; the - control: Manta's file alone lacks the CNVpytor deletion; - 4. og sv add-callers, Delly's read depth (FORMAT RDCN): a 50 Mb PASS - deletion with RDCN 2 is set aside and one with RDCN 1 added, a long - duplication with RDCN 2 set aside and one with RDCN 3 added, a short - precise deletion with RDCN 2 added, a long one with RDCN -1 set aside; - on chrX outside the PARs a deletion with RDCN 1 is set aside for a - male and added with no sex, a duplication with RDCN 2 added for a male - and set aside for a female; in the PAR a male's RDCN 1 deletion is - added; the summary and the ##svcallers_added header count what was set - aside; the control: every planted call is PASS DEL or DUP, so without - the depth check each would be added. - -Run: OG=rust/target/release/og python3 tests/test_annotsv_sv_inputs.py -""" -import gzip -import os -import shutil -import subprocess -import sys -import tempfile - -REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) -OG = os.environ.get("OG", "") -FAILS = 0 - - -def check(desc, ok, detail=""): - global FAILS - if ok: - print("ok: " + desc) - else: - FAILS += 1 - print("FAIL: " + desc + (": " + str(detail) if detail else "")) - - -def add_callers(*args, stdin=""): - p = subprocess.run([OG, "sv", "add-callers"] + list(args), input=stdin, capture_output=True, text=True) - return p.returncode, p.stdout, p.stderr - - -def og(*args): - p = subprocess.run([OG] + list(args), capture_output=True, text=True) - return p.returncode, p.stdout, p.stderr - - -def records(text): - return [l.split("\t") for l in text.splitlines() if l and not l.startswith("#")] - - -def info(rec): - d = {} - for item in rec[7].split(";"): - k, _, v = item.partition("=") - d[k] = v if _ else True - return d - - -HEAD = ( - "##fileformat=VCFv4.1\n" - "##contig=\n" - "##contig=\n" - '##INFO=\n' - '##INFO=\n' - '##FORMAT=\n' - "#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT\ts1\n" -) - - -def bnd(chrom, pos, rid, alt, mate, extra=""): - return "{0}\t{1}\t{2}\tN\t{3}\t500\tPASS\tSVTYPE=BND;MATEID={4}{5}\tGT\t0/1\n".format( - chrom, pos, rid, alt, mate, extra) - - -def inv(chrom, pos, rid, end, side, extra=""): - return "{0}\t{1}\t{2}\tN\t\t500\tPASS\tEND={3};SVTYPE=INV;SVLEN={4}{5};INV{6}\tGT\t0/1\n".format( - chrom, pos, rid, end, end - pos, extra, side) - - -DEL = "chr20\t40000000\tMantaDEL:9:0:0:0:0:0\tN\t\t500\tPASS\tEND=40010000;SVTYPE=DEL;SVLEN=-10000\tGT\t0/1\n" - -# Manta's own file: each junction a pair of breakends. -RAW = HEAD + "".join([ - # reciprocal inversion chr20:1,000,000-2,000,000: INV3 and INV5 junctions - bnd("chr20", 1000000, "MantaBND:10:0:1:0:0:0:0", "N]chr20:2000000]", "MantaBND:10:0:1:0:0:0:1"), - bnd("chr20", 1000101, "MantaBND:10:0:2:0:0:0:0", "[chr20:2000101[N", "MantaBND:10:0:2:0:0:0:1"), - bnd("chr20", 2000000, "MantaBND:10:0:1:0:0:0:1", "N]chr20:1000000]", "MantaBND:10:0:1:0:0:0:0"), - bnd("chr20", 2000101, "MantaBND:10:0:2:0:0:0:1", "[chr20:1000101[N", "MantaBND:10:0:2:0:0:0:0"), - # one INV3 junction, 50 Mb - bnd("chr20", 5000000, "MantaBND:20:0:1:0:0:0:0", "N]chr20:55000000]", "MantaBND:20:0:1:0:0:0:1"), - bnd("chr20", 55000000, "MantaBND:20:0:1:0:0:0:1", "N]chr20:5000000]", "MantaBND:20:0:1:0:0:0:0"), - # one INV5 junction, 400 bp - bnd("chr20", 30000000, "MantaBND:30:0:1:0:0:0:0", "[chr20:30000400[N", "MantaBND:30:0:1:0:0:0:1"), - bnd("chr20", 30000400, "MantaBND:30:0:1:0:0:0:1", "[chr20:30000000[N", "MantaBND:30:0:1:0:0:0:0"), - # one INV3 junction, 100 kb - bnd("chr1", 3000000, "MantaBND:40:0:1:0:0:0:0", "N]chr1:3100000]", "MantaBND:40:0:1:0:0:0:1"), - bnd("chr1", 3100000, "MantaBND:40:0:1:0:0:0:1", "N]chr1:3000000]", "MantaBND:40:0:1:0:0:0:0"), - DEL, -]) -# What convertInversion.py writes for it (an INV5 record moves one base left). -EV = ";EVENT=MantaINV:50:0:1:0:0:0" -CONVERTED = HEAD + "".join([ - inv("chr1", 3000000, "MantaINV:40:0:1:0:0:0", 3100000, 3), - # two junctions 3 kb apart that Manta put in one EVENT - inv("chr1", 10000000, "MantaINV:50:0:1:0:0:0", 10500000, 3, EV), - inv("chr1", 10003000, "MantaINV:50:0:2:0:0:0", 10503000, 5, EV), - inv("chr20", 1000000, "MantaINV:10:0:1:0:0:0", 2000000, 3), - inv("chr20", 1000100, "MantaINV:10:0:2:0:0:0", 2000100, 5), - inv("chr20", 5000000, "MantaINV:20:0:1:0:0:0", 55000000, 3), - inv("chr20", 29999999, "MantaINV:30:0:1:0:0:0", 30000399, 5), - DEL, -]) - - -def test_inv_filter(w): - raw, conv = os.path.join(w, "raw.vcf.gz"), os.path.join(w, "conv.vcf") - with gzip.open(raw, "wt") as f: - f.write(RAW) - with open(conv, "w") as f: - f.write(CONVERTED) - - long_inv = [r for r in records(CONVERTED) if info(r).get("SVTYPE") == "INV" and int(info(r)["END"]) - int(r[1]) >= 1000000 - and r[2] == "MantaINV:20:0:1:0:0:0"] - check("control: the converted file holds the 50 Mb single-junction INV", len(long_inv) == 1) - - rc, out, err = og("sv", "inv-filter", raw, conv) - check("og sv inv-filter exits 0", rc == 0, err) - recs = records(out) - by_id = {r[2]: r for r in recs} - for rid in ("MantaINV:10:0:1:0:0:0", "MantaINV:10:0:2:0:0:0", "MantaINV:50:0:1:0:0:0", "MantaINV:50:0:2:0:0:0"): - check("both junctions present: {0} keeps its INV, INV_JUNCTIONS=2".format(rid), - rid in by_id and info(by_id[rid]).get("INV_JUNCTIONS") == "2", by_id.get(rid)) - check("the 50 Mb single junction is no INV record any more", "MantaINV:20:0:1:0:0:0" not in by_id) - mb_invs = [r[2] for r in recs if info(r).get("SVTYPE") == "INV" and int(info(r)["END"]) - int(r[1]) > 1000000] - check("no INV record longer than the largest paired inversion (1 Mb)", mb_invs == [], mb_invs) - for rid in ("MantaBND:20:0:1:0:0:0:0", "MantaBND:20:0:1:0:0:0:1"): - check("its breakend {0} is back, INV_JUNCTIONS=1".format(rid), - rid in by_id and info(by_id[rid]).get("SVTYPE") == "BND" and info(by_id[rid]).get("INV_JUNCTIONS") == "1", - by_id.get(rid)) - r = by_id.get("MantaINV:40:0:1:0:0:0") - check("the 100 kb single junction stays an INV, INV_JUNCTIONS=1, no INVDUP_CANDIDATE", - r is not None and info(r).get("INV_JUNCTIONS") == "1" and "INVDUP_CANDIDATE" not in info(r), r) - r = by_id.get("MantaINV:30:0:1:0:0:0") - check("the 400 bp single junction gets INVDUP_CANDIDATE", - r is not None and info(r).get("INV_JUNCTIONS") == "1" and info(r).get("INVDUP_CANDIDATE") is True, r) - check("the deletion passes unchanged", DEL.rstrip("\n").split("\t") in recs) - order = [(r[0], int(r[1])) for r in recs] - check("records sorted by the ##contig order, then POS", - order == sorted(order, key=lambda x: (["chr1", "chr20"].index(x[0]), x[1])), order) - check("the header declares INV_JUNCTIONS and INVDUP_CANDIDATE", - "##INFO=\n" - '##INFO=\n' - "#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT\ts1\n" -) - - -def test_svlen(w): - ref137, alt2 = "A" * 137, "AC" - rows = [ - ("chr20", "1005400", "complex", ref137, alt2, "END=1005536;SVTYPE=DEL;SVLEN=-136"), - ("chr20", "1100000", "pure", ref137, "A", "END=1100136;SVTYPE=DEL;SVLEN=-136"), - ("chr20", "1200000", "ins", "ACG", "A" + "T" * 59, "END=1200002;SVTYPE=INS;SVLEN=59"), - ("chr20", "1300000", "sym", "N", "", "END=1310000;SVTYPE=DEL;SVLEN=-10000"), - ("chr20", "1400000", "bnd", "N", "N]chr20:1500000]", "SVTYPE=BND"), - ("chr20", "1600000", "multi", ref137, "A,AC", "END=1600136;SVTYPE=DEL;SVLEN=-136"), - ] - vcf = SV + "".join("\t".join(r[:5] + ("50", "PASS", r[5], "GT", "0/1")) + "\n" for r in rows) - src = os.path.join(w, "sv.vcf") - with open(src, "w") as f: - f.write(vcf) - by_id = {r[2]: r for r in records(vcf)} - check("control: the input gives the REF 137 / ALT 2 record SVLEN -136", info(by_id["complex"])["SVLEN"] == "-136") - - rc, out, err = og("sv", "svlen-fix", src) - check("og sv svlen-fix exits 0", rc == 0, err) - by_id = {r[2]: r for r in records(out)} - c = info(by_id["complex"]) - check("REF 137 / ALT 2: SVLEN -135, END POS+134 (an interval of 135 bp for AnnotSV)", - c.get("SVLEN") == "-135" and c.get("END") == "1005534", c) - check("REF 137 / ALT 2: the old values kept", c.get("ORIG_SVLEN") == "-136" and c.get("ORIG_END") == "1005536", c) - p = info(by_id["pure"]) - check("pure deletion: SVLEN stays -136, END becomes POS+135", p.get("SVLEN") == "-136" and p.get("END") == "1100135" - and "ORIG_SVLEN" not in p, p) - i = info(by_id["ins"]) - check("net insertion: SVLEN 57, END kept", i.get("SVLEN") == "57" and i.get("END") == "1200002", i) - for rid in ("sym", "bnd", "multi"): - check("{0} passes unchanged".format(rid), by_id[rid] == [x for x in records(vcf) if x[2] == rid][0]) - check("the header declares ORIG_SVLEN and ORIG_END", "##INFO=\t50\tPASS\tEND=1310000;SVTYPE=DEL\tGT\t0/0\t0/1\n") - rc, out3, err3 = og("sv", "svlen-fix", two) - check("a VCF with two sample columns stops (exit 2) and names them", - rc == 2 and "holds 2 samples (s1, s2)" in err3 and not records(out3), (rc, err3)) - - -BASE = ( - "##fileformat=VCFv4.1\n##contig=\n##contig=\n" - '##INFO=\n' - '##INFO=\n' - '##FORMAT=\n' - "#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT\ts1\n" - "chr20\t1000000\tMantaDEL:1\tN\t\t500\tPASS\tEND=1100000;SVTYPE=DEL;SVLEN=-100000\tGT\t0/1\n" - "chr20\t3000000\tMantaINV:2\tN\t\t500\tPASS\tEND=4000000;SVTYPE=INV;SVLEN=1000000;INV3\tGT\t0/1\n" -) -DELLY = ( - "##fileformat=VCFv4.2\n#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT\ts1\n" - "chr20\t1000500\tDEL00000001\tN\t\t60\tPASS\tIMPRECISE;SVTYPE=DEL;END=1099000;CIPOS=-50,50;CIEND=-50,50\tGT:GQ\t0/1:40\n" - "chr20\t2000000\tDEL00000002\tN\t\t60\tPASS\tPRECISE;SVTYPE=DEL;END=2050000\tGT:GQ\t1/1:40\n" - "chr20\t2500000\tDEL00000003\tN\t\t10\tLowQual\tIMPRECISE;SVTYPE=DEL;END=2600000\tGT\t0/1\n" - "chr20\t3000000\tINV00000004\tN\t\t60\tPASS\tIMPRECISE;SVTYPE=INV;END=4000000;CT=3to3\tGT\t0/1\n" -) - - -def cnv(pos, rid, alt, end, p1, q0, gt): - t = alt[1:4] - return ("chr{0}\t{1}\t{2}\tN\t{3}\t.\tPASS\tEND={4};IMPRECISE;SVLEN={5};SVTYPE={6};pytorRD=0.5;pytorP1={7};" - "pytorP2=0;pytorP3=0;pytorP4=0;pytorQ0={8};pytorPN=0;pytorDG=1000;pytorCL=rd_mean_shift\tGT:CN\t{9}:1\n" - ).format(pos[0], pos[1], rid, alt, end, end - pos[1], t, p1, q0, gt) - - -CNVPYTOR = "##fileformat=VCFv4.2\n##source=CNVpytor\n#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT\ts1\n" + "".join([ - cnv(("20", 2001000), "CNVpytor_del0", "", 2049000, "1e-06", "0.01", "1/1"), # Delly's added call has it - cnv(("20", 3200000), "CNVpytor_del1", "", 3700000, "1e-08", "0.02", "0/1"), # inside Manta's INV: added - cnv(("20", 5000000), "CNVpytor_del2", "", 5100000, "0.5", "0.0", "0/1"), # e-val too high - cnv(("20", 6000000), "CNVpytor_del3", "", 6100000, "1e-08", "0.9", "0/1"), # q0 too high - cnv(("20", 7000000), "CNVpytor_loh4", "", 7100000, "1e-08", "0.0", "0/1"), # LOH - cnv(("1", 500000), "CNVpytor_dup5", "", 600000, "1e-05", "0.0", "0/1"), # added -]) - - -def test_extra_callers(w): - base, delly, cnvp = os.path.join(w, "base.vcf"), os.path.join(w, "delly.vcf.gz"), os.path.join(w, "cnv.vcf.gz") - with open(base, "w") as f: - f.write(BASE) - with gzip.open(delly, "wt") as f: - f.write(DELLY) - with gzip.open(cnvp, "wt") as f: - f.write(CNVPYTOR) - check("control: Manta's file alone lacks the CNVpytor deletion", "CNVpytor_del1" not in BASE) - - rc, out, err = add_callers("--base", base, "--delly", delly, "--cnvpytor", cnvp) - check("og sv add-callers exits 0", rc == 0, err) - recs = records(out) - ids = [r[2] for r in recs] - check("the added records are exactly Delly's DEL00000002 and CNVpytor's del1 and dup5", - ids == ["CNVpytor_dup5", "MantaDEL:1", "DEL00000002", "MantaINV:2", "CNVpytor_del1"], ids) - by_id = {r[2]: r for r in recs} - check("Manta's records name their caller", all(info(by_id[i]).get("SVCALLER") == "manta" for i in ("MantaDEL:1", "MantaINV:2"))) - d = by_id.get("DEL00000002") - check("Delly's call: SVCALLER=delly, , END kept, its genotype", - d is not None and info(d).get("SVCALLER") == "delly" and d[4] == "" and info(d).get("END") == "2050000" - and d[8:] == ["GT", "1/1"], d) - c = by_id.get("CNVpytor_del1") - check("CNVpytor's deletion: SVCALLER=cnvpytor, END, no SVLEN (its positive one), IMPRECISE, its genotype", - c is not None and info(c).get("SVCALLER") == "cnvpytor" and info(c).get("END") == "3700000" - and "SVLEN" not in info(c) and info(c).get("IMPRECISE") is True and c[8:] == ["GT", "0/1"], c) - c = by_id.get("CNVpytor_dup5") - check("CNVpytor's duplication is a ", c is not None and c[4] == "" and info(c).get("SVTYPE") == "DUP", c) - check("the header declares SVCALLER", "##INFO=\t500\t{0}\tEND=2049900;SVTYPE=DEL;SVLEN=-49800\tGT\t{1}\n" - .format(filt, gt)) - rc, out, err = add_callers("--base", b, "--delly", delly) - ids = [r[2] for r in records(out)] - check("{0} Manta deletion stays and does not hide Delly's DEL00000002".format(label), - rc == 0 and "MantaDEL:9" in ids and "DEL00000002" in ids, (ids, err)) - with open(b, "w") as f: - f.write(BASE + "chr20\t2000100\tMantaDEL:9\tN\t\t500\tPASS\tEND=2049900;SVTYPE=DEL;SVLEN=-49800\tGT\t0|1\n") - rc, out, err = add_callers("--base", b, "--delly", delly) - check("control: a PASS 0|1 Manta deletion still covers Delly's DEL00000002", - rc == 0 and "DEL00000002" not in [r[2] for r in records(out)], err) - sites = "".join("\t".join(l.split("\t")[:8]) + "\n" for l in (BASE + "chr20\t2000100\tMantaDEL:9\tN\t\t500\tPASS\tEND=2049900;SVTYPE=DEL\tGT\t0/0\n").splitlines()) - rc, out, err = add_callers("--base", "-", "--delly", delly, stdin=sites) - check("a sites-only base deletion covers Delly's DEL00000002 (no genotype to read)", - rc == 0 and "DEL00000002" not in [r[2] for r in records(out)], (records(out), err)) - - # A Delly PASS deletion genotyped 0/0 or ./. is no call either: it is not - # added, and CNVpytor's deletion at the same place is added instead. - for gt in ("0/0", "./."): - with gzip.open(delly, "wt") as f: - f.write(DELLY.replace("\t1/1:40\n", "\t{0}:40\n".format(gt))) - rc, out, err = add_callers("--base", base, "--delly", delly, "--cnvpytor", cnvp) - ids = [r[2] for r in records(out)] - check("a {0} Delly deletion is not added and does not hide CNVpytor_del0".format(gt), - rc == 0 and "DEL00000002" not in ids and "CNVpytor_del0" in ids, (ids, err)) - with gzip.open(delly, "wt") as f: - f.write(DELLY) - - rc, out, err = add_callers("--base", "-", "--base-name", "sv_vcf", stdin=BASE) - check("no other caller: stdin in, the base out with SVCALLER=sv_vcf", - rc == 0 and "none given" in err and [info(r).get("SVCALLER") for r in records(out)] == ["sv_vcf"] * 2, err) - - -DELLY_FMT = "GT:GL:GQ:FT:RCL:RC:RCR:RDCN:DR:DV:RR:RV" - - -def delly_rdcn(chrom, pos, rid, kind, end, rdcn, precise=False): - flag = "PRECISE" if precise else "IMPRECISE" - return ("{0}\t{1}\t{2}\tN\t<{3}>\t60\tPASS\t{4};SVTYPE={3};END={5}\t" + DELLY_FMT + - "\t0/1:-10,0,-10:40:PASS:100:100:100:{6}:10:10:0:0\n").format(chrom, pos, rid, kind, flag, end, rdcn) - - -DEPTH = "##fileformat=VCFv4.2\n#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT\ts1\n" + "".join([ - delly_rdcn("chr20", 5000000, "DEL_rd2", "DEL", 55000000, "2"), # 50 Mb, no loss: set aside - delly_rdcn("chr1", 10000000, "DEL_rd1", "DEL", 60000000, "1"), # 50 Mb, one copy: added - delly_rdcn("chr1", 100000000, "DUP_rd2", "DUP", 100500000, "2"), # no gain: set aside - delly_rdcn("chr1", 120000000, "DUP_rd3", "DUP", 120500000, "3"), # a gain: added - delly_rdcn("chr1", 150000000, "DEL_short", "DEL", 150000500, "2", precise=True), # 500 bp: added - delly_rdcn("chr1", 160000000, "DEL_rdneg", "DEL", 161000000, "-1"), # no estimate: set aside - delly_rdcn("chrX", 50000000, "DEL_x1", "DEL", 51000000, "1"), # male: normal, set aside - delly_rdcn("chrX", 60000000, "DUP_x2", "DUP", 60500000, "2"), # male: a gain; female: none - delly_rdcn("chrX", 1000000, "DEL_par1", "DEL", 1500000, "1"), # PAR: a loss for anyone -]) -PAR_BED = os.path.join(REPO, "assets", "par_grch38.bed") - - -def test_delly_depth(w): - base, delly = os.path.join(w, "base.vcf"), os.path.join(w, "delly_rdcn.vcf.gz") - with open(base, "w") as f: - f.write(BASE) - with gzip.open(delly, "wt") as f: - f.write(DEPTH) - planted = records(DEPTH) - check("control: every planted Delly call is PASS DEL or DUP", - all(r[6] == "PASS" and r[4] in ("", "") for r in planted) and len(planted) == 9) - - def added(*extra): - rc, out, err = add_callers("--base", base, "--delly", delly, *extra) - ids = {r[2] for r in records(out) if info(r).get("SVCALLER") == "delly"} - check("og sv add-callers {0}exits 0 and adds Delly calls".format(" ".join(extra[:2]) + " " if extra else ""), - rc == 0 and ids, err) - return rc, ids, out, err - - # No sex and no PAR file first: the arguments step 05 gave before. - rc, ids, _, err = added() - check("no sex: the 50 Mb deletion with RDCN 2 is set aside", "DEL_rd2" not in ids, ids) - check("no sex: the duplication with RDCN 2 is set aside", "DUP_rd2" not in ids, ids) - check("no sex: the long deletion with RDCN -1 is set aside", "DEL_rdneg" not in ids, ids) - check("no sex: the deletion with RDCN 1, the duplication with RDCN 3 and the short call are added", - {"DEL_rd1", "DUP_rd3", "DEL_short"} <= ids, ids) - check("no sex: the chrX deletion with RDCN 1 is added", "DEL_x1" in ids, ids) - check("no sex: the chrX duplication with RDCN 2 is added", "DUP_x2" in ids, ids) - - rc, ids, out, err = added("--sex", "male", "--par-bed", PAR_BED) - check("male: the 50 Mb deletion with RDCN 2 is set aside", "DEL_rd2" not in ids, ids) - check("male: the 50 Mb deletion with RDCN 1 is added", "DEL_rd1" in ids, ids) - check("male: the duplication with RDCN 2 is set aside", "DUP_rd2" not in ids, ids) - check("male: the duplication with RDCN 3 is added", "DUP_rd3" in ids, ids) - check("male: the short precise deletion with RDCN 2 is added", "DEL_short" in ids, ids) - check("male: the long deletion with RDCN -1 is set aside", "DEL_rdneg" not in ids, ids) - check("male: the chrX deletion with RDCN 1 is set aside (one copy is his normal)", "DEL_x1" not in ids, ids) - check("male: the chrX duplication with RDCN 2 is added", "DUP_x2" in ids, ids) - check("male: the PAR deletion with RDCN 1 is added", "DEL_par1" in ids, ids) - check("male: the summary counts the 4 calls set aside", - "delly: 5 PASS DEL/DUP calls, 5 added, and 4 PASS calls longer than 1000 bp set aside" in err, err) - check("male: the input VCF's header states the same, set-aside count included", - '##svcallers_added="Callers beside manta: delly: 5 PASS DEL/DUP calls, 5 added, and 4 PASS calls ' - 'longer than 1000 bp set aside' in out and out.index("##svcallers_added=") < out.index("#CHROM"), out[:400]) - - rc, ids, _, err = added("--sex", "female", "--par-bed", PAR_BED) - check("female: the chrX duplication with RDCN 2 is set aside", "DUP_x2" not in ids, ids) - check("female: the chrX deletion with RDCN 1 is added", "DEL_x1" in ids, ids) - - -def main(): - w = tempfile.mkdtemp() - try: - if OG: - test_inv_filter(w) - test_svlen(w) - test_extra_callers(w) - test_delly_depth(w) - else: - check("OG names an og binary", not os.environ.get("OG_REQUIRED"), "OG_REQUIRED is set") - print("skip: og sv inv-filter, svlen-fix and add-callers: OG is not set") - finally: - shutil.rmtree(w) - print("\nRESULT:", "ALL PASS" if FAILS == 0 else "{0} FAILED".format(FAILS)) - return 1 if FAILS else 0 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/tests/test_clinvar_join_adjacent.py b/tests/test_clinvar_join_adjacent.py deleted file mode 100755 index 733fa4828..000000000 --- a/tests/test_clinvar_join_adjacent.py +++ /dev/null @@ -1,427 +0,0 @@ -#!/usr/bin/env python3 -# og-leg: OG_REQUIRED -"""og clinvar join-adjacent, the combined-allele joiner of step 06 and the -CLINVAR_SCREEN module, on hand-made records at real GRCh38 chr22 positions. - -Checks 1 to 7 run og (OG, an og binary: the rust job of guard.yml sets it to -bin/og after scripts/setup.sh --og; without OG they are skipped and say so), -in its prepare and finish modes around the fill below, and in build mode, -which reads the reference itself and must write the same records: - 1. two records within 10 bp, both het with the same unphased GT, are joined - into one REF/ALT, the bases between them taken from the reference; - 0/1 and 1/0 are the same het call; - 2. phased records are joined only in the same phase set (FORMAT/PS) with - the same GT: other phase sets and the other haplotype are not joined; - 3. a het next to a hom, an unphased half call (./1), overlapping records - and records 11 bp apart are not joined; 10 bp apart are; a phased half - call (1|.) joins a record with its ALT on the same haplotype and keeps - "." on the other; - 4. a single SNV gives no combined record; - 5. finish drops a combined record whose part disagrees with the reference; - 6. the combined records come out in POS order; - 7. the tag pass marks each part of a matched combined allele "part of", - flags two hits in one gene that can be one allele, and leaves a single - SNV hit, hits in two genes and hits on two haplotypes unchanged; - 8. with docker (always in CI; elsewhere with PGP_TEST_DOCKER=1), the real - scripts/06-clinvar-screen.sh on the fixture records in - tests/fixtures/vcf/combined_allele_*.vcf: the 2-star ClinVar record of the combined allele is a hit, its two 1-star - parts are hits marked as its parts, and the single SNV hit is unchanged. - The delins written as two records is matched once, as the one combined - record (four hits in all), so this check is the proof that a delins split - differently from ClinVar's record is matched exactly once; - 9. with docker, the same step on a second sample: a 1/2 record at a poly-A - tract (CAAAA to C or CAA) whose first alt is a ClinVar allele is a hit - with GT 1/., worded "one copy, other copy not called ref", never het (on - the step before --multi-overlaps . it was 1/0, het); a plain 0/1 SNV hit - beside it stays het; and the always-report list (CLINVAR_ALWAYS_REPORT, - four sites) gives every site a call: het with its GQ, "ref, GQ 50" from a - gVCF block, the 1/. call, and "no call" with the reason. - -The window is 10 bp; set WINDOW to 0 in rust/crates/og-clinvar/src/join.rs -and checks 1, 2, 3, 6, 7 and 8 fail. - -Run: python3 tests/test_clinvar_join_adjacent.py -""" -import gzip -import os -import shutil -import subprocess -import sys -import tempfile - -REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) -OG = os.environ.get("OG") -FIXTURES = os.path.join(REPO, "tests", "fixtures", "vcf") - -# GRCh38 chr22 (the same in the NCBI no-ALT analysis set), two short stretches. -REF = {30000000: "AGACTCTGTCGCAAAAAAAAAATTAATGTAACCATTTAGCCATTACCCAGTTCCTTAAAAT", - 30000200: "GCCTGTAATCCCAGCACTTTGGGAGGCTGAG"} - -HEAD = ("##fileformat=VCFv4.2\n##contig=\n" - '##FORMAT=\n' - '##FORMAT=\n' - "#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT\tS\n") - -FAILS = 0 - - -def check(desc, ok, detail=""): - global FAILS - print(f"[{'PASS' if ok else 'FAIL'}] {desc}{'' if ok else ' -- ' + str(detail)}") - if not ok: - FAILS += 1 - - -def ref_at(pos, n): - for start, seq in REF.items(): - if start <= pos and pos + n <= start + len(seq): - return seq[pos - start:pos - start + n] - raise KeyError(f"chr22:{pos} (+{n}) is outside the test's reference stretches") - - -def rec(pos, ref, alt, gt, ps=None): - fmt, smp = ("GT", gt) if ps is None else ("GT:PS", f"{gt}:{ps}") - return f"chr22\t{pos}\t.\t{ref}\t{alt}\t50\tPASS\t.\t{fmt}\t{smp}\n" - - -def og_run(args, stdin=""): - p = subprocess.run([OG, "clinvar", "join-adjacent"] + args, input=stdin, capture_output=True, text=True) - return p.returncode, p.stdout, p.stderr - - -def write_reference(path): - """chr22 up to the last stretch, N everywhere else, 60 bases a line, with its .fai.""" - length = max(s + len(q) for s, q in REF.items()) + 50 - seq = bytearray(b"N" * length) - for start, q in REF.items(): - seq[start - 1:start - 1 + len(q)] = q.encode() - with open(path, "wb") as f: - f.write(b">chr22\n") - for i in range(0, length, 60): - f.write(bytes(seq[i:i + 60]) + b"\n") - with open(path + ".fai", "w") as f: - f.write(f"chr22\t{length}\t7\t60\t61\n") - - -def fill(vcf): - """What `bcftools +fill-from-fasta -- -c INFO/PGP_JOIN_SEQ` does: the - reference bases from POS for the length of REF.""" - out = [] - for line in vcf.splitlines(True): - if line.startswith("#"): - out.append(line) - continue - f = line.rstrip("\n").split("\t") - f[7] = f[7].replace("PGP_JOIN_SEQ=.", "PGP_JOIN_SEQ=" + ref_at(int(f[1]), len(f[3])), 1) - out.append("\t".join(f) + "\n") - return "".join(out) - - -def join(fasta, records, fill_fn=fill): - """prepare | fill | finish; the records out, as lists of fields. build - must write the same records.""" - rc, prep, err = og_run(["--mode", "prepare"], HEAD + "".join(records)) - if rc != 0: - return None, f"prepare exit {rc}: {err}" - rc, done, err = og_run(["--mode", "finish"], fill_fn(prep)) - if rc != 0: - return None, f"finish exit {rc}: {err}" - rc, built, err = og_run(["--mode", "build", "--fasta", fasta], HEAD + "".join(records)) - body = [l for l in done.splitlines() if l and not l.startswith("#")] - if rc != 0 or [l for l in built.splitlines() if l and not l.startswith("#")] != body: - return None, f"build exit {rc}: {err}, wrote {built!r}, not the records of prepare and finish {done!r}" - return [l.split("\t") for l in body], done - - -def core(r): - return (r[1], r[3], r[4], r[8], r[9]) - - -def og_checks(name, fasta, work): - # 1. unphased het pair, 3 bp apart: AGC>A (a 2 bp deletion) and T>TGGG - out, raw = join(fasta, [rec(30000037, "AGC", "A", "0/1"), rec(30000042, "T", "TGGG", "0/1")]) - check(f"{name}: two het records 3 bp apart become one REF/ALT with the reference bases between them", - out is not None and [core(r) for r in out] == [("30000037", "AGCCAT", "ACATGGG", "GT", "0/1")], raw) - check(f"{name}: the combined record names its two parts", - out is not None and len(out) == 1 - and out[0][7] == "PGP_JOINED_FROM=chr22:30000037:AGC:A|chr22:30000042:T:TGGG", raw) - check(f"{name}: the header keeps PGP_JOINED_FROM and drops the temporary PGP_JOIN_SEQ", - "ID=PGP_JOINED_FROM" in raw and "ID=PGP_JOIN_SEQ" not in raw, raw) - out, raw = join(fasta, [rec(30000203, "T", "A", "1/0"), rec(30000208, "T", "C", "0/1")]) - check(f"{name}: 1/0 and 0/1 (unphased) are the same het call and are joined", - out is not None and [core(r) for r in out] == [("30000203", "TGTAAT", "AGTAAC", "GT", "0/1")], raw) - - # 2. phase sets - out, raw = join(fasta, [rec(30000203, "T", "A", "0|1", 100), rec(30000208, "T", "C", "0|1", 100)]) - check(f"{name}: phased, same phase set and GT: joined, with GT and PS", - out is not None and [core(r) for r in out] == [("30000203", "TGTAAT", "AGTAAC", "GT:PS", "0|1:100")], raw) - out, raw = join(fasta, [rec(30000203, "T", "A", "0|1", 100), rec(30000208, "T", "C", "0|1", 200)]) - check(f"{name}: phased on different phase sets: not joined", out == [], raw) - out, raw = join(fasta, [rec(30000203, "T", "A", "0|1", 100), rec(30000208, "T", "C", "1|0", 100)]) - check(f"{name}: phased on the two haplotypes of one phase set: not joined", out == [], raw) - out, raw = join(fasta, [rec(30000203, "T", "A", "0|1", 100), rec(30000208, "T", "C", "0/1")]) - check(f"{name}: one phased and one unphased record: not joined", out == [], raw) - - # 3. what is not one allele, and the window's edge - out, raw = join(fasta, [rec(30000203, "T", "A", "0/1"), rec(30000208, "T", "C", "1/1")]) - check(f"{name}: a het next to a hom: not joined", out == [], raw) - out, raw = join(fasta, [rec(30000203, "T", "A", "./1"), rec(30000208, "T", "C", "./1")]) - check(f"{name}: unphased half calls (./1): not joined", out == [], raw) - out, raw = join(fasta, [rec(30000203, "T", "A", "1|.", 100), rec(30000208, "T", "C", "1|0", 100)]) - check(f"{name}: a phased half call (1|.) and 1|0 in one phase set: joined, '.' kept on the other haplotype", - out is not None and [core(r) for r in out] == [("30000203", "TGTAAT", "AGTAAC", "GT:PS", "1|.:100")], raw) - out, raw = join(fasta, [rec(30000203, "T", "A", "1|0", 100), rec(30000208, "T", "C", "1|.", 100)]) - check(f"{name}: 1|0 then a phased half call (1|.): joined, '.' kept on the other haplotype", - out is not None and [core(r) for r in out] == [("30000203", "TGTAAT", "AGTAAC", "GT:PS", "1|.:100")], raw) - out, raw = join(fasta, [rec(30000203, "T", "A", ".|1", 100), rec(30000208, "T", "C", ".|1", 100)]) - check(f"{name}: two phased half calls on one haplotype: joined", - out is not None and [core(r) for r in out] == [("30000203", "TGTAAT", "AGTAAC", "GT:PS", ".|1:100")], raw) - out, raw = join(fasta, [rec(30000203, "T", "A", "1|.", 100), rec(30000208, "T", "C", "0|1", 100)]) - check(f"{name}: a phased half call and an ALT on the other haplotype: not joined", out == [], raw) - out, raw = join(fasta, [rec(30000203, "TGT", "T", "0/1"), rec(30000204, "G", "A", "0/1")]) - check(f"{name}: overlapping records: not joined", out == [], raw) - out, raw = join(fasta, [rec(30000203, "T", "A", "0/1"), rec(30000213, "G", "T", "0/1")]) - check(f"{name}: 10 bp apart: joined", - out is not None and [core(r)[:3] for r in out] == [("30000203", "TGTAATCCCAG", "AGTAATCCCAT")], raw) - out, raw = join(fasta, [rec(30000203, "T", "A", "0/1"), rec(30000214, "C", "A", "0/1")]) - check(f"{name}: 11 bp apart: not joined", out == [], raw) - - # 4. a single SNV - out, raw = join(fasta, [rec(30000210, "C", "T", "0/1")]) - check(f"{name}: a single SNV gives no combined record", out == [], raw) - - # 5. a part whose REF is not the reference's base (30000203 is T) - out, raw = join(fasta, [rec(30000203, "C", "A", "0/1"), rec(30000208, "T", "C", "0/1")]) - check(f"{name}: a part that disagrees with the reference: dropped by finish", out == [], raw) - - # 6. POS order: a 12 bp deletion that ends after the two records inside it - recs = [rec(30000201, "CCTGTAATCCCA", "C", "0/1"), rec(30000205, "T", "G", "0/1"), - rec(30000208, "T", "C", "0/1"), rec(30000214, "C", "A", "0/1")] - out, raw = join(fasta, recs) - pos = [int(r[1]) for r in out] if out else [] - check(f"{name}: four combined records, in POS order", pos == [30000201, 30000205, 30000205, 30000208], raw) - - # mode is required - rc, _, err = og_run([], HEAD) - check(f"{name}: no mode: exit 2 with a message", rc == 2 and "mode" in err, f"exit {rc}: {err}") - - # 7. the tag pass on a hits VCF - hits = os.path.join(work, f"hits-{name.replace(' ', '_')}.vcf") - cv = ";GENEINFO={g};CLNSIG=Pathogenic;CLNREVSTAT=criteria_provided,_single_submitter" - lines = [ - "chr22\t30000037\t9900001\tAGC\tA\t50\tPASS\t" + cv.format(g="SYNTH1:1")[1:] + "\tGT\t0/1", - "chr22\t30000038\t9900003\tGCCAT\tCATGGG\t.\tPASS\tPGP_JOINED_FROM=chr22:30000037:AGC:A|chr22:30000042:T:TGGG" - + cv.format(g="SYNTH1:1").replace("single_submitter", "multiple_submitters,_no_conflicts") + "\tGT\t0/1", - "chr22\t30000042\t9900002\tT\tTGGG\t50\tPASS\t" + cv.format(g="SYNTH1:1")[1:] + "\tGT\t0/1", - "chr22\t30000210\t9900004\tC\tT\t50\tPASS\t" + cv.format(g="SYNTH2:2")[1:] + "\tGT\t0/1", - "chr22\t30000300\t9900005\tA\tG\t50\tPASS\t" + cv.format(g="SYNTH3:3")[1:] + "\tGT\t0/1", - "chr22\t30000305\t9900006\tC\tT\t50\tPASS\t" + cv.format(g="SYNTH3:3")[1:] + "\tGT\t0/1", - "chr22\t30000400\t9900007\tA\tG\t50\tPASS\t" + cv.format(g="SYNTH4:4")[1:] + "\tGT\t0/1", - "chr22\t30000404\t9900008\tC\tT\t50\tPASS\t" + cv.format(g="SYNTH5:5")[1:] + "\tGT\t0/1", - "chr22\t30000500\t9900009\tA\tG\t50\tPASS\t" + cv.format(g="SYNTH6:6")[1:] + "\tGT:PS\t0|1:7", - "chr22\t30000504\t9900010\tC\tT\t50\tPASS\t" + cv.format(g="SYNTH6:6")[1:] + "\tGT:PS\t1|0:7", - "chr22\t30000600\t9900011\tA\tG\t50\tPASS\t" + cv.format(g="SYNTH7:7")[1:] + "\tGT:PS\t1|.:8", - "chr22\t30000605\t9900012\tC\tT\t50\tPASS\t" + cv.format(g="SYNTH7:7")[1:] + "\tGT:PS\t1|0:8", - ] - with open(hits, "w") as f: - f.write(HEAD + "\n".join(lines) + "\n") - rc, out, err = og_run(["--mode", "tag", hits]) - got = [l for l in out.splitlines() if l and not l.startswith("#")] - info = {tuple(l.split("\t")[1:2]): l.split("\t")[7] for l in got} - check(f"{name}: tag: exits 0, header and every record kept", - rc == 0 and len(got) == len(lines) and out.startswith("##fileformat"), f"exit {rc}: {err}") - part = "PGP_PART_OF=chr22:30000038:GCCAT:CATGGG" - check(f"{name}: tag: both parts are marked part of the combined allele", - info.get(("30000037",), "").endswith(part) and info.get(("30000042",), "").endswith(part), info) - check(f"{name}: tag: the combined hit itself gets no part mark", - "PGP_PART_OF" not in info.get(("30000038",), "") and "PGP_ONE_ALLELE" not in info.get(("30000038",), ""), - info) - check(f"{name}: tag: the single SNV hit is unchanged", lines[3] in got, info) - check(f"{name}: tag: two hits in one gene, 5 bp apart, no combined record: flagged both ways", - info.get(("30000300",), "").endswith("PGP_ONE_ALLELE=chr22:30000305:C:T") - and info.get(("30000305",), "").endswith("PGP_ONE_ALLELE=chr22:30000300:A:G"), info) - check(f"{name}: tag: hits in two genes, or on the two haplotypes of a phase set, are unchanged", - all(x in got for x in lines[6:10]), info) - check(f"{name}: tag: a phased half call (1|.) and 1|0 in one gene and phase set: flagged both ways", - info.get(("30000600",), "").endswith("PGP_ONE_ALLELE=chr22:30000605:C:T") - and info.get(("30000605",), "").endswith("PGP_ONE_ALLELE=chr22:30000600:A:G"), info) - - -POLY_HEAD = ("##fileformat=VCFv4.2\n##contig=\n" - '##INFO=\n' - '##FORMAT=\n' - '##FORMAT=\n' - '##FORMAT=\n' - "#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT\tS2\n") - - -def polyt_checks(gd, image, env): - """Check 9: a split multiallelic poly-A call, and the always-report list.""" - vcf = os.path.join(gd, "S2", "vcf") - os.makedirs(vcf, exist_ok=True) - tract = ref_at(30000011, 5) - check("poly-A: the reference holds C and a run of A at chr22:30000011", tract == "CAAAA", tract) - with open(os.path.join(vcf, "S2.vcf"), "w") as f: - f.write(POLY_HEAD + f"chr22\t30000011\t.\t{tract}\tC,CAA\t50\tPASS\t.\tGT:GQ:DP\t1/2:45:30\n" - "chr22\t30000210\t.\tC\tT\t50\tPASS\t.\tGT:GQ:DP\t0/1:40:30\n") - with open(os.path.join(vcf, "S2.g.vcf"), "w") as f: - f.write(POLY_HEAD + "chr22\t30000001\t.\tA\t<*>\t0\t.\tEND=30000010\tGT:GQ:DP\t0/0:50:30\n" - f"chr22\t30000011\t.\t{tract}\tC,CAA,<*>\t50\tPASS\t.\tGT:GQ:DP\t1/2:45:30\n" - "chr22\t30000016\t.\tA\t<*>\t0\t.\tEND=30000209\tGT:GQ:DP\t0/0:50:30\n" - "chr22\t30000210\t.\tC\tT,<*>\t50\tPASS\t.\tGT:GQ:DP\t0/1:40:30\n") - # ClinVar's allele: the 4-base deletion of the tract, first among the records. - cv = os.path.join(gd, "clinvar", "clinvar_pathogenic_chr.vcf") - lines = open(cv).readlines() - first = next(i for i, l in enumerate(lines) if not l.startswith("#")) - lines.insert(first, f"chr22\t30000011\t9900005\t{tract}\tC\t.\t.\tGENEINFO=SYNTH4:990004;CLNSIG=Pathogenic;" - "CLNREVSTAT=criteria_provided,_single_submitter\n") - with open(cv, "w") as f: - f.writelines(lines) - sites = os.path.join(gd, "sites.tsv") - with open(sites, "w") as f: - f.write("# four sites of the test\nchrom\tpos\tref\talt\tgene\thgvs\tclinvar_id\n" - "chr1\t100\tA\tG\tSYNTHA\tc.1A>G\t1\n" - f"chr22\t30000011\t{tract}\tC\tSYNTH4\tc.2del\t9900005\n" - "chr22\t30000030\tA\tG\tSYNTHB\tc.3A>G\t2\n" - "chr22\t30000210\tC\tT\tSYNTH2\tc.4C>T\t9900004\n") - uid = f"{os.getuid()}:{os.getgid()}" - p = subprocess.run(["docker", "run", "--rm", "--user", uid, "-v", f"{gd}:/g", image, "sh", "-c", - "set -e; for v in S2.vcf S2.g.vcf; do bcftools view -Oz -o /g/S2/vcf/$v.gz /g/S2/vcf/$v;" - " bcftools index -f -t /g/S2/vcf/$v.gz; done;" - " bcftools view -Oz -o /g/clinvar/clinvar_pathogenic_chr.vcf.gz /g/clinvar/clinvar_pathogenic_chr.vcf;" - " bcftools index -f -t /g/clinvar/clinvar_pathogenic_chr.vcf.gz"], - capture_output=True, text=True) - check("poly-A: the files compress and index", p.returncode == 0, p.stderr) - e = dict(env, CLINVAR_ALWAYS_REPORT=sites) - p = subprocess.run(["bash", os.path.join(REPO, "scripts", "06-clinvar-screen.sh"), "S2"], - capture_output=True, text=True, env=e) - log = p.stdout + p.stderr - print(" | " + log.rstrip("\n").replace("\n", "\n | ")) - check("poly-A: step 06 exits 0", p.returncode == 0, f"exit {p.returncode}") - hits_path = os.path.join(gd, "S2", "clinvar", "S2_clinvar_hits.vcf") - rows = [] - if os.path.isfile(hits_path): - rows = [l.rstrip("\n").split("\t") for l in open(hits_path) if not l.startswith("#")] - by = {(r[1], r[3], r[4]): r for r in rows} - poly = by.get(("30000011", tract, "C")) - check("poly-A: the ClinVar allele of the 1/2 record is one hit, with GT 1/. (the other copy holds CAA)", - poly is not None and poly[9].split(":")[0] == "1/." and len(rows) == 2, rows) - words = subprocess.run([OG, "clinvar", "hits", hits_path], - capture_output=True, text=True).stdout if os.path.isfile(hits_path) else "" - zyg = {r.split("\t")[2]: r.split("\t")[5] for r in words.splitlines()} - check("poly-A: the hit is not worded het", zyg.get("30000011") == "one copy, other copy not called ref (1/.)", zyg) - check("poly-A: the plain 0/1 SNV hit beside it is still het", zyg.get("30000210") == "het", zyg) - check("poly-A: the step's log does not call the tract hit het", - not any("30000011" in l and " het " in l for l in log.splitlines()), "") - deep = os.path.join(gd, "S2", "clinvar", "S2_clinvar_deep_intronic.tsv") - got = {} - if os.path.isfile(deep): - got = {l.split("\t")[1]: tuple(l.rstrip("\n").split("\t")[7:9]) for l in list(open(deep))[1:]} - check("always-report: one row per site, all four", - sorted(got) == ["100", "30000011", "30000030", "30000210"], got) - check("always-report: the carried SNV gives het with its GQ and DP", got.get("30000210") == ("het", "GQ 40, DP 30"), got) - check("always-report: a site inside a gVCF reference block gives ref with the block's GQ", - got.get("30000030") == ("ref", "GQ 50 (gVCF block)"), got) - check("always-report: the poly-A site gives the 1/. call, not het", - got.get("30000011") == ("one copy, other copy not called ref (1/.)", "GQ 45, DP 30"), got) - check("always-report: a site no record covers gives no call, and why", - got.get("100") == ("no call", "no record in the VCF or the gVCF covers the site"), got) - check("always-report: the log lists the sites", "SYNTHB c.3A>G chr22:30000030 A>G: ref, GQ 50 (gVCF block)" in log, "") - - -def docker_checks(work): - """Check 8: the real step 06 with the real bcftools image.""" - env = dict(os.environ) - versions = {} - with open(os.path.join(REPO, "versions.env")) as f: - for line in f: - if line.startswith("BCFTOOLS_IMAGE="): - versions["BCFTOOLS_IMAGE"] = line.split("=", 1)[1].split("#")[0].strip().strip('"') - image = versions["BCFTOOLS_IMAGE"] - gd = os.path.join(work, "genome") - for d in ("reference", "clinvar", "S1/vcf"): - os.makedirs(os.path.join(gd, d), exist_ok=True) - write_reference(os.path.join(gd, "reference", "GRCh38_no_alt_analysis_set.fasta")) - shutil.copy(os.path.join(FIXTURES, "combined_allele_sample.vcf"), os.path.join(gd, "S1", "vcf", "S1.vcf")) - shutil.copy(os.path.join(FIXTURES, "combined_allele_clinvar.vcf"), - os.path.join(gd, "clinvar", "clinvar_pathogenic_chr.vcf")) - uid = f"{os.getuid()}:{os.getgid()}" - p = subprocess.run(["docker", "run", "--rm", "--user", uid, "-v", f"{gd}:/g", image, "sh", "-c", - "set -e; bcftools view -Oz -o /g/S1/vcf/S1.vcf.gz /g/S1/vcf/S1.vcf;" - " bcftools index -t /g/S1/vcf/S1.vcf.gz;" - " bcftools view -Oz -o /g/clinvar/clinvar_pathogenic_chr.vcf.gz /g/clinvar/clinvar_pathogenic_chr.vcf;" - " bcftools index -t /g/clinvar/clinvar_pathogenic_chr.vcf.gz"], - capture_output=True, text=True) - check("step 06: the fixture files compress and index", p.returncode == 0, p.stderr) - env["GENOME_DIR"] = gd - p = subprocess.run(["bash", os.path.join(REPO, "scripts", "06-clinvar-screen.sh"), "S1"], - capture_output=True, text=True, env=env) - log = p.stdout + p.stderr - print(" | " + log.rstrip("\n").replace("\n", "\n | ")) - check("step 06: exits 0", p.returncode == 0, f"exit {p.returncode}") - hits_path = os.path.join(gd, "S1", "clinvar", "S1_clinvar_hits.vcf") - rows = [] - if os.path.isfile(hits_path): - with open(hits_path) as f: - rows = [l.rstrip("\n").split("\t") for l in f if not l.startswith("#")] - by = {(r[1], r[3], r[4]): r for r in rows} - comb = by.get(("30000038", "GCCAT", "CATGGG")) - check("step 06: the 2-star ClinVar record of the combined allele is a hit", - comb is not None and comb[2] == "9900003" - and "CLNREVSTAT=criteria_provided,_multiple_submitters,_no_conflicts" in comb[7] - and "PGP_JOINED_FROM=chr22:30000037:AGC:A|chr22:30000042:T:TGGG" in comb[7], rows) - parts = [by.get(("30000037", "AGC", "A")), by.get(("30000042", "T", "TGGG"))] - check("step 06: both 1-star parts are hits, marked part of the combined allele", - all(x is not None and "PGP_PART_OF=chr22:30000038:GCCAT:CATGGG" in x[7] for x in parts), rows) - snv = by.get(("30000210", "C", "T")) - check("step 06: the single SNV hit is unchanged (its ClinVar fields, no combined-allele tag)", - snv is not None and snv[2] == "9900004" and "PGP_" not in snv[7], rows) - check("step 06: four hits in all", len(rows) == 4, rows) - tsv = os.path.join(gd, "S1", "clinvar", "S1_clinvar_hits.tsv") - n_tsv = sum(1 for _ in open(tsv)) - 1 if os.path.isfile(tsv) else -1 - check("step 06: the hits TSV has the same four rows", n_tsv == 4, n_tsv) - check("step 06: the log says which hits are parts of the combined allele", - "part of a combined allele: chr22:30000038 GCCAT>CATGGG" in log - and "combined allele of chr22:30000037 AGC>A + chr22:30000042 T>TGGG" in log, "") - polyt_checks(gd, image, env) - # Files written by the containers belong to the caller (run_in --user). - subprocess.run(["docker", "run", "--rm", "-v", f"{work}:/w", image, "sh", "-c", "rm -rf /w/genome"], - capture_output=True) - - -def main(): - work = tempfile.mkdtemp() - try: - if not OG: - print("[SKIP] OG is not set: og clinvar join-adjacent is checked by the rust job of guard.yml" - " (OG=bin/og after scripts/setup.sh --og)") - return 0 - fasta = os.path.join(work, "ref.fa") - write_reference(fasta) - og_checks("og", fasta, work) - # Check 8 pulls and runs BCFTOOLS_IMAGE, and runs bin/og inside it: - # always in CI, elsewhere only when asked for, so a quick local run - # stays light. - if os.environ.get("CI") or os.environ.get("PGP_TEST_DOCKER") == "1": - # OG can be a host og (unit-tests sets one); step 06 needs bin/.og. - if subprocess.run([os.path.join(REPO, "bin", "og"), "version"], capture_output=True).returncode != 0: - if os.environ.get("OG_REQUIRED"): - check("bin/og is built (check 8 runs the real step 06 with it)", False, "run scripts/setup.sh --og") - else: - print("[SKIP] check 8: og is not built for this machine, and step 06 runs it" - " (scripts/setup.sh --og): the rust job of guard.yml runs it") - elif shutil.which("docker"): - docker_checks(work) - else: - check("docker is installed (check 8 runs the real step 06)", False, "no docker on PATH") - else: - print("[SKIP] check 8: set PGP_TEST_DOCKER=1 to run the real step 06 here (CI always runs it)") - finally: - shutil.rmtree(work, ignore_errors=True) - print() - print("RESULT: ALL PASS" if FAILS == 0 else f"RESULT: {FAILS} FAILED") - return 1 if FAILS else 0 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/tests/test_str_catalog.py b/tests/test_str_catalog.py deleted file mode 100644 index 051ceb690..000000000 --- a/tests/test_str_catalog.py +++ /dev/null @@ -1,379 +0,0 @@ -#!/usr/bin/env python3 -# og-leg: OG_REQUIRED -"""og str catalog, and Stranger on the catalog it builds. - -Reads the two STRchive files versions.env pins (STRCHIVE_VERSION and the -two sha256), downloaded once into a cache directory (STRCHIVE_DIR, default -/pgp-strchive-) and checked against their sha256, and checks: - - 1. the files: a copy with one byte flipped fails the sha256 check (the - control that the check can fail); - 2. the converter: one entry per STRchive ExpansionHunter locus (82), no - name twice, every ExpansionHunter LocusId traced to one entry by its - STRchive id; the report's key loci keep their names (HTT, FMR1, - C9ORF72, ATXN1, DMPK, RFC1); the thresholds of HTT, FMR1 and the FAME - locus SAMD12; the loci without thresholds and why; and a planted extra - ExpansionHunter locus with no STRchive record stops it (exit 2, and - neither the catalog nor the ranges table is written); - 3. (moved) the motif count on the catalog's motifs is og str motif now; - tests/e2e/str-catalog-1-strchive.sh runs it through step 9b on the - real catalog (RFC1 pathogenic_motif with the built-in shares, SAMD12 - pathogenic_motif for TTTCA reads and benign_motif for TTTTA reads, the - reference reads not counted), and rust/crates/og-bam/tests/parity.rs - holds og to the Python it replaced on a made-up catalog; - 4. Stranger (STRANGER_IMAGE, through docker) on the catalog: HTT at its - benign maximum, one above it and at its pathogenic minimum gives - normal, pre_mutation and full_mutation; SAMD12 at the reference count - normal and one above pre_mutation; MUC1 no status; and the control: - Stranger's bundled GRCh38 catalog does not call HTT 27 pre_mutation, so - the check tells the two catalogs apart; - 5. the repeats STRchive leaves out (assets/str_extra_loci.json, --extra): - the merged catalog is STRchive's 82 entries plus every extra locus, - copied as it is, with ranges rows and no Stranger thresholds; each - extra region is, in GRCh38 (NCBI's GRCh38.p14 chromosome sequence), a - whole run of its motif that the bases beside it do not continue; and - the controls: an extra catalog missing UGT1A1 gives a merged catalog - the missing-locus check flags, and a planted entry with no - NoThresholds, one whose region is not a whole number of motifs, one - named like a STRchive locus and one with Stranger thresholds each stop - the converter (exit 2, nothing written), as does a region shifted by - one base in the sequence check. - -og str catalog runs from OG (the og binary, as in the other og tests); -without OG the test is skipped with a note, and with OG_REQUIRED set too it -fails. The parity cases of tests/parity (tool str-catalog) compare og with -the Python it replaced. - -Run: OG=rust/target/release/og python3 tests/test_str_catalog.py -""" -import hashlib -import json -import os -import re -import shutil -import subprocess -import sys -import tempfile -import time -import urllib.request - -REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) -OG = os.environ.get("OG", "") - - -class sc: # noqa: N801 (the module name the checks below were written against) - """og str catalog, and the two catalog readers the checks use.""" - - @staticmethod - def main(argv): - return subprocess.run([OG, "str", "catalog", *argv]).returncode - - @staticmethod - def motifs_of(structure): - return re.findall(r"\(([^)]*)\)\*", structure) - - @staticmethod - def as_list(x): - return x if isinstance(x, list) else [x] - - @staticmethod - def region(text): - chrom, span = text.split(":") - a, b = span.split("-") - return chrom, int(a), int(b) - - -def og_or_skip(): - """None when OG names an og binary; else the exit code: 0 (skip, with a - note), or 1 with OG_REQUIRED set.""" - if OG and os.access(OG, os.X_OK): - return None - print("skip: OG is not set (OG=rust/target/release/og)") - return 1 if os.environ.get("OG_REQUIRED") else 0 - -FAILS = [] - - -def check(desc, ok, detail=""): - print(f"{'PASS' if ok else 'FAIL'} {desc}" + ("" if ok else f": {detail}")) - if not ok: - FAILS.append(desc) - - -def env(name): - for line in open(os.path.join(REPO, "versions.env")): - if line.startswith(name + "="): - return line.split("=", 1)[1].split("#")[0].strip().strip('"') - raise SystemExit(f"FAIL no {name} line in versions.env") - - -def sha256(path): - h = hashlib.sha256() - with open(path, "rb") as f: - for block in iter(lambda: f.read(1 << 20), b""): - h.update(block) - return h.hexdigest() - - -def files(): - """(release, EH catalog path, loci path), downloaded and checked.""" - rel = env("STRCHIVE_VERSION") - d = os.environ.get("STRCHIVE_DIR") or os.path.join(tempfile.gettempdir(), f"pgp-strchive-{rel}") - os.makedirs(d, exist_ok=True) - out = [] - for name, var in ((f"STRchive-disease-loci-{rel}.hg38.expansionhunter.json", "STRCHIVE_EH_SHA256"), - (f"STRchive-loci-{rel}.json", "STRCHIVE_LOCI_SHA256")): - path = os.path.join(d, name) - if not (os.path.exists(path) and sha256(path) == env(var)): - url = f"https://github.com/dashnowlab/STRchive/releases/download/{rel}/{name}" - for i in range(3): - try: - with urllib.request.urlopen(url, timeout=60) as r, open(path + ".part", "wb") as f: - shutil.copyfileobj(r, f) - break - except OSError as e: - print(f"download {i + 1}/3 of {url} failed: {e}") - time.sleep(5 * (i + 1)) - os.replace(path + ".part", path) - if sha256(path) != env(var): - raise SystemExit(f"FAIL {name} does not have the sha256 {var} pins") - out.append(path) - return rel, out[0], out[1] - - -def build_catalog(): - """The converted catalog (list of entries) and its path, in a temp file; the - ranges table (--ranges) is str_ranges.tsv beside it.""" - rel, eh, loci = files() - out = os.path.join(tempfile.mkdtemp(), "catalog.json") - rc = sc.main(["--eh", eh, "--loci", loci, "--release", rel, "--out", out, - "--ranges", os.path.join(os.path.dirname(out), "str_ranges.tsv")]) - if rc != 0: - raise SystemExit("FAIL the converter failed on the pinned files") - return json.load(open(out)), out - - -def part_files(rel, eh, loci): - with tempfile.TemporaryDirectory() as tmp: - bad = os.path.join(tmp, "flipped.json") - data = bytearray(open(eh, "rb").read()) - data[len(data) // 2] ^= 1 - open(bad, "wb").write(data) - check("control: a copy with one byte flipped fails the sha256 check", - sha256(bad) != env("STRCHIVE_EH_SHA256")) - - -def part_convert(rel, eh, loci, cat, cat_path): - src = json.load(open(eh)) - check("STRchive's ExpansionHunter catalog has 82 loci", len(src) == 82, len(src)) - check("the catalog has one entry per ExpansionHunter locus", len(cat) == len(src), len(cat)) - by_src = {} - for e in cat: - by_src.setdefault(e["SourceId"], []).append(e["LocusId"]) - check("every ExpansionHunter LocusId has exactly one entry", - sorted(by_src) == sorted(x["LocusId"] for x in src) and all(len(v) == 1 for v in by_src.values()), - {k: v for k, v in by_src.items() if len(v) != 1}) - names = [v for e in cat for v in (e.get("VariantId") if isinstance(e.get("VariantId"), list) else [e["LocusId"]])] - check("no variant name twice", len(names) == len(set(names)), sorted(n for n in names if names.count(n) > 1)) - by = {e["LocusId"]: e for e in cat} - check("the report's key loci and RFC1 keep their names", - {"HTT", "FMR1", "C9ORF72", "ATXN1", "DMPK", "RFC1"} <= set(by), sorted(by)) - check("one gene with two loci keeps STRchive's ids", {"EIEE1_ARX", "PRTS_ARX", "HFG_HOXA13-I"} <= set(by)) - htt = by["HTT"] - check("HTT: normal to 26, pathogenic from 36 (STRchive), the CAG repeat is REPID HTT", - (htt.get("NormalMax"), htt.get("PathologicMin"), htt.get("VariantId")) == (26, 36, ["HTT", "HTT_CCG"]), htt) - check("FMR1: normal to 44, pathogenic from 201", (by["FMR1"].get("NormalMax"), by["FMR1"].get("PathologicMin")) - == (44, 201), by["FMR1"]) - s12 = by["SAMD12"] - check("SAMD12 (FAME1): no benign maximum in STRchive, so normal is the reference count (1 TGAAA), pathogenic " - "from 105; TTTTA benign, TTTCA pathogenic", - (s12.get("NormalMax"), s12.get("PathologicMin"), s12.get("BenignMotifs"), s12.get("PathogenicMotifs"), - s12.get("VariantId")) == (1, 105, ["TAAAA"], ["TGAAA"], ["SAMD12_TAAAA", "SAMD12"]), s12) - check("RFC1: AAAGG is counted benign, CAGGG pathogenic", - (by["RFC1"].get("BenignMotifs"), by["RFC1"].get("PathogenicMotifs")) - == (["AAAAG", "AAAGG"], ["AAGGG", "ACAGG", "CAGGG"]), by["RFC1"]) - check("motif loci: RFC1, the FAME loci, SCA31 and SCA37", - {"RFC1", "SAMD12", "STARD7", "YEATS2", "MARCHF6", "RAPGEF2", "TNRC6A", "RAI1", "BEAN1", "DAB1"} - <= {e["LocusId"] for e in cat if "PathogenicMotifs" in e}) - none = {e["LocusId"]: e["NoThresholds"] for e in cat if "NoThresholds" in e} - check("no thresholds: the four repeats longer than two reads, two without a pathogenic size, one contraction", - sorted(none) == ["CEL", "MIR7-2", "MUC1", "PLIN4", "POLG", "TYMS", "VWA1"], none) - check("a locus without thresholds has no NormalMax, so Stranger gives it no status", - not any("NormalMax" in by[k] or "PathologicMin" in by[k] for k in none)) - check("every threshold pair is normal below pathogenic", - all(e["NormalMax"] < e["PathologicMin"] for e in cat if "NormalMax" in e)) - check("the release is in SourceDisplay", by["HTT"].get("SourceDisplay") == f"STRchive-{rel}", by["HTT"]) - with tempfile.TemporaryDirectory() as tmp: - planted = os.path.join(tmp, "eh.json") - json.dump(src + [{"LocusId": "PLANTED_GENE", "LocusStructure": "(CAG)*", - "ReferenceRegion": "chr1:1000-1030", "VariantType": "Repeat"}], open(planted, "w")) - out, rng = os.path.join(tmp, "out.json"), os.path.join(tmp, "ranges.tsv") - rc = sc.main(["--eh", planted, "--loci", loci, "--release", rel, "--out", out, "--ranges", rng]) - check("control: a planted ExpansionHunter locus with no STRchive record stops the converter (exit 2, " - "no catalog and no ranges table)", rc == 2 and not os.path.exists(out) and not os.path.exists(rng), rc) - - -# --- 4. Stranger on the catalog ----------------------------------------------------------- -FMT = "GT:SO:REPCN:REPCI:ADSP:ADFL:ADIR:LC" - - -def stranger(tmp, catalog, records): - """STR_STATUS per record id from STRANGER_IMAGE on a VCF of (id, chrom, pos, end, count) records.""" - head = [line for line in open(os.path.join(REPO, "tests", "smoke", "stranger_input.vcf.in")) if line.startswith("##")] - alts = sorted({n for *_, n in records}) - head += [f'##ALT=\n' for a in alts] - head += [f"##contig=\n" for c in sorted({r[1] for r in records})] - with open(os.path.join(tmp, "in.vcf"), "w") as f: - f.writelines(head) - f.write("#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT\tS\n") - for i, (rid, chrom, pos, end, n) in enumerate(records): - f.write(f"{chrom}\t{pos}\t{rid}.{i}\tA\t\t.\tPASS\tSVTYPE=STR;END={end};REF=1;RL=3;RU=CAG;" - f"VARID={rid};REPID={rid}\t{FMT}\t" - f"1/1:SPANNING/SPANNING:{n}/{n}:{n}-{n}/{n}-{n}:20/20:0/0:0/0:30.0\n") - # Without a catalog: the GRCh38 catalog Stranger bundles, as step 9b passes it. - run = (["stranger", "--repeats-file", f"/w/{os.path.basename(catalog)}", "/w/in.vcf"] if catalog else - ["sh", "-c", 'exec stranger --repeats-file "$(python -c \'import os, stranger; print(os.path.join(' - 'os.path.dirname(stranger.__file__), "resources", "variant_catalog_grch38.json"))\')" /w/in.vcf']) - cmd = ["docker", "run", "--rm", "--network", "none", "-u", f"{os.getuid()}:{os.getgid()}", "-v", f"{tmp}:/w", - env("STRANGER_IMAGE")] + run - print("+ " + " ".join(cmd), flush=True) - try: - p = subprocess.run(cmd, capture_output=True, text=True) - except FileNotFoundError: - return None - if p.returncode != 0: - print(p.stderr) - return None - out = {} - for line in p.stdout.splitlines(): - if line.startswith("#"): - continue - f = line.split("\t") - info = dict(x.split("=", 1) if "=" in x else (x, "") for x in f[7].split(";")) - out[f[2]] = info.get("STR_STATUS", "") - return out - - -def part_stranger(cat_path): - recs = [("HTT", "chr4", 3074877, 3074933, 26), ("HTT", "chr4", 3074877, 3074933, 27), - ("HTT", "chr4", 3074877, 3074933, 36), ("SAMD12", "chr8", 118366913, 118366918, 1), - ("SAMD12", "chr8", 118366913, 118366918, 2), ("MUC1", "chr1", 155188506, 155192239, 60)] - with tempfile.TemporaryDirectory() as tmp: - os.chmod(tmp, 0o755) - shutil.copy(cat_path, os.path.join(tmp, "catalog.json")) - got = stranger(tmp, os.path.join(tmp, "catalog.json"), recs) - if got is None: - check("Stranger runs on the catalog (docker and STRANGER_IMAGE)", False, "it did not run") - return - want = {"HTT.0": "normal", "HTT.1": "pre_mutation", "HTT.2": "full_mutation", - "SAMD12.3": "normal", "SAMD12.4": "pre_mutation", "MUC1.5": ""} - check("Stranger on the catalog: HTT 26 normal, 27 pre_mutation, 36 full_mutation; SAMD12 1 normal, 2 " - "pre_mutation; MUC1 no status", got == want, got) - bundled = stranger(tmp, None, recs[:3]) - check("control: Stranger's bundled GRCh38 catalog grades HTT 27 otherwise", - bundled is not None and bundled.get("HTT.1") != "pre_mutation", bundled) - - -# --- 5. the extra repeats --------------------------------------------------------------- -EXTRA = os.path.join(REPO, "assets", "str_extra_loci.json") -NCBI = "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?db=nuccore&rettype=fasta" -ACCESSION = {"chr2": "NC_000002.12", "chr7": "NC_000007.14"} # GRCh38.p14, as the no-ALT set's chr2 and chr7 - - -def grch38(chrom, start, end): - """GRCh38 bases [start, end) (0-based) of chrom, from NCBI.""" - url = f"{NCBI}&id={ACCESSION[chrom]}&seq_start={start + 1}&seq_stop={end}" - for i in range(3): - try: - with urllib.request.urlopen(url, timeout=60) as r: - text = r.read().decode() - return "".join(text.splitlines()[1:]).upper() - except OSError as e: - print(f"NCBI fetch {i + 1}/3 failed: {e}") - time.sleep(5 * (i + 1)) - raise SystemExit(f"FAIL could not read {chrom}:{start}-{end} from NCBI") - - -def region_errors(entry, shift=0): - """Why each region of an extra entry is not a whole run of its motif in GRCh38 (empty when it is).""" - errs = [] - for m, r in zip(sc.motifs_of(entry["LocusStructure"]), sc.as_list(entry["ReferenceRegion"])): - chrom, a, b = sc.region(r) - a, b = a + shift, b + shift - k = len(m) - seq = grch38(chrom, a - k, b + k) - inner, left, right = seq[k:-k], seq[:k], seq[-k:] - if inner != m * ((b - a) // k): - errs.append(f"{r}: GRCh38 has {inner}, not {m} repeated") - elif left == m or right == m: - errs.append(f"{r}: the {m} run goes on past the region ({left}|...|{right})") - return errs - - -def missing_extra(extra, cat): - """The LocusIds of the extra catalog that the merged catalog lacks.""" - have = {e["LocusId"] for e in cat} - return [e["LocusId"] for e in extra if e["LocusId"] not in have] - - -def part_extra(rel, eh, loci): - extra = json.load(open(EXTRA)) - check("the extra catalog names CFTR (TG and T) and UGT1A1 (TA)", - [(e["LocusId"], e["LocusStructure"]) for e in extra] == [("CFTR", "(TG)*(T)*"), ("UGT1A1", "(TA)*")], - [e.get("LocusId") for e in extra]) - with tempfile.TemporaryDirectory() as tmp: - out, rng = os.path.join(tmp, "catalog.json"), os.path.join(tmp, "ranges.tsv") - rc = sc.main(["--eh", eh, "--loci", loci, "--release", rel, "--out", out, "--ranges", rng, "--extra", EXTRA]) - cat = json.load(open(out)) if rc == 0 else [] - check("--extra: STRchive's 82 entries and the extra loci", rc == 0 and len(cat) == 82 + len(extra), (rc, len(cat))) - check("every extra locus is in the merged catalog", rc == 0 and not missing_extra(extra, cat), - missing_extra(extra, cat)) - check("the extra loci are copied as they are, last", cat[-len(extra):] == extra, cat[-len(extra):]) - check("no extra locus has Stranger thresholds, each says why", - all("NormalMax" not in e and "PathologicMin" not in e and e.get("NoThresholds") for e in cat[-len(extra):])) - rows = [l.split("\t") for l in open(rng).read().splitlines()[2:]] if rc == 0 else [] - check("the ranges table has a row per entry, the extra ones without STRchive ranges", - len(rows) == len(cat) and [r[:6] for r in rows[-len(extra):]] - == [[e["LocusId"], e["SourceId"], "", "", "", ""] for e in extra], rows[-len(extra):]) - short = [e for e in extra if e["LocusId"] != "UGT1A1"] - json.dump(short, open(os.path.join(tmp, "short.json"), "w")) - sc.main(["--eh", eh, "--loci", loci, "--release", rel, "--out", out, "--extra", os.path.join(tmp, "short.json")]) - check("control: an extra catalog without UGT1A1 gives a merged catalog the check flags", - missing_extra(extra, json.load(open(out))) == ["UGT1A1"]) - cftr = extra[0] - planted = {"no NoThresholds": dict(cftr, NoThresholds=""), - "a region that is not a whole number of motifs": dict( - cftr, ReferenceRegion=["chr7:117548606-117548627", "chr7:117548627-117548635"]), - "a STRchive locus name": dict(cftr, LocusId="HTT"), - "Stranger thresholds": dict(cftr, NormalMax=11, PathologicMin=13)} - for desc, bad in planted.items(): - p, o, g = (os.path.join(tmp, x) for x in ("bad.json", "bad_out.json", "bad_ranges.tsv")) - json.dump([bad], open(p, "w")) - rc = sc.main(["--eh", eh, "--loci", loci, "--release", rel, "--out", o, "--ranges", g, "--extra", p]) - check(f"control: an extra entry with {desc} stops the converter (exit 2, nothing written)", - rc == 2 and not os.path.exists(o) and not os.path.exists(g), rc) - for e in extra: - errs = region_errors(e) - check(f"{e['LocusId']}: every region is a whole run of its motif in GRCh38", not errs, errs) - check("control: the UGT1A1 region shifted by one base fails the sequence check", bool(region_errors(extra[1], 1))) - - -def main(): - skip = og_or_skip() - if skip is not None: - return skip - rel, eh, loci = files() - part_files(rel, eh, loci) - cat, cat_path = build_catalog() - part_convert(rel, eh, loci, cat, cat_path) - part_stranger(cat_path) - part_extra(rel, eh, loci) - if FAILS: - print(f"{len(FAILS)} check(s) failed") - return 1 - print("all checks passed") - return 0 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/tests/test_str_ranges.py b/tests/test_str_ranges.py index b842e9633..d1a5e13e2 100644 --- a/tests/test_str_ranges.py +++ b/tests/test_str_ranges.py @@ -3,7 +3,8 @@ """The repeat ranges: the generated str_ranges.tsv, the report's cited subset and the docs. 1. str_ranges.tsv, which og str catalog writes beside the STRchive - catalog (built here from the pinned files by tests/test_str_catalog.py): + catalog (built here with og from the pinned files, as + rust/crates/og-setup/tests/strchive_catalog.rs builds it): one row per ExpansionHunter locus of the catalog (82), each traced to its STRchive id; normal_max and pathogenic_min are the catalog's NormalMax and PathologicMin, empty where the catalog has no thresholds and the reason in @@ -24,15 +25,19 @@ Run: python3 tests/test_str_ranges.py """ +import hashlib import json import os import re +import shutil +import subprocess import sys +import tempfile +import time +import urllib.request REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) -sys.path.insert(0, os.path.join(REPO, "bin")) -sys.path.insert(0, os.path.join(REPO, "tests")) -import test_str_catalog # noqa: E402 +OG = os.environ.get("OG", "") TSV = os.path.join(REPO, "bin", "str_intermediate_ranges.tsv") DOC = os.path.join(REPO, "docs", "interpreting-results.md") @@ -51,6 +56,68 @@ def check(desc, ok, detail=""): FAILS.append(desc) +def og_or_skip(): + """None when OG names an og binary; else the exit code: 0 (skip, with a + note), or 1 with OG_REQUIRED set.""" + if OG and os.access(OG, os.X_OK): + return None + print("skip: OG is not set (OG=rust/target/release/og)") + return 1 if os.environ.get("OG_REQUIRED") else 0 + + +def env(name): + for line in open(os.path.join(REPO, "versions.env")): + if line.startswith(name + "="): + return line.split("=", 1)[1].split("#")[0].strip().strip('"') + raise SystemExit(f"FAIL no {name} line in versions.env") + + +def sha256(path): + h = hashlib.sha256() + with open(path, "rb") as f: + for block in iter(lambda: f.read(1 << 20), b""): + h.update(block) + return h.hexdigest() + + +def files(): + """(release, EH catalog path, loci path), downloaded and checked.""" + rel = env("STRCHIVE_VERSION") + d = os.environ.get("STRCHIVE_DIR") or os.path.join(tempfile.gettempdir(), f"pgp-strchive-{rel}") + os.makedirs(d, exist_ok=True) + out = [] + for name, var in ((f"STRchive-disease-loci-{rel}.hg38.expansionhunter.json", "STRCHIVE_EH_SHA256"), + (f"STRchive-loci-{rel}.json", "STRCHIVE_LOCI_SHA256")): + path = os.path.join(d, name) + if not (os.path.exists(path) and sha256(path) == env(var)): + url = f"https://github.com/dashnowlab/STRchive/releases/download/{rel}/{name}" + for i in range(3): + try: + with urllib.request.urlopen(url, timeout=60) as r, open(path + ".part", "wb") as f: + shutil.copyfileobj(r, f) + break + except OSError as e: + print(f"download {i + 1}/3 of {url} failed: {e}") + time.sleep(5 * (i + 1)) + os.replace(path + ".part", path) + if sha256(path) != env(var): + raise SystemExit(f"FAIL {name} does not have the sha256 {var} pins") + out.append(path) + return rel, out[0], out[1] + + +def build_catalog(): + """The converted catalog (list of entries) and its path, in a temp file; the + ranges table (--ranges) is str_ranges.tsv beside it.""" + rel, eh, loci = files() + out = os.path.join(tempfile.mkdtemp(), "catalog.json") + rc = subprocess.run([OG, "str", "catalog", "--eh", eh, "--loci", loci, "--release", rel, "--out", out, + "--ranges", os.path.join(os.path.dirname(out), "str_ranges.tsv")]).returncode + if rc != 0: + raise SystemExit("FAIL the converter failed on the pinned files") + return json.load(open(out)), out + + def table(path): head, out = None, [] for line in open(path): @@ -115,7 +182,7 @@ def part_generated(cat, cat_path): check("control: a table with one row dropped is caught", bool(range_problems(rows[1:], cat))) check("control: a table with one row twice is caught", bool(range_problems(rows + rows[:1], cat))) by = {r[0]: dict(zip(RANGE_COLUMNS, r)) for r in rows} - loci = test_str_catalog.files()[2] + loci = files()[2] rec = {x["id"]: x for x in json.load(open(loci))} same = all((num(d["intermediate_min"]), num(d["intermediate_max"])) == (rec[d["source_id"]].get("intermediate_min"), rec[d["source_id"]].get("intermediate_max")) @@ -205,10 +272,10 @@ def part_docs(cat): def main(): - skip = test_str_catalog.og_or_skip() # the catalog is built with og (OG) + skip = og_or_skip() # the catalog is built with og (OG) if skip is not None: return skip - cat, cat_path = test_str_catalog.build_catalog() + cat, cat_path = build_catalog() gen = part_generated(cat, cat_path) part_subset(gen) part_docs(cat)