diff --git a/CHANGELOG.md b/CHANGELOG.md index b3e8e402..d6ee405e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,10 +4,10 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n ## Unreleased +- The checks of the HBA copy number, the blood groups and the traits table run in `cargo test`, and four Python tests are gone: `tests/test_hba_copy_number.py`, `test_bloodtype_report.py`, `test_bloodtype_hpa_hna.py` and `test_traits_table.py`. Every check they made is kept, through the `og cn`, `og bloodtype` and `og traits` command lines. The traits table's source check is a new verb, `og dev traits-sources`: `--fasta` checks each reference base against a FASTA and `--online` checks each PubMed id and rsID against PubMed and dbSNP. The e2e case `traits-panel-1` and the curation steps in the traits page use it. No pipeline output changes. - ROH summaries now divide F_ROH by the GRCh38 length outside the bands of the assessed autosomes. Per-site rows identify those chromosomes, including chromosomes with no ROH. A new `roh.data.scope` records the chromosomes, denominator and scope source. Relationship labels and UPD flags require sites on all 22 autosomes and the full unmasked autosome length. Segment-only files use their observed chromosomes as an unverified lower bound and get neither label. Empty input has null F_ROH. Segment counts and the step 11 summary table keep their format. - A one-chromosome ROH flag now names regions and citations without syndrome names. Its note includes deletion, acquired loss of heterozygosity and shared ancestry as alternatives. The maternal and paternal JSON keys remain as `"-"` for reader compatibility. The attributed `assets/imprinted_regions.tsv` is embedded at build time, replacing the duplicated Rust table. The container builder mounts it, and the source stamp includes its blob hash, so a table edit makes an old binary stale. The stamp format and `tree-hash` command name stay the same. Every listed chromosome uses the same lookup, including chr11 and chr15. A region hint does not establish overlap with its imprinted locus or parental origin. Broad cytoband names remain reference labels, not diagnostic intervals. - The ROH collector now streams per-site rows instead of keeping the whole file in memory. On 1.1 million synthetic site rows, peak RSS fell from 65.7-67.0 MB to 3.0-3.5 MB. CPU time stayed near 0.2 seconds. Under shared host load, wall time ranged from 1.40-21.46 seconds before and 7.89-38.16 seconds after; these timings do not establish a speedup. - - The reports now get summary.json from the Rust `og` binary. Step 24, the Nextflow HTML_REPORT process and `generate-report.sh` collect it with `og summary collect` (through `bin/render_report.py`), and a run collects each sample once: `generate-report.sh` renders the text report from the summary.json step 24 wrote, and collects again when a file it reads is newer than that summary.json, or a file was added to or removed from a step folder since, or a file a section names as its source, the run status or a table in `bin/` or `assets/` is gone. When it renders from step 24's summary.json, the text report's generated time is step 24's, the same as the HTML report's; it was the minute of its own collect. When it collects again, the text report carries the new collect's values and time, and the HTML report keeps step 24's until step 24 runs again, as before. The deliberate fixes the `og summary collect` entries below list now reach the reports; apart from them, nothing in summary.json or either report changes. `bin/collect_summary.py` keeps only what the report renderer (`bin/render_report.py`), the couple report, the results index and the live view import from it. The Python modules that only the old collector imported are gone: `bin/pgx_outside_calls.py`, `ancestry_fine_report.py`, `clip_check.py`, `annotsv_postfilter.py`, `chip_report.py`, `traits_report.py`, `run_delta.py` and `genotype_rules_indels.py`. On the synthetic demo sample (4.7 million VCF records) the collect takes 1.1 s of CPU and 12-15 MB, where the Python took 7.8-8.0 s and 34-37 MB. What changes on purpose: - The walk of the sample folder leaves out `/nextflow`. `run-all.sh` launches Nextflow there, and its `work/` holds every task's files and staged links. No section reads a file there. With 200,000 files under it, the collect took 67 MB walked and 14 MB skipped. - The sections other than the sample VCF are read one at a time, while the sample VCF is read on its own thread, so the peak memory is near the biggest section's and not the sum of all of them. With every side table 50 times its demo size, the peak footprint fell from 119-130 MB to 105-112 MB (the Python: 100-102 MB), with the same CPU time. diff --git a/assets/traits.tsv b/assets/traits.tsv index ed2a927a..bea62616 100644 --- a/assets/traits.tsv +++ b/assets/traits.tsv @@ -2,7 +2,7 @@ # Hand-curated from the primary papers in the pmids column; no text is taken from SNPedia or any # other database. GRCh38; ref is the GRCh38 base, alt the allele the readings are about. # hom_ref, het, hom_alt: the reading for ref/ref, ref/alt and alt/alt. ancestry: yes when the -# reading depends on ancestry (the note says how). tests/test_traits_table.py checks the schema. +# reading depends on ancestry (the note says how). og traits check checks the schema. chrom pos rsid gene ref alt trait hom_ref het hom_alt ancestry note pmids chr2 135851076 rs4988235 MCM6 G A Lactase persistence (digesting milk sugar as an adult) No copy of -13910T, the European lactase persistence allele: lactase activity usually falls after childhood, unless another row below carries a persistence allele. One copy of -13910T: lactase activity usually stays into adulthood. Two copies of -13910T: lactase activity usually stays into adulthood. yes Lactase persistence is dominant: one persistence allele at any of the five lactase rows is usually enough. -13910T explains it in Europeans; in Africa other alleles do (rows -14010, -13915, -13907), so read the five rows together. 11788828 chr2 135859184 rs182549 MCM6 C T Lactase persistence (digesting milk sugar as an adult) No copy of -22018A, which travels with the -13910T persistence allele in Europeans. One copy of -22018A; read it with row rs4988235, which predicts persistence better. Two copies of -22018A; read it with row rs4988235, which predicts persistence better. yes A weaker marker than -13910: it matched lactase non-persistence in 229 of 236 cases of the original study, -13910 in all of them. Lactase persistence is dominant: one persistence allele at any of the five lactase rows is usually enough. -13910T explains it in Europeans; in Africa other alleles do (rows -14010, -13915, -13907), so read the five rows together. 11788828 diff --git a/docs/35-paralogs.md b/docs/35-paralogs.md index 1a1b0ae0..88ab4329 100644 --- a/docs/35-paralogs.md +++ b/docs/35-paralogs.md @@ -237,7 +237,7 @@ The ratio of a site is its estimate over the median of the sites, so one copy co - `tests/test_paralog_loci.py` checks the model list (167 genes in `assets/parascopy_models_v1.7.txt`, each with one main region) and every catalog row (a model on the list, a GRCh38 region inside the model's main region, a known tag and cross-check, a table name used once, the SMN1 row), and that planted faults are refused: a model not on the list (`GBA1.gz`, and `CYP21A2.gz`, which the data has no model for), a bad region, a region outside its model, a repeated table name, a catalog without the SMN1 row. It checks one owner per locus: no catalog model overlaps a depth-engine row of `bin/cn_loci.tsv` but RHCE (without that exemption RHCE is refused, and a planted model over HBA is refused), and every catalog gene `assets/callability_genes.tsv` lists has the owner `parascopy` there (a planted `.` is refused). The catalog sets owners in that file but adds no rows to it: its rows come from GIAB's lists, gene body by gene body, and a model can cover only part of a gene. So 17 catalog genes are not there: seven pseudogenes (CELP, GBAP1, NCF1B, NCF1C, OTOAP1, PMS2CL, STRCP1), which it leaves out as not protein-coding, and ten genes to which none of its reasons applies (ABCC6, CFH, CFHR4, CHRNA7, DEFA3, KRT86, NEB, PIK3CA, PLG, VWF). This section reads them all. The test holds that list and fails when it changes. It also runs the step's own lines that write a not-assessed SMN table and checks that the report, the Paraphase side-by-side and the couple screen each read it as not assessed, with the reason, and that a catalog table without `agCN_filter` or `agCN_qual` reads as unreadable, not as not resolved. - `tests/fake-docker/paralog-catalog-loop.sh` runs the step on a catalog of three rows against a fake Docker: three `parascopy cn-using` calls with the right model files, in order, one table each; another population's files with `PARASCOPY_POPULATION`; no call and a not-assessed table for a region without reads, a model that is not installed and a Parascopy run that fails, while the other rows run; a rerun with fewer rows removes the tables of the dropped rows and keeps the HBA and Paraphase tables; a bad row, or a region off chr1-22, X and Y or with its start past its end, stops the step before any container. - The e2e case `paralog-catalog-1-gba` adds a GBA row (`GBA.gz`) to a copy of the catalog and runs the step on the fixture's slice plus GIAB's HG002 reads over GBA1 and GBAP1, sampled to about 30x: the GBA table must hold an integer agCN with its quality. On the fixture's slice alone, which has no read at GBA1, the same row must be not assessed. The case also checks that the committed models are the names in the installed `list.txt` of each population, that the SMN table is byte for byte what one `parascopy cn-using` of `SMN1.gz` writes, as the step did before the catalog, and that the loop of the Nextflow module, cut from the file, writes the same two tables in the same image. -- `tests/test_copy_number.py` checks the HBA row and that these depth tables give, through the engine, the same tables as the code before it. `tests/test_hba_copy_number.py` gives `og cn call --locus HBA` made-up depth tables: four genes at even depth, three with HBA2 at half depth, two, a triplication, an estimate between two whole numbers, low flanks, flanks that disagree, and no read of mapping quality 0 over the genes (the depths of the novoalign BAM, and a clean three), which must be not assessed. +- `tests/test_copy_number.py` checks the HBA row and that these depth tables give, through the engine, the same tables as the code before it. `cargo test -p og-cn --test port_tests_cn_traits` gives `og cn call --locus HBA` made-up depth tables: four genes at even depth, three with HBA2 at half depth, two, a triplication, an estimate between two whole numbers, low flanks, flanks that disagree, and no read of mapping quality 0 over the genes (the depths of the novoalign BAM, and a clean three), which must be not assessed. - The e2e case `paralog-checks-and-doc-fixes-1-hba` writes synthetic 30x reads over chr16:95,001-235,000 for two samples, one with every second read over HBA2 left out (a planted one-gene loss), runs step 35 on each and checks four and three genes. A synthetic read whose bases occur more than once in that stretch gets mapping quality 0, as an aligner gives it. The HBA block of `modules/local/parascopy/main.nf`, cut from the file, then runs on the same BAMs in the same image and must write the same table. Last, the block runs on HG002's real Illumina reads at the cluster, three ways: GIAB's 60x novoalign BAM sampled to about 30x must be not assessed, naming the missing mapping quality 0 reads; Google's 35x NovaSeq BWA-MEM BAM must count four; and the same reads realigned with `minimap2 -a -x sr` to chr16:1-300,000 must count four. The truth is four, from Paraphase on HG002's HiFi reads. The run's notes record each estimate, the four depths and the share of mapping quality 0. - The e2e case `paralog-rows-1` runs the whole catalog on the fixture's slice plus GIAB's HG002 reads over every catalog locus on the fixture's chromosomes, sampled to about 30x, timed (it fails past 30 minutes): every row's table must hold an integer agCN or say not assessed with the reason, and a header-only table fails that check. It then runs the step with a catalog that has no SMN1 row and checks that both reports still show the HBA count and the other loci. `tests/test_collect_summary.py` holds the same case on made-up tables, with the loci read in the catalog's order. - `tests/test_copy_number.py` section 10 runs the CYP21A2 row on made-up bedcov and mpileup tables (`tests/fixtures/cn/CYP21A2_*`): unmodified, 2 copies and no finding; a planted chimeric deletion, 1 copy with the module total 3; a planted In2G conversion on one copy, a ratio of 0.5 at In2G and a possible conversion there. The same depths with every site even give no finding. The deletion's tables with no MAPQ 0 read (a share under 0.02) read not assessed (`low_mapq0_share`), and with a share of 0.021 or HG002's 0.1 still give the deletion. The CYP21A2 block of step 35, cut out, writes those tables, a not-assessed one when mpileup fails and a `low_mapq0_share` one when its MAPQ 1 pass equals the first, and the couple screen and the report read them. diff --git a/docs/42-traits.md b/docs/42-traits.md index 90a25c9c..6d37e304 100644 --- a/docs/42-traits.md +++ b/docs/42-traits.md @@ -73,7 +73,7 @@ Variants whose effect allele the papers we could read did not state were left ou ### Adding a row -Read the primary paper first and write the reading from it, in your own words. Take the GRCh38 position, the reference base and the allele from dbSNP's record of the rsID, and the direction of the effect from the paper, checking the strand: an allele named on the gene's strand of a gene on the minus strand is the complement on GRCh38. Then run `python3 tests/test_traits_table.py --online`, which looks up every PubMed id and checks each rsID's GRCh38 position and alleles against dbSNP. +Read the primary paper first and write the reading from it, in your own words. Take the GRCh38 position, the reference base and the allele from dbSNP's record of the rsID, and the direction of the effect from the paper, checking the strand: an allele named on the gene's strand of a gene on the minus strand is the complement on GRCh38. Then run `bin/og traits check --table assets/traits.tsv`, which checks the schema, and `bin/og dev traits-sources --online`, which looks up every PubMed id and checks each rsID's GRCh38 position and alleles against dbSNP (add `--fasta` with the reference to check each reference base too). ## Genotype Rules diff --git a/docs/cn-loci.md b/docs/cn-loci.md index 71f2ef3e..618288d6 100644 --- a/docs/cn-loci.md +++ b/docs/cn-loci.md @@ -128,7 +128,7 @@ pypgx owns the genes it calls from the BAM, the `bam` rows of `assets/pgx_genes. - The e2e case `cn-intervals-1-catalog` rebuilds the interval rows from the pinned downloads and fails on any difference with the committed catalog. `cn-intervals-2-hg002` runs the interval blocks of step 35 and of the PARASCOPY module on HG002's real reads at the AZF genes, their controls and 22q11.2: a male with one copy of AZFa and AZFb, AZFc_DAZ not assessed (its estimate, between 3.3 and 3.9, is no whole count of the four DAZ copies, so the row says `between_counts` rather than a call), no change at 22q11.2 A-B, no call on a row off chrY and chr22 (they have no read; the chr22 rows share A-B's control window, so they are left out of that check), and the AZF rows not applicable when the same reads are read as female. - The e2e case `followups-cn-1-couple-report` runs step 39, COUPLE_CARRIER and HTML_REPORT on a made-up couple, plus a stub run of `main.nf`, and checks that the interval table and the CNVpytor calls reach the couple report and `summary.json`. Each wiring line has a red control: the case runs again without it and checks that the rows are gone. -- `tests/test_copy_number.py` checks every row and refuses broken ones. For the three gene-dosage rows it calls made-up depths at 0 to 4 copies, cuts the dosage block out of step 35 and runs it on the made-up bedcov tables in `tests/fixtures/cn/` (HP 3, LCE3BC 1, CCL3L1 5 copies), checks that a row shifted by 1 kb no longer matches its table, and that the PARASCOPY module runs the same list. It runs the owner check with planted AMY1C, C4B and DEFB4 rows. It feeds made-up bedcov tables at 0, 1, 2, 3 and 4 copies and a MAPQ 0 share over the ceiling. It checks that a control with no read stops the run with `BROKEN_CONTROL`. With `OG` set (Guard's rust job sets it) it also checks that the depth tables of `tests/test_hba_copy_number.py` and `tests/test_lpa_kiv2.py` give, through `og cn`, the same tables the Python code before it wrote, and that `og cn` refuses the broken rows with the same message as the catalog reader. +- `tests/test_copy_number.py` checks every row and refuses broken ones. For the three gene-dosage rows it calls made-up depths at 0 to 4 copies, cuts the dosage block out of step 35 and runs it on the made-up bedcov tables in `tests/fixtures/cn/` (HP 3, LCE3BC 1, CCL3L1 5 copies), checks that a row shifted by 1 kb no longer matches its table, and that the PARASCOPY module runs the same list. It runs the owner check with planted AMY1C, C4B and DEFB4 rows. It feeds made-up bedcov tables at 0, 1, 2, 3 and 4 copies and a MAPQ 0 share over the ceiling. It checks that a control with no read stops the run with `BROKEN_CONTROL`. With `OG` set (Guard's rust job sets it) it also checks that the depth tables of the HBA cases (`cargo test -p og-cn --test port_tests_cn_traits`) and of `tests/test_lpa_kiv2.py` give, through `og cn`, the same tables the Python code before it wrote, and that `og cn` refuses the broken rows with the same message as the catalog reader. - The fake-docker case `cn-engine-calls` runs steps 35, 41 and 46 with made-up depths. Each step calls `og cn` once for its regions, step 35 and 46 once more for the call, step 46 reads the KIV-2 4925G>A marker in the same call, and no step calls a Python copy-number script. A control exon with no read fails step 46 with `BROKEN_CONTROL`. The case needs `og`, so Guard's rust job runs it after the build. - `tests/test_copy_number.py` section 10 checks the CYP21A2 row: its six sites, the pileup parser, and the fixtures `tests/fixtures/cn/CYP21A2_{normal,deletion,conversion}` (2 copies and no finding; 1 copy with the module total 3; an In2G ratio of 0.5 and a possible conversion), with the conversion's depths read with every site even as the red control (no finding). The e2e case `cyp21a2-short-reads-1-hg002` runs the step's and the module's blocks on HG002's reads, unmodified and with a planted conversion and a planted deletion. The same section feeds the deletion's tables with MAPQ 0 shares of 0, 0.019, 0.021 and 0.1 (not assessed twice, then the deletion), and `cn-guard-1-mapq0` runs both blocks on HG002's reads with every MAPQ 0 read removed: not assessed (`low_mapq0_share`), never a deletion. - `tests/parity` runs `og cn` against the Python engine it replaced on every command and the tables of the three per-gene loci (`scripts/ci/oracle-diff.sh`, [docs/rust.md](rust.md)); the deliberate fixes are its goldens. diff --git a/rust/crates/og-bloodtype/tests/port_tests_cn_traits.rs b/rust/crates/og-bloodtype/tests/port_tests_cn_traits.rs new file mode 100644 index 00000000..ee7da774 --- /dev/null +++ b/rust/crates/og-bloodtype/tests/port_tests_cn_traits.rs @@ -0,0 +1,1899 @@ +//! `og bloodtype report`: the blood group, platelet (HPA) and neutrophil +//! (HNA) antigens a genome predicts (step 41), through the command line. +//! +//! Each case writes a synthetic one-sample VCF (only the records it plants) +//! and, when the case has a BAM, a `samtools bedcov` table over +//! `assets/bloodtype_markers.tsv` and the RhD regions with chosen depths, +//! runs `og bloodtype report`, and checks rows of the table it writes. The +//! marker table is read here on its own, not through the crate's reader, so +//! a reader bug cannot hide a table bug. +//! +//! Red cells (this was `tests/test_bloodtype_report.py`): +//! 1. the table and the code agree (`og bloodtype check-markers`); a table +//! missing a rule's site fails the check +//! 2. nothing planted, reads everywhere: the reference alleles at +//! confidence high; without the BAM the same calls at medium, and RhD +//! not determined +//! 3. ABO: A1, A2, B, AB, O.02, and markers that do not add up +//! 4. RhD from depth: two copies, one, none, the RHD pseudogene, a hybrid +//! gene, low flank depth +//! 5. Duffy and Kell flip with planted variants +//! 6. the VCF: too few reads, a FILTER-failed call, a RefCall 0/0, contigs +//! without chr, a multi-allelic record, a REF that is not the marker's, +//! a second sample +//! 7. a BAM without some contigs +//! 8. each null and HPA-7 to -35 row's REF and ALT read its c. change on +//! the gene's strand +//! 9. every such row, positive and negative +//! 10. gVCF records and reference blocks +//! 11. null alleles in detail +//! 12. common variants are not null alleles +//! 13. another ALT at a null row's position +//! 14. two null copies need no pair site +//! +//! Platelets and neutrophils (this was `tests/test_bloodtype_hpa_hna.py`): +//! 15. every HPA and HNA system has its site(s), at the GRCh38 position and +//! REF base Ensembl gave for its dbSNP id +//! 16. each two-allele antigen for 0/0, 0/1 and 1/1 +//! 17. the meaning a person who lacks an antigen gets in the note +//! 18. HNA-1: FCGR3B*01, *02 and *03, a c.233 no call, markers that +//! disagree, c.194 that disagrees, and confidence always low + +use indexmap::IndexMap; +use std::collections::{BTreeSet, HashMap}; +use std::fmt::Debug; +use std::io::Write; +use std::path::PathBuf; +use std::sync::LazyLock; +use std::sync::atomic::{AtomicU64, Ordering}; + +const REPO: &str = concat!(env!("CARGO_MANIFEST_DIR"), "/../../.."); + +fn markers_path() -> String { + format!("{REPO}/assets/bloodtype_markers.tsv") +} + +fn cn_loci() -> String { + format!("{REPO}/bin/cn_loci.tsv") +} + +/// One marker row: its columns by name. +type Site = IndexMap; + +struct Table { + /// The marker table's sites by name, in table order. + sites: IndexMap, + /// The RhD depth regions `og cn bed --locus RHD` prints, by name: + /// (chrom, start, end). + regions: IndexMap, +} + +static TABLE: LazyLock = LazyLock::new(|| { + let text = std::fs::read_to_string(markers_path()).unwrap(); + let mut lines = text + .lines() + .filter(|l| !l.starts_with('#') && !l.trim().is_empty()); + let header: Vec<&str> = lines.next().unwrap().split('\t').collect(); + let mut sites = IndexMap::new(); + for l in lines { + let cols: Vec<&str> = l.split('\t').collect(); + let row: Site = header + .iter() + .enumerate() + .map(|(i, h)| { + ( + (*h).to_owned(), + cols.get(i).copied().unwrap_or("").to_owned(), + ) + }) + .collect(); + sites.insert(row["name"].clone(), row); + } + let args: Vec = ["bed", "--catalog", &cn_loci(), "--locus", "RHD"] + .iter() + .map(|a| (*a).to_owned()) + .collect(); + let (mut out, mut err) = (Vec::new(), Vec::new()); + assert_eq!(og_cn::cli::main(&args, &mut out, &mut err), 0); + let regions = String::from_utf8(out) + .unwrap() + .lines() + .map(|l| { + let c: Vec<&str> = l.split('\t').collect(); + ( + c[3].to_owned(), + ( + c[0].to_owned(), + c[1].parse().unwrap(), + c[2].parse().unwrap(), + ), + ) + }) + .collect(); + Table { sites, regions } +}); + +fn site(name: &str) -> &'static Site { + TABLE + .sites + .get(name) + .unwrap_or_else(|| panic!("no marker {name}")) +} + +fn start(s: &Site) -> i64 { + s["start"].parse().unwrap() +} + +/// The VCF position: the BED end. +fn pos(s: &Site) -> i64 { + s["end"].parse().unwrap() +} + +fn nulls() -> BTreeSet { + TABLE + .sites + .iter() + .filter(|(_, r)| r["alt_means"] == "null") + .map(|(n, _)| n.clone()) + .collect() +} + +/// Collects failed checks and reports them all at once. +#[derive(Default)] +struct Checks(Vec); + +impl Checks { + fn eq(&mut self, desc: &str, got: T, want: T) { + if got != want { + self.0 + .push(format!("FAIL {desc}: got {got:?}, want {want:?}")); + } + } + fn done(self) { + assert!(self.0.is_empty(), "\n{}", self.0.join("\n")); + } +} + +/// A VCF record at the marker `name`, with the options a case changes. +#[derive(Clone, Copy)] +struct Line<'a> { + filt: &'a str, + chrom: Option<&'a str>, + alt: Option<&'a str>, + ref_: Option<&'a str>, +} + +const PLAIN: Line<'static> = Line { + filt: "PASS", + chrom: None, + alt: None, + ref_: None, +}; + +fn vcf_with(name: &str, gt: &str, o: Line<'_>) -> String { + let r = site(name); + [ + o.chrom.unwrap_or(&r["chrom"]), + &pos(r).to_string(), + ".", + o.ref_.unwrap_or(&r["ref"]), + o.alt.unwrap_or(&r["alt"]), + "50", + o.filt, + ".", + "GT", + gt, + ] + .join("\t") + + "\n" +} + +fn vl(name: &str, gt: &str) -> String { + vcf_with(name, gt, PLAIN) +} + +fn raw_line(chrom: &str, pos: i64, ref_: &str, alt: &str, gt: &str) -> String { + format!("{chrom}\t{pos}\t.\t{ref_}\t{alt}\t50\tPASS\t.\tGT\t{gt}\n") +} + +/// Mean depth per region name, `*` for every other one. +type Depth = HashMap; + +fn depth_of(pairs: &[(&str, f64)]) -> Depth { + pairs.iter().map(|(k, v)| ((*k).to_owned(), *v)).collect() +} + +/// Depth means for an RHD copy number (2 = the flanks' depth), with some +/// exons at other copy numbers. +fn rhd_ex(copies: f64, flank: f64, exons: &[(i64, f64)]) -> Depth { + let mut d = depth_of(&[ + ("*", 30.0), + ("RHD_FLANK_UP", flank), + ("RHD_FLANK_DOWN", flank), + ("RHD_GENE", flank * copies / 2.0), + ]); + for n in TABLE.regions.keys() { + if let Some(e) = n.strip_prefix("RHD_EX") { + let e: i64 = e.parse().unwrap(); + let c = exons + .iter() + .find(|(x, _)| *x == e) + .map_or(copies, |(_, c)| *c); + d.insert(n.clone(), flank * c / 2.0); + } + } + d +} + +fn rhd(copies: f64) -> Depth { + rhd_ex(copies, 30.0, &[]) +} + +fn with(mut d: Depth, name: &str, mean: f64) -> Depth { + d.insert(name.to_owned(), mean); + d +} + +/// The shape of the marker rows of a depth table. +#[derive(Clone, Copy)] +enum Cols { + /// The table's 16 columns, as step 41 gives samtools bedcov. + Wide, + /// The first six columns. + Narrow, +} + +fn depth_table(depth: &Depth, drop: &[&str], cols: Cols) -> String { + let mean = |name: &str| { + depth + .get(name) + .or_else(|| depth.get("*")) + .copied() + .unwrap_or(30.0) + }; + let mut text = String::new(); + for (name, r) in &TABLE.sites { + if drop.contains(&r["chrom"].as_str()) { + continue; // a contig the BAM has no @SQ line for: step 41 leaves its rows out + } + let (s, e) = (start(r), pos(r)); + let total = (mean(name) * (e - s) as f64) as i64; + let mut c: Vec = ["chrom", "start", "end", "kind", "system", "name"] + .iter() + .map(|k| r[*k].clone()) + .collect(); + if let Cols::Wide = cols { + c.extend(std::iter::repeat_n(String::new(), 10)); + } + c.push(total.to_string()); + text += &(c.join("\t") + "\n"); + } + for (name, (chrom, s, e)) in &TABLE.regions { + if drop.contains(&chrom.as_str()) { + continue; + } + let total = (mean(name) * (e - s) as f64) as i64; + text += &format!("{chrom}\t{s}\t{e}\t{name}\t{total}\n"); + } + text +} + +static SEQ: AtomicU64 = AtomicU64::new(0); + +/// A fresh folder for one run. +fn scratch() -> PathBuf { + let d = std::env::temp_dir().join(format!( + "og-bloodtype-port-{}-{}", + std::process::id(), + SEQ.fetch_add(1, Ordering::Relaxed) + )); + let _ = std::fs::remove_dir_all(&d); + std::fs::create_dir_all(&d).unwrap(); + d +} + +/// A run's options beyond its records. +#[derive(Clone, Copy)] +struct Opts<'a> { + depth: Option<&'a Depth>, + cols: Cols, + gz: bool, + samples: &'a [&'a str], + drop: &'a [&'a str], +} + +const NO_BAM: Opts<'static> = Opts { + depth: None, + cols: Cols::Wide, + gz: false, + samples: &["S"], + drop: &[], +}; + +/// The report's rows in table order, or the error text of a refused run. +type Rows = Result, String>; + +fn report(records: &[String], o: Opts<'_>) -> Rows { + let d = scratch(); + let text = format!( + "##fileformat=VCFv4.2\n#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT\t{}\n{}", + o.samples.join("\t"), + records.concat() + ); + let vcf = d.join(if o.gz { "s.vcf.gz" } else { "s.vcf" }); + if o.gz { + let mut g = flate2::write::GzEncoder::new( + std::fs::File::create(&vcf).unwrap(), + flate2::Compression::default(), + ); + g.write_all(text.as_bytes()).unwrap(); + g.finish().unwrap(); + } else { + std::fs::write(&vcf, text).unwrap(); + } + let p = |n: &str| d.join(n).to_string_lossy().into_owned(); + let mut args = vec![ + "--cn-loci".to_owned(), + cn_loci(), + "--sample".to_owned(), + "S".to_owned(), + "--vcf".to_owned(), + vcf.to_string_lossy().into_owned(), + "--markers".to_owned(), + markers_path(), + "--out".to_owned(), + p("t.tsv"), + "--evidence".to_owned(), + p("e.tsv"), + ]; + if let Some(depth) = o.depth { + std::fs::write(d.join("d.tsv"), depth_table(depth, o.drop, o.cols)).unwrap(); + args.extend(["--depth".to_owned(), p("d.tsv")]); + } + let (mut out, mut err) = (Vec::new(), Vec::new()); + let code = og_bloodtype::cli("report", &args, &mut out, &mut err); + let result = if code == 0 { + let t = std::fs::read_to_string(p("t.tsv")).unwrap(); + let mut lines = t.lines(); + let header: Vec<&str> = lines.next().unwrap().split('\t').collect(); + Ok(lines + .map(|l| { + header + .iter() + .zip(l.split('\t')) + .map(|(h, v)| ((*h).to_owned(), v.to_owned())) + .collect() + }) + .collect()) + } else { + let e = String::from_utf8_lossy(&err).trim().to_owned(); + Err(if e.is_empty() { + format!("exit {code}") + } else { + e + }) + }; + let _ = std::fs::remove_dir_all(&d); + result +} + +/// `{system: row}` of a run. +struct T(Rows); + +impl T { + fn get(&self, system: &str, col: &str) -> String { + match &self.0 { + Err(e) => e.clone(), + Ok(rows) => rows + .iter() + .find(|r| r["system"] == system) + .map_or_else(|| format!("no {system} row"), |r| r[col].clone()), + } + } + fn ph(&self, system: &str) -> String { + self.get(system, "phenotype") + } + fn cf(&self, system: &str) -> String { + self.get(system, "confidence") + } + fn gt(&self, system: &str) -> String { + self.get(system, "genotype") + } + fn note(&self, system: &str) -> String { + self.get(system, "note") + } + fn order(&self) -> Vec { + match &self.0 { + Ok(rows) => rows.iter().map(|r| r["system"].clone()).collect(), + Err(e) => vec![e.clone()], + } + } +} + +/// A run with reads (the table's 16 columns in the depth rows). +fn run(records: &[String], depth: &Depth) -> T { + T(report( + records, + Opts { + depth: Some(depth), + ..NO_BAM + }, + )) +} + +/// A run with 30 reads everywhere and six-column depth rows. +fn run30(records: &[String]) -> T { + static FLAT: LazyLock = LazyLock::new(|| depth_of(&[("*", 30.0)])); + T(report( + records, + Opts { + depth: Some(&FLAT), + cols: Cols::Narrow, + ..NO_BAM + }, + )) +} + +/// The genotype's first part, before " (RHD copies ...)". +fn head(s: &str) -> String { + s.split(" (").next().unwrap_or("").to_owned() +} + +fn has(s: &str, part: &str) -> bool { + s.contains(part) +} + +fn check_markers(path: &str) -> (u8, String) { + let args = [path.to_owned(), "--cn-loci".to_owned(), cn_loci()]; + let (mut out, mut err) = (Vec::new(), Vec::new()); + let code = og_bloodtype::cli("check-markers", &args, &mut out, &mut err); + (code, String::from_utf8_lossy(&err).into_owned()) +} + +#[test] +fn s01_the_marker_table_and_the_rules_agree() { + let mut c = Checks::default(); + c.eq( + "the marker table and the rules agree", + check_markers(&markers_path()), + (0, String::new()), + ); + let d = scratch(); + let broken = d.join("m.tsv"); + let text: String = std::fs::read_to_string(markers_path()) + .unwrap() + .split_inclusive('\n') + .filter(|l| !l.contains("\tKEL_841\t")) + .collect(); + std::fs::write(&broken, text).unwrap(); + let (code, err) = check_markers(&broken.to_string_lossy()); + c.eq( + "control: a table without KEL_841 fails the check", + (code, has(&err, "rule sites the table lacks: KEL_841")), + (1, true), + ); + let _ = std::fs::remove_dir_all(&d); + c.done(); +} + +#[test] +fn s02_nothing_planted() { + let mut c = Checks::default(); + let r = run(&[], &rhd(2.0)); + for (system, want) in [ + ("ABO", "O"), + ("RhD", "D+"), + ("RhCE", "C-c+ E-e+"), + ("Kell", "K-k+ Kp(a-b+) Js(a-b+)"), + ("Duffy", "Fy(a+b-)"), + ("Kidd", "Jk(a+b-)"), + ("MNS", "M-N+ S-s+"), + ("Lutheran", "Lu(a-b+)"), + ("Diego", "Di(a-b+)"), + ("Dombrock", "Do(a-b+)"), + ("Colton", "Co(a+b-)"), + ("HPA-1", "HPA-1(a+b-)"), + ("HPA-2", "HPA-2(a+b-)"), + ("HPA-3", "HPA-3(a+b-)"), + ("HPA-4", "HPA-4(a+b-)"), + ("HPA-5", "HPA-5(a+b-)"), + ("HPA-15", "HPA-15(a-b+)"), + ] { + c.eq( + &format!("reference alleles, reads everywhere: {system}"), + r.ph(system).as_str(), + want, + ); + } + c.eq("ABO genotype", r.gt("ABO").as_str(), "O.01/O.01"); + c.eq( + "ABO confidence high with reads", + r.cf("ABO").as_str(), + "high", + ); + c.eq( + "RhCE low: c.307 sits in RHCE's RHD-like exon 2", + r.cf("RhCE").as_str(), + "low", + ); + c.eq("Kell high", r.cf("Kell").as_str(), "high"); + let nobam = T(report(&[], NO_BAM)); + c.eq( + "without the BAM, ABO is still O", + nobam.ph("ABO").as_str(), + "O", + ); + c.eq( + "without the BAM, confidence medium", + nobam.cf("ABO").as_str(), + "medium", + ); + c.eq( + "without the BAM, RhD is not determined", + nobam.ph("RhD").as_str(), + "not determined", + ); + c.eq( + "without the BAM, the note says why", + has(&nobam.note("RhD"), "deletes the whole RHD gene"), + true, + ); + c.done(); +} + +#[test] +fn s03_abo() { + let mut c = Checks::default(); + let d2 = rhd(2.0); + c.eq( + "A1/O: one c.261 insertion", + run(&[vl("ABO_261", "0/1")], &d2).ph("ABO").as_str(), + "A1", + ); + let a2 = run(&[vl("ABO_261", "1/1"), vl("ABO_1061", "1/1")], &d2); + c.eq( + "A2/A2: c.1061delC on both", + (a2.ph("ABO"), a2.gt("ABO")), + ("A2".into(), "A2/A2".into()), + ); + c.eq( + "A1/A2 reads A1", + run(&[vl("ABO_261", "1/1"), vl("ABO_1061", "0/1")], &d2) + .ph("ABO") + .as_str(), + "A1", + ); + let b = run( + &[ + vl("ABO_261", "0/1"), + vl("ABO_803", "0/1"), + vl("ABO_796", "0/1"), + ], + &d2, + ); + c.eq( + "B/O", + (b.ph("ABO"), b.gt("ABO")), + ("B".into(), "B/O.01".into()), + ); + let ab = run( + &[ + vl("ABO_261", "1/1"), + vl("ABO_803", "0/1"), + vl("ABO_796", "0/1"), + ], + &d2, + ); + c.eq( + "A1B", + (ab.ph("ABO"), ab.cf("ABO")), + ("A1B".into(), "high".into()), + ); + c.eq( + "the A1 or A2 caveat is in the note", + has(&ab.note("ABO"), "c.1061delC, a deletion in a run of C"), + true, + ); + let o2 = run(&[vl("ABO_261", "0/1"), vl("ABO_802", "0/1")], &d2); + c.eq( + "O.01/O.02 is O", + (o2.ph("ABO"), o2.gt("ABO")), + ("O".into(), "O.01/O.02".into()), + ); + c.eq( + "two B markers on one functional allele: not determined", + run(&[vl("ABO_261", "0/1"), vl("ABO_803", "1/1")], &d2) + .ph("ABO") + .as_str(), + "not determined", + ); + let odd = run(&[vl("ABO_261", "0/1"), vl("ABO_803", "0/1")], &d2); + c.eq( + "c.803 without c.796: B, confidence low", + (odd.ph("ABO"), odd.cf("ABO")), + ("B".into(), "low".into()), + ); + c.done(); +} + +#[test] +fn s04_rhd_from_depth() { + let mut c = Checks::default(); + c.eq( + "two copies: D+", + run(&[], &rhd(2.0)).ph("RhD").as_str(), + "D+", + ); + let het = run(&[], &rhd(1.0)); + c.eq( + "one copy (one RHD deleted): D+", + (het.ph("RhD"), head(&het.gt("RhD"))), + ("D+".into(), "RHD/RHD deleted".into()), + ); + let gone = run(&[], &rhd(0.0)); + c.eq( + "no copy (RHD deleted on both): D-", + (gone.ph("RhD"), gone.cf("RhD")), + ("D-".into(), "high".into()), + ); + c.eq( + "the deleted genotype", + head(&gone.gt("RhD")).as_str(), + "RHD deleted/RHD deleted", + ); + let psi = [vl("RHD_PSI_DUP", "0/1"), vl("RHD_PSI_609", "0/1")]; + c.eq( + "one RHD copy and it is the pseudogene: D-", + run(&psi, &rhd(1.0)).ph("RhD").as_str(), + "D-", + ); + c.eq( + "two copies, one of them the pseudogene: D+", + run(&psi, &rhd(2.0)).ph("RhD").as_str(), + "D+", + ); + let exons: Vec<(i64, f64)> = (3..=7).map(|e| (e, 0.0)).collect(); + let hyb = run(&[], &rhd_ex(1.6, 30.0, &exons)); + c.eq( + "exons 3 to 7 without reads: a hybrid, RhD not determined", + hyb.ph("RhD").as_str(), + "not determined", + ); + c.eq( + "the hybrid note names the exons", + has(&hyb.note("RhD"), "exons 3, 4, 5, 6, 7"), + true, + ); + c.eq( + "flanks under 10x: not determined", + run(&[], &rhd_ex(2.0, 6.0, &[])).ph("RhD").as_str(), + "not determined", + ); + c.eq( + "near a boundary (1.45 copies): confidence low", + run(&[], &rhd(1.45)).cf("RhD").as_str(), + "low", + ); + c.done(); +} + +#[test] +fn s05_duffy_and_kell_flip() { + let mut c = Checks::default(); + let d2 = rhd(2.0); + c.eq( + "Fyb het: Fy(a+b+)", + run(&[vl("FY_125", "0/1")], &d2).ph("Duffy").as_str(), + "Fy(a+b+)", + ); + let gata = run(&[vl("FY_125", "0/1"), vl("FY_GATA", "0/1")], &d2); + c.eq( + "Fyb het with the GATA box: Fy(a+b-)", + (gata.ph("Duffy"), gata.gt("Duffy")), + ("Fy(a+b-)".into(), "FY*A/FY*B GATA".into()), + ); + c.eq( + "Fyb hom, GATA hom: Fy(a-b-)", + run(&[vl("FY_125", "1/1"), vl("FY_GATA", "1/1")], &d2) + .ph("Duffy") + .as_str(), + "Fy(a-b-)", + ); + c.eq( + "FY*X: Fy(a-b+w)", + run(&[vl("FY_125", "1/1"), vl("FY_265", "1/1")], &d2) + .ph("Duffy") + .as_str(), + "Fy(a-b+w)", + ); + let k = run(&[vl("KEL_578", "0/1")], &d2); + c.eq( + "K het: K+k+", + (k.ph("Kell"), k.gt("Kell")), + ( + "K+k+ Kp(a-b+) Js(a-b+)".into(), + "K/k Kpb/Kpb Jsb/Jsb".into(), + ), + ); + c.eq( + "K hom: K+k-", + run(&[vl("KEL_578", "1/1")], &d2) + .ph("Kell") + .split(' ') + .next() + .unwrap_or(""), + "K+k-", + ); + c.eq( + "HPA-15: the reference is 15b, the alt 15a", + run(&[vl("HPA15_2108", "1/1")], &d2).ph("HPA-15").as_str(), + "HPA-15(a+b-)", + ); + c.done(); +} + +#[test] +fn s06_the_vcf() { + let mut c = Checks::default(); + let d2 = rhd(2.0); + let low = run(&[], &with(rhd(2.0), "KEL_578", 3.0)); + c.eq( + "a missing site with 3 reads: that pair is not determined", + (low.ph("Kell"), low.cf("Kell")), + ("K/k not determined Kp(a-b+) Js(a-b+)".into(), "low".into()), + ); + let lowqual = Line { + filt: "LowQual", + ..PLAIN + }; + c.eq( + "a FILTER-failed call is no call", + run(&[vcf_with("JK_838", "1/1", lowqual)], &d2) + .ph("Kidd") + .as_str(), + "not determined", + ); + let refcall = Line { + filt: "RefCall", + ..PLAIN + }; + c.eq( + "a RefCall 0/0 is a reference call", + run(&[vcf_with("JK_838", "0/0", refcall)], &d2) + .ph("Kidd") + .as_str(), + "Jk(a+b-)", + ); + let nochr = Line { + chrom: Some("18"), + ..PLAIN + }; + let gz = T(report( + &[vcf_with("JK_838", "1/1", nochr)], + Opts { + depth: Some(&d2), + gz: true, + ..NO_BAM + }, + )); + c.eq( + "contigs without chr, gzip", + gz.ph("Kidd").as_str(), + "Jk(a-b+)", + ); + let multi = run( + &[vcf_with( + "KEL_578", + "1/2", + Line { + alt: Some("C,A"), + ..PLAIN + }, + )], + &d2, + ); + c.eq( + "a multi-allelic record: the second ALT is K", + multi.ph("Kell").split(' ').next().unwrap_or(""), + "K+k+", + ); + c.eq( + "another allele at the site lowers the confidence", + multi.cf("Kell").as_str(), + "low", + ); + let mism = run( + &[vcf_with( + "KEL_578", + "1/1", + Line { + ref_: Some("C"), + alt: Some("T"), + ..PLAIN + }, + )], + &d2, + ); + let words: Vec = mism + .ph("Kell") + .split(' ') + .take(3) + .map(str::to_owned) + .collect(); + c.eq( + "a record whose REF is not the marker's REF: that pair is not determined", + (words, mism.cf("Kell")), + ( + vec!["K/k".into(), "not".into(), "determined".into()], + "low".into(), + ), + ); + c.eq( + "and the note names both REFs", + has( + &mism.note("Kell"), + "REF C at this position, the marker's REF is G", + ), + true, + ); + let span = run( + &[vcf_with( + "KEL_578", + "0/0", + Line { + ref_: Some("GT"), + alt: Some("G"), + ..PLAIN + }, + )], + &d2, + ); + c.eq( + "a reference call over a longer REF that starts with the marker's REF is reference", + span.ph("Kell").split(' ').next().unwrap_or(""), + "K-k+", + ); + c.eq( + "a no-call genotype", + run(&[vl("JK_838", "./.")], &d2).ph("Kidd").as_str(), + "not determined", + ); + let two = report( + &[], + Opts { + samples: &["S", "T"], + ..NO_BAM + }, + ); + c.eq( + "two samples stop the report", + matches!(&two, Err(e) if e.contains("2 samples")), + true, + ); + c.done(); +} + +#[test] +fn s07_a_bam_without_some_contigs() { + let mut c = Checks::default(); + let d2 = rhd(2.0); + // A BAM without chr1 and chr9: no depth rows for RH, Duffy and ABO. + let part = T(report( + &[vl("JK_838", "1/1")], + Opts { + depth: Some(&d2), + drop: &["chr1", "chr9"], + ..NO_BAM + }, + )); + c.eq( + "no depth over RHD: RhD not determined", + part.ph("RhD").as_str(), + "not determined", + ); + c.eq( + "the RhD note names the missing contig", + has(&part.note("RhD"), "the BAM has no chr1 contig"), + true, + ); + c.eq( + "ABO with no record and no depth row: not determined", + part.ph("ABO").as_str(), + "not determined", + ); + c.eq( + "the ABO note says why", + has(&part.note("ABO"), "no depth row"), + true, + ); + c.eq( + "Duffy (chr1) not determined", + part.ph("Duffy").as_str(), + "not determined", + ); + c.eq( + "Kidd (chr18, called) is read as usual", + (part.ph("Kidd"), part.cf("Kidd")), + ("Jk(a-b+)".into(), "high".into()), + ); + c.eq( + "Kell (chr7, covered) is read as usual", + (part.ph("Kell"), part.cf("Kell")), + ("K-k+ Kp(a-b+) Js(a-b+)".into(), "high".into()), + ); + let contigs: Vec<&str> = TABLE + .sites + .values() + .map(|r| r["chrom"].as_str()) + .collect::>() + .into_iter() + .collect(); + let empty_depth = Depth::new(); + let empty = T(report( + &[], + Opts { + depth: Some(&empty_depth), + drop: &contigs, + ..NO_BAM + }, + )); + c.eq( + "an empty depth file (no marker contig in the BAM): RhD not determined", + empty.ph("RhD").as_str(), + "not determined", + ); + c.eq( + "and a system with no record is not determined", + empty.ph("Kidd").as_str(), + "not determined", + ); + c.done(); +} + +// 8. The orientation of the null and HPA-7 to -35 rows: each row's REF and +// ALT are its c. change on the gene's strand (Ensembl GRCh38, 2026-11-03). +// A row with REF and ALT swapped, or on the wrong strand, fails here. + +fn strand(gene: &str) -> i8 { + match gene { + "SMIM1" | "FUT2" | "SLC14A1" | "ITGB3" | "GP1BB" | "ITGA2" | "GP9" => 1, + "ABCB6" | "ABCG2" | "FUT1" | "KEL" | "ITGA2B" => -1, + other => panic!("no strand for gene {other}: add it to strand()"), + } +} + +fn is_bases(s: &str) -> bool { + !s.is_empty() && s.bytes().all(|b| b"ACGT".contains(&b)) +} + +/// A c. position: digits with an optional leading `-` or `*` and `+` +/// offsets (`[-*0-9+]+`). +fn is_cpos(s: &str) -> bool { + !s.is_empty() && s.bytes().all(|b| b"-*+0123456789".contains(&b)) +} + +/// `P` or `P_P`: the one or two positions of a change. +fn positions(s: &str) -> Option<(&str, Option<&str>)> { + match s.split_once('_') { + Some((a, b)) if is_cpos(a) && is_cpos(b) => Some((a, Some(b))), + None if is_cpos(s) => Some((s, None)), + _ => None, + } +} + +fn span(a: &str, b: Option<&str>) -> usize { + let at = |x: &str| -> i64 { + let (base, off) = x.split_once('+').unwrap_or((x, "")); + base.parse::().unwrap() + + if off.is_empty() { + 0 + } else { + off.parse::().unwrap() + } + }; + b.map_or(1, |b| usize::try_from(at(b) - at(a) + 1).unwrap()) +} + +/// The VCF REF/ALT agree with one c. change on the gene's strand: the bases +/// of a substitution or a delins, the length of a deletion, duplication or +/// insertion. +fn oriented(gene: &str, ref_: &str, alt: &str, change: &str) -> bool { + let flip = |s: &str| -> String { + if strand(gene) == -1 { + s.bytes() + .rev() + .map(|b| match b { + b'A' => 'T', + b'C' => 'G', + b'G' => 'C', + b'T' => 'A', + o => o as char, + }) + .collect() + } else { + s.to_owned() + } + }; + let Some(c) = change.strip_prefix("c.") else { + return false; + }; + // A substitution: c.[-*0-9+_]+X>Y + if let Some((left, y)) = c.split_once('>') { + let Some(x) = left.chars().last().map(|x| x.to_string()) else { + return false; + }; + let p = &left[..left.len() - 1]; + let ok = !p.is_empty() && p.bytes().all(|b| b"-*+_0123456789".contains(&b)); + return ok + && is_bases(&x) + && y.len() == 1 + && is_bases(y) + && (flip(ref_), flip(alt)) == (x, y.to_owned()); + } + if let Some((p, bases)) = c.split_once("delins") { + return match positions(p) { + Some((a, b)) if is_bases(bases) => ref_.len() == span(a, b) && flip(alt) == bases, + _ => false, + }; + } + for kind in ["del", "dup"] { + if let Some((p, bases)) = c.split_once(kind) { + let Some((a, b)) = positions(p) else { + return false; + }; + if !(bases.is_empty() || is_bases(bases)) { + return false; + } + let n = if bases.is_empty() { + span(a, b) + } else { + bases.len() + }; + let d = if kind == "del" { + ref_.len() as i64 - alt.len() as i64 + } else { + alt.len() as i64 - ref_.len() as i64 + }; + if d != n as i64 || ref_.as_bytes().first() != alt.as_bytes().first() { + return false; + } + // Bases given: the VCF's deleted or added bases on the gene's + // strand are those bases, or a rotation of them (VCF left-aligns + // in a repeat, HGVS shifts 3'). + let moved = flip(&(if kind == "del" { ref_ } else { alt })[1..]); + return bases.is_empty() || (moved.len() == n && bases.repeat(2).contains(&moved)); + } + } + if let Some((p, bases)) = c.split_once("ins") { + let digits = |s: &str| !s.is_empty() && s.bytes().all(|b| b.is_ascii_digit()); + return match p.split_once('_') { + Some((a, b)) if digits(a) && digits(b) && is_bases(bases) => { + alt.len() as i64 - ref_.len() as i64 == bases.len() as i64 + && ref_.as_bytes().first() == alt.as_bytes().first() + } + _ => false, + }; + } + false +} + +/// The null rows and HPA-7 to -35 but HPA-15: one site each, the a antigen +/// on the reference allele. +fn new_rows() -> Vec { + let rare = TABLE.sites.keys().filter(|n| { + n.strip_prefix("HPA").is_some_and(|rest| { + rest.split('_').next().unwrap().parse::().unwrap() >= 7 && !n.starts_with("HPA15_") + }) + }); + let mut all: BTreeSet = nulls(); + all.extend(rare.cloned()); + all.into_iter().collect() +} + +#[test] +fn s08_null_and_rare_hpa_rows_read_their_c_change_on_the_gene_strand() { + let mut c = Checks::default(); + let rows = new_rows(); + assert!(rows.len() > 20, "too few null and rare HPA rows: {rows:?}"); + for n in &rows { + let r = site(n); + let (gene, changes) = r["hgvs"].split_once(' ').unwrap_or((&r["hgvs"], "")); + let changes: Vec<&str> = changes.split(", ").collect(); + let alts: Vec<&str> = r["alt"].split(',').collect(); + c.eq( + &format!("{n}: one c. change per ALT"), + changes.len(), + alts.len(), + ); + let wrong: Vec<&str> = alts + .iter() + .zip(&changes) + .filter(|(a, ch)| !oriented(gene, &r["ref"], a, ch)) + .map(|(_, ch)| *ch) + .collect(); + c.eq( + &format!("{n}: REF and ALT read the c. change on the {gene} strand"), + wrong, + vec![], + ); + } + c.eq( + "control: a swapped REF and ALT fails the orientation", + oriented("KEL", "A", "G", "c.578C>T"), + false, + ); + c.eq( + "control: the right one passes", + oriented("KEL", "G", "A", "c.578C>T"), + true, + ); + c.eq( + "control: a deletion one base short fails", + oriented("FUT2", "CG", "C", "c.719_720delGT"), + false, + ); + c.eq( + "control: a deletion of the right length but other bases fails", + oriented("FUT2", "CAC", "C", "c.719_720delGT"), + false, + ); + c.eq( + "control: a left-aligned deletion in a repeat passes", + oriented("FUT2", "CTG", "C", "c.719_720delGT"), + true, + ); + c.done(); +} + +#[test] +fn s09_every_null_and_rare_hpa_row_positive_and_negative() { + let mut c = Checks::default(); + let neg = |s: &str| match s { + "Vel" => "Vel-".to_owned(), + "Langereis" => "Lan-".to_owned(), + "Junior" => "Jr(a-)".to_owned(), + "H" => "H- (Bombay, Oh)".to_owned(), + "Secretor" => "non-secretor".to_owned(), + "Kell" => "K-k- Kp(a-b-) Js(a-b-)".to_owned(), + "Kidd" => "Jk(a-b-)".to_owned(), + s => format!("{s}(a-b+)"), + }; + let pos_ = |s: &str| match s { + "Vel" => "Vel+".to_owned(), + "Langereis" => "Lan+".to_owned(), + "Junior" => "Jr(a+)".to_owned(), + "H" => "H+".to_owned(), + "Secretor" => "secretor".to_owned(), + "Kell" => "K-k+ Kp(a-b+) Js(a-b+)".to_owned(), + "Kidd" => "Jk(a+b-)".to_owned(), + s => format!("{s}(a+b-)"), + }; + let base = run30(&[]); + for n in new_rows() { + let r = site(&n); + let s = r["system"].as_str(); + c.eq( + &format!("{n}: the reference genome is {}", pos_(s)), + base.ph(s), + pos_(s), + ); + for a in r["alt"].split(',') { + let mut recs = vec![vcf_with( + &n, + "1/1", + Line { + alt: Some(a), + ..PLAIN + }, + )]; + if s == "H" { + recs.push(vl("FUT2_01N02", "1/1")); // FUT1 null with a non-secretor: Bombay + } + c.eq( + &format!("{n} {}>{a} on both chromosomes: {}", r["ref"], neg(s)), + run30(&recs).ph(s), + neg(s), + ); + } + } + c.eq( + "a reference genome lacks no high-prevalence antigen: ABO is the first row", + base.order().into_iter().take(3).collect::>(), + vec!["ABO".to_owned(), "RhD".into(), "RhCE".into()], + ); + c.done(); +} + +#[test] +fn s10_gvcf_records_and_reference_blocks() { + let mut c = Checks::default(); + // Jk(a-b-) from SLC14A1 c.342-1G>A and HPA-21b/21b, each with the <*> + // allele DeepVariant's gVCF adds. + let (jk, hpa) = (site("JK_01N06"), site("HPA21_1960")); + let gv = vec![ + format!( + "{}\t{}\t.\tG\tA,<*>\t40\tPASS\t.\tGT:GQ\t1/1:40\n", + jk["chrom"], + pos(jk) + ), + format!( + "{}\t{}\t.\t{}\t{},<*>\t40\tPASS\t.\tGT:GQ\t1/1:40\n", + hpa["chrom"], + pos(hpa), + hpa["ref"], + hpa["alt"] + ), + ]; + let g = run30(&gv); + c.eq( + "gVCF: c.342-1G>A 1/1 is Jk(a-b-), lacking Jk3", + ( + g.ph("Kidd"), + g.gt("Kidd"), + has(&g.note("Kidd"), "lacks Jk3, carried by most people"), + ), + ("Jk(a-b-)".into(), "null/null".into(), true), + ); + c.eq( + "gVCF: HPA-21b/21b", + (g.ph("HPA-21"), g.gt("HPA-21")), + ("HPA-21(a-b+)".into(), "21b/21b".into()), + ); + c.eq( + "the systems that lack a high-prevalence antigen come first", + g.order().into_iter().take(2).collect::>(), + vec!["Kidd".to_owned(), "HPA-21".into()], + ); + // A reference block at the sites gives the common phenotypes. + let blocks: Vec = [jk, hpa] + .iter() + .map(|r| { + format!( + "{}\t{}\t.\t{}\t<*>\t0\t.\tEND={}\tGT:GQ\t0/0:50\n", + r["chrom"], + pos(r), + &r["ref"][..1], + pos(r) + 5 + ) + }) + .collect(); + let gb = run30(&blocks); + c.eq( + "gVCF reference blocks: Jk(a+b-) and HPA-21(a+b-), confidence high", + (gb.ph("Kidd"), gb.ph("HPA-21"), gb.cf("Kidd")), + ("Jk(a+b-)".into(), "HPA-21(a+b-)".into(), "high".into()), + ); + c.done(); +} + +#[test] +fn s11_null_alleles_in_detail() { + let mut c = Checks::default(); + let one = run30(&[vl("VEL_01N01", "0/1")]); + c.eq( + "one SMIM1 null copy: Vel+, Vel/null, the note names it", + ( + one.ph("Vel"), + one.gt("Vel"), + has(&one.note("Vel"), "SMIM1 c.64_80del"), + ), + ("Vel+".into(), "Vel/null".into(), true), + ); + let cpd = run30(&[vl("LAN_01N02", "0/1"), vl("LAN_01N13", "0/1")]); + c.eq( + "two different ABCB6 nulls: Lan-, confidence low (no phase), the note says so", + ( + cpd.ph("Langereis"), + cpd.cf("Langereis"), + has(&cpd.note("Langereis"), "phase is not read"), + ), + ("Lan-".into(), "low".into(), true), + ); + let kk = run30(&[vl("KEL_578", "0/1"), vl("KEL_02N03", "0/1")]); + c.eq( + "K/k with one KEL null: the null goes on the k chromosome, K+k-", + (kk.ph("Kell"), kk.gt("Kell")), + ( + "K+k- Kp(a-b+) Js(a-b+)".into(), + "K/null Kpb/null Jsb/null".into(), + ), + ); + c.eq( + "and that lacks k, a high-prevalence antigen, so Kell comes first", + ( + kk.order().first().cloned().unwrap_or_default(), + has( + &kk.note("Kell"), + "lacks k, carried by most people (high prevalence)", + ), + ), + ("Kell".into(), true), + ); + let jb = run30(&[vl("JK_838", "0/1"), vl("JK_02N06", "0/1")]); + c.eq( + "Jka/Jkb with JK*02N.06 (c.871T>C): the null goes on Jkb, Jk(a+b-)", + (jb.ph("Kidd"), jb.gt("Kidd")), + ("Jk(a+b-)".into(), "Jka/null".into()), + ); + let para = run30(&[vl("FUT1_01N08", "1/1"), vl("ABO_261", "0/1")]); + c.eq( + "FUT1 null with a working FUT2: para-Bombay", + para.ph("H").as_str(), + "H- (para-Bombay)", + ); + c.eq( + "and the ABO row says the red cells group as O", + has(¶.note("ABO"), "the cells usually group as O"), + true, + ); + let se = run30(&[vl("FUT2_01N02", "1/1")]); + let kept: Vec<(String, String)> = se + .order() + .into_iter() + .filter(|s| ["H", "ABO", "Secretor"].contains(&s.as_str())) + .map(|s| { + let p = se.ph(&s); + (s, p) + }) + .collect(); + c.eq( + "a non-secretor alone leaves H and ABO as they are", + kept, + [("ABO", "O"), ("H", "H+"), ("Secretor", "non-secretor")] + .map(|(s, p)| (s.to_owned(), p.to_owned())) + .to_vec(), + ); + c.eq( + "the only Vel null row with 2 reads: not determined", + run(&[], &with(rhd(2.0), "VEL_01N01", 2.0)) + .ph("Vel") + .as_str(), + "not determined", + ); + let lowk = run(&[], &with(rhd(2.0), "KEL_02N03", 2.0)); + c.eq( + "one KEL null row with 2 reads: Kell still read, confidence medium", + (lowk.ph("Kell"), lowk.cf("Kell")), + ("K-k+ Kp(a-b+) Js(a-b+)".into(), "medium".into()), + ); + let multi_null = run30(&[vcf_with( + "KEL_02N01", + "1/1", + Line { + alt: Some("A"), + ..PLAIN + }, + )]); + c.eq( + "a null row with several ALTs counts each (c.223+1G>T, the third)", + multi_null.ph("Kell").as_str(), + "K-k- Kp(a-b-) Js(a-b-)", + ); + c.done(); +} + +#[test] +fn s12_common_variants_are_not_null_alleles() { + // FUT2 c.772G>A (rs602662, about half of all chromosomes in gnomAD) and + // c.335C>T (rs200157007, 0.14 in South Asians) ride on the c.461G>A + // haplotype, and SLC14A1 c.810G>A (rs17675299, 0.08 in Finns) is far + // too common for a Jk null. As null rows they turned Se/se carriers into + // non-secretors and Jk(a+b+) into Jk(a+b-) at confidence high. + let mut c = Checks::default(); + for (desc, recs, system, want) in [ + ( + "FUT2 c.461G>A 0/1 with c.772G>A 0/1: secretor", + vec![ + vl("FUT2_01N02", "0/1"), + raw_line("chr19", 48_703_728, "G", "A", "0/1"), + ], + "Secretor", + "secretor", + ), + ( + "FUT2 c.772G>A 1/1 alone: secretor", + vec![raw_line("chr19", 48_703_728, "G", "A", "1/1")], + "Secretor", + "secretor", + ), + ( + "FUT2 c.461G>A 0/1 with c.335C>T 0/1: secretor", + vec![ + vl("FUT2_01N02", "0/1"), + raw_line("chr19", 48_703_291, "C", "T", "0/1"), + ], + "Secretor", + "secretor", + ), + ( + "JK_838 0/1 with SLC14A1 c.810G>A 0/1: Jk(a+b+)", + vec![ + vl("JK_838", "0/1"), + raw_line("chr18", 45_739_309, "G", "A", "0/1"), + ], + "Kidd", + "Jk(a+b+)", + ), + ] { + c.eq(desc, run30(&recs).ph(system).as_str(), want); + } + let common = [ + ("chr19", 48_703_728), + ("chr19", 48_703_291), + ("chr18", 45_739_309), + ]; + c.eq( + "those three positions are no null row", + nulls() + .into_iter() + .filter(|n| { + let r = site(n); + common.contains(&(r["chrom"].as_str(), pos(r))) + }) + .collect::>(), + vec![], + ); + c.done(); +} + +#[test] +fn s13_another_alt_at_a_null_row_is_not_that_null_allele() { + let mut c = Checks::default(); + // KEL c.1546C>T's position, G>T: the antigens stay fully read at + // confidence high, and the note names it. + let kel = site("KEL_02N17"); + let oth = run30(&[raw_line(&kel["chrom"], pos(kel), "G", "T", "0/1")]); + c.eq( + "another ALT at a KEL null row: K-k+, confidence high, the note names it", + ( + oth.ph("Kell"), + oth.cf("Kell"), + has(&oth.note("Kell"), "another allele at this position"), + ), + ("K-k+ Kp(a-b+) Js(a-b+)".into(), "high".into(), true), + ); + c.done(); +} + +#[test] +fn s14_two_null_copies_need_no_pair_site() { + let mut c = Checks::default(); + let nc = run30(&[vl("JK_01N06", "1/1"), vl("JK_838", "./.")]); + c.eq( + "Jk null 1/1 with c.838 a no call: Jk(a-b-), null/null", + (nc.ph("Kidd"), nc.gt("Kidd")), + ("Jk(a-b-)".into(), "null/null".into()), + ); + let kn = run30(&[vl("KEL_02N03", "1/1"), vl("KEL_578", "./.")]); + c.eq( + "KEL null 1/1 with c.578 a no call: K-k- Kp(a-b-) Js(a-b-), null/null", + (kn.ph("Kell"), kn.gt("Kell")), + ("K-k- Kp(a-b-) Js(a-b-)".into(), "null/null".into()), + ); + let two = run30(&[ + vl("JK_838", "0/1"), + vl("JK_02N21", "0/1"), + vl("JK_01N32", "0/1"), + ]); + c.eq( + "two Kidd nulls with different null_on: the note names each allele's placement", + ( + has(&two.note("Kidd"), "SLC14A1 c.118G>A: Jkb"), + has(&two.note("Kidd"), "SLC14A1 c.151+1G>C: Jka"), + ), + (true, true), + ); + c.done(); +} + +// Platelets (HPA) and neutrophils (HNA). + +/// (system, chrom, GRCh38 position, REF, ALT, rsid) +type Want = ( + &'static str, + &'static str, + i64, + &'static str, + &'static str, + &'static str, +); + +/// name: [`Want`] +const HPA_HNA_SITES: [(&str, Want); 16] = [ + ( + "HPA1_176", + ("HPA-1", "chr17", 47_283_364, "T", "C", "rs5918"), + ), + ( + "HPA2_482", + ("HPA-2", "chr17", 4_933_086, "C", "T", "rs6065"), + ), + ( + "HPA3_2621", + ("HPA-3", "chr17", 44_375_697, "A", "C", "rs5911"), + ), + ( + "HPA4_506", + ("HPA-4", "chr17", 47_284_587, "G", "A", "rs5917"), + ), + ( + "HPA5_1600", + ("HPA-5", "chr5", 53_062_927, "G", "A", "rs1801106"), + ), + ( + "HPA6_1544", + ("HPA-6", "chr17", 47_292_422, "G", "A", "rs13306487"), + ), + ( + "HPA15_2108", + ("HPA-15", "chr6", 73_783_709, "A", "C", "rs10455097"), + ), + ( + "FCGR3B_108", + ("HNA-1", "chr1", 161_629_989, "G", "C", "rs200688856"), + ), + ( + "FCGR3B_114", + ("HNA-1", "chr1", 161_629_983, "A", "G", "rs527909462"), + ), + ( + "FCGR3B_194", + ("HNA-1", "chr1", 161_629_903, "T", "C", "rs448740"), + ), + ( + "FCGR3B_233", + ("HNA-1", "chr1", 161_629_864, "G", "T", "rs5030738"), + ), + ( + "FCGR3B_244", + ("HNA-1", "chr1", 161_629_853, "T", "C", "rs147574249"), + ), + ( + "FCGR3B_316", + ("HNA-1", "chr1", 161_629_781, "T", "C", "rs2290834"), + ), + ( + "HNA3_461", + ("HNA-3", "chr19", 10_631_494, "A", "G", "rs2288904"), + ), + ( + "HNA4_230", + ("HNA-4", "chr16", 31_265_490, "G", "A", "rs1143679"), + ), + ( + "HNA5_2372", + ("HNA-5", "chr16", 30_506_720, "G", "C", "rs2230433"), + ), +]; + +fn is_hpa_hna(system: &str) -> bool { + system.starts_with("HPA-") || system.starts_with("HNA-") +} + +#[test] +fn s15_every_hpa_and_hna_site_is_at_its_grch38_position() { + let mut c = Checks::default(); + for (n, w) in HPA_HNA_SITES { + let got = TABLE.sites.get(n).map(|r| { + ( + r["system"].clone(), + r["chrom"].clone(), + pos(r), + r["ref"].clone(), + r["alt"].clone(), + r["rsid"].clone(), + ) + }); + let want = ( + w.0.to_owned(), + w.1.to_owned(), + w.2, + w.3.to_owned(), + w.4.to_owned(), + w.5.to_owned(), + ); + c.eq(&format!("{n} at its GRCh38 position"), got, Some(want)); + } + let (mut out, mut err) = (Vec::new(), Vec::new()); + c.eq( + "og bloodtype systems exits 0", + og_bloodtype::cli("systems", &[], &mut out, &mut err), + 0, + ); + let out = String::from_utf8(out).unwrap(); + let systems: Vec<&str> = out.split('\n').collect(); + for s in ["HPA-6", "HNA-1", "HNA-3", "HNA-4", "HNA-5"] { + c.eq( + &format!("{s} is a system of the table"), + systems.contains(&s), + true, + ); + } + let names = |want_meaning: bool| -> Vec<&String> { + TABLE + .sites + .iter() + .filter(|(_, r)| is_hpa_hna(&r["system"]) == want_meaning) + .filter(|(_, r)| r["meaning"].is_empty() == want_meaning) + .map(|(n, _)| n) + .collect() + }; + c.eq("every HPA and HNA site has a meaning", names(true), vec![]); + c.eq("no red cell site has a meaning", names(false), vec![]); + c.eq( + "every FCGR3B site carries the FCGR3A and copy-number caveat", + HPA_HNA_SITES + .iter() + .map(|(n, _)| *n) + .filter(|n| { + n.starts_with("FCGR3B") + && !TABLE + .sites + .get(*n) + .is_some_and(|r| r["caveat"].contains("copy number")) + }) + .collect::>(), + vec![], + ); + c.done(); +} + +#[test] +fn s16_each_two_allele_antigen_at_0_0_0_1_and_1_1() { + let mut c = Checks::default(); + // name: (system, phenotype at 0/0, 0/1, 1/1, genotype at 0/1) + let pairs = [ + ( + "HPA1_176", + ( + "HPA-1", + "HPA-1(a+b-)", + "HPA-1(a+b+)", + "HPA-1(a-b+)", + "1a/1b", + ), + ), + ( + "HPA2_482", + ( + "HPA-2", + "HPA-2(a+b-)", + "HPA-2(a+b+)", + "HPA-2(a-b+)", + "2a/2b", + ), + ), + ( + "HPA3_2621", + ( + "HPA-3", + "HPA-3(a+b-)", + "HPA-3(a+b+)", + "HPA-3(a-b+)", + "3a/3b", + ), + ), + ( + "HPA4_506", + ( + "HPA-4", + "HPA-4(a+b-)", + "HPA-4(a+b+)", + "HPA-4(a-b+)", + "4a/4b", + ), + ), + ( + "HPA5_1600", + ( + "HPA-5", + "HPA-5(a+b-)", + "HPA-5(a+b+)", + "HPA-5(a-b+)", + "5a/5b", + ), + ), + ( + "HPA6_1544", + ( + "HPA-6", + "HPA-6(a+b-)", + "HPA-6(a+b+)", + "HPA-6(a-b+)", + "6a/6b", + ), + ), + ( + "HPA15_2108", + ( + "HPA-15", + "HPA-15(a-b+)", + "HPA-15(a+b+)", + "HPA-15(a+b-)", + "15a/15b", + ), + ), + ( + "HNA3_461", + ( + "HNA-3", + "HNA-3(a-b+)", + "HNA-3(a+b+)", + "HNA-3(a+b-)", + "3a/3b", + ), + ), + ( + "HNA4_230", + ( + "HNA-4", + "HNA-4(a+b-)", + "HNA-4(a+b+)", + "HNA-4(a-b+)", + "4a/4b", + ), + ), + ( + "HNA5_2372", + ( + "HNA-5", + "HNA-5(a+b-)", + "HNA-5(a+b+)", + "HNA-5(a-b+)", + "5a/5b", + ), + ), + ]; + let none = run30(&[]); + for (n, (system, p0, p1, p2, g1)) in pairs { + let het = run30(&[vl(n, "0/1")]); + let hom = run30(&[vl(n, "1/1")]); + c.eq( + &format!("{system}: 0/0, 0/1, 1/1"), + (none.ph(system), het.ph(system), hom.ph(system)), + (p0.into(), p1.into(), p2.into()), + ); + c.eq( + &format!("{system}: the 0/1 genotype at confidence high"), + (het.gt(system), het.cf(system)), + (g1.into(), "high".into()), + ); + c.eq( + &format!("{system}: both antigens, no meaning in the note"), + has(&het.note(system), "lacks "), + false, + ); + } + c.done(); +} + +#[test] +fn s17_a_person_who_lacks_an_antigen_gets_its_meaning() { + let mut c = Checks::default(); + let none = run30(&[]); + let mum = run30(&[vl("HPA1_176", "1/1")]); + let note = mum.note("HPA-1"); + c.eq( + "HPA-1b/1b: the note says the person lacks HPA-1a", + note.starts_with("lacks HPA-1a: "), + true, + ); + c.eq( + "and names FNAIT and anti-HPA-1a", + has(¬e, "FNAIT") && has(¬e, "anti-HPA-1a"), + true, + ); + c.eq( + "HPA-1a/1a lacks HPA-1b", + none.note("HPA-1").starts_with("lacks HPA-1b: "), + true, + ); + let hna3 = none.note("HNA-3"); + c.eq( + "GRCh38 carries HNA-3b: no record lacks HNA-3a, with the neutrophil meaning", + ( + hna3.starts_with("lacks HNA-3a: "), + has(&hna3, "neonatal alloimmune neutropenia"), + ), + (true, true), + ); + c.eq( + "a red cell system never gets a meaning", + has(&mum.note("Kidd"), "lacks "), + false, + ); + c.done(); +} + +#[test] +fn s18_hna_1() { + let mut c = Checks::default(); + let a4 = ["FCGR3B_108", "FCGR3B_114", "FCGR3B_244", "FCGR3B_316"]; + let all = |gt: &str, extra: &[(&str, &str)]| -> Vec { + a4.iter() + .map(|n| vl(n, gt)) + .chain(extra.iter().map(|(n, g)| vl(n, g))) + .collect() + }; + let b = run30(&[]); + c.eq( + "the GRCh38 bases (FCGR3B*02 at four markers): HNA-1(a-b+c-), FCGR3B*02 twice", + (b.ph("HNA-1"), b.gt("HNA-1")), + ("HNA-1(a-b+c-)".into(), "FCGR3B*02/FCGR3B*02".into()), + ); + c.eq( + "confidence is always low (FCGR3A, copy number)", + b.cf("HNA-1").as_str(), + "low", + ); + c.eq( + "c.194 that does not match is named in the note", + has(&b.note("HNA-1"), "FCGR3B c.194A>G (x0) does not match"), + true, + ); + c.eq( + "the note lacks HNA-1a and HNA-1c", + has(&b.note("HNA-1"), "lacks HNA-1a, HNA-1c: "), + true, + ); + let ab = run30(&all("0/1", &[("FCGR3B_194", "0/1")])); + c.eq( + "HNA-1a/1b", + (ab.ph("HNA-1"), ab.gt("HNA-1")), + ("HNA-1(a+b+c-)".into(), "FCGR3B*01/FCGR3B*02".into()), + ); + c.eq( + "c.194 that matches: no note about it", + has(&ab.note("HNA-1"), "does not match"), + false, + ); + let aa = run30(&all("1/1", &[("FCGR3B_194", "1/1")])); + c.eq( + "HNA-1a/1a", + (aa.ph("HNA-1"), aa.gt("HNA-1")), + ("HNA-1(a+b-c-)".into(), "FCGR3B*01/FCGR3B*01".into()), + ); + c.eq( + "HNA-1a/1a lacks HNA-1b", + has(&aa.note("HNA-1"), "lacks HNA-1b, HNA-1c: "), + true, + ); + let cc = run30(&[vl("FCGR3B_233", "0/1"), vl("FCGR3B_194", "1/1")]); + c.eq( + "HNA-1c on an FCGR3B*02: HNA-1(a-b+c+), FCGR3B*02/FCGR3B*03", + (cc.ph("HNA-1"), cc.gt("HNA-1")), + ("HNA-1(a-b+c+)".into(), "FCGR3B*02/FCGR3B*03".into()), + ); + let ac = run30(&all("1/1", &[("FCGR3B_194", "1/1"), ("FCGR3B_233", "0/1")])); + c.eq( + "HNA-1c with two FCGR3B*01: b+ (FCGR3B*03 carries HNA-1b) and the extra-copy note", + (ac.ph("HNA-1"), has(&ac.note("HNA-1"), "extra FCGR3B copy")), + ("HNA-1(a+b+c+)".into(), true), + ); + let nc = run30(&[vl("FCGR3B_233", "./."), vl("FCGR3B_194", "1/1")]); + c.eq( + "c.233 with no call: the HNA-1b alleles read *02 or *03, HNA-1c not determined", + ( + nc.ph("HNA-1"), + nc.gt("HNA-1"), + has(&nc.note("HNA-1"), "HNA-1c not determined"), + ), + ( + "HNA-1(a-b+)".into(), + "FCGR3B*02 or *03/FCGR3B*02 or *03".into(), + true, + ), + ); + let mixed = run30(&[vl("FCGR3B_108", "1/1"), vl("FCGR3B_114", "0/1")]); + c.eq( + "markers that disagree: not determined", + mixed.ph("HNA-1").as_str(), + "not determined", + ); + c.eq( + "and the note says why", + has(&mixed.note("HNA-1"), "do not agree on HNA-1a or HNA-1b"), + true, + ); + c.done(); +} diff --git a/rust/crates/og-cn/tests/port_tests_cn_traits.rs b/rust/crates/og-cn/tests/port_tests_cn_traits.rs new file mode 100644 index 00000000..d637b3d3 --- /dev/null +++ b/rust/crates/og-cn/tests/port_tests_cn_traits.rs @@ -0,0 +1,454 @@ +//! `og cn call --locus HBA`: step 35's HBA1/HBA2 count from short-read +//! depth, through the `og cn` command line, on `samtools bedcov` tables made +//! up for each case (summed depth = mean depth x length). +//! +//! At 30x in the flanks, with a quarter of the gene-body depth from reads of +//! MAPQ 0 (as BWA-MEM and minimap2 give on HG002) unless a case says +//! otherwise: +//! 1. no deletion: 30x over HBA2 and HBA1 is four alpha-globin genes; +//! 2. a planted -a3.7-like loss (HBA2 at half depth) is three, a silent +//! carrier, marked unconfirmed; +//! 3. both genes at half depth (--SEA-like) is two, with the warning that +//! depth cannot tell --/aa from -a/-a; +//! 4. a triplication (HBA2 at 1.5x) is five; +//! 5. an estimate between two whole numbers, flanks under 8x, and flanks +//! that disagree are each "not assessed", with the reason; +//! 6. no read of MAPQ 0 over the genes (GIAB's novoalign BAM of HG002) is +//! "not assessed", naming the cause; 1% is too, 3% is called; no gene at +//! all (no reads to judge by) is still 0; +//! 7. the command line: `og cn bed --locus HBA` prints the four regions, +//! `og cn call --table` writes the metric table and exits 0; a table +//! with other regions, or a short row, in either input exits 2, and so +//! does a missing `--bedcov-mapq1`; +//! 8. the demo sample's HBA table (`tests/demo/make_demo_sample.py`) is the +//! one `og cn call` writes for the demo's depths. +//! +//! This was `tests/test_hba_copy_number.py`. + +use std::collections::BTreeMap; +use std::path::{Path, PathBuf}; + +const REPO: &str = concat!(env!("CARGO_MANIFEST_DIR"), "/../../.."); + +/// Step 35's HBA table, in its order. +const ORDER: [&str; 9] = [ + "status", + "alpha_globin_genes", + "estimate", + "hba2_depth", + "hba1_depth", + "flank_left_depth", + "flank_right_depth", + "mapq0_share", + "message", +]; + +fn catalog() -> String { + format!("{REPO}/bin/cn_loci.tsv") +} + +/// `og cn ARGS`: (exit code, stdout, stderr). +fn og_cn(args: &[&str]) -> (u8, String, String) { + let args: Vec = args.iter().map(|a| (*a).to_owned()).collect(); + let (mut out, mut err) = (Vec::new(), Vec::new()); + let code = og_cn::cli::main(&args, &mut out, &mut err); + ( + code, + String::from_utf8(out).unwrap(), + String::from_utf8(err).unwrap(), + ) +} + +/// The HBA row's regions as `og cn bed --locus HBA` prints them: +/// (chrom, start, end, name). +fn regions() -> Vec<(String, i64, i64, String)> { + let (code, out, err) = og_cn(&["bed", "--catalog", &catalog(), "--locus", "HBA"]); + assert_eq!(code, 0, "og cn bed: {err}"); + out.lines() + .map(|l| { + let c: Vec<&str> = l.split('\t').collect(); + ( + c[0].to_owned(), + c[1].parse().unwrap(), + c[2].parse().unwrap(), + c[3].to_owned(), + ) + }) + .collect() +} + +/// A fresh folder for one case. +fn dir(name: &str) -> PathBuf { + let d = std::env::temp_dir().join(format!("og-cn-hba-{}-{name}", std::process::id())); + let _ = std::fs::remove_dir_all(&d); + std::fs::create_dir_all(&d).unwrap(); + d +} + +type Depth = BTreeMap<&'static str, f64>; + +fn depths(hba2: f64, hba1: f64, left: f64, right: f64) -> Depth { + BTreeMap::from([ + ("flank_left", left), + ("HBA2", hba2), + ("HBA1", hba1), + ("flank_right", right), + ]) +} + +fn at(hba2: f64, hba1: f64) -> Depth { + depths(hba2, hba1, 30.0, 30.0) +} + +/// The depths of reads of MAPQ 1 or more: the flanks whole, the genes +/// without the given share. +fn mapq1(depth: &Depth, share: f64) -> Depth { + depth + .iter() + .map(|(n, v)| { + let v = if matches!(*n, "HBA2" | "HBA1") { + v * (1.0 - share) + } else { + *v + }; + (*n, v) + }) + .collect() +} + +/// A bedcov table with the given mean depth per region name. +fn bedcov(path: &Path, depth: &Depth) { + let mut text = String::new(); + for (c, s, e, n) in regions() { + // Truncated toward zero, as samtools' integer sum. + let sum = (depth[n.as_str()] * (e - s) as f64) as i64; + text.push_str(&format!("{c}\t{s}\t{e}\t{n}\t{sum}\n")); + } + std::fs::write(path, text).unwrap(); +} + +fn s(p: &Path) -> String { + p.to_string_lossy().into_owned() +} + +/// The command line of one call on the tables in `d`. +fn call_args(d: &Path, with_mapq1: bool) -> Vec { + let mut a = vec![ + "call".to_owned(), + "--catalog".to_owned(), + catalog(), + "--locus".to_owned(), + "HBA".to_owned(), + "--bedcov".to_owned(), + s(&d.join("cov.tsv")), + ]; + if with_mapq1 { + a.extend(["--bedcov-mapq1".to_owned(), s(&d.join("cov_q1.tsv"))]); + } + a.extend(["--table".to_owned(), s(&d.join("hba.tsv"))]); + a +} + +fn call(args: &[String]) -> (u8, String, String) { + og_cn(&args.iter().map(String::as_str).collect::>()) +} + +/// The metric table `og cn call` wrote, in its order; empty when it wrote none. +fn table(path: &Path) -> Vec<(String, String)> { + let Ok(text) = std::fs::read_to_string(path) else { + return Vec::new(); + }; + text.lines() + .skip(1) + .map(|l| { + let (k, v) = l.split_once('\t').unwrap(); + (k.to_owned(), v.to_owned()) + }) + .collect() +} + +/// Step 35's HBA table for these depths, as `og cn call` writes it. +fn run_share(name: &str, depth: &Depth, share: f64) -> BTreeMap { + let d = dir(name); + bedcov(&d.join("cov.tsv"), depth); + bedcov(&d.join("cov_q1.tsv"), &mapq1(depth, share)); + let (code, _, err) = call(&call_args(&d, true)); + assert_eq!(code, 0, "{name}: og cn call: {err}"); + let t = table(&d.join("hba.tsv")).into_iter().collect(); + let _ = std::fs::remove_dir_all(&d); + t +} + +fn run(name: &str, depth: &Depth) -> BTreeMap { + run_share(name, depth, 0.25) +} + +fn get<'a>(t: &'a BTreeMap, k: &str) -> &'a str { + t.get(k).map_or("", String::as_str) +} + +#[test] +fn no_deletion_is_four_genes_called() { + let r = run("four", &at(30.0, 30.0)); + assert_eq!( + ( + get(&r, "status"), + get(&r, "alpha_globin_genes"), + get(&r, "estimate") + ), + ("called", "4", "4.00"), + "{r:?}" + ); + assert!(get(&r, "message").contains("no deletion seen"), "{r:?}"); +} + +#[test] +fn hba2_at_half_depth_is_three_genes_a_silent_carrier() { + let r = run("three", &at(15.0, 30.0)); + assert_eq!( + (get(&r, "status"), get(&r, "alpha_globin_genes")), + ("called", "3"), + "{r:?}" + ); + assert!(get(&r, "message").contains("silent carrier"), "{r:?}"); +} + +#[test] +fn a_count_under_four_says_it_is_unconfirmed_and_four_does_not() { + let r = run("unconfirmed", &at(15.0, 30.0)); + let m = get(&r, "message"); + assert!( + m.contains("unconfirmed") && m.contains("needs a clinical test"), + "{r:?}" + ); + for (name, d) in [("four-u", at(30.0, 30.0)), ("five-u", at(45.0, 30.0))] { + let r = run(name, &d); + assert!(!get(&r, "message").contains("unconfirmed"), "{name}: {r:?}"); + } +} + +#[test] +fn both_genes_at_half_depth_is_two_and_depth_cannot_tell_which() { + let r = run("two", &at(15.0, 15.0)); + assert_eq!( + (get(&r, "status"), get(&r, "alpha_globin_genes")), + ("called", "2"), + "{r:?}" + ); + assert!( + get(&r, "message").contains("cannot tell these apart"), + "{r:?}" + ); +} + +#[test] +fn hba2_at_one_and_a_half_times_is_five_a_triplication() { + let r = run("five", &at(45.0, 30.0)); + assert_eq!(get(&r, "alpha_globin_genes"), "5", "{r:?}"); + assert!(get(&r, "message").contains("triplication"), "{r:?}"); +} + +#[test] +fn an_estimate_of_3_50_is_not_assessed_naming_both_whole_numbers() { + let r = run("between", &at(22.5, 30.0)); + assert_eq!( + (get(&r, "status"), get(&r, "estimate")), + ("not assessed", "3.50"), + "{r:?}" + ); + assert!(get(&r, "message").contains("between 3 and 4"), "{r:?}"); +} + +#[test] +fn flanks_under_8x_are_not_assessed_too_low() { + let r = run("low", &depths(5.0, 5.0, 6.0, 6.0)); + assert_eq!( + (get(&r, "status"), get(&r, "alpha_globin_genes")), + ("not assessed", "."), + "{r:?}" + ); + assert!(get(&r, "message").contains("too low"), "{r:?}"); +} + +#[test] +fn flanks_that_disagree_are_not_assessed() { + let r = run("disagree", &depths(30.0, 30.0, 30.0, 15.0)); + assert_eq!(get(&r, "status"), "not assessed", "{r:?}"); + assert!(get(&r, "message").contains("disagree"), "{r:?}"); +} + +#[test] +fn no_mapq0_read_over_the_genes_is_not_assessed_naming_the_aligner() { + // The depths of GIAB's novoalign BAM of HG002, which gives no read MAPQ 0. + let r = run_share("novo", &depths(15.5, 25.68, 30.11, 31.25), 0.0); + assert_eq!( + ( + get(&r, "status"), + get(&r, "alpha_globin_genes"), + get(&r, "mapq0_share") + ), + ("not assessed", ".", "0.000"), + "{r:?}" + ); + let m = get(&r, "message"); + assert!( + m.contains("mapping quality 0") && m.contains("novoalign"), + "{r:?}" + ); +} + +#[test] +fn no_mapq0_read_with_an_estimate_of_3_is_not_assessed_never_three() { + let r = run_share("novo-three", &at(15.0, 30.0), 0.0); + assert_eq!( + (get(&r, "status"), get(&r, "estimate")), + ("not assessed", "3.00"), + "{r:?}" + ); + assert!(get(&r, "message").contains("mapping quality 0"), "{r:?}"); +} + +#[test] +fn one_percent_at_mapq0_is_not_assessed_three_percent_is_called() { + let r = run_share("one-pct", &at(30.0, 30.0), 0.01); + assert_eq!(get(&r, "status"), "not assessed", "{r:?}"); + assert!(get(&r, "message").contains("1.0%"), "{r:?}"); + let r = run_share("three-pct", &at(30.0, 30.0), 0.03); + assert_eq!( + ( + get(&r, "status"), + get(&r, "alpha_globin_genes"), + get(&r, "mapq0_share") + ), + ("called", "4", "0.030"), + "{r:?}" + ); +} + +#[test] +fn no_gene_at_all_is_still_called_0() { + let r = run_share("none", &at(0.0, 0.0), 0.0); + assert_eq!( + (get(&r, "status"), get(&r, "alpha_globin_genes")), + ("called", "0"), + "{r:?}" + ); + assert!(get(&r, "message").contains("Bart"), "{r:?}"); +} + +#[test] +fn bed_prints_four_regions_the_genes_between_the_flanks() { + let names: Vec = regions().into_iter().map(|r| r.3).collect(); + assert_eq!(names, ["flank_left", "HBA2", "HBA1", "flank_right"]); +} + +#[test] +fn call_exits_0_and_writes_the_table_in_its_order() { + let d = dir("cli"); + bedcov(&d.join("cov.tsv"), &at(15.0, 30.0)); + bedcov(&d.join("cov_q1.tsv"), &mapq1(&at(15.0, 30.0), 0.25)); + let (code, out, err) = call(&call_args(&d, true)); + let t = table(&d.join("hba.tsv")); + assert_eq!(code, 0, "{err}"); + let keys: Vec<&str> = t.iter().map(|(k, _)| k.as_str()).collect(); + assert_eq!(keys, ORDER); + let m: BTreeMap<_, _> = t.into_iter().collect(); + assert_eq!( + ( + get(&m, "status"), + get(&m, "alpha_globin_genes"), + get(&m, "mapq0_share") + ), + ("called", "3", "0.250") + ); + assert!(out.contains("HBA1/HBA2: three"), "{out}"); + let _ = std::fs::remove_dir_all(&d); +} + +#[test] +fn call_without_bedcov_mapq1_exits_2() { + let d = dir("no-q1"); + bedcov(&d.join("cov.tsv"), &at(15.0, 30.0)); + let (code, _, err) = call(&call_args(&d, false)); + assert_eq!(code, 2, "{err}"); + assert!(err.contains("--bedcov-mapq1"), "{err}"); + let _ = std::fs::remove_dir_all(&d); +} + +#[test] +fn call_refuses_other_regions_or_a_short_row_in_either_table() { + for (name, file) in [("all reads", "cov.tsv"), ("MAPQ 1 or more", "cov_q1.tsv")] { + let d = dir(&format!("broken-{file}")); + let rewrite = |d: &Path| { + bedcov(&d.join("cov.tsv"), &at(15.0, 30.0)); + bedcov(&d.join("cov_q1.tsv"), &mapq1(&at(15.0, 30.0), 0.25)); + }; + rewrite(&d); + let p = d.join(file); + let mut text = std::fs::read_to_string(&p).unwrap(); + text.push_str("chr16\t1\t100\tother\t500\n"); + std::fs::write(&p, text).unwrap(); + let (code, _, err) = call(&call_args(&d, true)); + assert_eq!(code, 2, "a table ({name}) with other regions: {err}"); + assert!(err.contains("differ"), "{name}: {err}"); + rewrite(&d); + std::fs::write(&p, "chr16\t100000\t140000\n").unwrap(); + let (code, _, err) = call(&call_args(&d, true)); + assert_eq!(code, 2, "a short row ({name}): {err}"); + assert!(err.contains("expected chrom"), "{name}: {err}"); + let _ = std::fs::remove_dir_all(&d); + } +} + +/// The `{"name": number, ...}` literal assigned to `var` in `src`. +fn py_dict(src: &str, var: &str) -> Depth { + let line = src + .lines() + .find(|l| l.starts_with(&format!("{var} = {{"))) + .unwrap_or_else(|| panic!("the demo builder has no {var} line")); + let body = line.split_once('{').unwrap().1.rsplit_once('}').unwrap().0; + body.split(',') + .map(|kv| { + let (k, v) = kv.split_once(':').unwrap(); + let k = k.trim().trim_matches('"'); + let name = ["flank_left", "HBA2", "HBA1", "flank_right"] + .into_iter() + .find(|n| *n == k) + .unwrap_or_else(|| panic!("{var}: unknown region {k}")); + (name, v.trim().parse().unwrap()) + }) + .collect() +} + +/// The string the parenthesised literals assigned to `var` in `src` join to. +fn py_str(src: &str, var: &str) -> String { + let start = src + .find(&format!("\n{var} = (")) + .unwrap_or_else(|| panic!("the demo builder has no {var} literal")); + let rest = &src[start..]; + let body = &rest[rest.find('(').unwrap() + 1..rest.find(")\n").unwrap()]; + let mut out = String::new(); + for piece in body.split('"').skip(1).step_by(2) { + out.push_str(&piece.replace("\\t", "\t").replace("\\n", "\n")); + } + out +} + +#[test] +fn the_demo_hba_table_is_what_og_cn_call_writes_for_its_depths() { + let src = std::fs::read_to_string(format!("{REPO}/tests/demo/make_demo_sample.py")).unwrap(); + let depth = py_dict(&src, "HBA_DEPTH"); + let q1 = py_dict(&src, "HBA_Q1"); + let want = py_str(&src, "HBA_TABLE"); + assert!(want.starts_with("metric\tvalue\nstatus\t"), "{want:?}"); + let d = dir("demo"); + bedcov(&d.join("cov.tsv"), &depth); + bedcov( + &d.join("cov_q1.tsv"), + &depth.iter().map(|(n, v)| (*n, v * q1[n])).collect(), + ); + let (code, _, err) = call(&call_args(&d, true)); + assert_eq!(code, 0, "{err}"); + assert_eq!(std::fs::read_to_string(d.join("hba.tsv")).unwrap(), want); + let _ = std::fs::remove_dir_all(&d); +} diff --git a/rust/crates/og-dev/src/lib.rs b/rust/crates/og-dev/src/lib.rs index d2f16ba8..27067a8d 100644 --- a/rust/crates/og-dev/src/lib.rs +++ b/rust/crates/og-dev/src/lib.rs @@ -40,6 +40,9 @@ //! ([`licences`]), `og dev docs-check` ([`docs_checks`]), //! `og dev stub-http` ([`stub_http`]) and `og dev steps-check` //! ([`steps_check`]). +//! - `og dev traits-sources` ([`traits_sources`]): the traits panel's +//! reference bases against a FASTA, and its PubMed ids and rsIDs against +//! PubMed and dbSNP. pub mod build; pub mod contract; @@ -68,13 +71,14 @@ pub mod shlex; pub mod smoke; pub mod steps_check; pub mod stub_http; +pub mod traits_sources; pub mod versions; pub mod versions_env; mod zip; use std::io::Write; -const USAGE: &str = "usage: og dev parity-compare KIND ORACLE NEW\n og dev parity-compare --self-test\n og dev contract --static [--repo DIR] [--contract FILE]\n og dev contract --outputs DIR --leg bash|nextflow --sample S [--observe]\n og dev schema SCHEMA.json DOC.json\n og dev report-only [--repo DIR]\n og dev remnants [--list FILE] [--root DIR]\n og dev fixture comphet|trim-pharmcat [args]\n og dev evidence clashes|hg002-gvcf [args]\n og dev renovate-check LOG [--pins FILE] [--repo DIR]\n og dev helpers --list|--static|--images|--orad-calls [--root DIR]\n og dev helpers --orad --help-file FILE [--orad-version V] [--root DIR]\n og dev build cn-intervals|callability|sv-rule-regions|roh-af|acmg-lof [args]\n og dev scan personal-data|images [args]\n og dev versions check-run|expand|coupled [args]\n og dev pgs trait|label-match|metric-match [args]\n\n og dev pharmcat diplotypes REPORT [--called|--called-lines]\n og dev nf-trace requests|attempts TRACE [options]\n og dev smoke mini-ref|long-reads|eh-catalog|csv-cell|check [args]\n og dev demo-sample [args]\n og dev demo-events [args]\n og dev params-doc [args]\n og dev python-gate [args]\n og dev everything-check [args]\n og dev licences [args]\n og dev docs-check [args]\n og dev stub-http [args]\n og dev steps-check [args]"; +const USAGE: &str = "usage: og dev parity-compare KIND ORACLE NEW\n og dev parity-compare --self-test\n og dev contract --static [--repo DIR] [--contract FILE]\n og dev contract --outputs DIR --leg bash|nextflow --sample S [--observe]\n og dev schema SCHEMA.json DOC.json\n og dev report-only [--repo DIR]\n og dev remnants [--list FILE] [--root DIR]\n og dev fixture comphet|trim-pharmcat [args]\n og dev evidence clashes|hg002-gvcf [args]\n og dev renovate-check LOG [--pins FILE] [--repo DIR]\n og dev helpers --list|--static|--images|--orad-calls [--root DIR]\n og dev helpers --orad --help-file FILE [--orad-version V] [--root DIR]\n og dev build cn-intervals|callability|sv-rule-regions|roh-af|acmg-lof [args]\n og dev scan personal-data|images [args]\n og dev versions check-run|expand|coupled [args]\n og dev pgs trait|label-match|metric-match [args]\n\n og dev pharmcat diplotypes REPORT [--called|--called-lines]\n og dev nf-trace requests|attempts TRACE [options]\n og dev smoke mini-ref|long-reads|eh-catalog|csv-cell|check [args]\n og dev demo-sample [args]\n og dev demo-events [args]\n og dev params-doc [args]\n og dev python-gate [args]\n og dev everything-check [args]\n og dev licences [args]\n og dev docs-check [args]\n og dev stub-http [args]\n og dev steps-check [args]\n og dev traits-sources [--table T] [--fasta F [--min-fasta-rows N]] [--online]"; /// Python's `str(OSError)`: `[Errno 2] No such file or directory: 'path'`. pub(crate) fn py_os_error(e: &std::io::Error, path: &str) -> String { @@ -207,6 +211,7 @@ pub fn main(args: &[String]) -> u8 { Some("docs-check") => run(docs_checks::cli, &args[1..]), Some("stub-http") => run(stub_http::cli, &args[1..]), Some("steps-check") => run(steps_check::cli, &args[1..]), + Some("traits-sources") => run(traits_sources::cli, &args[1..]), Some("-h" | "--help") => { print!("{USAGE}"); 0 diff --git a/rust/crates/og-dev/src/traits_sources.rs b/rust/crates/og-dev/src/traits_sources.rs new file mode 100644 index 00000000..9f3e99df --- /dev/null +++ b/rust/crates/og-dev/src/traits_sources.rs @@ -0,0 +1,728 @@ +//! `og dev traits-sources`: the sources of the curated traits panel +//! (`assets/traits.tsv`, step 42) against the reference and the public +//! records it was curated from. +//! +//! ```text +//! og dev traits-sources [--table TABLE] [--fasta FASTA [--min-fasta-rows N]] [--online] +//! ``` +//! +//! - `--fasta` reads each row's base from FASTA through its `.fai` and +//! names every row whose `ref` differs. Rows on a contig the FASTA lacks +//! are not checked; fewer than `--min-fasta-rows` (default 1) checked rows +//! is a failure, so a FASTA with none of the table's contigs cannot pass. +//! - `--online` looks every PubMed id up in PubMed (NCBI E-utilities) and +//! every rsID up in dbSNP (NCBI Variation Services), with curl: an id +//! that does not resolve, an rsID that is merged or withdrawn, or one +//! whose GRCh38 position, reference base or alleles differ from the row +//! is a failure. +//! +//! Exit 0 when every check passed, 1 when one failed, 2 for a refused +//! command line. The table's schema is `og traits check`'s to judge; run it +//! first. This was the `--fasta` and `--online` part of +//! `tests/test_traits_table.py`. + +use std::collections::BTreeSet; +use std::io::{Read, Seek, SeekFrom, Write}; +use std::path::Path; +use std::time::Duration; + +use pycompat::json::{Value, loads}; +use pycompat::value::truthy; + +const USAGE: &str = "usage: og dev traits-sources [--table TABLE] [--fasta FASTA [--min-fasta-rows N]] [--online]\n"; +const TABLE: &str = "assets/traits.tsv"; +const PUBMED: &str = + "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=pubmed&retmode=json&id="; +const DBSNP: &str = "https://api.ncbi.nlm.nih.gov/variation/v0/refsnp/"; + +/// One row of the table: the columns these checks read. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct Row { + pub chrom: String, + pub pos: u64, + pub rsid: String, + pub ref_: String, + pub alt: String, + pub pmids: String, +} + +/// The rows of a traits table: `#` lines and blank lines skipped, the +/// first other line the header. +pub fn read_table(text: &str) -> Result, String> { + let mut lines = text + .lines() + .enumerate() + .filter(|(_, l)| !l.starts_with('#') && !l.trim().is_empty()); + let Some((_, header)) = lines.next() else { + return Err("no header line".into()); + }; + let header: Vec<&str> = header.split('\t').collect(); + let col = |name: &str| { + header + .iter() + .position(|h| *h == name) + .ok_or_else(|| format!("no {name} column")) + }; + let [chrom, pos, rsid, ref_, alt, pmids] = + ["chrom", "pos", "rsid", "ref", "alt", "pmids"].map(col); + let (chrom, pos, rsid, ref_, alt, pmids) = (chrom?, pos?, rsid?, ref_?, alt?, pmids?); + lines + .map(|(i, l)| { + let c: Vec<&str> = l.split('\t').collect(); + let get = |k: usize| c.get(k).copied().unwrap_or("").to_owned(); + Ok(Row { + chrom: get(chrom), + pos: get(pos) + .parse() + .map_err(|_| format!("line {}: pos '{}' is not a number", i + 1, get(pos)))?, + rsid: get(rsid), + ref_: get(ref_), + alt: get(alt), + pmids: get(pmids), + }) + }) + .collect() +} + +/// (problems, rows checked) of the reference bases against FASTA, read +/// through its `.fai`. +pub fn fasta_errors(fasta: &Path, rows: &[Row]) -> Result<(Vec, usize), String> { + let fai_path = format!("{}.fai", fasta.display()); + let fai = std::fs::read_to_string(&fai_path).map_err(|e| format!("{fai_path}: {e}"))?; + let mut index = std::collections::HashMap::new(); + for l in fai.lines().filter(|l| !l.is_empty()) { + let c: Vec<&str> = l.split('\t').collect(); + let num = |k: usize| -> Result { + c.get(k) + .and_then(|v| v.parse().ok()) + .ok_or_else(|| format!("{fai_path}: not a .fai line: {l}")) + }; + index.insert(c[0], (num(1)?, num(2)?, num(3)?, num(4)?)); + } + let mut fh = std::fs::File::open(fasta).map_err(|e| format!("{}: {e}", fasta.display()))?; + let (mut errs, mut checked) = (Vec::new(), 0); + for r in rows { + let Some(&(length, offset, bases, width)) = index.get(r.chrom.as_str()) else { + continue; + }; + checked += 1; + let i = r.pos.checked_sub(1).ok_or("pos 0")?; + let got = if i >= length || bases == 0 { + "past the end of the contig".to_owned() + } else { + let mut b = [0u8]; + fh.seek(SeekFrom::Start(offset + (i / bases) * width + i % bases)) + .and_then(|_| fh.read_exact(&mut b)) + .map_err(|e| format!("{}: {e}", fasta.display()))?; + (b[0].to_ascii_uppercase() as char).to_string() + }; + if got != r.ref_ { + errs.push(format!( + "{} {}:{}: table ref {}, FASTA {got}", + r.rsid, r.chrom, r.pos, r.ref_ + )); + } + } + Ok((errs, checked)) +} + +fn field<'a>(v: &'a Value, key: &str) -> Option<&'a Value> { + match v { + Value::Object(o) => o.get(key), + _ => None, + } +} + +fn text(v: Option<&Value>) -> String { + match v { + Some(Value::Str(s)) => s.clone(), + Some(Value::Int(i)) => i.to_string(), + Some(Value::Null) | None => String::new(), + Some(other) => format!("{other:?}"), + } +} + +/// The sorted, distinct PubMed ids of the table. +pub fn pmids(rows: &[Row]) -> Vec { + let ids: BTreeSet<(u64, String)> = rows + .iter() + .flat_map(|r| r.pmids.split(',')) + .map(str::trim) + .filter(|p| !p.is_empty()) + .map(|p| (p.parse().unwrap_or(u64::MAX), p.to_owned())) + .collect(); + ids.into_iter().map(|(_, p)| p).collect() +} + +/// The PubMed ids that do not resolve in an esummary answer. +pub fn pubmed_errors(answer: &Value, ids: &[String]) -> Vec { + let result = field(answer, "result"); + ids.iter() + .filter_map(|p| { + let rec = result.and_then(|r| field(r, p)); + let error = rec.and_then(|r| field(r, "error")); + let titled = rec.and_then(|r| field(r, "title")).is_some_and(truthy); + (error.is_some() || !titled).then(|| { + let why = if error.is_some() { + text(error) + } else { + "no record".into() + }; + format!("PMID {p} does not resolve in PubMed ({why})") + }) + }) + .collect() +} + +/// `NC_000001` to `NC_000022` as `chr1` to `chr22`. +fn contig(seq_id: &str) -> String { + let acc = seq_id.split('.').next().unwrap_or(seq_id); + acc.strip_prefix("NC_0000") + .and_then(|n| n.parse::().ok()) + .filter(|n| (1..=22).contains(n) && acc.len() == 9) + .map_or_else(|| seq_id.to_owned(), |n| format!("chr{n}")) +} + +/// What is wrong with a row against its dbSNP record, if anything. +pub fn dbsnp_error(r: &Row, snp: &Value) -> Option { + let Some(psd) = field(snp, "primary_snapshot_data").filter(|v| truthy(v)) else { + return Some(format!( + "{}: not a current dbSNP record (merged or withdrawn)", + r.rsid + )); + }; + let placements = match field(psd, "placements_with_allele") { + Some(Value::Array(a)) => a.as_slice(), + _ => &[], + }; + let grch38 = |pl: &&Value| { + let first = match field(pl, "placement_annot") + .and_then(|a| field(a, "seq_id_traits_by_assembly")) + { + Some(Value::Array(t)) => t.first(), + _ => None, + }; + field(pl, "is_ptlp").is_some_and(truthy) + && first + .and_then(|t| field(t, "assembly_name")) + .is_some_and(|n| matches!(n, Value::Str(s) if s.starts_with("GRCh38"))) + }; + let Some(place) = placements.iter().find(grch38) else { + return Some(format!("{}: no GRCh38 placement in dbSNP", r.rsid)); + }; + let spdi: Vec<&Value> = match field(place, "alleles") { + Some(Value::Array(a)) => a + .iter() + .filter_map(|a| field(a, "allele").and_then(|x| field(x, "spdi"))) + .collect(), + _ => vec![], + }; + let Some(first) = spdi.first() else { + return Some(format!("{}: no allele in dbSNP's GRCh38 placement", r.rsid)); + }; + let chrom = contig(&text(field(place, "seq_id"))); + let pos = match field(first, "position") { + Some(Value::Int(p)) => p + 1, + _ => return Some(format!("{}: dbSNP's allele has no position", r.rsid)), + }; + let ref_ = text(field(first, "deleted_sequence")); + let alts: BTreeSet = spdi + .iter() + .map(|s| { + ( + text(field(s, "deleted_sequence")), + text(field(s, "inserted_sequence")), + ) + }) + .filter(|(d, i)| d != i) + .map(|(_, i)| i) + .collect(); + if (chrom.as_str(), pos, ref_.as_str()) + != (r.chrom.as_str(), i128::from(r.pos), r.ref_.as_str()) + { + Some(format!( + "{}: table {}:{} {}, dbSNP GRCh38 {chrom}:{pos} {ref_}", + r.rsid, r.chrom, r.pos, r.ref_ + )) + } else if !alts.contains(&r.alt) { + Some(format!( + "{}: alt {} is not one of dbSNP's alleles {alts:?}", + r.rsid, r.alt + )) + } else { + None + } +} + +/// A JSON answer by HTTP GET. +pub type Fetch<'a> = dyn FnMut(&str) -> Result + 'a; + +/// Problems with the sources: PubMed ids that do not resolve, and rsIDs +/// whose GRCh38 placement, reference base or alleles differ from dbSNP's. +/// `pause` is the wait between two dbSNP requests (NCBI allows three a +/// second without a key). +pub fn online_errors( + rows: &[Row], + get: &mut Fetch<'_>, + pause: Duration, + out: &mut dyn Write, +) -> Vec { + let ids = pmids(rows); + let mut errs = match get(&format!("{PUBMED}{}", ids.join(","))) { + Ok(answer) => pubmed_errors(&answer, &ids), + Err(e) => vec![format!("lookup failed: {e}")], + }; + let _ = writeln!(out, " {} PubMed ids looked up", ids.len()); + for r in rows { + std::thread::sleep(pause); + let id = r.rsid.strip_prefix("rs").unwrap_or(&r.rsid); + match get(&format!("{DBSNP}{id}")) { + Ok(snp) => errs.extend(dbsnp_error(r, &snp)), + Err(e) => errs.push(format!("{}: dbSNP lookup failed: {e}", r.rsid)), + } + } + let _ = writeln!(out, " {} rsIDs looked up in dbSNP", rows.len()); + errs +} + +/// GET `url` with curl, four tries, and parse the answer as JSON. +fn curl_json(url: &str) -> Result { + let mut last = String::new(); + for t in 0..4u64 { + if t > 0 { + std::thread::sleep(Duration::from_secs(5 * t)); + } + match std::process::Command::new("curl") + .args(["-sS", "--fail", "--max-time", "60", url]) + .output() + { + Ok(o) if o.status.success() => match loads(&String::from_utf8_lossy(&o.stdout)) { + Ok(v) => return Ok(v), + Err(e) => last = format!("not JSON: {e:?}"), + }, + Ok(o) => last = String::from_utf8_lossy(&o.stderr).trim().to_owned(), + Err(e) => last = format!("curl: {e}"), + } + } + Err(format!("{url}: {last}")) +} + +struct Opts { + table: String, + fasta: Option, + min_fasta_rows: usize, + online: bool, +} + +fn parse(args: &[String]) -> Result { + let mut o = Opts { + table: TABLE.into(), + fasta: None, + min_fasta_rows: 1, + online: false, + }; + let mut it = args.iter(); + while let Some(a) = it.next() { + let mut value = |name: &str| { + it.next() + .cloned() + .ok_or_else(|| format!("argument {name}: expected one argument")) + }; + match a.as_str() { + "--table" => o.table = value("--table")?, + "--fasta" => o.fasta = Some(value("--fasta")?), + "--min-fasta-rows" => { + let v = value("--min-fasta-rows")?; + o.min_fasta_rows = v + .parse() + .map_err(|_| format!("argument --min-fasta-rows: invalid int value: '{v}'"))?; + } + "--online" => o.online = true, + other => return Err(format!("unrecognized arguments: {other}")), + } + } + if o.fasta.is_none() && !o.online { + return Err("nothing to check: give --fasta, --online or both".into()); + } + Ok(o) +} + +/// `og dev traits-sources ARGS`. +pub fn cli(args: &[String], out: &mut dyn Write, err: &mut dyn Write) -> u8 { + if args.iter().any(|a| a == "-h" || a == "--help") { + let _ = out.write_all(USAGE.as_bytes()); + return 0; + } + let o = match parse(args) { + Ok(o) => o, + Err(e) => { + let _ = writeln!(err, "{USAGE}og dev traits-sources: error: {e}"); + return 2; + } + }; + let rows = match std::fs::read_to_string(&o.table) + .map_err(|e| e.to_string()) + .and_then(|t| read_table(&t)) + { + Ok(r) => r, + Err(e) => { + let _ = writeln!(err, "ERROR: {}: {e}", o.table); + return 1; + } + }; + let mut fails = 0usize; + let mut check = |out: &mut dyn Write, desc: &str, problems: &[String]| { + if problems.is_empty() { + let _ = writeln!(out, "[PASS] {desc}"); + } else { + fails += 1; + let _ = writeln!(out, "[FAIL] {desc}"); + for p in problems { + let _ = writeln!(out, " {p}"); + } + } + }; + if let Some(fasta) = &o.fasta { + match fasta_errors(Path::new(fasta), &rows) { + Ok((errs, n)) => { + let _ = writeln!( + out, + " ref checked against {fasta}: {n} rows (the others are on contigs it lacks)" + ); + check( + out, + &format!("every reference base matches the FASTA ({n} rows checked)"), + &errs, + ); + let few = (n < o.min_fasta_rows) + .then(|| format!("only {n} rows were checked")) + .into_iter() + .collect::>(); + check( + out, + &format!( + "at least {} rows were checked against the FASTA", + o.min_fasta_rows + ), + &few, + ); + } + Err(e) => check(out, "the FASTA could be read", &[e]), + } + } + if o.online { + let errs = online_errors(&rows, &mut curl_json, Duration::from_millis(400), out); + check( + out, + "every PMID resolves and every rsID matches dbSNP's GRCh38 record", + &errs, + ); + } + let _ = writeln!( + out, + "\nRESULT: {}", + if fails == 0 { + "ALL PASS".to_owned() + } else { + format!("{fails} FAILED") + } + ); + u8::from(fails > 0) +} + +#[cfg(test)] +mod tests { + use super::*; + use std::path::PathBuf; + + const REPO: &str = concat!(env!("CARGO_MANIFEST_DIR"), "/../../.."); + + fn shipped() -> Vec { + read_table(&std::fs::read_to_string(format!("{REPO}/{TABLE}")).unwrap()).unwrap() + } + + fn scratch(name: &str) -> PathBuf { + let d = + std::env::temp_dir().join(format!("og-traits-sources-{}-{name}", std::process::id())); + let _ = std::fs::remove_dir_all(&d); + std::fs::create_dir_all(&d).unwrap(); + d + } + + /// A FASTA whose chr2, 60 bases a line, holds the lactase row's base at + /// its position and A everywhere else. + fn lactase_fasta(d: &Path) -> (PathBuf, Row) { + let lct = shipped() + .into_iter() + .find(|r| r.rsid == "rs4988235") + .unwrap(); + let n = usize::try_from(lct.pos).unwrap(); + let mut seq = vec![b'A'; n]; + seq[n - 1] = lct.ref_.as_bytes()[0]; + let body: Vec = seq + .chunks(60) + .flat_map(|c| c.iter().copied().chain([b'\n'])) + .collect(); + let fa = d.join("ref.fasta"); + std::fs::write(&fa, [b">chr2\n".as_slice(), &body].concat()).unwrap(); + std::fs::write(d.join("ref.fasta.fai"), format!("chr2\t{n}\t6\t60\t61\n")).unwrap(); + (fa, lct) + } + + #[test] + fn the_shipped_table_reads() { + let rows = shipped(); + assert!(rows.len() >= 30, "{}", rows.len()); + assert!( + rows.iter() + .all(|r| r.rsid.starts_with("rs") && !r.pmids.is_empty()) + ); + } + + #[test] + fn fasta_check_the_right_base_passes() { + let d = scratch("right"); + let (fa, lct) = lactase_fasta(&d); + assert_eq!(fasta_errors(&fa, &[lct]).unwrap(), (vec![], 1)); + let _ = std::fs::remove_dir_all(&d); + } + + #[test] + fn fasta_check_control_a_planted_wrong_base_fails_naming_the_row() { + let d = scratch("wrong"); + let (fa, lct) = lactase_fasta(&d); + let other = ["A", "C", "G", "T"] + .into_iter() + .find(|b| *b != lct.ref_) + .unwrap(); + let wrong = Row { + ref_: other.into(), + ..lct.clone() + }; + let (errs, n) = fasta_errors(&fa, &[wrong]).unwrap(); + assert_eq!(n, 1); + assert_eq!( + errs, + [format!( + "rs4988235 chr2:{}: table ref {other}, FASTA {}", + lct.pos, lct.ref_ + )] + ); + let _ = std::fs::remove_dir_all(&d); + } + + #[test] + fn fasta_check_a_row_on_a_contig_the_fasta_lacks_is_not_checked() { + let d = scratch("contig"); + let (fa, _) = lactase_fasta(&d); + let other = shipped().into_iter().find(|r| r.chrom != "chr2").unwrap(); + assert_eq!(fasta_errors(&fa, &[other]).unwrap(), (vec![], 0)); + let _ = std::fs::remove_dir_all(&d); + } + + #[test] + fn the_command_line_fails_on_a_wrong_base_and_too_few_rows() { + let d = scratch("cli"); + let (fa, _) = lactase_fasta(&d); + let fa = fa.to_string_lossy().into_owned(); + let lct = shipped() + .into_iter() + .find(|r| r.rsid == "rs4988235") + .unwrap(); + // The shipped table without its other chr2 rows: one row on the + // FASTA's contig, every other row on a contig it lacks. With + // `wrong`, the lactase row's ref is that base. + let text = std::fs::read_to_string(format!("{REPO}/{TABLE}")).unwrap(); + let write = |name: &str, wrong: Option<&str>| { + let kept: String = text + .lines() + .filter_map(|l| { + let mut c: Vec<&str> = l.split('\t').collect(); + let lactase = c.get(2) == Some(&"rs4988235"); + if c.first() == Some(&"chr2") && !lactase { + return None; + } + if let (true, Some(w)) = (lactase, wrong) { + c[4] = w; + } + Some(c.join("\t") + "\n") + }) + .collect(); + let p = d.join(name); + std::fs::write(&p, kept).unwrap(); + p.to_string_lossy().into_owned() + }; + let table = write("one.tsv", None); + let run = |args: &[&str]| { + let args: Vec = args.iter().map(|a| (*a).to_owned()).collect(); + let (mut out, mut err) = (Vec::new(), Vec::new()); + let code = cli(&args, &mut out, &mut err); + ( + code, + String::from_utf8(out).unwrap(), + String::from_utf8(err).unwrap(), + ) + }; + let (code, out, _) = run(&["--table", &table, "--fasta", &fa]); + assert_eq!(code, 0, "{out}"); + assert!( + out.contains(&format!("ref checked against {fa}: 1 rows")), + "{out}" + ); + let (code, out, _) = run(&["--table", &table, "--fasta", &fa, "--min-fasta-rows", "2"]); + assert_eq!(code, 1, "{out}"); + assert!(out.contains("only 1 rows were checked"), "{out}"); + let wrong = if lct.ref_ == "C" { "T" } else { "C" }; + let planted = write("planted.tsv", Some(wrong)); + let (code, out, _) = run(&["--table", &planted, "--fasta", &fa]); + assert_eq!(code, 1, "{out}"); + assert!( + out.contains(&format!( + "rs4988235 chr2:{}: table ref {wrong}, FASTA {}", + lct.pos, lct.ref_ + )), + "{out}" + ); + assert!(out.contains("RESULT: 1 FAILED"), "{out}"); + let (code, _, err) = run(&["--table", &table]); + assert_eq!(code, 2); + assert!(err.contains("nothing to check"), "{err}"); + let (code, _, err) = run(&["--fasta", &fa, "--bogus"]); + assert_eq!(code, 2); + assert!(err.contains("unrecognized arguments: --bogus"), "{err}"); + let _ = std::fs::remove_dir_all(&d); + } + + fn row(chrom: &str, pos: u64, ref_: &str, alt: &str) -> Row { + Row { + chrom: chrom.into(), + pos, + rsid: "rs1801133".into(), + ref_: ref_.into(), + alt: alt.into(), + pmids: "7647779".into(), + } + } + + /// A dbSNP refsnp answer with one GRCh38 placement (0-based position) + /// and one GRCh37 placement before it. + fn snp(seq_id: &str, pos0: i64, del: &str, ins: &[&str]) -> Value { + snp_on("GRCh38.p14", seq_id, pos0, del, ins) + } + + fn snp_on(assembly: &str, seq_id: &str, pos0: i64, del: &str, ins: &[&str]) -> Value { + let alleles: Vec = [del] + .iter() + .chain(ins) + .map(|i| { + format!( + r#"{{"allele": {{"spdi": {{"seq_id": "{seq_id}", "position": {pos0}, "deleted_sequence": "{del}", "inserted_sequence": "{i}"}}}}}}"# + ) + }) + .collect(); + loads(&format!( + r#"{{"primary_snapshot_data": {{"placements_with_allele": [ + {{"seq_id": "NC_000001.10", "is_ptlp": false, "placement_annot": {{"seq_id_traits_by_assembly": [{{"assembly_name": "GRCh37.p13"}}]}}, "alleles": []}}, + {{"seq_id": "{seq_id}", "is_ptlp": true, "placement_annot": {{"seq_id_traits_by_assembly": [{{"assembly_name": "{assembly}"}}]}}, "alleles": [{}]}}]}}}}"#, + alleles.join(", ") + )) + .unwrap() + } + + #[test] + fn dbsnp_a_matching_record_passes() { + let s = snp("NC_000001.11", 11_796_320, "G", &["A"]); + assert_eq!(dbsnp_error(&row("chr1", 11_796_321, "G", "A"), &s), None); + } + + #[test] + fn dbsnp_controls_another_position_base_or_allele_fails() { + let s = snp("NC_000001.11", 11_796_320, "G", &["A"]); + assert_eq!( + dbsnp_error(&row("chr1", 11_796_322, "G", "A"), &s).as_deref(), + Some("rs1801133: table chr1:11796322 G, dbSNP GRCh38 chr1:11796321 G") + ); + assert!(dbsnp_error(&row("chr1", 11_796_321, "C", "A"), &s).is_some()); + assert_eq!( + dbsnp_error(&row("chr1", 11_796_321, "G", "T"), &s).as_deref(), + Some("rs1801133: alt T is not one of dbSNP's alleles {\"A\"}") + ); + let merged = loads(r#"{"merged_snapshot_data": {}}"#).unwrap(); + assert_eq!( + dbsnp_error(&row("chr1", 1, "G", "A"), &merged).as_deref(), + Some("rs1801133: not a current dbSNP record (merged or withdrawn)") + ); + let only37 = snp_on("GRCh37.p13", "NC_000001.10", 11_856_377, "G", &["A"]); + assert_eq!( + dbsnp_error(&row("chr1", 1, "G", "A"), &only37).as_deref(), + Some("rs1801133: no GRCh38 placement in dbSNP") + ); + } + + #[test] + fn contigs_are_named_as_the_table_names_them() { + assert_eq!(contig("NC_000001.11"), "chr1"); + assert_eq!(contig("NC_000022.11"), "chr22"); + assert_eq!(contig("NC_000023.11"), "NC_000023.11"); + assert_eq!(contig("NT_187361.1"), "NT_187361.1"); + } + + #[test] + fn pubmed_an_id_without_a_record_or_with_an_error_fails() { + let answer = loads( + r#"{"result": {"uids": ["7647779", "99999999"], "7647779": {"title": "A candidate genetic risk factor"}, + "99999999": {"uid": "99999999", "error": "cannot get document summary"}}}"#, + ) + .unwrap(); + let ids = vec!["7647779".to_owned(), "99999999".into(), "123".into()]; + assert_eq!( + pubmed_errors(&answer, &ids), + [ + "PMID 99999999 does not resolve in PubMed (cannot get document summary)", + "PMID 123 does not resolve in PubMed (no record)" + ] + ); + } + + #[test] + fn online_asks_pubmed_once_and_dbsnp_per_row() { + let rows = vec![ + row("chr1", 11_796_321, "G", "A"), + Row { + rsid: "rs671".into(), + pmids: "99999999,7647779".into(), + ..row("chr12", 111_803_962, "G", "A") + }, + ]; + let mut asked = Vec::new(); + let mut get = |url: &str| -> Result { + asked.push(url.to_owned()); + if url.starts_with(PUBMED) { + return loads(r#"{"result": {"7647779": {"title": "t"}}}"#) + .map_err(|e| format!("{e:?}")); + } + if url.ends_with("/671") { + return Err("timeout".into()); + } + Ok(snp("NC_000001.11", 11_796_320, "G", &["A"])) + }; + let mut out = Vec::new(); + let errs = online_errors(&rows, &mut get, Duration::ZERO, &mut out); + assert_eq!( + errs, + [ + "PMID 99999999 does not resolve in PubMed (no record)", + "rs671: dbSNP lookup failed: timeout" + ] + ); + assert_eq!(asked[0], format!("{PUBMED}7647779,99999999")); + assert_eq!( + &asked[1..], + [format!("{DBSNP}1801133"), format!("{DBSNP}671")] + ); + let out = String::from_utf8(out).unwrap(); + assert!( + out.contains("2 PubMed ids looked up") && out.contains("2 rsIDs looked up in dbSNP") + ); + } +} diff --git a/rust/crates/og-steps/tests/traits_table.rs b/rust/crates/og-steps/tests/traits_table.rs new file mode 100644 index 00000000..7dac1a0e --- /dev/null +++ b/rust/crates/og-steps/tests/traits_table.rs @@ -0,0 +1,65 @@ +//! `assets/traits.tsv`, the curated traits panel of step 42: it holds at +//! least 30 rows and passes `og traits check`, and a copy with a planted +//! wrong row fails it. The row-by-row schema rules are the unit tests in +//! `og_steps::traits`; the reference bases, PubMed ids and dbSNP records are +//! `og dev traits-sources`, which the e2e case `traits-panel-1` runs on the +//! fixture reference. +//! +//! This was the schema part of `tests/test_traits_table.py`. + +use std::path::PathBuf; + +const REPO: &str = concat!(env!("CARGO_MANIFEST_DIR"), "/../../.."); + +fn table() -> String { + format!("{REPO}/assets/traits.tsv") +} + +/// The table's data rows: no `#` line, no blank line, no header. +fn rows(text: &str) -> Vec<&str> { + text.lines() + .filter(|l| !l.starts_with('#') && !l.trim().is_empty()) + .skip(1) + .collect() +} + +fn check(path: &str) -> u8 { + og_steps::traits::cli(&["check".to_owned(), "--table".to_owned(), path.to_owned()]) +} + +fn scratch(name: &str) -> PathBuf { + let d = std::env::temp_dir().join(format!("og-traits-table-{}-{name}", std::process::id())); + let _ = std::fs::remove_dir_all(&d); + std::fs::create_dir_all(&d).unwrap(); + d +} + +#[test] +fn the_table_has_at_least_30_rows() { + let text = std::fs::read_to_string(table()).unwrap(); + let n = rows(&text).len(); + assert!(n >= 30, "assets/traits.tsv has {n} rows"); +} + +#[test] +fn the_table_passes_og_traits_check() { + assert_eq!(check(&table()), 0); +} + +#[test] +fn control_a_planted_bad_row_fails_og_traits_check() { + let text = std::fs::read_to_string(table()).unwrap(); + let lactase = rows(&text) + .into_iter() + .find(|l| l.split('\t').nth(2) == Some("rs4988235")) + .expect("the lactase row rs4988235"); + // The same rsID twice and a reference base that is no base. + let mut cols: Vec<&str> = lactase.split('\t').collect(); + cols[4] = "N"; + let planted = format!("{text}{}\n", cols.join("\t")); + let d = scratch("planted"); + let p = d.join("traits.tsv"); + std::fs::write(&p, planted).unwrap(); + assert_eq!(check(&p.to_string_lossy()), 1); + let _ = std::fs::remove_dir_all(&d); +} diff --git a/rust/og/src/main.rs b/rust/og/src/main.rs index c20fad8a..2ff4c9be 100644 --- a/rust/og/src/main.rs +++ b/rust/og/src/main.rs @@ -129,6 +129,7 @@ const USAGE: &str = "usage: og [args] og dev docs-check [args] og dev stub-http [args] og dev steps-check [args] + og dev traits-sources [args] og dev freshness [--out FILE] [--sections a,b] | --update-issue BODY --repo O/N | --self-test og version og --tree-hash diff --git a/tests/demo/make_demo_sample.py b/tests/demo/make_demo_sample.py index c6efbb91..d24b77f9 100755 --- a/tests/demo/make_demo_sample.py +++ b/tests/demo/make_demo_sample.py @@ -29,7 +29,7 @@ og paraphase table, step 39's from bin/couple_report.py and step 42's from og traits table, each fed made-up inputs; step 46's table and step 35's HBA1/HBA2 table are what og cn call writes for made-up depths - (HBA_TABLE and LPA_TABLE; tests/test_hba_copy_number.py and + (HBA_TABLE and LPA_TABLE; og-cn's cargo test port_tests_cn_traits and tests/test_lpa_kiv2.py check that og still writes them), step 49's depth rows with og chip's detection limit, at made-up depths; the tables of steps 02, 40, 41, 45, 47, 48, 50, 51, 52 and 53 are written diff --git a/tests/e2e/traits-panel-1.sh b/tests/e2e/traits-panel-1.sh index e6cb5676..50b5fd9c 100644 --- a/tests/e2e/traits-panel-1.sh +++ b/tests/e2e/traits-panel-1.sh @@ -3,7 +3,7 @@ # Nextflow mirror: # - assets/traits.tsv: every reference base is the fixture reference's base # (on the contigs it has), every PubMed id resolves and every rsID's GRCh38 -# position and alleles are dbSNP's (tests/test_traits_table.py --fasta +# position and alleles are dbSNP's (og dev traits-sources --fasta # --online); control: a copy with one wrong reference base and one PubMed # id that does not exist must fail, naming both; # - scripts/42-traits.sh: one row per table row plus the APOE line; the @@ -28,10 +28,11 @@ GVCF="${SAMPLE}/vcf/${SAMPLE}.g.vcf.gz" check "case 21 wrote a gVCF" vcf_ok "$GVCF" # --- the table: reference bases, PubMed ids, dbSNP -------------------------------- -python3 "${REPO}/tests/test_traits_table.py" --fasta "$REF" --min-fasta-rows 20 --online > "${CASE_TMP}/table.log" 2>&1 +OG_BIN="${OG:-${REPO}/bin/og}" +"$OG_BIN" dev traits-sources --table "$TABLE" --fasta "$REF" --min-fasta-rows 20 --online > "${CASE_TMP}/table.log" 2>&1 TRC=$? cat "${CASE_TMP}/table.log" -check_eq "tests/test_traits_table.py --fasta --online exits 0" "$TRC" 0 +check_eq "og dev traits-sources --fasta --online exits 0" "$TRC" 0 check "the reference bases were checked on the fixture contigs" has 'ref checked against .*: [0-9]+ rows' "$(cat "${CASE_TMP}/table.log")" # Control: one wrong reference base (rs1801133, chr1, a fixture contig) and one @@ -40,7 +41,7 @@ BAD="${CASE_TMP}/traits-planted.tsv" awk -F'\t' -v OFS='\t' '$3 == "rs1801133" { $5 = "C" } $3 == "rs671" { $13 = "99999999" } { print }' "$TABLE" > "$BAD" check_eq "control: the planted copy differs from the table in two lines" \ "$(diff "$TABLE" "$BAD" | grep -c '^>' || true)" 2 -python3 "${REPO}/tests/test_traits_table.py" --table "$BAD" --fasta "$REF" --online > "${CASE_TMP}/table-bad.log" 2>&1 +"$OG_BIN" dev traits-sources --table "$BAD" --fasta "$REF" --online > "${CASE_TMP}/table-bad.log" 2>&1 BRC=$? BADLOG=$(cat "${CASE_TMP}/table-bad.log") check "control: the planted copy fails (exit ${BRC})" test "$BRC" -ne 0 diff --git a/tests/test_bloodtype_hpa_hna.py b/tests/test_bloodtype_hpa_hna.py deleted file mode 100644 index 8484f038..00000000 --- a/tests/test_bloodtype_hpa_hna.py +++ /dev/null @@ -1,205 +0,0 @@ -#!/usr/bin/env python3 -"""og bloodtype report: the platelet (HPA) and neutrophil (HNA) antigens of step 41. - -Each case writes a synthetic one-sample VCF with planted genotypes at the HPA and -HNA sites of assets/bloodtype_markers.tsv, a samtools bedcov file with 30 reads -everywhere (so a site with no record is the reference allele, confidence high), -runs og bloodtype report and checks the rows it writes: - - 1. the table: every HPA and HNA system has its site(s), at the GRCh38 position - and REF base Ensembl gave for its dbSNP id (FCGR3B's on FCGR3B, not FCGR3A) - 2. each two-allele antigen (HPA-1 to -6, -15, HNA-3, -4, -5) for 0/0, 0/1 and - 1/1 of its planted variant: phenotype and genotype - 3. the meaning: a person who lacks an antigen gets the table's meaning in the - note (HPA-1b/1b: FNAIT and anti-HPA-1a), a person with both antigens none - 4. HNA-1: FCGR3B*01 and *02 from the four markers, *03 (HNA-1c, which carries - HNA-1b too), c.233 with no call (*02 or *03), markers that disagree (not - determined), c.194 that disagrees (a note), and the confidence is always - low (FCGR3A, copy number) - -og is found by tests/og_binary.py (OG, or the release build of rust/). -Run: OG=rust/target/release/og python3 tests/test_bloodtype_hpa_hna.py -""" -import csv -import os -import subprocess -import sys -import tempfile - -REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) -MARKERS = os.path.join(REPO, "assets", "bloodtype_markers.tsv") -CN_LOCI = os.path.join(REPO, "bin", "cn_loci.tsv") -sys.path.insert(0, os.path.join(REPO, "tests")) -from og_binary import og_binary # noqa: E402 - -OG = og_binary() -FAILS = [] - - -def read_markers(): - """The marker table's sites by name (pos is the VCF position), and the - RhD depth regions og cn bed --locus RHD prints, by name.""" - with open(MARKERS) as f: - lines = [ln for ln in f if not ln.startswith("#") and ln.strip()] - sites = {} - for r in csv.DictReader(lines, delimiter="\t"): - r["start"], r["end"] = int(r["start"]), int(r["end"]) - r["pos"] = r["end"] - sites[r["name"]] = r - bed = subprocess.run([OG, "cn", "bed", "--catalog", CN_LOCI, "--locus", "RHD"], capture_output=True, text=True, - check=True).stdout - regions = {} - for ln in bed.splitlines(): - chrom, start, end, name = ln.split("\t")[:4] - regions[name] = {"chrom": chrom, "start": int(start), "end": int(end), "name": name} - return sites, regions - - -SITES, REGIONS = read_markers() - - -def check(desc, got, want): - if got == want: - print(f"PASS {desc}") - else: - print(f"FAIL {desc}: got {got!r}, want {want!r}") - FAILS.append(desc) - - -def rec(name, gt): - r = SITES[name] - return "\t".join([r["chrom"], str(r["pos"]), ".", r["ref"], r["alt"], "50", "PASS", ".", "GT", gt]) + "\n" - - -def run(records=()): - """{system: row} of the table the script writes, or the error text.""" - with tempfile.TemporaryDirectory() as tmp: - vcf, dp = os.path.join(tmp, "s.vcf"), os.path.join(tmp, "d.tsv") - with open(vcf, "w") as f: - f.write("##fileformat=VCFv4.2\n#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT\tS\n" - + "".join(records)) - with open(dp, "w") as f: - for r in SITES.values(): - cols = [r["chrom"], str(r["start"]), str(r["end"]), r["kind"], r["system"], r["name"]] - f.write("\t".join(cols + [str(30 * (r["end"] - r["start"]))]) + "\n") - for r in REGIONS.values(): - cols = [r["chrom"], str(r["start"]), str(r["end"]), r["name"]] - f.write("\t".join(cols + [str(30 * (r["end"] - r["start"]))]) + "\n") - p = subprocess.run([OG, "bloodtype", "report", "--cn-loci", CN_LOCI, "--sample", "S", "--vcf", vcf, - "--markers", MARKERS, "--depth", dp, "--out", os.path.join(tmp, "t.tsv"), - "--evidence", os.path.join(tmp, "e.tsv")], capture_output=True, text=True) - if p.returncode != 0: - return p.stderr.strip() or f"exit {p.returncode}" - with open(os.path.join(tmp, "t.tsv")) as f: - return {r["system"]: r for r in csv.DictReader(f, delimiter="\t")} - - -def cell(t, system, col): - if not isinstance(t, dict): - return t - return t[system][col] if system in t else f"no {system} row" - - -# 1. the table -want_sites = { - # name: (system, chrom, GRCh38 position, REF, ALT, rsid) - "HPA1_176": ("HPA-1", "chr17", 47283364, "T", "C", "rs5918"), - "HPA2_482": ("HPA-2", "chr17", 4933086, "C", "T", "rs6065"), - "HPA3_2621": ("HPA-3", "chr17", 44375697, "A", "C", "rs5911"), - "HPA4_506": ("HPA-4", "chr17", 47284587, "G", "A", "rs5917"), - "HPA5_1600": ("HPA-5", "chr5", 53062927, "G", "A", "rs1801106"), - "HPA6_1544": ("HPA-6", "chr17", 47292422, "G", "A", "rs13306487"), - "HPA15_2108": ("HPA-15", "chr6", 73783709, "A", "C", "rs10455097"), - "FCGR3B_108": ("HNA-1", "chr1", 161629989, "G", "C", "rs200688856"), - "FCGR3B_114": ("HNA-1", "chr1", 161629983, "A", "G", "rs527909462"), - "FCGR3B_194": ("HNA-1", "chr1", 161629903, "T", "C", "rs448740"), - "FCGR3B_233": ("HNA-1", "chr1", 161629864, "G", "T", "rs5030738"), - "FCGR3B_244": ("HNA-1", "chr1", 161629853, "T", "C", "rs147574249"), - "FCGR3B_316": ("HNA-1", "chr1", 161629781, "T", "C", "rs2290834"), - "HNA3_461": ("HNA-3", "chr19", 10631494, "A", "G", "rs2288904"), - "HNA4_230": ("HNA-4", "chr16", 31265490, "G", "A", "rs1143679"), - "HNA5_2372": ("HNA-5", "chr16", 30506720, "G", "C", "rs2230433"), -} -for n, w in want_sites.items(): - r = SITES.get(n) - got = (r["system"], r["chrom"], r["pos"], r["ref"], r["alt"], r["rsid"]) if r else None - check(f"1 {n} at its GRCh38 position", got, w) -systems = subprocess.run([OG, "bloodtype", "systems"], capture_output=True, text=True, check=True).stdout.split("\n") -for s in ("HPA-6", "HNA-1", "HNA-3", "HNA-4", "HNA-5"): - check(f"1 {s} is a system of the table", s in systems, True) -check("1 every HPA and HNA site has a meaning", - sorted(n for n, r in SITES.items() if r["system"][:4] in ("HPA-", "HNA-") and not r.get("meaning")), []) -check("1 no red cell site has a meaning", - sorted(n for n, r in SITES.items() if r["system"][:4] not in ("HPA-", "HNA-") and r.get("meaning")), []) -check("1 every FCGR3B site carries the FCGR3A and copy-number caveat", - sorted(n for n in want_sites if n.startswith("FCGR3B") and "copy number" not in SITES.get(n, {}).get("caveat", "")), []) - -# 2. the two-allele antigens, 0/0, 0/1, 1/1 -pairs = { - # name: (system, phenotype at 0/0, 0/1, 1/1, genotype at 0/1) - "HPA1_176": ("HPA-1", "HPA-1(a+b-)", "HPA-1(a+b+)", "HPA-1(a-b+)", "1a/1b"), - "HPA2_482": ("HPA-2", "HPA-2(a+b-)", "HPA-2(a+b+)", "HPA-2(a-b+)", "2a/2b"), - "HPA3_2621": ("HPA-3", "HPA-3(a+b-)", "HPA-3(a+b+)", "HPA-3(a-b+)", "3a/3b"), - "HPA4_506": ("HPA-4", "HPA-4(a+b-)", "HPA-4(a+b+)", "HPA-4(a-b+)", "4a/4b"), - "HPA5_1600": ("HPA-5", "HPA-5(a+b-)", "HPA-5(a+b+)", "HPA-5(a-b+)", "5a/5b"), - "HPA6_1544": ("HPA-6", "HPA-6(a+b-)", "HPA-6(a+b+)", "HPA-6(a-b+)", "6a/6b"), - "HPA15_2108": ("HPA-15", "HPA-15(a-b+)", "HPA-15(a+b+)", "HPA-15(a+b-)", "15a/15b"), - "HNA3_461": ("HNA-3", "HNA-3(a-b+)", "HNA-3(a+b+)", "HNA-3(a+b-)", "3a/3b"), - "HNA4_230": ("HNA-4", "HNA-4(a+b-)", "HNA-4(a+b+)", "HNA-4(a-b+)", "4a/4b"), - "HNA5_2372": ("HNA-5", "HNA-5(a+b-)", "HNA-5(a+b+)", "HNA-5(a-b+)", "5a/5b"), -} -none = run() -for n, (system, p0, p1, p2, g1) in pairs.items(): - het, hom = run([rec(n, "0/1")]), run([rec(n, "1/1")]) - check(f"2 {system}: 0/0, 0/1, 1/1", (cell(none, system, "phenotype"), cell(het, system, "phenotype"), - cell(hom, system, "phenotype")), (p0, p1, p2)) - check(f"2 {system}: the 0/1 genotype at confidence high", - (cell(het, system, "genotype"), cell(het, system, "confidence")), (g1, "high")) - check(f"2 {system}: both antigens, no meaning in the note", "lacks " in cell(het, system, "note"), False) - -# 3. the meaning -mum = run([rec("HPA1_176", "1/1")]) -note = cell(mum, "HPA-1", "note") -check("3 HPA-1b/1b: the note says the person lacks HPA-1a", note.startswith("lacks HPA-1a: "), True) -check("3 and names FNAIT and anti-HPA-1a", "FNAIT" in note and "anti-HPA-1a" in note, True) -check("3 HPA-1a/1a lacks HPA-1b", cell(none, "HPA-1", "note").startswith("lacks HPA-1b: "), True) -check("3 GRCh38 carries HNA-3b: no record lacks HNA-3a, with the neutrophil meaning", - (cell(none, "HNA-3", "note").startswith("lacks HNA-3a: "), "neonatal alloimmune neutropenia" - in cell(none, "HNA-3", "note")), (True, True)) -check("3 a red cell system never gets a meaning", "lacks " in cell(mum, "Kidd", "note"), False) - -# 4. HNA-1 -A4 = ["FCGR3B_108", "FCGR3B_114", "FCGR3B_244", "FCGR3B_316"] -b = none -check("4 the GRCh38 bases (FCGR3B*02 at four markers): HNA-1(a-b+c-), FCGR3B*02 twice", - (cell(b, "HNA-1", "phenotype"), cell(b, "HNA-1", "genotype")), ("HNA-1(a-b+c-)", "FCGR3B*02/FCGR3B*02")) -check("4 confidence is always low (FCGR3A, copy number)", cell(b, "HNA-1", "confidence"), "low") -check("4 c.194 that does not match is named in the note", - "FCGR3B c.194A>G (x0) does not match" in cell(b, "HNA-1", "note"), True) -check("4 the note lacks HNA-1a and HNA-1c", "lacks HNA-1a, HNA-1c: " in cell(b, "HNA-1", "note"), True) -ab = run([rec(n, "0/1") for n in A4] + [rec("FCGR3B_194", "0/1")]) -check("4 HNA-1a/1b", (cell(ab, "HNA-1", "phenotype"), cell(ab, "HNA-1", "genotype")), - ("HNA-1(a+b+c-)", "FCGR3B*01/FCGR3B*02")) -check("4 c.194 that matches: no note about it", "does not match" in cell(ab, "HNA-1", "note"), False) -aa = run([rec(n, "1/1") for n in A4] + [rec("FCGR3B_194", "1/1")]) -check("4 HNA-1a/1a", (cell(aa, "HNA-1", "phenotype"), cell(aa, "HNA-1", "genotype")), - ("HNA-1(a+b-c-)", "FCGR3B*01/FCGR3B*01")) -check("4 HNA-1a/1a lacks HNA-1b", "lacks HNA-1b, HNA-1c: " in cell(aa, "HNA-1", "note"), True) -c = run([rec("FCGR3B_233", "0/1")] + [rec("FCGR3B_194", "1/1")]) -check("4 HNA-1c on an FCGR3B*02: HNA-1(a-b+c+), FCGR3B*02/FCGR3B*03", - (cell(c, "HNA-1", "phenotype"), cell(c, "HNA-1", "genotype")), ("HNA-1(a-b+c+)", "FCGR3B*02/FCGR3B*03")) -ac = run([rec(n, "1/1") for n in A4] + [rec("FCGR3B_194", "1/1"), rec("FCGR3B_233", "0/1")]) -check("4 HNA-1c with two FCGR3B*01: b+ (FCGR3B*03 carries HNA-1b) and the extra-copy note", - (cell(ac, "HNA-1", "phenotype"), "extra FCGR3B copy" in cell(ac, "HNA-1", "note")), ("HNA-1(a+b+c+)", True)) -nc = run([rec("FCGR3B_233", "./."), rec("FCGR3B_194", "1/1")]) -check("4 c.233 with no call: the HNA-1b alleles read *02 or *03, HNA-1c not determined", - (cell(nc, "HNA-1", "phenotype"), cell(nc, "HNA-1", "genotype"), "HNA-1c not determined" in cell(nc, "HNA-1", "note")), - ("HNA-1(a-b+)", "FCGR3B*02 or *03/FCGR3B*02 or *03", True)) -mixed = run([rec("FCGR3B_108", "1/1"), rec("FCGR3B_114", "0/1")]) -check("4 markers that disagree: not determined", cell(mixed, "HNA-1", "phenotype"), "not determined") -check("4 and the note says why", "do not agree on HNA-1a or HNA-1b" in cell(mixed, "HNA-1", "note"), True) - -if FAILS: - print(f"{len(FAILS)} check(s) failed") - sys.exit(1) -print("all checks passed") diff --git a/tests/test_bloodtype_report.py b/tests/test_bloodtype_report.py deleted file mode 100644 index 1b7e9a73..00000000 --- a/tests/test_bloodtype_report.py +++ /dev/null @@ -1,463 +0,0 @@ -#!/usr/bin/env python3 -"""og bloodtype report: the blood group antigens a genome predicts (step 41). - -Each case writes a synthetic one-sample VCF (only the records it plants) and, -when the case has a BAM, a samtools bedcov file over assets/bloodtype_markers.tsv -with chosen depths, runs og bloodtype report, and checks rows of the table it writes: - - 1. the table and the code agree (og bloodtype check-markers): every site has - a rule, every rule a site, every phenotype template names both antigens - of its site; a table missing a rule's site fails the check - 2. nothing planted, reads everywhere: the reference alleles (O, k, Fy(a+b-) ...) - at confidence high; without the BAM the same calls at medium, and RhD - not determined - 3. ABO: A1, A2, B, AB, O.02, and markers that do not add up - 4. RhD from depth: two copies, one, none (the RHD deletion), the RHD - pseudogene, a hybrid gene (exons 3 to 7 without reads), low flank depth - 5. Duffy and Kell flip with planted variants: Fyb, the GATA box, FY*X; K - 6. the VCF: a site with too few reads, a FILTER-failed call, a RefCall 0/0, - contigs without chr, a multi-allelic record, a record whose REF is not the - marker's REF, a second sample - 7. a BAM without some contigs (no depth rows there): those systems and RhD - are not determined, the others are read as usual - -og is found by tests/og_binary.py (OG, or the release build of rust/). -Run: OG=rust/target/release/og python3 tests/test_bloodtype_report.py -""" -import csv -import gzip -import os -import subprocess -import sys -import tempfile - -REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) -MARKERS = os.path.join(REPO, "assets", "bloodtype_markers.tsv") -CN_LOCI = os.path.join(REPO, "bin", "cn_loci.tsv") -sys.path.insert(0, os.path.join(REPO, "tests")) -from og_binary import og_binary # noqa: E402 - -OG = og_binary() - - -def read_markers(): - """The marker table's sites by name (pos is the VCF position), and the - RhD depth regions og cn bed --locus RHD prints, by name.""" - with open(MARKERS) as f: - lines = [ln for ln in f if not ln.startswith("#") and ln.strip()] - sites = {} - for r in csv.DictReader(lines, delimiter="\t"): - r["start"], r["end"] = int(r["start"]), int(r["end"]) - r["pos"] = r["end"] - sites[r["name"]] = r - bed = subprocess.run([OG, "cn", "bed", "--catalog", CN_LOCI, "--locus", "RHD"], capture_output=True, text=True, - check=True).stdout - regions = {} - for ln in bed.splitlines(): - chrom, start, end, name = ln.split("\t")[:4] - regions[name] = {"chrom": chrom, "start": int(start), "end": int(end), "name": name} - return sites, regions - - -def report(argv): - """og bloodtype report ARGV (with this checkout's catalog): the process.""" - return subprocess.run([OG, "bloodtype", "report", "--cn-loci", CN_LOCI] + argv, capture_output=True, text=True) - - -FAILS = [] -SITES, REGIONS = read_markers() - - -def check(desc, got, want): - if got == want: - print(f"PASS {desc}") - else: - print(f"FAIL {desc}: got {got!r}, want {want!r}") - FAILS.append(desc) - - -def vcf_line(name, gt, filt="PASS", chrom=None, alt=None, ref=None): - r = SITES[name] - return "\t".join([chrom or r["chrom"], str(r["pos"]), ".", ref or r["ref"], alt or r["alt"], "50", filt, ".", - "GT", gt]) + "\n" - - -def run(records=(), depth=None, gz=False, samples=("S",), raw=None, drop=()): - """{system: row} of the table the script writes, or the error text.""" - with tempfile.TemporaryDirectory() as tmp: - vcf = os.path.join(tmp, "s.vcf.gz" if gz else "s.vcf") - text = ("##fileformat=VCFv4.2\n#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT\t" - + "\t".join(samples) + "\n" + "".join(records) + (raw or "")) - if gz: - with gzip.open(vcf, "wt") as f: - f.write(text) - else: - with open(vcf, "w") as f: - f.write(text) - cmd = ["--sample", "S", "--vcf", vcf, "--markers", MARKERS, - "--out", os.path.join(tmp, "t.tsv"), "--evidence", os.path.join(tmp, "e.tsv")] - if depth is not None: - dp = os.path.join(tmp, "d.tsv") - with open(dp, "w") as f: - for r in list(SITES.values()) + list(REGIONS.values()): - if r["chrom"] in drop: - continue # a contig the BAM has no @SQ line for: step 41 leaves its rows out - mean = depth.get(r["name"], depth.get("*", 30.0)) - total = str(int(mean * (r["end"] - r["start"]))) - if r["name"] in REGIONS: # og cn bed's 4-column BED - cols = [r["chrom"], str(r["start"]), str(r["end"]), r["name"], total] - else: # a marker row: the table's 16 columns - cols = [r["chrom"], str(r["start"]), str(r["end"]), r["kind"], r["system"], r["name"], - "", "", "", "", "", "", "", "", "", "", total] - f.write("\t".join(cols) + "\n") - cmd += ["--depth", dp] - p = report(cmd) - if p.returncode != 0: - return p.stderr.strip() or f"exit {p.returncode}" - with open(os.path.join(tmp, "t.tsv")) as f: - return {r["system"]: r for r in csv.DictReader(f, delimiter="\t")} - - -def rhd(copies, flank=30.0, exons=None): - """Depth means for an RHD copy number (2 = the flanks' depth).""" - d = {"*": 30.0, "RHD_FLANK_UP": flank, "RHD_FLANK_DOWN": flank, "RHD_GENE": flank * copies / 2} - for n in REGIONS: - if n.startswith("RHD_EX"): - d[n] = flank * (exons or {}).get(int(n[6:]), copies) / 2 - return d - - -def ph(t, system): - return t[system]["phenotype"] if isinstance(t, dict) else t - - -def cf(t, system): - return t[system]["confidence"] if isinstance(t, dict) else t - - -# 1. table and code: og bloodtype check-markers -chk = subprocess.run([OG, "bloodtype", "check-markers", MARKERS, "--cn-loci", CN_LOCI], capture_output=True, text=True) -check("1 the marker table and the rules agree", (chk.returncode, chk.stderr), (0, "")) -NULLS = {n for n, r in SITES.items() if r["alt_means"] == "null"} -with tempfile.TemporaryDirectory() as tmp: - broken = os.path.join(tmp, "m.tsv") - with open(MARKERS) as f, open(broken, "w") as g: - g.writelines(ln for ln in f if "\tKEL_841\t" not in ln) - chk = subprocess.run([OG, "bloodtype", "check-markers", broken, "--cn-loci", CN_LOCI], capture_output=True, - text=True) - check("1 control: a table without KEL_841 fails the check", - (chk.returncode, "rule sites the table lacks: KEL_841" in chk.stderr), (1, True)) - -# 2. nothing planted -ref = run(depth=rhd(2)) -want = {"ABO": "O", "RhD": "D+", "RhCE": "C-c+ E-e+", "Kell": "K-k+ Kp(a-b+) Js(a-b+)", "Duffy": "Fy(a+b-)", - "Kidd": "Jk(a+b-)", "MNS": "M-N+ S-s+", "Lutheran": "Lu(a-b+)", "Diego": "Di(a-b+)", "Dombrock": "Do(a-b+)", - "Colton": "Co(a+b-)", "HPA-1": "HPA-1(a+b-)", "HPA-2": "HPA-2(a+b-)", "HPA-3": "HPA-3(a+b-)", - "HPA-4": "HPA-4(a+b-)", "HPA-5": "HPA-5(a+b-)", "HPA-15": "HPA-15(a-b+)"} -check("2 reference alleles, reads everywhere: every system", {k: ph(ref, k) for k in want}, want) -check("2 ABO genotype", ref["ABO"]["genotype"] if isinstance(ref, dict) else ref, "O.01/O.01") -check("2 ABO confidence high with reads", cf(ref, "ABO"), "high") -check("2 RhCE low: c.307 sits in RHCE's RHD-like exon 2", cf(ref, "RhCE"), "low") -check("2 Kell high", cf(ref, "Kell"), "high") -nobam = run() -check("2 without the BAM, ABO is still O", ph(nobam, "ABO"), "O") -check("2 without the BAM, confidence medium", cf(nobam, "ABO"), "medium") -check("2 without the BAM, RhD is not determined", ph(nobam, "RhD"), "not determined") -check("2 without the BAM, the note says why", "deletes the whole RHD gene" in nobam["RhD"]["note"], True) - -# 3. ABO -check("3 A1/O: one c.261 insertion", ph(run([vcf_line("ABO_261", "0/1")], rhd(2)), "ABO"), "A1") -a2 = run([vcf_line("ABO_261", "1/1"), vcf_line("ABO_1061", "1/1")], rhd(2)) -check("3 A2/A2: c.1061delC on both", (ph(a2, "ABO"), a2["ABO"]["genotype"]), ("A2", "A2/A2")) -check("3 A1/A2 reads A1", ph(run([vcf_line("ABO_261", "1/1"), vcf_line("ABO_1061", "0/1")], rhd(2)), "ABO"), "A1") -b = run([vcf_line("ABO_261", "0/1"), vcf_line("ABO_803", "0/1"), vcf_line("ABO_796", "0/1")], rhd(2)) -check("3 B/O", (ph(b, "ABO"), b["ABO"]["genotype"]), ("B", "B/O.01")) -ab = run([vcf_line("ABO_261", "1/1"), vcf_line("ABO_803", "0/1"), vcf_line("ABO_796", "0/1")], rhd(2)) -check("3 A1B", (ph(ab, "ABO"), cf(ab, "ABO")), ("A1B", "high")) -check("3 the A1 or A2 caveat is in the note", "c.1061delC, a deletion in a run of C" in ab["ABO"]["note"], True) -o2 = run([vcf_line("ABO_261", "0/1"), vcf_line("ABO_802", "0/1")], rhd(2)) -check("3 O.01/O.02 is O", (ph(o2, "ABO"), o2["ABO"]["genotype"]), ("O", "O.01/O.02")) -bad = run([vcf_line("ABO_261", "0/1"), vcf_line("ABO_803", "1/1")], rhd(2)) -check("3 two B markers on one functional allele: not determined", ph(bad, "ABO"), "not determined") -odd = run([vcf_line("ABO_261", "0/1"), vcf_line("ABO_803", "0/1")], rhd(2)) -check("3 c.803 without c.796: B, confidence low", (ph(odd, "ABO"), cf(odd, "ABO")), ("B", "low")) - -# 4. RhD from depth -check("4 two copies: D+", ph(run(depth=rhd(2)), "RhD"), "D+") -het = run(depth=rhd(1)) -check("4 one copy (one RHD deleted): D+", (ph(het, "RhD"), het["RhD"]["genotype"].split(" (")[0]), - ("D+", "RHD/RHD deleted")) -gone = run(depth=rhd(0)) -check("4 no copy (RHD deleted on both): D-", (ph(gone, "RhD"), cf(gone, "RhD")), ("D-", "high")) -check("4 the deleted genotype", gone["RhD"]["genotype"].split(" (")[0] if isinstance(gone, dict) else gone, - "RHD deleted/RHD deleted") -psi = run([vcf_line("RHD_PSI_DUP", "0/1"), vcf_line("RHD_PSI_609", "0/1")], rhd(1)) -check("4 one RHD copy and it is the pseudogene: D-", ph(psi, "RhD"), "D-") -psi2 = run([vcf_line("RHD_PSI_DUP", "0/1"), vcf_line("RHD_PSI_609", "0/1")], rhd(2)) -check("4 two copies, one of them the pseudogene: D+", ph(psi2, "RhD"), "D+") -hyb = run(depth=rhd(1.6, exons={3: 0, 4: 0, 5: 0, 6: 0, 7: 0})) -check("4 exons 3 to 7 without reads: a hybrid, RhD not determined", ph(hyb, "RhD"), "not determined") -check("4 the hybrid note names the exons", "exons 3, 4, 5, 6, 7" in hyb["RhD"]["note"], True) -check("4 flanks under 10x: not determined", ph(run(depth=rhd(2, flank=6.0)), "RhD"), "not determined") -check("4 near a boundary (1.45 copies): confidence low", cf(run(depth=rhd(1.45)), "RhD"), "low") - -# 5. Duffy and Kell flip -check("5 Fyb het: Fy(a+b+)", ph(run([vcf_line("FY_125", "0/1")], rhd(2)), "Duffy"), "Fy(a+b+)") -gata = run([vcf_line("FY_125", "0/1"), vcf_line("FY_GATA", "0/1")], rhd(2)) -check("5 Fyb het with the GATA box: Fy(a+b-)", (ph(gata, "Duffy"), gata["Duffy"]["genotype"]), ("Fy(a+b-)", "FY*A/FY*B GATA")) -check("5 Fyb hom, GATA hom: Fy(a-b-)", - ph(run([vcf_line("FY_125", "1/1"), vcf_line("FY_GATA", "1/1")], rhd(2)), "Duffy"), "Fy(a-b-)") -check("5 FY*X: Fy(a-b+w)", - ph(run([vcf_line("FY_125", "1/1"), vcf_line("FY_265", "1/1")], rhd(2)), "Duffy"), "Fy(a-b+w)") -k = run([vcf_line("KEL_578", "0/1")], rhd(2)) -check("5 K het: K+k+", (ph(k, "Kell"), k["Kell"]["genotype"]), ("K+k+ Kp(a-b+) Js(a-b+)", "K/k Kpb/Kpb Jsb/Jsb")) -check("5 K hom: K+k-", ph(run([vcf_line("KEL_578", "1/1")], rhd(2)), "Kell").split()[0], "K+k-") -check("5 HPA-15: the reference is 15b, the alt 15a", - ph(run([vcf_line("HPA15_2108", "1/1")], rhd(2)), "HPA-15"), "HPA-15(a+b-)") - -# 6. the VCF -low = run(depth=dict(rhd(2), KEL_578=3.0)) -check("6 a missing site with 3 reads: that pair is not determined", - (ph(low, "Kell"), cf(low, "Kell")), ("K/k not determined Kp(a-b+) Js(a-b+)", "low")) -check("6 a FILTER-failed call is no call", - ph(run([vcf_line("JK_838", "1/1", filt="LowQual")], rhd(2)), "Kidd"), "not determined") -check("6 a RefCall 0/0 is a reference call", - (ph(run([vcf_line("JK_838", "0/0", filt="RefCall")], rhd(2, flank=30.0)), "Kidd")), "Jk(a+b-)") -check("6 contigs without chr, gzip", - ph(run([vcf_line("JK_838", "1/1", chrom="18")], rhd(2), gz=True), "Kidd"), "Jk(a-b+)") -multi = run([vcf_line("KEL_578", "1/2", alt="C,A")], rhd(2)) -check("6 a multi-allelic record: the second ALT is K", ph(multi, "Kell").split()[0], "K+k+") -check("6 another allele at the site lowers the confidence", cf(multi, "Kell"), "low") -mism = run([vcf_line("KEL_578", "1/1", ref="C", alt="T")], rhd(2)) -check("6 a record whose REF is not the marker's REF: that pair is not determined", - (ph(mism, "Kell").split()[0:3], cf(mism, "Kell")), (["K/k", "not", "determined"], "low")) -check("6 and the note names both REFs", "REF C at this position, the marker's REF is G" in mism["Kell"]["note"], True) -span = run([vcf_line("KEL_578", "0/0", ref="GT", alt="G")], rhd(2)) -check("6 a reference call over a longer REF that starts with the marker's REF is reference", - ph(span, "Kell").split()[0], "K-k+") -check("6 a no-call genotype", ph(run([vcf_line("JK_838", "./.")], rhd(2)), "Kidd"), "not determined") -two = run(samples=("S", "T")) -check("6 two samples stop the script", isinstance(two, str) and "2 samples" in two, True) - -# 7. a BAM without chr1 and chr9: no depth rows for RH, Duffy and ABO -part = run([vcf_line("JK_838", "1/1")], rhd(2), drop=("chr1", "chr9")) -check("7 no depth over RHD: RhD not determined", ph(part, "RhD"), "not determined") -check("7 the RhD note names the missing contig", "the BAM has no chr1 contig" in part["RhD"]["note"], True) -check("7 ABO with no record and no depth row: not determined", ph(part, "ABO"), "not determined") -check("7 the ABO note says why", "no depth row" in part["ABO"]["note"], True) -check("7 Duffy (chr1) not determined", ph(part, "Duffy"), "not determined") -check("7 Kidd (chr18, called) is read as usual", (ph(part, "Kidd"), cf(part, "Kidd")), ("Jk(a-b+)", "high")) -check("7 Kell (chr7, covered) is read as usual", (ph(part, "Kell"), cf(part, "Kell")), - ("K-k+ Kp(a-b+) Js(a-b+)", "high")) -empty = run(depth={}, drop=tuple({r["chrom"] for r in SITES.values()})) -check("7 an empty depth file (no marker contig in the BAM): RhD not determined", ph(empty, "RhD"), "not determined") -check("7 and a system with no record is not determined", ph(empty, "Kidd"), "not determined") - -# 8. the orientation of the null and HPA-7 to -35 rows: each row's REF and ALT are -# its c. change on the gene's strand (Ensembl GRCh38, 2026-11-03). A row with REF -# and ALT swapped, or on the wrong strand, fails here. -STRAND = {"SMIM1": 1, "ABCB6": -1, "ABCG2": -1, "FUT1": -1, "FUT2": 1, "KEL": -1, "SLC14A1": 1, - "ITGB3": 1, "ITGA2B": -1, "GP1BB": 1, "ITGA2": 1, "GP9": 1} -COMP = str.maketrans("ACGT", "TGCA") - - -def oriented(gene, ref, alt, change): - """The VCF REF/ALT agree with one c. change on the gene's strand: the bases - of a substitution or a delins, the length of a deletion, duplication or insertion.""" - import re - flip = (lambda s: s.translate(COMP)[::-1]) if STRAND[gene] == -1 else (lambda s: s) - m = re.fullmatch(r"c\.[-*0-9+_]+([ACGT])>([ACGT])", change) - if m: - return (flip(ref), flip(alt)) == m.groups() - m = re.fullmatch(r"c\.([-*0-9+]+)(?:_([-*0-9+]+))?delins([ACGT]+)", change) - if m: - return len(ref) == span(m.group(1), m.group(2)) and flip(alt) == m.group(3) - m = re.fullmatch(r"c\.([-*0-9+]+)(?:_([-*0-9+]+))?(del|dup)([ACGT]*)", change) - if m: - n = len(m.group(4)) or span(m.group(1), m.group(2)) - d = len(ref) - len(alt) if m.group(3) == "del" else len(alt) - len(ref) - if d != n or ref[0] != alt[0]: - return False - # Bases given: the VCF's deleted or added bases on the gene's strand are - # those bases, or a rotation of them (VCF left-aligns in a repeat, HGVS - # shifts 3'). - moved = flip((ref if m.group(3) == "del" else alt)[1:]) - return not m.group(4) or (len(moved) == n and moved in m.group(4) * 2) - m = re.fullmatch(r"c\.[0-9]+_[0-9]+ins([ACGT]+)", change) - if m: - return len(alt) - len(ref) == len(m.group(1)) and ref[0] == alt[0] - return False - - -def span(a, b): - def pos(x): - base, _, off = x.partition("+") - return int(base) + (int(off) if off else 0) - return 1 if b is None else pos(b) - pos(a) + 1 - - -# HPA-7 to -35 but HPA-15: one site each, the a antigen on the reference allele -HPA_RARE = {n for n in SITES if n.startswith("HPA") and int(n[3:].split("_")[0]) >= 7 and not n.startswith("HPA15_")} -NEW = sorted(NULLS | HPA_RARE) -for n in NEW: - r = SITES[n] - gene, _, changes = r["hgvs"].partition(" ") - changes = changes.split(", ") - alts = r["alt"].split(",") - check(f"8 {n}: one c. change per ALT", len(changes), len(alts)) - check(f"8 {n}: REF and ALT read the c. change on the {gene} strand", - [c for a, c in zip(alts, changes) if not oriented(gene, r["ref"], a, c)], []) -check("8 control: a swapped REF and ALT fails the orientation", oriented("KEL", "A", "G", "c.578C>T"), False) -check("8 control: the right one passes", oriented("KEL", "G", "A", "c.578C>T"), True) -check("8 control: a deletion one base short fails", oriented("FUT2", "CG", "C", "c.719_720delGT"), False) -check("8 control: a deletion of the right length but other bases fails", oriented("FUT2", "CAC", "C", "c.719_720delGT"), False) -check("8 control: a left-aligned deletion in a repeat passes", oriented("FUT2", "CTG", "C", "c.719_720delGT"), True) - - -# 9. every new row, positive and negative: a reference genome carries the antigen -# (the a antigen for HPA-7 to -35), the row's ALT on both chromosomes lacks it. -def run_inproc(records): - """Like run(), with reads everywhere: the rows in table order.""" - with tempfile.TemporaryDirectory() as tmp: - vcf, dp, out = os.path.join(tmp, "s.vcf"), os.path.join(tmp, "d.tsv"), os.path.join(tmp, "t.tsv") - with open(vcf, "w") as f: - f.write("##fileformat=VCFv4.2\n#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT\tS\n" + "".join(records)) - with open(dp, "w") as f: - for r in SITES.values(): - f.write("\t".join([r["chrom"], str(r["start"]), str(r["end"]), r["kind"], r["system"], r["name"], - str(30 * (r["end"] - r["start"]))]) + "\n") - for r in REGIONS.values(): - f.write("\t".join([r["chrom"], str(r["start"]), str(r["end"]), r["name"], - str(30 * (r["end"] - r["start"]))]) + "\n") - p = report(["--sample", "S", "--vcf", vcf, "--markers", MARKERS, "--depth", dp, "--out", out, - "--evidence", os.path.join(tmp, "e.tsv")]) - if p.returncode: - return f"exit {p.returncode}" - with open(out) as f: - return [dict(r) for r in csv.DictReader(f, delimiter="\t")] - - -NEG = {"Vel": "Vel-", "Langereis": "Lan-", "Junior": "Jr(a-)", "H": "H- (Bombay, Oh)", - "Secretor": "non-secretor", "Kell": "K-k- Kp(a-b-) Js(a-b-)", "Kidd": "Jk(a-b-)"} -POS = {"Vel": "Vel+", "Langereis": "Lan+", "Junior": "Jr(a+)", "H": "H+", "Secretor": "secretor", - "Kell": "K-k+ Kp(a-b+) Js(a-b+)", "Kidd": "Jk(a+b-)"} -base = {r["system"]: r for r in run_inproc([])} -for n in NEW: - r = SITES[n] - s = r["system"] - want_pos = POS.get(s) or f"{s}(a+b-)" - check(f"9 {n}: the reference genome is {want_pos}", base[s]["phenotype"], want_pos) - for a in r["alt"].split(","): - recs = [vcf_line(n, "1/1", alt=a)] - if s == "H": - recs.append(vcf_line("FUT2_01N02", "1/1")) # FUT1 null with a non-secretor: Bombay - t = run_inproc(recs) - got = {x["system"]: x for x in t} if isinstance(t, list) else {} - want_neg = NEG.get(s) or f"{s}(a-b+)" - check(f"9 {n} {r['ref']}>{a} on both chromosomes: {want_neg}", - got.get(s, {}).get("phenotype"), want_neg) -check("9 a reference genome lacks no high-prevalence antigen: ABO is the first row", - [x["system"] for x in run_inproc([])][:3], ["ABO", "RhD", "RhCE"]) - -# 10. synthetic gVCF records: Jk(a-b-) from SLC14A1 c.342-1G>A and HPA-21b/21b, -# each with the <*> allele DeepVariant's gVCF adds; a reference block at the -# sites gives the common phenotypes. -jk, hpa = SITES["JK_01N06"], SITES["HPA21_1960"] -gv = [f"{jk['chrom']}\t{jk['pos']}\t.\tG\tA,<*>\t40\tPASS\t.\tGT:GQ\t1/1:40\n", - f"{hpa['chrom']}\t{hpa['pos']}\t.\t{hpa['ref']}\t{hpa['alt']},<*>\t40\tPASS\t.\tGT:GQ\t1/1:40\n"] -g = {x["system"]: x for x in run_inproc(gv)} -check("10 gVCF: c.342-1G>A 1/1 is Jk(a-b-), lacking Jk3", - (g["Kidd"]["phenotype"], g["Kidd"]["genotype"], "lacks Jk3, carried by most people" in g["Kidd"]["note"]), - ("Jk(a-b-)", "null/null", True)) -check("10 gVCF: HPA-21b/21b", (g["HPA-21"]["phenotype"], g["HPA-21"]["genotype"]), ("HPA-21(a-b+)", "21b/21b")) -order = [x["system"] for x in run_inproc(gv)] -check("10 the systems that lack a high-prevalence antigen come first", order[:2], ["Kidd", "HPA-21"]) -blocks = [f"{r['chrom']}\t{r['pos']}\t.\t{r['ref'][0]}\t<*>\t0\t.\tEND={r['pos'] + 5}\tGT:GQ\t0/0:50\n" for r in (jk, hpa)] -gb = {x["system"]: x for x in run_inproc(blocks)} -check("10 gVCF reference blocks: Jk(a+b-) and HPA-21(a+b-), confidence high", - (gb["Kidd"]["phenotype"], gb["HPA-21"]["phenotype"], gb["Kidd"]["confidence"]), - ("Jk(a+b-)", "HPA-21(a+b-)", "high")) - -# 11. null alleles in detail -one = {x["system"]: x for x in run_inproc([vcf_line("VEL_01N01", "0/1")])} -check("11 one SMIM1 null copy: Vel+, Vel/null, the note names it", - (one["Vel"]["phenotype"], one["Vel"]["genotype"], "SMIM1 c.64_80del" in one["Vel"]["note"]), ("Vel+", "Vel/null", True)) -cpd = {x["system"]: x for x in run_inproc([vcf_line("LAN_01N02", "0/1"), vcf_line("LAN_01N13", "0/1")])} -check("11 two different ABCB6 nulls: Lan-, confidence low (no phase), the note says so", - (cpd["Langereis"]["phenotype"], cpd["Langereis"]["confidence"], "phase is not read" in cpd["Langereis"]["note"]), - ("Lan-", "low", True)) -kk = {x["system"]: x for x in run_inproc([vcf_line("KEL_578", "0/1"), vcf_line("KEL_02N03", "0/1")])} -check("11 K/k with one KEL null: the null goes on the k chromosome, K+k-", - (kk["Kell"]["phenotype"], kk["Kell"]["genotype"]), ("K+k- Kp(a-b+) Js(a-b+)", "K/null Kpb/null Jsb/null")) -check("11 and that lacks k, a high-prevalence antigen, so Kell comes first", - (run_inproc([vcf_line("KEL_578", "0/1"), vcf_line("KEL_02N03", "0/1")])[0]["system"], - "lacks k, carried by most people (high prevalence)" in kk["Kell"]["note"]), ("Kell", True)) -jb = {x["system"]: x for x in run_inproc([vcf_line("JK_838", "0/1"), vcf_line("JK_02N06", "0/1")])} -check("11 Jka/Jkb with JK*02N.06 (c.871T>C): the null goes on Jkb, Jk(a+b-)", - (jb["Kidd"]["phenotype"], jb["Kidd"]["genotype"]), ("Jk(a+b-)", "Jka/null")) -para = {x["system"]: x for x in run_inproc([vcf_line("FUT1_01N08", "1/1"), vcf_line("ABO_261", "0/1")])} -check("11 FUT1 null with a working FUT2: para-Bombay", para["H"]["phenotype"], "H- (para-Bombay)") -check("11 and the ABO row says the red cells group as O", "the cells usually group as O" in para["ABO"]["note"], True) -check("11 a non-secretor alone leaves H and ABO as they are", - [(x["system"], x["phenotype"]) for x in run_inproc([vcf_line("FUT2_01N02", "1/1")]) - if x["system"] in ("H", "ABO", "Secretor")], [("ABO", "O"), ("H", "H+"), ("Secretor", "non-secretor")]) -lowd = run(depth=dict(rhd(2), VEL_01N01=2.0)) -check("11 the only Vel null row with 2 reads: not determined", ph(lowd, "Vel"), "not determined") -lowk = run(depth=dict(rhd(2), KEL_02N03=2.0)) -check("11 one KEL null row with 2 reads: Kell still read, confidence medium", - (ph(lowk, "Kell"), cf(lowk, "Kell")), ("K-k+ Kp(a-b+) Js(a-b+)", "medium")) -multi_null = {x["system"]: x for x in run_inproc([vcf_line("KEL_02N01", "1/1", alt="A")])} -check("11 a null row with several ALTs counts each (c.223+1G>T, the third)", - multi_null["Kell"]["phenotype"], "K-k- Kp(a-b-) Js(a-b-)") - - -# 12. common variants are not null alleles. FUT2 c.772G>A (rs602662, about half of -# all chromosomes in gnomAD) and c.335C>T (rs200157007, 0.14 in South Asians) ride on -# the c.461G>A haplotype, and SLC14A1 c.810G>A (rs17675299, 0.08 in Finns) is far -# too common for a Jk null. As null rows they turned Se/se carriers into -# non-secretors and Jk(a+b+) into Jk(a+b-) at confidence high. -def raw_line(chrom, pos, ref, alt, gt): - return "\t".join([chrom, str(pos), ".", ref, alt, "50", "PASS", ".", "GT", gt]) + "\n" - - -for desc, recs, system, want in [ - ("FUT2 c.461G>A 0/1 with c.772G>A 0/1: secretor", - [vcf_line("FUT2_01N02", "0/1"), raw_line("chr19", 48703728, "G", "A", "0/1")], "Secretor", "secretor"), - ("FUT2 c.772G>A 1/1 alone: secretor", - [raw_line("chr19", 48703728, "G", "A", "1/1")], "Secretor", "secretor"), - ("FUT2 c.461G>A 0/1 with c.335C>T 0/1: secretor", - [vcf_line("FUT2_01N02", "0/1"), raw_line("chr19", 48703291, "C", "T", "0/1")], "Secretor", "secretor"), - ("JK_838 0/1 with SLC14A1 c.810G>A 0/1: Jk(a+b+)", - [vcf_line("JK_838", "0/1"), raw_line("chr18", 45739309, "G", "A", "0/1")], "Kidd", "Jk(a+b+)")]: - got = {x["system"]: x for x in run_inproc(recs)} - check(f"12 {desc}", got[system]["phenotype"], want) -check("12 those three positions are no null row", - sorted(n for n in NULLS if (SITES[n]["chrom"], SITES[n]["pos"]) in - {("chr19", 48703728), ("chr19", 48703291), ("chr18", 45739309)}), []) - -# 14. two null copies need no pair site: with c.838 a no call, SLC14A1 c.342-1G>A -# 1/1 is still Jk(a-b-). Two Kidd nulls placed on different alleles name each. -nc = {x["system"]: x for x in run_inproc([vcf_line("JK_01N06", "1/1"), vcf_line("JK_838", "./.")])} -check("14 Jk null 1/1 with c.838 a no call: Jk(a-b-), null/null", - (nc["Kidd"]["phenotype"], nc["Kidd"]["genotype"]), ("Jk(a-b-)", "null/null")) -kn = {x["system"]: x for x in run_inproc([vcf_line("KEL_02N03", "1/1"), vcf_line("KEL_578", "./.")])} -check("14 KEL null 1/1 with c.578 a no call: K-k- Kp(a-b-) Js(a-b-), null/null", - (kn["Kell"]["phenotype"], kn["Kell"]["genotype"]), ("K-k- Kp(a-b-) Js(a-b-)", "null/null")) -two = {x["system"]: x for x in run_inproc([vcf_line("JK_838", "0/1"), vcf_line("JK_02N21", "0/1"), - vcf_line("JK_01N32", "0/1")])} -check("14 two Kidd nulls with different null_on: the note names each allele's placement", - ("SLC14A1 c.118G>A: Jkb" in two["Kidd"]["note"], "SLC14A1 c.151+1G>C: Jka" in two["Kidd"]["note"]), - (True, True)) - -# 13. another ALT at a null row's position (KEL c.1546C>T's position, G>T) is not that -# null allele: the antigens stay fully read at confidence high, and the note names it. -kel = SITES["KEL_02N17"] -oth = {x["system"]: x for x in run_inproc([raw_line(kel["chrom"], kel["pos"], "G", "T", "0/1")])} -check("13 another ALT at a KEL null row: K-k+, confidence high, the note names it", - (oth["Kell"]["phenotype"], oth["Kell"]["confidence"], "another allele at this position" in oth["Kell"]["note"]), - ("K-k+ Kp(a-b+) Js(a-b+)", "high", True)) - -if FAILS: - print(f"{len(FAILS)} check(s) failed") - sys.exit(1) -print("all checks passed") diff --git a/tests/test_hba_copy_number.py b/tests/test_hba_copy_number.py deleted file mode 100755 index e9150c01..00000000 --- a/tests/test_hba_copy_number.py +++ /dev/null @@ -1,179 +0,0 @@ -#!/usr/bin/env python3 -# og-leg: OG_REQUIRED -"""og cn call --locus HBA: step 35's HBA1/HBA2 count from short-read depth. -It needs og: OG names the binary (OG=rust/target/release/og, or bin/og); -without it the test is skipped, and with OG_REQUIRED set too it fails. - -On `samtools bedcov` tables made up for each case (summed depth = mean depth -x length), at 30x in the flanks, with a quarter of the gene-body depth from -reads of MAPQ 0 (as BWA-MEM and minimap2 give on HG002) unless a case says -otherwise: - 1. no deletion: 30x over HBA2 and HBA1 is four alpha-globin genes; - 2. a planted -a3.7-like loss (HBA2 at half depth) is three, a silent carrier, - marked unconfirmed (the count is checked on one real genome only); - 3. both genes at half depth (--SEA-like) is two, with the warning that - depth cannot tell --/aa from -a/-a; - 4. a triplication (HBA2 at 1.5x) is five; - 5. an estimate between two whole numbers, flanks under 8x, and flanks that - disagree are each 'not assessed', with the reason; - 6. no read of MAPQ 0 over the genes (GIAB's novoalign BAM of HG002: HBA2 - 15.5x, HBA1 25.7x, flanks 30x and 31x, which main counted as three or - left between two and three) is 'not assessed', naming the cause; 1% - is too, 3% is called; no gene at all (no reads to judge by) is still 0; - 7. the CLI: `og cn bed --locus HBA` prints the four regions, `og cn call - --table` writes the metric table and exits 0; a table with other - regions, or a short row, in either input exits 2, and so does a missing - --bedcov-mapq1. - -Run: OG=rust/target/release/og python3 tests/test_hba_copy_number.py -""" -import os -import subprocess -import sys -import tempfile - -REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) -CATALOG = os.path.join(REPO, "bin", "cn_loci.tsv") -OG = os.environ.get("OG", "") -sys.path.insert(0, os.path.join(REPO, "bin")) -from test_copy_number import cn # noqa: E402 pinned differential oracle - -REGIONS = cn.regions(cn.load_catalog(CATALOG)["HBA"]) -# Step 35's HBA table, in its order (what bin/hba_copy_number.py wrote). -ORDER = ["status", "alpha_globin_genes", "estimate", "hba2_depth", "hba1_depth", "flank_left_depth", - "flank_right_depth", "mapq0_share", "message"] -FAILS = 0 - - -def check(desc, ok, detail=""): - global FAILS - print(f"[{'PASS' if ok else 'FAIL'}] {desc}" + ("" if ok else f" -- {detail}")) - if not ok: - FAILS += 1 - - -def bedcov(path, depth): - """A bedcov table with the given mean depth per region name.""" - with open(path, "w") as f: - for c, s, e, n in REGIONS: - f.write(f"{c}\t{s}\t{e}\t{n}\t{int(depth[n] * (e - s))}\n") - - -def mapq1(depth, share=0.25): - """The depths of reads of MAPQ 1 or more: the flanks whole, the genes - without the given share.""" - return {n: v * (1 - share) if n in ("HBA2", "HBA1") else v for n, v in depth.items()} - - -def og_cn(*args): - return subprocess.run([OG, "cn", *args], capture_output=True, text=True) - - -def run(depth, share=0.25): - """Step 35's HBA table for these depths, as og cn call writes it.""" - with tempfile.TemporaryDirectory() as t: - p, q, o = (os.path.join(t, x) for x in ("cov.tsv", "cov_q1.tsv", "hba.tsv")) - bedcov(p, depth) - bedcov(q, mapq1(depth, share)) - og_cn("call", "--catalog", CATALOG, "--locus", "HBA", "--bedcov", p, "--bedcov-mapq1", q, "--table", o) - return dict(x.split("\t", 1) for x in open(o).read().splitlines()[1:]) if os.path.exists(o) else {} - - -def depths(hba2, hba1, left=30.0, right=30.0): - return {"flank_left": left, "HBA2": hba2, "HBA1": hba1, "flank_right": right} - - -def main(): - if not OG: - print("skip: OG is not set (OG=rust/target/release/og)") - return 1 if os.environ.get("OG_REQUIRED") else 0 - r = run(depths(30, 30)) - check("no deletion: four alpha-globin genes, called", - (r["status"], r["alpha_globin_genes"], r["estimate"]) == ("called", "4", "4.00") - and "no deletion seen" in r["message"], r) - r = run(depths(15, 30)) - check("planted one-gene loss (HBA2 at half depth): three genes, a silent carrier", - (r["status"], r["alpha_globin_genes"]) == ("called", "3") and "silent carrier" in r["message"], r) - check("a count under four says it is unconfirmed; four does not", - "unconfirmed" in r["message"] and "needs a clinical test" in r["message"] - and "unconfirmed" not in run(depths(30, 30))["message"] - and "unconfirmed" not in run(depths(45, 30))["message"], r) - r = run(depths(15, 15)) - check("both genes at half depth: two, and depth cannot tell --/aa from -a/-a", - (r["status"], r["alpha_globin_genes"]) == ("called", "2") and "cannot tell these apart" in r["message"], r) - r = run(depths(45, 30)) - check("HBA2 at 1.5x: five genes, a triplication", - r["alpha_globin_genes"] == "5" and "triplication" in r["message"], r) - r = run(depths(22.5, 30)) - check("an estimate of 3.50 is not assessed, with both whole numbers named", - r["status"] == "not assessed" and r["estimate"] == "3.50" and "between 3 and 4" in r["message"], r) - r = run(depths(5, 5, 6, 6)) - check("flanks under 8x: not assessed, too low", - r["status"] == "not assessed" and r["alpha_globin_genes"] == "." and "too low" in r["message"], r) - r = run(depths(30, 30, 30, 15)) - check("flanks 30x and 15x: not assessed, they disagree", - r["status"] == "not assessed" and "disagree" in r["message"], r) - novo = depths(15.5, 25.68, 30.11, 31.25) - r = run(novo, share=0) - check("no read of MAPQ 0 over the genes (novoalign's HG002): not assessed, the aligner named", - r["status"] == "not assessed" and r["alpha_globin_genes"] == "." and r["mapq0_share"] == "0.000" - and "mapping quality 0" in r["message"] and "novoalign" in r["message"], r) - r = run(depths(15, 30), share=0) - check("no read of MAPQ 0 with an estimate of 3.00: not assessed either, never three", - r["status"] == "not assessed" and r["estimate"] == "3.00" and "mapping quality 0" in r["message"], r) - r = run(depths(30, 30), share=0.01) - check("1% of the gene depth at MAPQ 0: not assessed", r["status"] == "not assessed" - and "1.0%" in r["message"], r) - r = run(depths(30, 30), share=0.03) - check("3% at MAPQ 0: called, four", (r["status"], r["alpha_globin_genes"], r["mapq0_share"]) == ("called", "4", "0.030"), r) - r = run(depths(0, 0), share=0) - check("no gene at all: still called 0, with no reads to judge the aligner by", - (r["status"], r["alpha_globin_genes"]) == ("called", "0") and "Bart" in r["message"], r) - - with tempfile.TemporaryDirectory() as t: - out = og_cn("bed", "--catalog", CATALOG, "--locus", "HBA") - rows = [line.split("\t") for line in out.stdout.splitlines()] - check("bed: four regions, the two genes between the flanks", - out.returncode == 0 and [x[3] for x in rows] == ["flank_left", "HBA2", "HBA1", "flank_right"], out) - cov, covq, res = os.path.join(t, "cov.tsv"), os.path.join(t, "cov_q1.tsv"), os.path.join(t, "hba.tsv") - bedcov(cov, depths(15, 30)) - bedcov(covq, mapq1(depths(15, 30))) - cli = [OG, "cn", "call", "--catalog", CATALOG, "--locus", "HBA", "--bedcov", cov, "--bedcov-mapq1", covq, - "--table", res] - out = subprocess.run(cli, capture_output=True, text=True) - table = dict(line.split("\t", 1) for line in open(res).read().splitlines()[1:]) if os.path.exists(res) else {} - check("call: exit 0, the table has the status, the count and the message", - out.returncode == 0 and table.get("status") == "called" and table.get("alpha_globin_genes") == "3" - and table.get("mapq0_share") == "0.250" - and list(table) == ORDER and "HBA1/HBA2: three" in out.stdout, (out, table)) - out = subprocess.run(cli[:9] + cli[11:], capture_output=True, text=True) - check("call: without --bedcov-mapq1 exits 2", out.returncode == 2 and "--bedcov-mapq1" in out.stderr, out) - for name, path in (("all reads", cov), ("MAPQ 1 or more", covq)): - bedcov(cov, depths(15, 30)) - bedcov(covq, mapq1(depths(15, 30))) - with open(path, "a") as f: - f.write("chr16\t1\t100\tother\t500\n") - out = subprocess.run(cli, capture_output=True, text=True) - check(f"call: a table ({name}) with other regions exits 2", out.returncode == 2 and "differ" in out.stderr, out) - with open(path, "w") as f: - f.write("chr16\t100000\t140000\n") - out = subprocess.run(cli, capture_output=True, text=True) - check(f"call: a short row ({name}) exits 2", out.returncode == 2 and "expected chrom" in out.stderr, out) - - # The demo's HBA table (tests/demo/make_demo_sample.py) is what og writes. - sys.path.insert(0, os.path.join(REPO, "tests", "demo")) - import make_demo_sample as demo - with tempfile.TemporaryDirectory() as t: - cov, covq, res = os.path.join(t, "cov.tsv"), os.path.join(t, "cov_q1.tsv"), os.path.join(t, "hba.tsv") - bedcov(cov, demo.HBA_DEPTH) - bedcov(covq, {n: v * demo.HBA_Q1[n] for n, v in demo.HBA_DEPTH.items()}) - og_cn("call", "--catalog", CATALOG, "--locus", "HBA", "--bedcov", cov, "--bedcov-mapq1", covq, "--table", res) - got = open(res).read() if os.path.exists(res) else "" - check("the demo's HBA table is the one og cn call writes for its depths", got == demo.HBA_TABLE, got) - - print("\nRESULT:", "ALL PASS" if FAILS == 0 else f"{FAILS} FAILED") - return 1 if FAILS else 0 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/tests/test_traits_table.py b/tests/test_traits_table.py deleted file mode 100644 index f9bdeb84..00000000 --- a/tests/test_traits_table.py +++ /dev/null @@ -1,202 +0,0 @@ -#!/usr/bin/env python3 -# og-leg: OG_REQUIRED -"""Validate the curated traits catalog and its GRCh38 reference bases. - -Schema checks use og traits check. Rust unit tests plant invalid catalog -rows and rules. The optional --fasta and --online checks also verify the -FASTA bases, PubMed sources and dbSNP records. A synthetic FASTA provides -positive, wrong-base and missing-contig controls without network access. -""" -import argparse -import csv -import json -import os -import shutil -import subprocess -import sys -import tempfile -import time - -REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) -TABLE = os.path.join(REPO, "assets", "traits.tsv") -FAILS = 0 -NC = {f"NC_0000{n:02d}": f"chr{n}" for n in range(1, 23)} - - -def check(desc, ok, detail=""): - global FAILS - print(f"[{'PASS' if ok else 'FAIL'}] {desc}{'' if ok else ' -- ' + str(detail)[:800]}") - if not ok: - FAILS += 1 - - -def fasta_bases(fasta, rows): - """{rsid: base} from FASTA at each row's position, by the .fai; rows on a - contig the FASTA lacks are left out.""" - fai = {} - with open(fasta + ".fai") as f: - for line in f: - c = line.split("\t") - fai[c[0]] = (int(c[1]), int(c[2]), int(c[3]), int(c[4])) - out = {} - with open(fasta, "rb") as fh: - for r in rows: - if r["chrom"] not in fai: - continue - length, offset, bases, width = fai[r["chrom"]] - i = int(r["pos"]) - 1 - if i >= length: - out[r["rsid"]] = "past the end of the contig" - continue - fh.seek(offset + (i // bases) * width + i % bases) - out[r["rsid"]] = fh.read(1).decode().upper() - return out - - -def fasta_errors(fasta, rows): - """(problems, rows checked) of the reference bases against FASTA.""" - got = fasta_bases(fasta, rows) - errs = [f"{r['rsid']} {r['chrom']}:{r['pos']}: table ref {r['ref']}, FASTA {got[r['rsid']]}" - for r in rows if r["rsid"] in got and got[r["rsid"]] != r["ref"]] - return errs, len(got) - - -def curl_json(url, tries=4): - last = "" - for t in range(tries): - res = subprocess.run(["curl", "-sS", "--fail", "--max-time", "60", url], capture_output=True, text=True) - if res.returncode == 0: - try: - return json.loads(res.stdout) - except ValueError as e: - last = f"not JSON: {e}" - else: - last = res.stderr.strip() - time.sleep(5 * (t + 1)) - raise RuntimeError(f"{url}: {last}") - - -def online_errors(rows): - """Problems with the sources: PubMed ids that do not resolve, and rsIDs - whose GRCh38 placement, reference base or alleles differ from dbSNP's.""" - errs = [] - pmids = sorted({p for r in rows for p in r["pmids"].split(",")}, key=int) - d = curl_json("https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=pubmed&retmode=json&id=" - + ",".join(pmids)) - res = d.get("result", {}) - for p in pmids: - rec = res.get(p) or {} - if "error" in rec or not rec.get("title"): - errs.append(f"PMID {p} does not resolve in PubMed ({rec.get('error', 'no record')})") - print(f" {len(pmids)} PubMed ids looked up") - for r in rows: - time.sleep(0.4) # NCBI allows three requests a second without a key - try: - snp = curl_json(f"https://api.ncbi.nlm.nih.gov/variation/v0/refsnp/{r['rsid'][2:]}") - except RuntimeError as e: - errs.append(f"{r['rsid']}: dbSNP lookup failed: {e}") - continue - psd = snp.get("primary_snapshot_data") - if not psd: - errs.append(f"{r['rsid']}: not a current dbSNP record (merged or withdrawn)") - continue - place = None - for pl in psd.get("placements_with_allele", []): - traits = pl.get("placement_annot", {}).get("seq_id_traits_by_assembly", []) - if pl.get("is_ptlp") and traits and traits[0].get("assembly_name", "").startswith("GRCh38"): - place = pl - break - if place is None: - errs.append(f"{r['rsid']}: no GRCh38 placement in dbSNP") - continue - spdi = [a["allele"]["spdi"] for a in place["alleles"]] - chrom = NC.get(place["seq_id"].split(".")[0], place["seq_id"]) - pos, ref = spdi[0]["position"] + 1, spdi[0]["deleted_sequence"] - alts = {s["inserted_sequence"] for s in spdi if s["inserted_sequence"] != s["deleted_sequence"]} - if (chrom, str(pos), ref) != (r["chrom"], r["pos"], r["ref"]): - errs.append(f"{r['rsid']}: table {r['chrom']}:{r['pos']} {r['ref']}, dbSNP GRCh38 {chrom}:{pos} {ref}") - elif r["alt"] not in alts: - errs.append(f"{r['rsid']}: alt {r['alt']} is not one of dbSNP's alleles {sorted(alts)}") - print(f" {len(rows)} rsIDs looked up in dbSNP") - return errs - - -def read_table(path): - """The rows of a traits table as dicts (its schema is og traits check's to judge).""" - with open(path, encoding="utf-8") as f: - return list(csv.DictReader((line for line in f if line.strip() and not line.startswith("#")), - delimiter="\t", quoting=csv.QUOTE_NONE)) - - -def schema(path): - return subprocess.run([os.environ.get("OG", os.path.join(REPO, "bin", "og")), - "traits", "check", "--table", path], capture_output=True, text=True) - - -def schema_controls(): - good = read_table(TABLE) - check("the table has at least 30 rows", len(good) >= 30, len(good)) - work = tempfile.mkdtemp() - try: - # Schema controls are Rust unit tests and traits parity cases. - res = schema(TABLE) - check("assets/traits.tsv passes og traits check", res.returncode == 0, res.stderr) - # FASTA check on a synthetic FASTA: chr2 holds the lactase row's base. - lct = next(r for r in good if r["rsid"] == "rs4988235") - pos = int(lct["pos"]) - fa = os.path.join(work, "ref.fasta") - seq = ["A"] * pos - seq[pos - 1] = lct["ref"] - width = 60 - body = "".join("".join(seq[i:i + width]) + "\n" for i in range(0, pos, width)) - with open(fa, "w") as f: - f.write(">chr2\n" + body) - with open(fa + ".fai", "w") as f: - f.write(f"chr2\t{pos}\t6\t{width}\t{width + 1}\n") - e, n = fasta_errors(fa, [lct]) - check("FASTA check: the right base passes", not e and n == 1, (e, n)) - wrong = dict(lct, ref=next(b for b in "ACGT" if b != lct["ref"])) - e, n = fasta_errors(fa, [wrong]) - check("FASTA check control: a planted wrong reference base fails, naming the row", - len(e) == 1 and "rs4988235" in e[0], e) - other = next(r for r in good if r["chrom"] != "chr2") - e, n = fasta_errors(fa, [other]) - check("FASTA check: a row on a contig the FASTA lacks is not checked", not e and n == 0, (e, n)) - finally: - shutil.rmtree(work) - - -def main(argv=None): - ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0]) - ap.add_argument("--table", default=TABLE) - ap.add_argument("--fasta", help="check each row's ref against this FASTA (with its .fai)") - ap.add_argument("--min-fasta-rows", type=int, default=1, - help="fail unless at least this many rows were checked against the FASTA") - ap.add_argument("--online", action="store_true", help="look the PMIDs up in PubMed and the rsIDs in dbSNP") - a = ap.parse_args(argv) - if a.table == TABLE and not a.fasta and not a.online: - if not os.environ.get("OG") and not os.environ.get("OG_REQUIRED"): - print("skip: OG is not set: traits schema controls run in the Rust job") - return 0 - schema_controls() - else: - res = schema(a.table) - check(f"{a.table} passes the schema check", res.returncode == 0, res.stderr) - rows = read_table(a.table) if res.returncode == 0 else [] - if a.fasta and rows: - e, n = fasta_errors(a.fasta, rows) - print(f" ref checked against {a.fasta}: {n} rows (the others are on contigs it lacks)") - check(f"every reference base matches the FASTA ({n} rows checked)", not e, e) - check(f"at least {a.min_fasta_rows} rows were checked against the FASTA", n >= a.min_fasta_rows, n) - if a.online and rows: - try: - e = online_errors(rows) - except RuntimeError as x: - e = [f"lookup failed: {x}"] - check("every PMID resolves and every rsID matches dbSNP's GRCh38 record", not e, e) - print("\nRESULT:", "ALL PASS" if FAILS == 0 else f"{FAILS} FAILED") - return 1 if FAILS else 0 - - -if __name__ == "__main__": - sys.exit(main())