Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,10 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n

## Unreleased

- The checks of the HBA copy number, the blood groups and the traits table run in `cargo test`, and four Python tests are gone: `tests/test_hba_copy_number.py`, `test_bloodtype_report.py`, `test_bloodtype_hpa_hna.py` and `test_traits_table.py`. Every check they made is kept, through the `og cn`, `og bloodtype` and `og traits` command lines. The traits table's source check is a new verb, `og dev traits-sources`: `--fasta` checks each reference base against a FASTA and `--online` checks each PubMed id and rsID against PubMed and dbSNP. The e2e case `traits-panel-1` and the curation steps in the traits page use it. No pipeline output changes.
- ROH summaries now divide F_ROH by the GRCh38 length outside the bands of the assessed autosomes. Per-site rows identify those chromosomes, including chromosomes with no ROH. A new `roh.data.scope` records the chromosomes, denominator and scope source. Relationship labels and UPD flags require sites on all 22 autosomes and the full unmasked autosome length. Segment-only files use their observed chromosomes as an unverified lower bound and get neither label. Empty input has null F_ROH. Segment counts and the step 11 summary table keep their format.
- A one-chromosome ROH flag now names regions and citations without syndrome names. Its note includes deletion, acquired loss of heterozygosity and shared ancestry as alternatives. The maternal and paternal JSON keys remain as `"-"` for reader compatibility. The attributed `assets/imprinted_regions.tsv` is embedded at build time, replacing the duplicated Rust table. The container builder mounts it, and the source stamp includes its blob hash, so a table edit makes an old binary stale. The stamp format and `tree-hash` command name stay the same. Every listed chromosome uses the same lookup, including chr11 and chr15. A region hint does not establish overlap with its imprinted locus or parental origin. Broad cytoband names remain reference labels, not diagnostic intervals.
- The ROH collector now streams per-site rows instead of keeping the whole file in memory. On 1.1 million synthetic site rows, peak RSS fell from 65.7-67.0 MB to 3.0-3.5 MB. CPU time stayed near 0.2 seconds. Under shared host load, wall time ranged from 1.40-21.46 seconds before and 7.89-38.16 seconds after; these timings do not establish a speedup.

- The reports now get summary.json from the Rust `og` binary. Step 24, the Nextflow HTML_REPORT process and `generate-report.sh` collect it with `og summary collect` (through `bin/render_report.py`), and a run collects each sample once: `generate-report.sh` renders the text report from the summary.json step 24 wrote, and collects again when a file it reads is newer than that summary.json, or a file was added to or removed from a step folder since, or a file a section names as its source, the run status or a table in `bin/` or `assets/` is gone. When it renders from step 24's summary.json, the text report's generated time is step 24's, the same as the HTML report's; it was the minute of its own collect. When it collects again, the text report carries the new collect's values and time, and the HTML report keeps step 24's until step 24 runs again, as before. The deliberate fixes the `og summary collect` entries below list now reach the reports; apart from them, nothing in summary.json or either report changes. `bin/collect_summary.py` keeps only what the report renderer (`bin/render_report.py`), the couple report, the results index and the live view import from it. The Python modules that only the old collector imported are gone: `bin/pgx_outside_calls.py`, `ancestry_fine_report.py`, `clip_check.py`, `annotsv_postfilter.py`, `chip_report.py`, `traits_report.py`, `run_delta.py` and `genotype_rules_indels.py`. On the synthetic demo sample (4.7 million VCF records) the collect takes 1.1 s of CPU and 12-15 MB, where the Python took 7.8-8.0 s and 34-37 MB. What changes on purpose:
- The walk of the sample folder leaves out `<sample>/nextflow`. `run-all.sh` launches Nextflow there, and its `work/` holds every task's files and staged links. No section reads a file there. With 200,000 files under it, the collect took 67 MB walked and 14 MB skipped.
- The sections other than the sample VCF are read one at a time, while the sample VCF is read on its own thread, so the peak memory is near the biggest section's and not the sum of all of them. With every side table 50 times its demo size, the peak footprint fell from 119-130 MB to 105-112 MB (the Python: 100-102 MB), with the same CPU time.
Expand Down
2 changes: 1 addition & 1 deletion assets/traits.tsv
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
# Hand-curated from the primary papers in the pmids column; no text is taken from SNPedia or any
# other database. GRCh38; ref is the GRCh38 base, alt the allele the readings are about.
# hom_ref, het, hom_alt: the reading for ref/ref, ref/alt and alt/alt. ancestry: yes when the
# reading depends on ancestry (the note says how). tests/test_traits_table.py checks the schema.
# reading depends on ancestry (the note says how). og traits check checks the schema.
chrom pos rsid gene ref alt trait hom_ref het hom_alt ancestry note pmids
chr2 135851076 rs4988235 MCM6 G A Lactase persistence (digesting milk sugar as an adult) No copy of -13910T, the European lactase persistence allele: lactase activity usually falls after childhood, unless another row below carries a persistence allele. One copy of -13910T: lactase activity usually stays into adulthood. Two copies of -13910T: lactase activity usually stays into adulthood. yes Lactase persistence is dominant: one persistence allele at any of the five lactase rows is usually enough. -13910T explains it in Europeans; in Africa other alleles do (rows -14010, -13915, -13907), so read the five rows together. 11788828
chr2 135859184 rs182549 MCM6 C T Lactase persistence (digesting milk sugar as an adult) No copy of -22018A, which travels with the -13910T persistence allele in Europeans. One copy of -22018A; read it with row rs4988235, which predicts persistence better. Two copies of -22018A; read it with row rs4988235, which predicts persistence better. yes A weaker marker than -13910: it matched lactase non-persistence in 229 of 236 cases of the original study, -13910 in all of them. Lactase persistence is dominant: one persistence allele at any of the five lactase rows is usually enough. -13910T explains it in Europeans; in Africa other alleles do (rows -14010, -13915, -13907), so read the five rows together. 11788828
Expand Down
2 changes: 1 addition & 1 deletion docs/35-paralogs.md
Original file line number Diff line number Diff line change
Expand Up @@ -237,7 +237,7 @@ The ratio of a site is its estimate over the median of the sites, so one copy co
- `tests/test_paralog_loci.py` checks the model list (167 genes in `assets/parascopy_models_v1.7.txt`, each with one main region) and every catalog row (a model on the list, a GRCh38 region inside the model's main region, a known tag and cross-check, a table name used once, the SMN1 row), and that planted faults are refused: a model not on the list (`GBA1.gz`, and `CYP21A2.gz`, which the data has no model for), a bad region, a region outside its model, a repeated table name, a catalog without the SMN1 row. It checks one owner per locus: no catalog model overlaps a depth-engine row of `bin/cn_loci.tsv` but RHCE (without that exemption RHCE is refused, and a planted model over HBA is refused), and every catalog gene `assets/callability_genes.tsv` lists has the owner `parascopy` there (a planted `.` is refused). The catalog sets owners in that file but adds no rows to it: its rows come from GIAB's lists, gene body by gene body, and a model can cover only part of a gene. So 17 catalog genes are not there: seven pseudogenes (CELP, GBAP1, NCF1B, NCF1C, OTOAP1, PMS2CL, STRCP1), which it leaves out as not protein-coding, and ten genes to which none of its reasons applies (ABCC6, CFH, CFHR4, CHRNA7, DEFA3, KRT86, NEB, PIK3CA, PLG, VWF). This section reads them all. The test holds that list and fails when it changes. It also runs the step's own lines that write a not-assessed SMN table and checks that the report, the Paraphase side-by-side and the couple screen each read it as not assessed, with the reason, and that a catalog table without `agCN_filter` or `agCN_qual` reads as unreadable, not as not resolved.
- `tests/fake-docker/paralog-catalog-loop.sh` runs the step on a catalog of three rows against a fake Docker: three `parascopy cn-using` calls with the right model files, in order, one table each; another population's files with `PARASCOPY_POPULATION`; no call and a not-assessed table for a region without reads, a model that is not installed and a Parascopy run that fails, while the other rows run; a rerun with fewer rows removes the tables of the dropped rows and keeps the HBA and Paraphase tables; a bad row, or a region off chr1-22, X and Y or with its start past its end, stops the step before any container.
- The e2e case `paralog-catalog-1-gba` adds a GBA row (`GBA.gz`) to a copy of the catalog and runs the step on the fixture's slice plus GIAB's HG002 reads over GBA1 and GBAP1, sampled to about 30x: the GBA table must hold an integer agCN with its quality. On the fixture's slice alone, which has no read at GBA1, the same row must be not assessed. The case also checks that the committed models are the names in the installed `list.txt` of each population, that the SMN table is byte for byte what one `parascopy cn-using` of `SMN1.gz` writes, as the step did before the catalog, and that the loop of the Nextflow module, cut from the file, writes the same two tables in the same image.
- `tests/test_copy_number.py` checks the HBA row and that these depth tables give, through the engine, the same tables as the code before it. `tests/test_hba_copy_number.py` gives `og cn call --locus HBA` made-up depth tables: four genes at even depth, three with HBA2 at half depth, two, a triplication, an estimate between two whole numbers, low flanks, flanks that disagree, and no read of mapping quality 0 over the genes (the depths of the novoalign BAM, and a clean three), which must be not assessed.
- `tests/test_copy_number.py` checks the HBA row and that these depth tables give, through the engine, the same tables as the code before it. `cargo test -p og-cn --test port_tests_cn_traits` gives `og cn call --locus HBA` made-up depth tables: four genes at even depth, three with HBA2 at half depth, two, a triplication, an estimate between two whole numbers, low flanks, flanks that disagree, and no read of mapping quality 0 over the genes (the depths of the novoalign BAM, and a clean three), which must be not assessed.
- The e2e case `paralog-checks-and-doc-fixes-1-hba` writes synthetic 30x reads over chr16:95,001-235,000 for two samples, one with every second read over HBA2 left out (a planted one-gene loss), runs step 35 on each and checks four and three genes. A synthetic read whose bases occur more than once in that stretch gets mapping quality 0, as an aligner gives it. The HBA block of `modules/local/parascopy/main.nf`, cut from the file, then runs on the same BAMs in the same image and must write the same table. Last, the block runs on HG002's real Illumina reads at the cluster, three ways: GIAB's 60x novoalign BAM sampled to about 30x must be not assessed, naming the missing mapping quality 0 reads; Google's 35x NovaSeq BWA-MEM BAM must count four; and the same reads realigned with `minimap2 -a -x sr` to chr16:1-300,000 must count four. The truth is four, from Paraphase on HG002's HiFi reads. The run's notes record each estimate, the four depths and the share of mapping quality 0.
- The e2e case `paralog-rows-1` runs the whole catalog on the fixture's slice plus GIAB's HG002 reads over every catalog locus on the fixture's chromosomes, sampled to about 30x, timed (it fails past 30 minutes): every row's table must hold an integer agCN or say not assessed with the reason, and a header-only table fails that check. It then runs the step with a catalog that has no SMN1 row and checks that both reports still show the HBA count and the other loci. `tests/test_collect_summary.py` holds the same case on made-up tables, with the loci read in the catalog's order.
- `tests/test_copy_number.py` section 10 runs the CYP21A2 row on made-up bedcov and mpileup tables (`tests/fixtures/cn/CYP21A2_*`): unmodified, 2 copies and no finding; a planted chimeric deletion, 1 copy with the module total 3; a planted In2G conversion on one copy, a ratio of 0.5 at In2G and a possible conversion there. The same depths with every site even give no finding. The deletion's tables with no MAPQ 0 read (a share under 0.02) read not assessed (`low_mapq0_share`), and with a share of 0.021 or HG002's 0.1 still give the deletion. The CYP21A2 block of step 35, cut out, writes those tables, a not-assessed one when mpileup fails and a `low_mapq0_share` one when its MAPQ 1 pass equals the first, and the couple screen and the report read them.
Expand Down
2 changes: 1 addition & 1 deletion docs/42-traits.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ Variants whose effect allele the papers we could read did not state were left ou

### Adding a row

Read the primary paper first and write the reading from it, in your own words. Take the GRCh38 position, the reference base and the allele from dbSNP's record of the rsID, and the direction of the effect from the paper, checking the strand: an allele named on the gene's strand of a gene on the minus strand is the complement on GRCh38. Then run `python3 tests/test_traits_table.py --online`, which looks up every PubMed id and checks each rsID's GRCh38 position and alleles against dbSNP.
Read the primary paper first and write the reading from it, in your own words. Take the GRCh38 position, the reference base and the allele from dbSNP's record of the rsID, and the direction of the effect from the paper, checking the strand: an allele named on the gene's strand of a gene on the minus strand is the complement on GRCh38. Then run `bin/og traits check --table assets/traits.tsv`, which checks the schema, and `bin/og dev traits-sources --online`, which looks up every PubMed id and checks each rsID's GRCh38 position and alleles against dbSNP (add `--fasta` with the reference to check each reference base too).

## Genotype Rules

Expand Down
2 changes: 1 addition & 1 deletion docs/cn-loci.md
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,7 @@ pypgx owns the genes it calls from the BAM, the `bam` rows of `assets/pgx_genes.
- The e2e case `cn-intervals-1-catalog` rebuilds the interval rows from the pinned downloads and fails on any difference with the committed catalog. `cn-intervals-2-hg002` runs the interval blocks of step 35 and of the PARASCOPY module on HG002's real reads at the AZF genes, their controls and 22q11.2: a male with one copy of AZFa and AZFb, AZFc_DAZ not assessed (its estimate, between 3.3 and 3.9, is no whole count of the four DAZ copies, so the row says `between_counts` rather than a call), no change at 22q11.2 A-B, no call on a row off chrY and chr22 (they have no read; the chr22 rows share A-B's control window, so they are left out of that check), and the AZF rows not applicable when the same reads are read as female.
- The e2e case `followups-cn-1-couple-report` runs step 39, COUPLE_CARRIER and HTML_REPORT on a made-up couple, plus a stub run of `main.nf`, and checks that the interval table and the CNVpytor calls reach the couple report and `summary.json`. Each wiring line has a red control: the case runs again without it and checks that the rows are gone.

- `tests/test_copy_number.py` checks every row and refuses broken ones. For the three gene-dosage rows it calls made-up depths at 0 to 4 copies, cuts the dosage block out of step 35 and runs it on the made-up bedcov tables in `tests/fixtures/cn/` (HP 3, LCE3BC 1, CCL3L1 5 copies), checks that a row shifted by 1 kb no longer matches its table, and that the PARASCOPY module runs the same list. It runs the owner check with planted AMY1C, C4B and DEFB4 rows. It feeds made-up bedcov tables at 0, 1, 2, 3 and 4 copies and a MAPQ 0 share over the ceiling. It checks that a control with no read stops the run with `BROKEN_CONTROL`. With `OG` set (Guard's rust job sets it) it also checks that the depth tables of `tests/test_hba_copy_number.py` and `tests/test_lpa_kiv2.py` give, through `og cn`, the same tables the Python code before it wrote, and that `og cn` refuses the broken rows with the same message as the catalog reader.
- `tests/test_copy_number.py` checks every row and refuses broken ones. For the three gene-dosage rows it calls made-up depths at 0 to 4 copies, cuts the dosage block out of step 35 and runs it on the made-up bedcov tables in `tests/fixtures/cn/` (HP 3, LCE3BC 1, CCL3L1 5 copies), checks that a row shifted by 1 kb no longer matches its table, and that the PARASCOPY module runs the same list. It runs the owner check with planted AMY1C, C4B and DEFB4 rows. It feeds made-up bedcov tables at 0, 1, 2, 3 and 4 copies and a MAPQ 0 share over the ceiling. It checks that a control with no read stops the run with `BROKEN_CONTROL`. With `OG` set (Guard's rust job sets it) it also checks that the depth tables of the HBA cases (`cargo test -p og-cn --test port_tests_cn_traits`) and of `tests/test_lpa_kiv2.py` give, through `og cn`, the same tables the Python code before it wrote, and that `og cn` refuses the broken rows with the same message as the catalog reader.
- The fake-docker case `cn-engine-calls` runs steps 35, 41 and 46 with made-up depths. Each step calls `og cn` once for its regions, step 35 and 46 once more for the call, step 46 reads the KIV-2 4925G>A marker in the same call, and no step calls a Python copy-number script. A control exon with no read fails step 46 with `BROKEN_CONTROL`. The case needs `og`, so Guard's rust job runs it after the build.
- `tests/test_copy_number.py` section 10 checks the CYP21A2 row: its six sites, the pileup parser, and the fixtures `tests/fixtures/cn/CYP21A2_{normal,deletion,conversion}` (2 copies and no finding; 1 copy with the module total 3; an In2G ratio of 0.5 and a possible conversion), with the conversion's depths read with every site even as the red control (no finding). The e2e case `cyp21a2-short-reads-1-hg002` runs the step's and the module's blocks on HG002's reads, unmodified and with a planted conversion and a planted deletion. The same section feeds the deletion's tables with MAPQ 0 shares of 0, 0.019, 0.021 and 0.1 (not assessed twice, then the deletion), and `cn-guard-1-mapq0` runs both blocks on HG002's reads with every MAPQ 0 read removed: not assessed (`low_mapq0_share`), never a deletion.
- `tests/parity` runs `og cn` against the Python engine it replaced on every command and the tables of the three per-gene loci (`scripts/ci/oracle-diff.sh`, [docs/rust.md](rust.md)); the deliberate fixes are its goldens.
Expand Down
Loading
Loading