Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,13 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n

## Unreleased

- Step 49 now states the CHIP read-support limit from each coding base's depth, not from a gene's mean depth. With one mean, a gene with a fifth of its coding bases unread read `covered`, with almost the limit of a gene read evenly, and the step and both reports said a clone above the limit "is found 8 times in 10". Now each gene's limit is the smallest VAF at which a clone shows 3 reads with the change 8 times in 10, averaged over its bases, each at its own depth. A gene where no VAF reaches that reads `partly_covered` (some bases have 3 reads) or `too_shallow` (none), with no limit. The stated limit of a run is the median of the covered genes' limits, in the step's line, in summary.json's `median_limit` and so in both reports; it was the limit at the median depth. What changes in the outputs:
- `chip/<sample>_chip_depth.tsv` gets two columns at the end: `callable_fraction`, the share of a gene's coding bases with 10 reads or more, and `assessable_fraction`, the share where a clone at the stated limit shows its reads. Its `detection_limit_vaf` is the per-base limit, and `status` can be `partly_covered`. The first seven columns keep their names and order.
- summary.json's chip `genes` carry the two new columns when the table has them, and `median_limit` is the median of the covered genes' limits. Uneven depth raises the limit: half the bases at 20x and half at 40x give about 16%, where the mean of 30x gave 14%. Even depth gives the same limit as before.
- The step's line says "Read-support limit", says that Mutect2 and its filters call fewer clones than that, and names each gene with no limit and the share of its bases read deep enough. On 2 Mb of synthetic depth up to 5,000x the report takes 0.5 s and 5.3 MB, as before (5.0 MB). Part of #411.
- ROH summaries now divide F_ROH by the GRCh38 length outside the bands of the assessed autosomes. Per-site rows identify those chromosomes, including chromosomes with no ROH. A new `roh.data.scope` records the chromosomes, denominator and scope source. Relationship labels and UPD flags require sites on all 22 autosomes and the full unmasked autosome length. Segment-only files use their observed chromosomes as an unverified lower bound and get neither label. Empty input has null F_ROH. Segment counts and the step 11 summary table keep their format.
- A one-chromosome ROH flag now names regions and citations without syndrome names. Its note includes deletion, acquired loss of heterozygosity and shared ancestry as alternatives. The maternal and paternal JSON keys remain as `"-"` for reader compatibility. The attributed `assets/imprinted_regions.tsv` is embedded at build time, replacing the duplicated Rust table. The container builder mounts it, and the source stamp includes its blob hash, so a table edit makes an old binary stale. The stamp format and `tree-hash` command name stay the same. Every listed chromosome uses the same lookup, including chr11 and chr15. A region hint does not establish overlap with its imprinted locus or parental origin. Broad cytoband names remain reference labels, not diagnostic intervals.
- The ROH collector now streams per-site rows instead of keeping the whole file in memory. On 1.1 million synthetic site rows, peak RSS fell from 65.7-67.0 MB to 3.0-3.5 MB. CPU time stayed near 0.2 seconds. Under shared host load, wall time ranged from 1.40-21.46 seconds before and 7.89-38.16 seconds after; these timings do not establish a speedup.

- The reports now get summary.json from the Rust `og` binary. Step 24, the Nextflow HTML_REPORT process and `generate-report.sh` collect it with `og summary collect` (through `bin/render_report.py`), and a run collects each sample once: `generate-report.sh` renders the text report from the summary.json step 24 wrote, and collects again when a file it reads is newer than that summary.json, or a file was added to or removed from a step folder since, or a file a section names as its source, the run status or a table in `bin/` or `assets/` is gone. When it renders from step 24's summary.json, the text report's generated time is step 24's, the same as the HTML report's; it was the minute of its own collect. When it collects again, the text report carries the new collect's values and time, and the HTML report keeps step 24's until step 24 runs again, as before. The deliberate fixes the `og summary collect` entries below list now reach the reports; apart from them, nothing in summary.json or either report changes. `bin/collect_summary.py` keeps only what the report renderer (`bin/render_report.py`), the couple report, the results index and the live view import from it. The Python modules that only the old collector imported are gone: `bin/pgx_outside_calls.py`, `ancestry_fine_report.py`, `clip_check.py`, `annotsv_postfilter.py`, `chip_report.py`, `traits_report.py`, `run_delta.py` and `genotype_rules_indels.py`. On the synthetic demo sample (4.7 million VCF records) the collect takes 1.1 s of CPU and 12-15 MB, where the Python took 7.8-8.0 s and 34-37 MB. What changes on purpose:
- The walk of the sample folder leaves out `<sample>/nextflow`. `run-all.sh` launches Nextflow there, and its `work/` holds every task's files and staged links. No section reads a file there. With 200,000 files under it, the collect took 67 MB walked and 14 MB skipped.
- The sections other than the sample VCF are read one at a time, while the sample VCF is read on its own thread, so the peak memory is near the biggest section's and not the sum of all of them. With every side table 50 times its demo size, the peak footprint fell from 119-130 MB to 105-112 MB (the Python: 100-102 MB), with the same CPU time.
Expand Down
32 changes: 26 additions & 6 deletions docs/49-chip.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,22 +76,42 @@ With `run-all.sh`: `TOOLS=...,chip`. It is opt-in, so a run lists step 49 only w

Synonymous, intronic and UTR changes are left out.

## The detection limit
## The read-support limit

The step counts the depth over the same exons with bcftools mpileup at Mutect2's default read filters (mapping quality 20 and up, base quality 10 and up, duplicates left out), and gives each gene a detection limit: the smallest VAF at which a clone shows at least 3 reads with the change 8 times in 10 at that gene's mean depth (a binomial count). At 20x that is about 20%, at 30x about 14%, at 40x about 10%, at 60x about 7% and at 100x about 4%.
The step counts the depth of each coding base with bcftools mpileup, at Mutect2's default read filters (mapping quality 20 and up, base quality 10 and up, duplicates left out). A base with no pileup line has depth 0.

So at the depth of a usual 30x genome, a clone has to hold about 14% of the alleles to be found reliably. CHIP starts at 2%, so a clone between 2% and the limit is likely missed: a result with no call does not rule out CHIP. A gene whose depth is far below the genome's (reads that map with low quality there) gets a higher limit, or `too_shallow` when it has fewer than 3 reads.
Each gene then gets a read-support limit: the smallest VAF at which a clone shows at least 3 reads with the change 8 times in 10, averaged over the gene's coding bases, each at its own depth (a binomial count per base). A base the run did not read counts as a base where no clone can show. With every base at one depth, the limit is about 20% at 20x, 14% at 30x, 10% at 40x, 7% at 60x and 4% at 100x. Uneven depth raises it: half the bases at 20x and half at 40x give about 16%, where the mean of 30x would say 14%.

It is a read count, not the caller's sensitivity. Mutect2's model and FilterMutectCalls can still reject a change that 3 reads carry, so they find fewer clones than the limit says.

The status of each gene:

| Status | Rule |
|---|---|
| `covered` | some VAF reaches 8 in 10; `detection_limit_vaf` holds it |
| `partly_covered` | some coding bases have 3 reads or more, but too few: no VAF reaches 8 in 10 (a fifth of the bases or more under 3 reads is enough). No limit |
| `too_shallow` | no coding base has 3 reads. No limit |
| `not_on_reference` | none of its coding bases is on the reference |

The stated limit of the run is the median of the covered genes' limits. The step's log line, the summary (`median_limit`) and both reports give that one figure, with the median of the covered genes' mean depths.

Two more columns help to find a gene with a region the run did not read:

- `callable_fraction`: the share of the gene's coding bases with 10 reads or more.
- `assessable_fraction`: the share of its coding bases where a clone at the stated limit shows 3 reads 8 times in 10. The limit is an average over bases, so a gene read near the median depth has only part of its bases at full power, often under half. Compare genes by it: a gene far below the others has bases the run did not read. Empty when no gene is covered.

So at the depth of a usual 30x genome, a clone has to hold about 14% of the alleles to show its reads reliably. CHIP starts at 2%, so a clone between 2% and the limit is likely missed: a result with no call does not rule out CHIP. A gene whose depth is far below the genome's (reads that map with low quality there) gets a higher limit, `partly_covered` when part of it has no reads, or `too_shallow`.

## Output

| File | Contents |
|---|---|
| `chip/${SAMPLE}_chip.tsv` | one row per protein change in a CHIP gene: `gene`, `chrom`, `pos`, `ref`, `alt`, `consequence`, `protein_change` (for example `R882H`, `Q1200fs`), `exon`, `vaf`, `alt_reads`, `depth`, `filter`, `status`, `class` (truncating or missense, for a change in the gene's class), `note` |
| `chip/${SAMPLE}_chip_depth.tsv` | one row per gene: `transcript`, `chrom`, `coding_bases`, `mean_depth`, `detection_limit_vaf`, `status` (covered, too_shallow, not_on_reference) |
| `chip/${SAMPLE}_chip_depth.tsv` | one row per gene: `transcript`, `chrom`, `coding_bases`, `mean_depth`, `detection_limit_vaf` (the read-support limit), `status` (covered, partly_covered, too_shallow, not_on_reference), `callable_fraction`, `assessable_fraction` |
| `chip/${SAMPLE}_chip_filtered.vcf.gz` | Mutect2's calls after FilterMutectCalls |
| `chip/${SAMPLE}_chip.gff3`, `chip/${SAMPLE}_chip.bed` | the transcripts and exons it called on (the bash step) |

Both reports show a "Clonal Haematopoiesis (CHIP)" section: the calls, the detection limit at the sample's depth, the genes too shallow to read, and how many other changes were listed.
Both reports show a "Clonal Haematopoiesis (CHIP)" section: the calls, the stated read-support limit, the genes that are not covered, and how many other changes were listed.

## Runtime

Expand All @@ -101,5 +121,5 @@ Not measured on a whole genome. Mutect2 reads about 35 kb of the BAM, which take

- A `chip` row is a finding worth showing a haematologist, together with a blood count. It does not mean a blood cancer, and the risks the studies report are averages over many people.
- Tumour-only calling cannot prove a change is acquired. The VAF rule removes most inherited variants, but a change at 35% or more is never counted, and a real large clone there is missed too.
- **U2AF1 on the default reference.** GRCh38 added to the short arm of chr21 a near copy of the U2AF1 locus that real genomes do not have, and the no-ALT analysis set keeps it. Reads from U2AF1 then fit both copies equally well and get mapping quality 0. Miller et al. 2022 (PMID 35041928) found that GRCh38 pipelines missed U2AF1 mutations for this reason, and that they were found again once the copy was masked. This step counts depth and Mutect2 reads only from mapping quality 20 up, so on the default reference expect U2AF1 to show `too_shallow` in `chip_depth.tsv`, or a detection limit far above the other genes'. Either way, no call at S34 or Q157 does not mean there is none. The fix is the masked reference, at the same coordinates: `setup.sh --masked-reference` builds it, with the false copies set to N as FixItFelix (Behera et al. 2023, PMID 36810122) and GIAB do, and a sample realigned to it gets its U2AF1 reads back ([the masked reference](realignment.md#the-masked-reference)). On chr21 of GIAB's HG002, step 16b reads U2AF1 on the masked reference and not on the default one (CI case `fixitfelix-chr21-1`); this step has not been measured on a whole genome either way.
- **U2AF1 on the default reference.** GRCh38 added to the short arm of chr21 a near copy of the U2AF1 locus that real genomes do not have, and the no-ALT analysis set keeps it. Reads from U2AF1 then fit both copies equally well and get mapping quality 0. Miller et al. 2022 (PMID 35041928) found that GRCh38 pipelines missed U2AF1 mutations for this reason, and that they were found again once the copy was masked. This step counts depth and Mutect2 reads only from mapping quality 20 up, so on the default reference expect U2AF1 to show `too_shallow` or `partly_covered` in `chip_depth.tsv`, or a read-support limit far above the other genes'. Either way, no call at S34 or Q157 does not mean there is none. The fix is the masked reference, at the same coordinates: `setup.sh --masked-reference` builds it, with the false copies set to N as FixItFelix (Behera et al. 2023, PMID 36810122) and GIAB do, and a sample realigned to it gets its U2AF1 reads back ([the masked reference](realignment.md#the-masked-reference)). On chr21 of GIAB's HG002, step 16b reads U2AF1 on the masked reference and not on the default one (CI case `fixitfelix-chr21-1`); this step has not been measured on a whole genome either way.
- Steps 29 and 49 call the same genes in two ways: step 29 calls gene bodies and leaves the calls unread, step 49 calls the coding exons and reads them.
2 changes: 1 addition & 1 deletion modules/local/chip/main.nf
Original file line number Diff line number Diff line change
Expand Up @@ -168,7 +168,7 @@ process CHIP_REPORT {
stub:
"""
printf 'gene\\tchrom\\tpos\\tref\\talt\\tconsequence\\tprotein_change\\texon\\tvaf\\talt_reads\\tdepth\\tfilter\\tstatus\\tclass\\tnote\\n' > ${meta.id}_chip.tsv
printf 'gene\\ttranscript\\tchrom\\tcoding_bases\\tmean_depth\\tdetection_limit_vaf\\tstatus\\n' > ${meta.id}_chip_depth.tsv
printf 'gene\\ttranscript\\tchrom\\tcoding_bases\\tmean_depth\\tdetection_limit_vaf\\tstatus\\tcallable_fraction\\tassessable_fraction\\n' > ${meta.id}_chip_depth.tsv
printf '"%s":\\n python: %s\\n' "${task.process}" "${task.container.replaceFirst(/^[^:@]+[:@]/, '')}" > versions.yml
"""
}
Loading
Loading