From e7fdd67cd17d8d62c59a650f5313ffd923f76588 Mon Sep 17 00:00:00 2001 From: GeiserX <9169332+GeiserX@users.noreply.github.com> Date: Sun, 15 Nov 2026 18:56:25 +0100 Subject: [PATCH 1/2] fix(mito): pin haplogrep fraction threshold and keep task working directory Document the 0.9 heterozygous FORMAT/AF threshold and the DeepVariant fallback. Classify from the Nextflow task directory and test threshold boundaries offline. Bead: lin-37 Bead: lin-40 Bead: D10 --- CHANGELOG.md | 2 + docs/12-mito-haplogroup.md | 12 +++--- modules/local/mito_haplogroup/main.nf | 16 +++----- scripts/12-mito-haplogroup.sh | 6 ++- tests/smoke/haplogrep3.sh | 56 +++++++++++++++++++++++---- 5 files changed, 67 insertions(+), 25 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d6409f33f..6df971bce 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,8 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n ## Unreleased +- Step 12 now states and explicitly passes haplogrep3's existing `--hetLevel 0.9`: heterozygous calls with FORMAT/AF below 0.9 do not enter the haplogroup profile. The docs distinguish this from haplocheck's contamination check, the FASTA's AF 0.5 rule and the DeepVariant fallback's unread FORMAT/VAF. The Nextflow task stays in its work directory and uses the pinned image's bundled config and trees offline. The image smoke test checks an empty work directory and retained and excluded heteroplasmic markers. No output name, format or classification threshold changes. Part of #411 (lin-37, lin-40, D10). + - The reports now get summary.json from the Rust `og` binary. Step 24, the Nextflow HTML_REPORT process and `generate-report.sh` collect it with `og summary collect` (through `bin/render_report.py`), and a run collects each sample once: `generate-report.sh` renders the text report from the summary.json step 24 wrote, and collects again when a file it reads is newer than that summary.json, or a file was added to or removed from a step folder since, or a file a section names as its source, the run status or a table in `bin/` or `assets/` is gone. When it renders from step 24's summary.json, the text report's generated time is step 24's, the same as the HTML report's; it was the minute of its own collect. When it collects again, the text report carries the new collect's values and time, and the HTML report keeps step 24's until step 24 runs again, as before. The deliberate fixes the `og summary collect` entries below list now reach the reports; apart from them, nothing in summary.json or either report changes. `bin/collect_summary.py` keeps only what the report renderer (`bin/render_report.py`), the couple report, the results index and the live view import from it. The Python modules that only the old collector imported are gone: `bin/pgx_outside_calls.py`, `ancestry_fine_report.py`, `clip_check.py`, `annotsv_postfilter.py`, `chip_report.py`, `traits_report.py`, `run_delta.py` and `genotype_rules_indels.py`. On the synthetic demo sample (4.7 million VCF records) the collect takes 1.1 s of CPU and 12-15 MB, where the Python took 7.8-8.0 s and 34-37 MB. What changes on purpose: - The walk of the sample folder leaves out `/nextflow`. `run-all.sh` launches Nextflow there, and its `work/` holds every task's files and staged links. No section reads a file there. With 200,000 files under it, the collect took 67 MB walked and 14 MB skipped. - The sections other than the sample VCF are read one at a time, while the sample VCF is read on its own thread, so the peak memory is near the biggest section's and not the sum of all of them. With every side table 50 times its demo size, the peak footprint fell from 119-130 MB to 105-112 MB (the Python: 100-102 MB), with the same CPU time. diff --git a/docs/12-mito-haplogroup.md b/docs/12-mito-haplogroup.md index 5fc8b210f..6e1e7645e 100644 --- a/docs/12-mito-haplogroup.md +++ b/docs/12-mito-haplogroup.md @@ -23,8 +23,10 @@ Pinned in `versions.env`; [Image versions](versions.md) lists the current tags. The step reads, first match: -1. `${SAMPLE}/mito/${SAMPLE}_chrM_filtered.vcf.gz`, step 20's Mutect2 calls in mitochondrial mode. Mutect2 is the caller made for chrM: it reports each allele's fraction, so haplogrep3 sees heteroplasmies and haplocheck can run. The step keeps the PASS records (possible NuMTs are filtered by step 20) and splits multi-allelic records. -2. Otherwise the chrM records of `${SAMPLE}/vcf/${SAMPLE}.vcf.gz` (step 03). DeepVariant calls chrM as a diploid nuclear contig, so this is the fallback, and haplocheck does not run on it. +1. `${SAMPLE}/mito/${SAMPLE}_chrM_filtered.vcf.gz`, step 20's Mutect2 calls in mitochondrial mode. Mutect2 reports each allele's fraction. haplocheck reads these fractions to check for contamination. The step keeps the PASS records (possible NuMTs are filtered by step 20) and splits multi-allelic records. +2. Otherwise the chrM records of `${SAMPLE}/vcf/${SAMPLE}.vcf.gz` (step 03). DeepVariant calls chrM as a diploid nuclear contig, so this is the fallback, and haplocheck does not run on it. Its heterozygous calls carry FORMAT/VAF, which haplogrep3 does not read. Only its homozygous ALT calls contribute variants to the profile. + +haplogrep3 classifies with `--hetLevel 0.9` in both the script and Nextflow. For heterozygous calls with FORMAT/AF, it keeps ALT alleles at a fraction of 0.9 or more. Lower fractions do not enter its haplogroup profile. Homozygous ALT calls are kept regardless of AF. The threshold does not change haplocheck's contamination check or step 20's heteroplasmy report. Run step 20 before step 12 for the Mutect2 input. The Nextflow pipeline does the same: with `mito_variants` in `--tools` (it is in the default set), `mito_haplogroup` reads `MITO_VARIANTS`' calls and runs `HAPLOCHECK`. @@ -46,7 +48,7 @@ docker run --rm --network none -v "${GENOME_DIR}:/genome" "${BCFTOOLS_IMAGE}" ba "bcftools view -f PASS ${M}/${SAMPLE}_chrM_filtered.vcf.gz | bcftools norm -m-any -Oz -o ${M}/${SAMPLE}_chrM_for_haplogroup.vcf.gz" docker run --rm --network none -v "${GENOME_DIR}:/genome" "${HAPLOGREP3_IMAGE}" \ - haplogrep3 classify --tree phylotree-fu-rcrs@1.2 \ + haplogrep3 classify --tree phylotree-fu-rcrs@1.2 --hetLevel 0.9 \ --input ${M}/${SAMPLE}_chrM_for_haplogroup.vcf.gz \ --output ${M}/${SAMPLE}_haplogroup.txt --extend-report @@ -65,7 +67,7 @@ docker run --rm --network none -v "${GENOME_DIR}:/genome" "${HAPLOCHECK_IMAGE}" Both reports (step 24 and the text report) show the haplogroup and the contamination status, or "not checked" when the input was the step 03 VCF, and the links of the next section built from the haplogroup. -The FASTA is the reference chrM (the rCRS, in GRCh38) with the sample's majority alleles applied: the Mutect2 calls with an allele fraction of 0.5 or more, or the haploid and homozygous ALT calls of the step 03 VCF. A position without a call reads as the reference, so the FASTA is only as complete as the chrM calls. A heteroplasmy below 0.5 is not in it. +The FASTA is the reference chrM (the rCRS, in GRCh38) with the sample's majority alleles applied: the Mutect2 calls with an allele fraction of 0.5 or more, or the haploid and homozygous ALT calls of the step 03 VCF. A position without a call reads as the reference, so the FASTA is only as complete as the chrM calls. A heteroplasmy below 0.5 is not in it. Mutect2 alleles at AF 0.5 to below 0.9 can therefore appear in the FASTA while absent from the haplogroup profile. ## Explore your branch further @@ -88,5 +90,5 @@ Names differ between trees. haplogrep3 uses PhyloTree 17 names (`phylotree-fu-rc ## Interpretation - Common European haplogroups: H, U, J, T, K, V, W, X - Output includes quality score (0-1): >0.9 = high confidence -- Discordant variants may indicate heteroplasmy (mixture of mtDNA types) +- Discordant variants alone do not establish heteroplasmy. Read step 20's allele fractions and haplocheck's contamination report alongside the haplogroup. - Contamination `YES` with a level of a few percent: compare with step 33's VerifyBamID2 estimate, which reads the nuclear genome. The two can differ when the mtDNA copy number differs between the two people's cells. diff --git a/modules/local/mito_haplogroup/main.nf b/modules/local/mito_haplogroup/main.nf index e15d83ea0..05093b9b8 100644 --- a/modules/local/mito_haplogroup/main.nf +++ b/modules/local/mito_haplogroup/main.nf @@ -92,20 +92,16 @@ process MITO_HAPLOGROUP { script: """ - # haplogrep3 reads haplogrep3.yaml and its trees from the working - # directory. In the task directory it finds neither and fetches the - # config from GitHub, which fails without network (and was an unpinned - # download before). So it runs in the directory of its binary, where the - # image keeps them, as tests/smoke/haplogrep3.sh does. - WD=\$PWD - cd "\$(dirname "\$(readlink -f "\$(command -v haplogrep3)")")" + # The pinned Bioconda build uses the config and trees beside its jar + # when the task directory has no haplogrep3.yaml. Stay in the task + # directory, with no network, as the image smoke test does. haplogrep3 classify \\ --tree phylotree-fu-rcrs@1.2 \\ - --input "\$WD/${chrm_vcf}" \\ - --output "\$WD/${meta.id}_haplogroup.txt" \\ + --hetLevel 0.9 \\ + --input "${chrm_vcf}" \\ + --output "${meta.id}_haplogroup.txt" \\ --extend-report REPORTED=\$( { haplogrep3 --version 2>&1 || true; } | awk '!v && match(\$0, /[0-9]+\\.[0-9]+(\\.[0-9]+)*/) { v = substr(\$0, RSTART, RLENGTH) } END { print (v != "" ? v : "unknown") }') - cd "\$WD" cat <<-END_VERSIONS > versions.yml "${task.process}": diff --git a/scripts/12-mito-haplogroup.sh b/scripts/12-mito-haplogroup.sh index f0380e986..7a1c685b0 100755 --- a/scripts/12-mito-haplogroup.sh +++ b/scripts/12-mito-haplogroup.sh @@ -9,8 +9,9 @@ # (docs/12-mito-haplogroup.md lists the routes; nothing is uploaded) # # Mutect2 in mitochondrial mode is the caller made for chrM: it reports each -# allele's fraction, so haplogrep3 sees heteroplasmies and haplocheck can look -# for two haplogroups mixed in one sample. DeepVariant calls chrM as a diploid +# allele's fraction. haplogrep3 keeps heterozygous calls with FORMAT/AF >= 0.9; +# haplocheck uses the heteroplasmies to look for two haplogroups mixed in one +# sample. DeepVariant calls chrM as a diploid # nuclear contig, so its calls are the fallback, without the contamination check. set -euo pipefail @@ -58,6 +59,7 @@ run_in \ "${HAPLOGREP3_IMAGE}" \ haplogrep3 classify \ --tree phylotree-fu-rcrs@1.2 \ + --hetLevel 0.9 \ --input "$(cpath "$CHRM")" \ --output "/genome/${SAMPLE}/mito/${SAMPLE}_haplogroup.txt" \ --extend-report diff --git a/tests/smoke/haplogrep3.sh b/tests/smoke/haplogrep3.sh index 7a086488c..3e686426d 100644 --- a/tests/smoke/haplogrep3.sh +++ b/tests/smoke/haplogrep3.sh @@ -1,11 +1,51 @@ # shellcheck shell=sh -# HAPLOGREP3_IMAGE row. haplogrep3 reads haplogrep3.yaml and its trees from -# the working directory; step 12 runs it in the image's own working -# directory, so the row moves to the directory of the binary first. -set -e -BIN=$(readlink -f "$(command -v haplogrep3)") -cd "$(dirname "$BIN")" +# HAPLOGREP3_IMAGE row. Start in an empty writable task directory. The pinned +# image finds its bundled config and trees beside the jar, without a network +# or a change into the binary directory (which can be read-only). +set -eu +WORK=$(mktemp -d /out/haplogrep3.XXXXXX) +trap 'rm -rf "$WORK"' EXIT +cd "$WORK" echo "running in $(pwd)" -haplogrep3 classify --tree phylotree-fu-rcrs@1.2 --input /in/mito.vcf.gz \ - --output /out/haplogroup.txt --extend-report +haplogrep3 classify --tree phylotree-fu-rcrs@1.2 --hetLevel 0.9 \ + --input /in/mito.vcf.gz --output /out/haplogroup.txt --extend-report cat /out/haplogroup.txt + +# Mutect2-style GT:AF calls: below, on and above the chosen threshold. +# A homozygous ALT call stays in the profile even with a low AF. +cat > fractions.vcf <<'VCF' +##fileformat=VCFv4.2 +##contig= +##FORMAT= +##FORMAT= +#CHROM POS ID REF ALT QUAL FILTER INFO FORMAT S1 +chrM 750 . A G 50 PASS . GT:AF 1/1:0.05 +chrM 3243 . A G 50 PASS . GT:AF 0/1:0.05 +chrM 5000 . C T 50 PASS . GT:AF 0/1:0.95 +chrM 9000 . T C 50 PASS . GT:AF 0/1:0.60 +chrM 10000 . A G 50 PASS . GT:AF 0/1:0.90 +VCF +haplogrep3 classify --tree phylotree-fu-rcrs@1.2 --hetLevel 0.9 \ + --input fractions.vcf --output fractions.txt --extend-report +awk -F '\t' ' + NR == 1 { + for (i = 1; i <= NF; i++) { + gsub(/"/, "", $i) + if ($i == "Input_Sample") col = i + } + next + } + NR == 2 && col { + profile = " " $col " " + gsub(/"/, "", profile) + ok = profile ~ / 750G / && profile ~ / 5000T / && profile ~ / 10000G / \ + && profile !~ / 3243G / && profile !~ / 9000C / + } + END { + if (!ok) { + print "FAIL: haplogrep3 profile does not obey hetLevel 0.9" > "/dev/stderr" + exit 1 + } + print "PASS: AF 0.90 and 0.95 retained; AF 0.05 and 0.60 excluded; homozygous ALT retained" + } +' fractions.txt From e309d017e1de75899032fc67123be1584a8e7abe Mon Sep 17 00:00:00 2001 From: GeiserX <9169332+GeiserX@users.noreply.github.com> Date: Sun, 15 Nov 2026 22:42:05 +0100 Subject: [PATCH 2/2] style(mito): rewrap the step 12 header comment --- scripts/12-mito-haplogroup.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/scripts/12-mito-haplogroup.sh b/scripts/12-mito-haplogroup.sh index 7a1c685b0..e533832f1 100755 --- a/scripts/12-mito-haplogroup.sh +++ b/scripts/12-mito-haplogroup.sh @@ -11,8 +11,8 @@ # Mutect2 in mitochondrial mode is the caller made for chrM: it reports each # allele's fraction. haplogrep3 keeps heterozygous calls with FORMAT/AF >= 0.9; # haplocheck uses the heteroplasmies to look for two haplogroups mixed in one -# sample. DeepVariant calls chrM as a diploid -# nuclear contig, so its calls are the fallback, without the contamination check. +# sample. DeepVariant calls chrM as a diploid nuclear contig, so its calls are +# the fallback, without the contamination check. set -euo pipefail SAMPLE=${1:?Usage: $0 }