Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,10 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n

## Unreleased

- Step 04c (Sniffles2) now refuses a short-read BAM. Sniffles2 finds no SV in short reads and still exits 0, so the step reported success with an empty VCF. It now reads the mean length of the first 10,000 primary reads with the pinned samtools image and stops with exit 1 below 1,000 bp, naming the long-read BAM to use. Part of #411.
- `setup.sh` installs GIAB's GRCh38 tandem-repeat stratification (v3.6, `TANDEM_REPEATS_URL` and `TANDEM_REPEATS_SHA256`) and keeps it unpacked as `reference/GRCh38_AllTandemRepeats.bed`. Step 04c passes it to Sniffles2 as `--tandem-repeats`, which improves breakpoints and merging inside repeats, and says so when it is missing. On a public HG002 HiFi slice of chr20 it found one more benchmark SV (50 of 64 instead of 49) at the same precision. Sniffles' own GRCh38 file comes from PacBio's pbsv under a licence limited to PacBio data, so we use GIAB's.
- The step's closing hint no longer says duphold adds a GQ to long-read SVs. Sniffles2 writes its own QUAL and GQ, and step 15 reads Manta's calls only. The long-read guide's duphold row and its uncited accuracy lines are corrected.
- The SV smoke rows of Sniffles2, Manta, Delly and TIDDIT now need one PASS deletion of the planted size, not any record near it, and Sniffles2's row fails on a PASS call elsewhere. A new Delly row checks that `--sex` gives a haploid genotype on a male chrX of GRCh38's length.
- ROH summaries now divide F_ROH by the GRCh38 length outside the bands of the assessed autosomes. Per-site rows identify those chromosomes, including chromosomes with no ROH. A new `roh.data.scope` records the chromosomes, denominator and scope source. Relationship labels and UPD flags require sites on all 22 autosomes and the full unmasked autosome length. Segment-only files use their observed chromosomes as an unverified lower bound and get neither label. Empty input has null F_ROH. Segment counts and the step 11 summary table keep their format.
- A one-chromosome ROH flag now names regions and citations without syndrome names. Its note includes deletion, acquired loss of heterozygosity and shared ancestry as alternatives. The maternal and paternal JSON keys remain as `"-"` for reader compatibility. The attributed `assets/imprinted_regions.tsv` is embedded at build time, replacing the duplicated Rust table. The container builder mounts it, and the source stamp includes its blob hash, so a table edit makes an old binary stale. The stamp format and `tree-hash` command name stay the same. Every listed chromosome uses the same lookup, including chr11 and chr15. A region hint does not establish overlap with its imprinted locus or parental origin. Broad cytoband names remain reference labels, not diagnostic intervals.
- The ROH collector now streams per-site rows instead of keeping the whole file in memory. On 1.1 million synthetic site rows, peak RSS fell from 65.7-67.0 MB to 3.0-3.5 MB. CPU time stayed near 0.2 seconds. Under shared host load, wall time ranged from 1.40-21.46 seconds before and 7.89-38.16 seconds after; these timings do not establish a speedup.
Expand Down
25 changes: 16 additions & 9 deletions docs/long-read-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ These tools are specifically designed for short-read data and will produce incor
| 4a | TIDDIT | Short-read coverage/discordance model | Sniffles2 |
| 4b | GRIDSS | Assembly-based, short-read specific | Sniffles2 |
| 9 | ExpansionHunter | Illumina short-read graph model; expects paired-end data | TRGT (PacBio) or Straglr (ONT), outside this pipeline (vendor workflows) |
| 15 | duphold | Re-genotypes SVs using short-read depth models | Not needed — Sniffles2 QUAL scores are reliable |
| 15 | duphold | Adds depth fold-change tags (DHFFC, DHBFC) to Manta's calls and filters on them; reads Manta's VCF and the short-read BAM only | Not needed: Sniffles2 writes its own QUAL and GQ |
| 18 | CNVpytor | Read-depth model calibrated for short reads | Sniffles2 detects CNVs natively |
| 19 | Delly | Paired-end and split-read model | Sniffles2 |
| 21 | Cyrius (CYP2D6) | Short-read depth-based star allele caller | Paraphase (long-read CYP2D6 resolver), outside this pipeline (vendor workflows) |
Expand Down Expand Up @@ -217,11 +217,18 @@ ALIGN_DIR=aligned_longread ./scripts/04c-sniffles2.sh your_sample

Output: `${GENOME_DIR}/your_sample/sv_sniffles/your_sample_sv.vcf.gz`

Sniffles2 excels at detecting all SV types from long reads. Typical output for a 30X long-read genome:
- 15,000-25,000 SVs (vs 7,000-9,000 from short-read Manta)
- Much better at insertions (short reads systematically miss these)
- Better breakpoint resolution (often single-nucleotide)
- Reliable in repetitive regions where short reads struggle
Before Sniffles2 runs, the step reads the first 10,000 primary reads of the BAM with the pinned samtools image. When their mean length is below 1,000 bp, it stops with exit 1 and names the long-read BAM to use. Sniffles2 finds no SV in short reads and still exits 0, so without this check a short-read BAM gave an empty VCF that looked like a finished run. Short-read SVs come from steps 04 (Manta), 04a, 04b and 19.

`setup.sh` installs a tandem-repeat map for Sniffles2: every GRCh38 tandem repeat in GIAB's genome stratifications v3.6 (`reference/GRCh38_AllTandemRepeats.bed`, pinned by sha256 in `TANDEM_REPEATS_URL` and `TANDEM_REPEATS_SHA256`; NIST data, free to reuse). When the map is there, the step passes it as `--tandem-repeats`, which gives SVs inside repeats better breakpoints and merging. When it is not, the step prints a note and runs without it. We do not use the GRCh38 file that ships with Sniffles: it comes from PacBio's pbsv under a licence that allows PacBio data only.

What we measured: the step on GIAB's public HG002 PacBio HiFi BAM (Sequel II, 15-20 kb, 10x), chr20:10-20 Mb, scored with truvari against GIAB's v5.0q SV benchmark inside its benchmark regions (chr20:10.1-19.9 Mb, PASS calls, 1 kb matching distance):

| | Calls on chr20 | Recall | Precision |
|---|---|---|---|
| Without the tandem-repeat map | 62 | 0.77 (49 of 64) | 0.96 |
| With the map | 63 | 0.78 (50 of 64) | 0.96 |

That is one 10 Mb window at low depth, not a genome-wide benchmark. For Sniffles2's own accuracy figures, see Smolka et al., Nature Biotechnology 2024 (https://doi.org/10.1038/s41587-023-02024-y). No CI case runs this step on real long reads; [Testing](testing.md) lists what checks it.

### Step 5: Continue with Standard Pipeline Steps

Expand Down Expand Up @@ -346,10 +353,10 @@ The Docker image `CLAIR3_IMAGE` bundles models at `/opt/models/`. If you get a m
2. Check available models: `docker run --rm "${CLAIR3_IMAGE}" ls /opt/models/` (after `source versions.env` in the repository root)
3. If your ONT chemistry is different (e.g., R9.4.1), look for a matching model in the container

### "Sniffles2 produces 0 SVs"
### "Sniffles2 needs long reads" or "Sniffles2 produces 0 SVs"

- Verify the BAM is from long-read alignment (not short-read): `samtools stats your.bam | grep "average length"`
- Average read length should be >1000bp for ONT or >5000bp for HiFi
- Step 04c stops when the first 10,000 primary reads of the BAM average below 1,000 bp. Point `ALIGN_DIR` at the long-read BAM (step 02b writes `aligned_longread/<sample>_sorted.bam`).
- A mean above that but still low (below about 5,000 bp for HiFi) usually means a filtered or fragmented run: check `samtools stats your.bam | grep "average length"`
- Check that the BAM is sorted and indexed

### "DeepVariant long-read model fails"
Expand Down
4 changes: 3 additions & 1 deletion docs/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -173,7 +173,7 @@ This is every step script that no e2e case runs, plus the full-size index build
| 01 (ORA to FASTQ) | needs Illumina's `orad` binary, which runs outside Docker | nothing |
| 02b (long-read alignment) | long reads; the e2e fixture has none | the minimap2 image's smoke test, which maps synthetic HiFi reads with `-x map-hifi`; nothing runs the script |
| 03e (Clair3) | long reads; the e2e fixture has none | the image's smoke test, with a recall check on synthetic HiFi reads and a check that the two model directories the script names exist; the fake-docker suite runs the script |
| 04c (Sniffles2) | long reads; the e2e fixture has none | the image's smoke test on the synthetic HiFi alignment; `tests/fake-docker/bash-step-partial-outputs.sh` runs the script |
| 04c (Sniffles2) | long reads; the e2e fixture has none | the image's two smoke rows on the synthetic HiFi alignment: exactly one PASS deletion of 2.7-3.3 kb at the planted one and no PASS call elsewhere, once as before and once with `--tandem-repeats`; `tests/fake-docker/bash-step-sv.sh` runs the script (a short-read BAM stops it before Sniffles2, and the tandem-repeat map is passed when installed and noted when not), and `tests/fake-docker/bash-step-partial-outputs.sh` reruns it after a failed run. No case runs it on real long reads; [the long-read guide](long-read-guide.md#step-4-structural-variant-calling) has a measured run on a public HG002 HiFi slice |
| 15 (duphold) | the bash step: no case yet | case `nf-wiring-2-sv-str-tools` runs the Nextflow `DUPHOLD` and `DUPHOLD_FILTER` with the real image on the fixture's Manta calls; the image's smoke test on a planted deletion; `tests/fake-docker/bash-step-sv.sh` runs the script |
| 18 (CNVpytor) | the bash step: no case yet | case `nf-wiring-2-sv-str-tools` runs the Nextflow `CNVPYTOR` and `CNVPYTOR_VCF` with the real image and the pinned GC and mask files; the image's smoke test; `tests/fake-docker/followups-ci-2-steps-03c-18.sh` runs the script (resource checks, mounts, commands); the SV merge case reads a planted CNVpytor VCF, not one CNVpytor wrote |
| 29 (somatic Mutect2) | no case yet | `tests/fake-docker/bash-step-somatic-mito.sh` checks the Mutect2 command line |
Expand All @@ -182,6 +182,8 @@ This is every step script that no e2e case runs, plus the full-size index build
| 04a (TIDDIT) | no case yet | the image's smoke test; settle-doubts question 4 runs the script with only a BWA-MEM2 index; case `bash-step-bash42` and `tests/fake-docker/bash-step-sv.sh` run it against the fake docker |
| chip-to-vcf (array data to VCF) on a vendor download | case `consumer-export-1` runs it only on the 23andMe and AncestryDNA files step 40 wrote | `tests/test_chip_to_vcf.sh` in the unit-tests job runs the script with the real bcftools and Picard images on two ten-row vendor files |

The SV callers' smoke rows (Sniffles2, Manta, Delly and TIDDIT) each need exactly one PASS deletion of 2.7-3.3 kb at the 3 kb deletion planted in the mini reference: a record of another type, size or filter there does not count. GRIDSS writes breakends, not deletion records, so its row still counts any record there. A second Delly row checks the `--sex` value that step 19 and the Nextflow `DELLY` process pass: it moves `mini.bam` onto a chrX of GRCh38's length past PAR1, where the deletion must come out haploid (`1`) with `--sex male` and `1/1` with `--sex female`, Delly's log must name the sex, and an unknown value must fail. Delly ignores `--sex` without a message on a chrX of any other length; the row goes red when that chrX is one base longer. `image-smoke.sh --self-test` plants records for each rule of these checks.

Case `nf-wiring-2-sv-str-tools` runs the Nextflow `EXPANSION_HUNTER` module with the real image on the fixture's BAM (the bundled catalog cut to the fixture's contigs); the bash step 09 runs in case `bash-step-09-str`.

## The settle-doubts job
Expand Down
50 changes: 46 additions & 4 deletions scripts/04c-sniffles2.sh
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,13 @@
# Sniffles2 writes the bgzipped VCF and its .tbi itself (an -v name ending in
# .vcf.gz), and --allow-overwrite lets a rerun replace the output of a run
# that failed half way.
# Known gap: Sniffles2 calls SVs in tandem repeats more accurately with a
# tandem-repeat BED (--tandem-repeats); none is passed here yet.
#
# Sniffles2 finds no SV in short reads and still exits 0 with an empty VCF,
# so the step first reads the mean length of the first 10,000 primary reads
# (SAMTOOLS_IMAGE) and stops below 1,000 bp. With the tandem-repeat map
# setup.sh installs (TANDEM_REPEATS_URL, unpacked in reference/), Sniffles2
# gets --tandem-repeats, which gives SVs inside repeats better breakpoints
# and merging; without it, the step runs as before and says so.
set -euo pipefail

SAMPLE=${1:?Usage: $0 <sample_name>}
Expand Down Expand Up @@ -41,6 +46,42 @@ for f in "$BAM" "${BAM}.bai" "$REF" "${REF}.fai"; do
fi
done

# Long reads only: the mean length of the first 10,000 primary reads with a
# sequence, as "<reads> <mean bp>".
MIN_MEAN_READ_LEN=1000
# shellcheck disable=SC2016 # the program and $1/$2 belong to the inner shell
LEN_AWK='$10 != "*" { n++; s += length($10); if (n == 10000) exit } END { printf "%d %d\n", n, (n ? s / n : 0) }'
# shellcheck disable=SC2016
PROBE=$(run_in "$SAMTOOLS_IMAGE" sh -c 'samtools view -F 0x900 "$1" | awk -F "\t" "$2"' sh \
"/genome/${SAMPLE}/${ALIGN_DIR}/${SAMPLE}_sorted.bam" "$LEN_AWK")
read -r N_READS MEAN_LEN _ <<< "$PROBE" || true
if ! [[ "${N_READS:-}" =~ ^[0-9]+$ && "${MEAN_LEN:-}" =~ ^[0-9]+$ ]]; then
echo "ERROR: could not read the read lengths of ${BAM} (samtools printed '${PROBE}')" >&2
exit 1
fi
if [ "$N_READS" -eq 0 ]; then
echo "ERROR: ${BAM} has no primary read with a sequence (see samtools's message above, if any); Sniffles2 has nothing to call SVs from." >&2
exit 1
fi
if [ "$MEAN_LEN" -lt "$MIN_MEAN_READ_LEN" ]; then
echo "ERROR: the first ${N_READS} primary reads of ${BAM} average ${MEAN_LEN} bp." >&2
echo " Sniffles2 needs long reads (ONT or PacBio HiFi, ${MIN_MEAN_READ_LEN} bp or more on average): on short reads it finds no SV and still exits 0." >&2
echo " Point ALIGN_DIR at the long-read BAM (step 02b writes aligned_longread/${SAMPLE}_sorted.bam). Short-read SVs come from steps 04 (Manta), 04a, 04b and 19." >&2
exit 1
fi
echo "Reads: the first ${N_READS} primary reads average ${MEAN_LEN} bp"

# The tandem-repeat map setup.sh installs: TANDEM_REPEATS_URL's file, unpacked.
TR_NAME=${TANDEM_REPEATS_URL##*/}
TR_BED="${GENOME_DIR}/reference/${TR_NAME%.gz}"
TR_ARGS=()
if [ -f "$TR_BED" ] && [ -s "$TR_BED" ]; then
TR_ARGS=(--tandem-repeats "$(cpath "$TR_BED")")
echo "Tandem repeats: ${TR_BED}"
else
echo "NOTE: no tandem-repeat map at ${TR_BED} (setup.sh installs it): Sniffles2 runs without --tandem-repeats, so SVs inside repeats get less precise breakpoints and merging."
fi

mkdir -p "$OUTPUT_DIR"

# Run Sniffles2
Expand All @@ -54,7 +95,8 @@ run_in --cpus "$THREADS" --memory 16g \
--reference "${REF_FASTA_C}" \
--threads "$THREADS" \
--sample-id "${SAMPLE}" \
--allow-overwrite
--allow-overwrite \
${TR_ARGS[@]+"${TR_ARGS[@]}"}

if ! wrote_vcf "${OUTPUT_DIR}/${SAMPLE}_sv.vcf.gz"; then
echo "ERROR: Sniffles2 exited without a ${OUTPUT_DIR}/${SAMPLE}_sv.vcf.gz" >&2
Expand Down Expand Up @@ -88,4 +130,4 @@ echo ""
echo "Next steps:"
echo " - Annotate SVs: ./scripts/05-annotsv.sh ${SAMPLE} (set SV_VCF to point at sv_sniffles)"
echo " - Merge with short-read SVs: ./scripts/22-survivor-merge.sh ${SAMPLE}"
echo " - Filter quality: duphold can re-annotate long-read SVs with GQ scores"
echo " - Quality: Sniffles2 writes its own QUAL and FORMAT/GQ; step 15 (duphold) reads Manta's short-read calls only"
Loading
Loading