diff --git a/CHANGELOG.md b/CHANGELOG.md index b3e8e4021..3ac8b2acf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,10 +4,10 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n ## Unreleased +- Fine ancestry (step 26b), ancient ancestry (step 44) and PRS (step 25) now keep only PASS or unfiltered records, in both bash and Nextflow. LowQual, RefCall and other filtered records stay missing instead of becoming genotypes. Fine ancestry's genotype and reference counts can fall, so its VCF reference-call requirement can reject an input it previously accepted. Ancient ancestry also requires GQ >= 20, including reference blocks; low or missing GQ stays missing. Its sites_used, PCA position and nearest groups can change. Output names and formats stay the same. Part of #411 (lin-50, lin-51). - ROH summaries now divide F_ROH by the GRCh38 length outside the bands of the assessed autosomes. Per-site rows identify those chromosomes, including chromosomes with no ROH. A new `roh.data.scope` records the chromosomes, denominator and scope source. Relationship labels and UPD flags require sites on all 22 autosomes and the full unmasked autosome length. Segment-only files use their observed chromosomes as an unverified lower bound and get neither label. Empty input has null F_ROH. Segment counts and the step 11 summary table keep their format. - A one-chromosome ROH flag now names regions and citations without syndrome names. Its note includes deletion, acquired loss of heterozygosity and shared ancestry as alternatives. The maternal and paternal JSON keys remain as `"-"` for reader compatibility. The attributed `assets/imprinted_regions.tsv` is embedded at build time, replacing the duplicated Rust table. The container builder mounts it, and the source stamp includes its blob hash, so a table edit makes an old binary stale. The stamp format and `tree-hash` command name stay the same. Every listed chromosome uses the same lookup, including chr11 and chr15. A region hint does not establish overlap with its imprinted locus or parental origin. Broad cytoband names remain reference labels, not diagnostic intervals. - The ROH collector now streams per-site rows instead of keeping the whole file in memory. On 1.1 million synthetic site rows, peak RSS fell from 65.7-67.0 MB to 3.0-3.5 MB. CPU time stayed near 0.2 seconds. Under shared host load, wall time ranged from 1.40-21.46 seconds before and 7.89-38.16 seconds after; these timings do not establish a speedup. - - The reports now get summary.json from the Rust `og` binary. Step 24, the Nextflow HTML_REPORT process and `generate-report.sh` collect it with `og summary collect` (through `bin/render_report.py`), and a run collects each sample once: `generate-report.sh` renders the text report from the summary.json step 24 wrote, and collects again when a file it reads is newer than that summary.json, or a file was added to or removed from a step folder since, or a file a section names as its source, the run status or a table in `bin/` or `assets/` is gone. When it renders from step 24's summary.json, the text report's generated time is step 24's, the same as the HTML report's; it was the minute of its own collect. When it collects again, the text report carries the new collect's values and time, and the HTML report keeps step 24's until step 24 runs again, as before. The deliberate fixes the `og summary collect` entries below list now reach the reports; apart from them, nothing in summary.json or either report changes. `bin/collect_summary.py` keeps only what the report renderer (`bin/render_report.py`), the couple report, the results index and the live view import from it. The Python modules that only the old collector imported are gone: `bin/pgx_outside_calls.py`, `ancestry_fine_report.py`, `clip_check.py`, `annotsv_postfilter.py`, `chip_report.py`, `traits_report.py`, `run_delta.py` and `genotype_rules_indels.py`. On the synthetic demo sample (4.7 million VCF records) the collect takes 1.1 s of CPU and 12-15 MB, where the Python took 7.8-8.0 s and 34-37 MB. What changes on purpose: - The walk of the sample folder leaves out `/nextflow`. `run-all.sh` launches Nextflow there, and its `work/` holds every task's files and staged links. No section reads a file there. With 200,000 files under it, the collect took 67 MB walked and 14 MB skipped. - The sections other than the sample VCF are read one at a time, while the sample VCF is read on its own thread, so the peak memory is near the biggest section's and not the sum of all of them. With every side table 50 times its demo size, the peak footprint fell from 119-130 MB to 105-112 MB (the Python: 100-102 MB), with the same CPU time. diff --git a/docs/25-prs.md b/docs/25-prs.md index 1e43a8926..cdbd5e5ee 100644 --- a/docs/25-prs.md +++ b/docs/25-prs.md @@ -112,8 +112,8 @@ Only additive scores are accepted (no `dosage_*_weight` columns, no `is_dominant 1. **Scoring files.** Downloads the GRCh38-harmonised file of each score from the PGS Catalog FTP when it is not in `${GENOME_DIR}/prs_scores/` yet, checked against the md5 the catalog publishes beside it. A file whose `#HmPOS_build` header is not `GRCh38` is refused. There is no fallback to the author-reported file, which is often GRCh37 or rsID-only and would score the wrong positions without any visible sign. 2. **Scores as pgsc_calc reads them.** `og prs format` (the Rust binary, `rust/crates/og-steps`) writes each file as a custom GRCh38 scoring file: `chr_name` and `chr_position` from the harmonised `hm_chr` and `hm_pos`, the effect allele, the other allele (from `other_allele`, else `hm_inferOtherAllele` when it names one allele), the weight, and the catalog's trait as its label. Rows the catalog could not place on GRCh38 are dropped, and so are rows on chrY or the mitochondrial genome. Rows on chrX are kept when you give the sex and dropped when you do not (the step prints how many of each): pgsc_calc converts your genotypes with plink2, which stops on a chrX record when it has no sex. Without the sex, PGS000662 (prostate cancer) loses 8 of its 269 rows this way; see [chrX and the sex](#chrx-and-the-sex). pgsc_calc then needs no network and no liftover. 3. **Genotypes.** - - **With step 3's gVCF** (the default since step 3 writes one): every score position, and with the panel every common panel SNV (the site list `setup.sh` writes), is genotyped from the gVCF. `bcftools convert --gvcf2vcf` turns each reference block over a position into a 0/0 call with the reference base; a position with no coverage (`./.`) or outside every block stays missing. A 0/0 record gets as its ALT the position's first allele (a score's effect or other allele, the panel's ALT) that is not the reference, so pgsc_calc can match it. These genotypes are kept as `prs/pgsc_calc/target.vcf.gz`. - - **Without a gVCF** (an older run): the variant-only VCF is cut to the same positions, so every site where you match the reference is missing (see below). When not one position is left, pgsc_calc is not started and every score is reported unmatched. + - **With step 3's gVCF** (the default since step 3 writes one): every score position, and with the panel every common panel SNV (the site list `setup.sh` writes), is genotyped from the gVCF. `bcftools convert --gvcf2vcf` turns each reference block over a position into a 0/0 call with the reference base; a position with no coverage (`./.`) or outside every block stays missing, and so does a record whose FILTER is not `PASS` or `.` (such as LowQual or RefCall). A 0/0 record gets as its ALT the position's first allele (a score's effect or other allele, the panel's ALT) that is not the reference, so pgsc_calc can match it. These genotypes are kept as `prs/pgsc_calc/target.vcf.gz`. + - **Without a gVCF** (an older run): the variant-only VCF is cut to the same positions and to its `PASS` or `.` records, so every site where you match the reference is missing (see below). When not one position is left, pgsc_calc is not started and every score is reported unmatched. Either way pgsc_calc gets only those positions, a small file to convert: the autosomes, and chrX when you gave the sex. 4. **pgsc_calc.** With the sex, the step writes it for plink2 (`pgsc_calc/sex/sex.tsv`: the VCF's sample name, 1 for male, 2 for female) and hands it to pgsc_calc's `PLINK2_VCF` conversion as `--update-sex`, beside the arguments pgsc_calc gives that process itself. Runs `pgsc_calc` (from `${GENOME_DIR}/tools/pgsc_calc-`, which `setup.sh` or the step itself unpacks from GitHub's archive of the release, checked against `PGSC_CALC_SHA256`) with the images of `versions.env`, offline, its containers without network. With the panel it adds `--run_ancestry`. pgsc_calc matches each score's variants to your genotypes (strand flips, ambiguous A/T and C/G pairs dropped, one best match per variant), scores them with plink2 and, with the panel, projects you onto the panel's principal components and compares your score with the reference group most similar to you. A score that matches under 75% of its variants is dropped by pgsc_calc and gets no sum. With `PRS_KEEP_AMBIGUOUS=true` it adds `--keep_ambiguous`, and the ambiguous pairs are kept where step 2 left them in (see [Strand-ambiguous rows](#strand-ambiguous-rows)). @@ -222,7 +222,7 @@ The check says the score was reported on the forward strand; it does not prove e ### Hom-ref sites come from the gVCF -The VCF from step 3 lists only sites where you differ from the reference. A scoring variant whose effect allele is the reference allele is missing from that VCF when you carry two copies of it, and adds nothing to your sum. Step 3 also writes a gVCF, which records where you match the reference, and step 25 reads those sites from it as 0/0. With the gVCF, a site is missing only when it was not covered. +The VCF from step 3 lists only sites where you differ from the reference. A scoring variant whose effect allele is the reference allele is missing from that VCF when you carry two copies of it, and adds nothing to your sum. Step 3 also writes a gVCF, which records where you match the reference, and step 25 reads those sites from it as 0/0. With the gVCF, a site is missing when it was not covered or its record is filtered (FILTER other than `PASS` or `.`, such as LowQual or RefCall). Without a gVCF (`Input` is `vcf`, a sample called by an older version), the sum misses the weight of every reference-allele effect allele you carry on two copies, and few scores reach pgsc_calc's 75% match rate. The step then prints: diff --git a/docs/26b-ancestry-fine.md b/docs/26b-ancestry-fine.md index a21ce1f75..fe2a5e74d 100644 --- a/docs/26b-ancestry-fine.md +++ b/docs/26b-ancestry-fine.md @@ -90,7 +90,7 @@ The Allen Ancient DNA Resource (AADR, release v66.p1, CC0 1.0 on the [Harvard Da ## What the Script Does Internally -1. Genotypes the panel's SNVs from the gVCF (`bcftools convert --gvcf2vcf`, as step 25 does) and keeps a record only with the panel's two alleles. +1. Genotypes the panel's SNVs from the gVCF (`bcftools convert --gvcf2vcf`, as step 25 does) and keeps a record only with the panel's two alleles. Both gVCF and VCF inputs keep only `FILTER=PASS` or `.`. LowQual, RefCall and other filtered records stay missing. Unfiltered reference blocks still count as reference. This can lower the genotype and reference counts. The VCF rule requiring one reference call in five uses those filtered counts. 2. Writes a one-person ADMIXTURE input on the SNVs ADMIXTURE uses that you have a genotype for (ADMIXTURE stops on a SNV with no genotype, so a missing one is left out), projects it on each level's frequencies, then runs the bootstrap replicates. 3. Chooses the chromosomes to paint: 100 genotyped SNVs or more spanning 5 cM or more. RFMix stops with a segmentation fault on a span of a few cM; on a whole genome every autosome spans more than 40 cM. 4. Phases those chromosomes with Beagle against the panel (`impute=false`, `seed=1`). diff --git a/docs/44-ancient.md b/docs/44-ancient.md index ec0c31253..946c490d7 100644 --- a/docs/44-ancient.md +++ b/docs/44-ancient.md @@ -22,7 +22,7 @@ Most people who look at their ancestry want to know where they fall among ancien ## Input -- `${SAMPLE}/vcf/${SAMPLE}.g.vcf.gz` with its `.tbi`: the gVCF of [step 3](03-variant-calling.md). A site where you match the reference reads as a real genotype and a site without reads stays missing; a variant-only VCF cannot tell those apart. The gVCF must hold one sample. +- `${SAMPLE}/vcf/${SAMPLE}.g.vcf.gz` with its `.tbi`: the gVCF of [step 3](03-variant-calling.md). A reference match counts only with `FILTER=PASS` or `.` and genotype quality `GQ >= 20`. LowQual, RefCall, no-calls, low GQ and missing GQ stay missing. Reference blocks must meet the same rule. A variant-only VCF cannot distinguish a reference match from a missing site. The gVCF must hold one sample and define FORMAT/GQ. - The panel `setup.sh --ancient-panel` builds in `reference/ancient/aadr_/` (`ANCIENT_PANEL_DIR` picks another folder inside `GENOME_DIR`, `ANCIENT_PANEL_DIR=none` skips the step). Without it the step prints one line saying so and exits 0. In the Nextflow pipeline (`ancient` in `--tools`, with `--ancient_panel `) the processes `ANCIENT_SITES`, `ANCIENT_GENO`, `ANCIENT_PCA` and `ANCIENT_REPORT` run the same commands for every sample with a gVCF; a sample without one is skipped with a warning. `run-all.sh` lists the step only when `TOOLS` names `ancient`, and passes the panel when its `DONE` file exists. @@ -40,7 +40,7 @@ In the Nextflow pipeline (`ancient` in `--tools`, with `--ancient_panel ## How Step 44 Works -1. bcftools reads your genotype at each panel site from the gVCF. +1. bcftools reads your genotype at each panel site from the gVCF. It keeps only `FILTER=PASS` or `.` with `GQ >= 20`, including expanded reference blocks. Other calls stay missing. The filter matches the archaic ancestry step's treatment of RefCall and LowQual. Fewer sites can count toward `sites_used`, which can change the PCA position and nearest groups. Output names and formats stay the same. 2. Your genotypes become one more column of the present-day panel's genotypes, counted as copies of the AADR's first allele; a site with no genotype, or with an allele the AADR does not list, is missing. 3. smartpca builds the axes from the same present-day people and projects you onto them (`lsqproject`). Each axis is then turned to the sign of the panel's run, through the present-day people both runs share, so your position and the ancient people's positions are on the same axes. The step stops if an axis disagrees (correlation below 0.999): the panel and the run do not match. 4. Each ancient group (the AADR's Group ID) gets the mean position of its people. The distance from you to that mean uses the first four PCs, each weighted by the square root of its eigenvalue, so an axis that explains more variation counts more. diff --git a/modules/local/ancestry_fine/main.nf b/modules/local/ancestry_fine/main.nf index 608b6866c..f293cb557 100644 --- a/modules/local/ancestry_fine/main.nf +++ b/modules/local/ancestry_fine/main.nf @@ -45,8 +45,8 @@ process ANCESTRY_FINE_GENOTYPE { script: def id = meta.id def extract = kind == 'gvcf' - ? "bcftools convert --gvcf2vcf -f ${reference} -R ${panel}/panel/positions.tsv -Ou ${vcf} | bcftools view -T ${panel}/panel/positions.tsv --trim-alt-alleles -i 'GT!=\"mis\"' -Ov" - : "bcftools view -T ${panel}/panel/positions.tsv -Ov ${vcf}" + ? "bcftools convert --gvcf2vcf -f ${reference} -R ${panel}/panel/positions.tsv -Ou ${vcf} | bcftools view -f PASS,. -T ${panel}/panel/positions.tsv --trim-alt-alleles -i 'GT!=\"mis\"' -Ov" + : "bcftools view -f PASS,. -T ${panel}/panel/positions.tsv -Ov ${vcf}" """ set -euo pipefail # As scripts/26b-ancestry-fine.sh: the awk reads the first sample column diff --git a/modules/local/ancient/main.nf b/modules/local/ancient/main.nf index a5b83de3b..9a93687b7 100644 --- a/modules/local/ancient/main.nf +++ b/modules/local/ancient/main.nf @@ -4,7 +4,8 @@ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ANCIENT_SITES genotypes the panel's GRCh38 sites from the gVCF: each reference block over a site becomes a record with the - reference base, and a site without reads has no line + reference base. Only PASS or unfiltered calls with + GQ >= 20 count; no-calls and missing GQ have no line ANCIENT_GENO the sample as one more column of the panel's present-day EIGENSTRAT genotypes (og ancient person-geno) ANCIENT_PCA smartpca: axes from the present-day West Eurasians, the @@ -47,7 +48,7 @@ process ANCIENT_SITES { fi awk -F'\\t' -v OFS='\\t' 'NR > 1 {print \$2, \$3}' ${panel}/sites_grch38.tsv > sites.tsv bcftools convert --gvcf2vcf -f ${reference} -R sites.tsv -Ou ${gvcf} \\ - | bcftools view -T sites.tsv --trim-alt-alleles -i 'GT!="mis"' -Ou \\ + | bcftools view -f PASS,. -T sites.tsv --trim-alt-alleles -i 'GT!="mis" && FMT/GQ>=20' -Ou \\ | bcftools query -f '%CHROM\\t%POS\\t%REF\\t%ALT\\t[%TGT]\\n' > ${meta.id}_ancient_genotypes.tsv rm -f sites.tsv diff --git a/modules/local/prs/main.nf b/modules/local/prs/main.nf index 04dd4e728..c0ad2f43f 100644 --- a/modules/local/prs/main.nf +++ b/modules/local/prs/main.nf @@ -105,10 +105,10 @@ process PRS_SCORE_SITES { fi if [ "${input_kind}" = vcf ]; then - bcftools view -T sites.tsv -Ov ${gvcf} | gzip -c > ${meta.id}_score_sites.vcf.gz + bcftools view -f PASS,. -T sites.tsv -Ov ${gvcf} | gzip -c > ${meta.id}_score_sites.vcf.gz else bcftools convert --gvcf2vcf -f ${reference} -R sites.tsv -Ou ${gvcf} \\ - | bcftools view -T sites.tsv --trim-alt-alleles -i 'GT!="mis"' -Ov \\ + | bcftools view -f PASS,. -T sites.tsv --trim-alt-alleles -i 'GT!="mis"' -Ov \\ | awk -F'\\t' -v OFS='\\t' -v alleles=alleles.tsv ' BEGIN { while ((getline l < alleles) > 0) { split(l, f, "\\t"); k = f[1] ":" f[2]; ea[k] = ea[k] " " f[3] } } /^#/ { print; next } diff --git a/scripts/25-prs.sh b/scripts/25-prs.sh index da0106f3e..75d64df76 100755 --- a/scripts/25-prs.sh +++ b/scripts/25-prs.sh @@ -227,7 +227,7 @@ if [ -f "$GVCF" ] && [ -f "${GVCF}.tbi" ]; then "${BCFTOOLS_IMAGE}" \ bash -euo pipefail -c ' bcftools convert --gvcf2vcf -f "$1" -R "$2" -Ou "$3" \ - | bcftools view -T "$2" --trim-alt-alleles -i "GT!=\"mis\"" -Ov' \ + | bcftools view -f PASS,. -T "$2" --trim-alt-alleles -i "GT!=\"mis\"" -Ov' \ _ "${REF_FASTA_C}" "$(cpath "$SITES")" "/genome/${SAMPLE}/vcf/${SAMPLE}.g.vcf.gz" \ | awk -F'\t' -v OFS='\t' -v alleles="$ALLELES" ' BEGIN { while ((getline l < alleles) > 0) { split(l, f, "\t"); k = f[1] ":" f[2]; ea[k] = ea[k] " " f[3] } } @@ -242,7 +242,7 @@ else echo "[3/5] No gVCF beside the VCF: scoring the variant sites only (sites where you match the reference are missing)..." # shellcheck disable=SC2016 # $1 and $2 belong to the inner bash run_in --cpus 1 --memory 4g "${BCFTOOLS_IMAGE}" \ - bash -euo pipefail -c 'bcftools view -T "$1" -Ov "$2"' \ + bash -euo pipefail -c 'bcftools view -f PASS,. -T "$1" -Ov "$2"' \ _ "$(cpath "$SITES")" "/genome/${SAMPLE}/vcf/${SAMPLE}.vcf.gz" | gzip -c > "${TARGET}.tmp" fi mv -f "${TARGET}.tmp" "$TARGET" diff --git a/scripts/26b-ancestry-fine.sh b/scripts/26b-ancestry-fine.sh index 1727d1552..8e891222e 100755 --- a/scripts/26b-ancestry-fine.sh +++ b/scripts/26b-ancestry-fine.sh @@ -82,13 +82,13 @@ if [ -f "$GVCF" ] && [ -f "${GVCF}.tbi" ]; then INPUT_KIND=gvcf echo "[1/5] Genotyping the panel's SNVs from the gVCF (${GVCF})..." # shellcheck disable=SC2016 # $1 to $3 belong to the inner bash - CMD=(bash -euo pipefail -c 'n=$(bcftools query -l "$3" | wc -l | tr -d " "); [ "$n" -eq 1 ] || { echo "ERROR: $3 holds $n samples; step 26b reads a one-sample file (keep one with bcftools view -s )." >&2; exit 1; }; bcftools convert --gvcf2vcf -f "$1" -R "$2" -Ou "$3" | bcftools view -T "$2" --trim-alt-alleles -i "GT!=\"mis\"" -Ov' + CMD=(bash -euo pipefail -c 'n=$(bcftools query -l "$3" | wc -l | tr -d " "); [ "$n" -eq 1 ] || { echo "ERROR: $3 holds $n samples; step 26b reads a one-sample file (keep one with bcftools view -s )." >&2; exit 1; }; bcftools convert --gvcf2vcf -f "$1" -R "$2" -Ou "$3" | bcftools view -f PASS,. -T "$2" --trim-alt-alleles -i "GT!=\"mis\"" -Ov' _ "$REF_FASTA_C" "${P}/panel/positions.tsv" "/genome/${SAMPLE}/vcf/${SAMPLE}.g.vcf.gz") else INPUT_KIND=vcf echo "[1/5] No gVCF beside the VCF: genotyping the panel's SNVs from the VCF (${VCF})..." # shellcheck disable=SC2016 # $1 and $2 belong to the inner bash - CMD=(bash -euo pipefail -c 'n=$(bcftools query -l "$2" | wc -l | tr -d " "); [ "$n" -eq 1 ] || { echo "ERROR: $2 holds $n samples; step 26b reads a one-sample file (keep one with bcftools view -s )." >&2; exit 1; }; bcftools view -T "$1" -Ov "$2"' _ "${P}/panel/positions.tsv" "/genome/${SAMPLE}/vcf/${SAMPLE}.vcf.gz") + CMD=(bash -euo pipefail -c 'n=$(bcftools query -l "$2" | wc -l | tr -d " "); [ "$n" -eq 1 ] || { echo "ERROR: $2 holds $n samples; step 26b reads a one-sample file (keep one with bcftools view -s )." >&2; exit 1; }; bcftools view -f PASS,. -T "$1" -Ov "$2"' _ "${P}/panel/positions.tsv" "/genome/${SAMPLE}/vcf/${SAMPLE}.vcf.gz") fi run_in --cpus 2 --memory 4g "$BCFTOOLS_IMAGE" "${CMD[@]}" \ | awk -F'\t' -v OFS='\t' -v sites="${PANEL}/panel/sites.tsv" -v counts="${WORK}/counts.tsv" ' diff --git a/scripts/44-ancient.sh b/scripts/44-ancient.sh index 3206fa8f4..00aba32af 100755 --- a/scripts/44-ancient.sh +++ b/scripts/44-ancient.sh @@ -13,8 +13,9 @@ # skip). Without it the step says so in one line and exits 0. # # Reads the gVCF of step 3 (vcf/.g.vcf.gz), so a site where the -# sample matches the reference is a real genotype and a site without reads is -# missing, never taken for the reference. The same commands run in the +# sample matches the reference can be a real genotype. Only PASS or unfiltered +# records with GQ >= 20 count; no-calls and missing GQ stay missing. +# The same commands run in the # Nextflow ANCIENT_SITES, ANCIENT_GENO, ANCIENT_PCA and ANCIENT_REPORT processes. # # Output in ancient/: @@ -62,7 +63,8 @@ rm -f "${OUTDIR}/${SAMPLE}"_ancient* echo "[1/4] Genotyping the panel's sites from the gVCF..." # gvcf2vcf expands each reference block over a site into a record with the # reference base, -T keeps the panel's sites, --trim-alt-alleles drops <*>, -# and a no-call is dropped, so a site without reads has no line. +# and only PASS or unfiltered calls with GQ >= 20 survive. No-calls, +# filtered records and missing GQ have no line. # shellcheck disable=SC2016 # $1 to $4 belong to the inner bash run_in --cpus 2 --memory 4g "$BCFTOOLS_IMAGE" \ bash -euo pipefail -c ' @@ -70,7 +72,7 @@ run_in --cpus 2 --memory 4g "$BCFTOOLS_IMAGE" \ [ "$n" -eq 1 ] || { echo "ERROR: $3 holds $n samples; step 44 reads a one-sample gVCF (keep one with bcftools view -s )." >&2; exit 1; } awk -F"\t" -v OFS="\t" "NR > 1 {print \$2, \$3}" "$2/sites_grch38.tsv" > "$4/sites.tsv" bcftools convert --gvcf2vcf -f "$1" -R "$4/sites.tsv" -Ou "$3" \ - | bcftools view -T "$4/sites.tsv" --trim-alt-alleles -i "GT!=\"mis\"" -Ou \ + | bcftools view -f PASS,. -T "$4/sites.tsv" --trim-alt-alleles -i "GT!=\"mis\" && FMT/GQ>=20" -Ou \ | bcftools query -f "%CHROM\t%POS\t%REF\t%ALT\t[%TGT]\n" > "$4/genotypes.tsv" rm -f "$4/sites.tsv"' \ _ "$REF_FASTA_C" "$P" "$(cpath "$GVCF")" "$W" \ diff --git a/scripts/ci/e2e-case-paths.tsv b/scripts/ci/e2e-case-paths.tsv index 721cc60eb..cca8e7ad9 100644 --- a/scripts/ci/e2e-case-paths.tsv +++ b/scripts/ci/e2e-case-paths.tsv @@ -181,3 +181,4 @@ y-tree-current-1-placement assets/live/index.html assets/viewer/index.html bin/o rust-z2e-readers bin/og tests/e2e/lib.sh scripts/ci/image-smoke.sh scripts/ci/nf-trace-requests.tsv rust-c2b-traits assets/traits.tsv assets/genotype_rules.tsv bin/og modules/local/traits/* scripts/42-traits.sh qc-3-cram-checksum modules/local/cram_archive/* scripts/34-cram-archive.sh scripts/lib/common.sh +gvcf-pass-gate scripts/26b-ancestry-fine.sh modules/local/ancestry_fine/main.nf scripts/44-ancient.sh modules/local/ancient/main.nf scripts/25-prs.sh modules/local/prs/main.nf diff --git a/scripts/ci/e2e-case-times.tsv b/scripts/ci/e2e-case-times.tsv index 0206946d3..c48ee868e 100644 --- a/scripts/ci/e2e-case-times.tsv +++ b/scripts/ci/e2e-case-times.tsv @@ -180,3 +180,4 @@ y-tree-current-1-placement shard 21 nextflow-hardening-2-nf rust-z2e-readers shard 1 - rust-c2b-traits shard 2 - qc-3-cram-checksum shard 34 - +gvcf-pass-gate shard 1 - diff --git a/tests/e2e/gvcf-pass-gate.sh b/tests/e2e/gvcf-pass-gate.sh new file mode 100755 index 000000000..cadec7390 --- /dev/null +++ b/tests/e2e/gvcf-pass-gate.sh @@ -0,0 +1,101 @@ +#!/usr/bin/env bash +# Synthetic controls for the actual bcftools view commands in both callers. +# No HG002 outputs or downloaded panels are needed. Stream the fixture into +# Docker so this case also works where Docker cannot mount the checkout. +set -euo pipefail +. "$(dirname "$0")/lib.sh" + +mkdir -p "${CASE_TMP}/input" +INPUT="${CASE_TMP}/input" +COMMANDS="${INPUT}/commands.tsv" +: > "$COMMANDS" +for step in 26b-ancestry-fine 44-ancient 25-prs; do + case "$step" in + 26b-*) module=ancestry_fine ;; + 44-*) module=ancient ;; + 25-*) module='prs' ;; + esac + for source in "scripts/${step}.sh" "modules/local/${module}/main.nf"; do + # Keep each view's options verbatim. Only replace its panel path, remove + # its filename (read stdin instead), and decode Groovy's escaped quotes. + awk -v source="$source" ' + /bcftools view .*-[T] / { + line = $0 + sub(/^.*bcftools view /, "bcftools view ", line) + if (line !~ /-O[uv]/) next + mode = line ~ /-Ou/ ? "-Ou" : "-Ov" + sub(/-T [^ ]+/, "-T sites.tsv", line) + sub(/-O[uv].*$/, mode, line) + if (source ~ /^modules\//) gsub(/\\"/, "\"", line) + kind = line ~ /--trim-alt-alleles/ ? "gvcf" : "vcf" + print source "\t" kind "\t" line + } + ' "${REPO}/${source}" >> "$COMMANDS" + done +done +check_eq "all ten extraction branches are exercised" "$(wc -l < "$COMMANDS" | tr -d ' ')" 10 + +cat > "${INPUT}/ref.fa" <<'EOF' +>chr1 +AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA +EOF +printf 'chr1\t100\t6\t100\t101\n' > "${INPUT}/ref.fa.fai" +for pos in 10 20 30 40 50 60 70 80 90; do + printf 'chr1\t%s\n' "$pos" +done > "${INPUT}/sites.tsv" +cat > "${INPUT}/sample.g.vcf" <<'EOF' +##fileformat=VCFv4.2 +##contig= +##FILTER= +##FILTER= +##INFO= +##FORMAT= +##FORMAT= +##FORMAT= +#CHROM POS ID REF ALT QUAL FILTER INFO FORMAT SYNTHETIC +chr1 10 . A G,<*> 40 PASS . GT:GQ:DP 0/1:40:30 +chr1 20 . A G,<*> 1 LowQual . GT:GQ:DP 1/1:1:2 +chr1 30 . A <*> 0 RefCall . GT:GQ:DP 0/0:40:30 +chr1 35 . A <*> . . END=45 GT:GQ:DP 0/0:40:30 +chr1 50 . A G,<*> . . . GT:GQ:DP 0/1:20:10 +chr1 55 . A <*> . . END=65 GT:GQ:DP ./.:0:0 +chr1 70 . A G,<*> 40 PASS . GT:GQ:DP 0/1:19:1 +chr1 75 . A <*> . . END=85 GT:GQ:DP 0/0:0:0 +chr1 90 . A G,<*> 40 PASS . GT:DP 0/1:30 +EOF +cat > "${INPUT}/run.sh" <<'EOF' +#!/usr/bin/env bash +set -euo pipefail +bcftools view -Oz -o sample.g.vcf.gz sample.g.vcf +bcftools index -t sample.g.vcf.gz +bcftools convert --gvcf2vcf -f ref.fa -R sites.tsv -Ob -o expanded.bcf sample.g.vcf.gz +bcftools view -e 'INFO/END>0' -Ob -o variants.bcf sample.g.vcf.gz +while IFS=$'\t' read -r source kind command; do + input=expanded.bcf + [ "$kind" = gvcf ] || input=variants.bcf + printf 'RESULT\t%s\t%s\t' "$source" "$kind" + bash -euo pipefail -c "$command" < "$input" \ + | bcftools query -f '%POS,' | tr -d '\n' + printf '\n' +done < commands.tsv +EOF + +LOG="${CASE_TMP}/extract.log" +if ! tar -C "$INPUT" -cf - . \ + | docker run --rm -i --network none --entrypoint bash "$BCFTOOLS_IMAGE" \ + -euo pipefail -c 'mkdir /w; cd /w; tar -xf -; bash run.sh' > "$LOG" 2>&1; then + cat "$LOG" + fail "the real bcftools extraction completed" +else + cat "$LOG" + while IFS=$'\t' read -r source kind _command; do + case "$source:$kind" in + *ancient*) expected=10,40,50, ;; + *:gvcf) expected=10,40,50,70,80,90, ;; + *:vcf) expected=10,50,70,90, ;; + esac + got=$(awk -F'\t' -v src="$source" -v k="$kind" '$1 == "RESULT" && $2 == src && $3 == k {print $4}' "$LOG") + check_eq "$source ($kind) keeps exactly the eligible panel sites" "$got" "$expected" + done < "$COMMANDS" +fi +finish