Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,10 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n

## Unreleased

- Fine ancestry (step 26b), ancient ancestry (step 44) and PRS (step 25) now keep only PASS or unfiltered records, in both bash and Nextflow. LowQual, RefCall and other filtered records stay missing instead of becoming genotypes. Fine ancestry's genotype and reference counts can fall, so its VCF reference-call requirement can reject an input it previously accepted. Ancient ancestry also requires GQ >= 20, including reference blocks; low or missing GQ stays missing. Its sites_used, PCA position and nearest groups can change. Output names and formats stay the same. Part of #411 (lin-50, lin-51).
- ROH summaries now divide F_ROH by the GRCh38 length outside the bands of the assessed autosomes. Per-site rows identify those chromosomes, including chromosomes with no ROH. A new `roh.data.scope` records the chromosomes, denominator and scope source. Relationship labels and UPD flags require sites on all 22 autosomes and the full unmasked autosome length. Segment-only files use their observed chromosomes as an unverified lower bound and get neither label. Empty input has null F_ROH. Segment counts and the step 11 summary table keep their format.
- A one-chromosome ROH flag now names regions and citations without syndrome names. Its note includes deletion, acquired loss of heterozygosity and shared ancestry as alternatives. The maternal and paternal JSON keys remain as `"-"` for reader compatibility. The attributed `assets/imprinted_regions.tsv` is embedded at build time, replacing the duplicated Rust table. The container builder mounts it, and the source stamp includes its blob hash, so a table edit makes an old binary stale. The stamp format and `tree-hash` command name stay the same. Every listed chromosome uses the same lookup, including chr11 and chr15. A region hint does not establish overlap with its imprinted locus or parental origin. Broad cytoband names remain reference labels, not diagnostic intervals.
- The ROH collector now streams per-site rows instead of keeping the whole file in memory. On 1.1 million synthetic site rows, peak RSS fell from 65.7-67.0 MB to 3.0-3.5 MB. CPU time stayed near 0.2 seconds. Under shared host load, wall time ranged from 1.40-21.46 seconds before and 7.89-38.16 seconds after; these timings do not establish a speedup.

- The reports now get summary.json from the Rust `og` binary. Step 24, the Nextflow HTML_REPORT process and `generate-report.sh` collect it with `og summary collect` (through `bin/render_report.py`), and a run collects each sample once: `generate-report.sh` renders the text report from the summary.json step 24 wrote, and collects again when a file it reads is newer than that summary.json, or a file was added to or removed from a step folder since, or a file a section names as its source, the run status or a table in `bin/` or `assets/` is gone. When it renders from step 24's summary.json, the text report's generated time is step 24's, the same as the HTML report's; it was the minute of its own collect. When it collects again, the text report carries the new collect's values and time, and the HTML report keeps step 24's until step 24 runs again, as before. The deliberate fixes the `og summary collect` entries below list now reach the reports; apart from them, nothing in summary.json or either report changes. `bin/collect_summary.py` keeps only what the report renderer (`bin/render_report.py`), the couple report, the results index and the live view import from it. The Python modules that only the old collector imported are gone: `bin/pgx_outside_calls.py`, `ancestry_fine_report.py`, `clip_check.py`, `annotsv_postfilter.py`, `chip_report.py`, `traits_report.py`, `run_delta.py` and `genotype_rules_indels.py`. On the synthetic demo sample (4.7 million VCF records) the collect takes 1.1 s of CPU and 12-15 MB, where the Python took 7.8-8.0 s and 34-37 MB. What changes on purpose:
- The walk of the sample folder leaves out `<sample>/nextflow`. `run-all.sh` launches Nextflow there, and its `work/` holds every task's files and staged links. No section reads a file there. With 200,000 files under it, the collect took 67 MB walked and 14 MB skipped.
- The sections other than the sample VCF are read one at a time, while the sample VCF is read on its own thread, so the peak memory is near the biggest section's and not the sum of all of them. With every side table 50 times its demo size, the peak footprint fell from 119-130 MB to 105-112 MB (the Python: 100-102 MB), with the same CPU time.
Expand Down
6 changes: 3 additions & 3 deletions docs/25-prs.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,8 +112,8 @@ Only additive scores are accepted (no `dosage_*_weight` columns, no `is_dominant
1. **Scoring files.** Downloads the GRCh38-harmonised file of each score from the PGS Catalog FTP when it is not in `${GENOME_DIR}/prs_scores/` yet, checked against the md5 the catalog publishes beside it. A file whose `#HmPOS_build` header is not `GRCh38` is refused. There is no fallback to the author-reported file, which is often GRCh37 or rsID-only and would score the wrong positions without any visible sign.
2. **Scores as pgsc_calc reads them.** `og prs format` (the Rust binary, `rust/crates/og-steps`) writes each file as a custom GRCh38 scoring file: `chr_name` and `chr_position` from the harmonised `hm_chr` and `hm_pos`, the effect allele, the other allele (from `other_allele`, else `hm_inferOtherAllele` when it names one allele), the weight, and the catalog's trait as its label. Rows the catalog could not place on GRCh38 are dropped, and so are rows on chrY or the mitochondrial genome. Rows on chrX are kept when you give the sex and dropped when you do not (the step prints how many of each): pgsc_calc converts your genotypes with plink2, which stops on a chrX record when it has no sex. Without the sex, PGS000662 (prostate cancer) loses 8 of its 269 rows this way; see [chrX and the sex](#chrx-and-the-sex). pgsc_calc then needs no network and no liftover.
3. **Genotypes.**
- **With step 3's gVCF** (the default since step 3 writes one): every score position, and with the panel every common panel SNV (the site list `setup.sh` writes), is genotyped from the gVCF. `bcftools convert --gvcf2vcf` turns each reference block over a position into a 0/0 call with the reference base; a position with no coverage (`./.`) or outside every block stays missing. A 0/0 record gets as its ALT the position's first allele (a score's effect or other allele, the panel's ALT) that is not the reference, so pgsc_calc can match it. These genotypes are kept as `prs/pgsc_calc/target.vcf.gz`.
- **Without a gVCF** (an older run): the variant-only VCF is cut to the same positions, so every site where you match the reference is missing (see below). When not one position is left, pgsc_calc is not started and every score is reported unmatched.
- **With step 3's gVCF** (the default since step 3 writes one): every score position, and with the panel every common panel SNV (the site list `setup.sh` writes), is genotyped from the gVCF. `bcftools convert --gvcf2vcf` turns each reference block over a position into a 0/0 call with the reference base; a position with no coverage (`./.`) or outside every block stays missing, and so does a record whose FILTER is not `PASS` or `.` (such as LowQual or RefCall). A 0/0 record gets as its ALT the position's first allele (a score's effect or other allele, the panel's ALT) that is not the reference, so pgsc_calc can match it. These genotypes are kept as `prs/pgsc_calc/target.vcf.gz`.
- **Without a gVCF** (an older run): the variant-only VCF is cut to the same positions and to its `PASS` or `.` records, so every site where you match the reference is missing (see below). When not one position is left, pgsc_calc is not started and every score is reported unmatched.

Either way pgsc_calc gets only those positions, a small file to convert: the autosomes, and chrX when you gave the sex.
4. **pgsc_calc.** With the sex, the step writes it for plink2 (`pgsc_calc/sex/sex.tsv`: the VCF's sample name, 1 for male, 2 for female) and hands it to pgsc_calc's `PLINK2_VCF` conversion as `--update-sex`, beside the arguments pgsc_calc gives that process itself. Runs `pgsc_calc` (from `${GENOME_DIR}/tools/pgsc_calc-<release>`, which `setup.sh` or the step itself unpacks from GitHub's archive of the release, checked against `PGSC_CALC_SHA256`) with the images of `versions.env`, offline, its containers without network. With the panel it adds `--run_ancestry`. pgsc_calc matches each score's variants to your genotypes (strand flips, ambiguous A/T and C/G pairs dropped, one best match per variant), scores them with plink2 and, with the panel, projects you onto the panel's principal components and compares your score with the reference group most similar to you. A score that matches under 75% of its variants is dropped by pgsc_calc and gets no sum. With `PRS_KEEP_AMBIGUOUS=true` it adds `--keep_ambiguous`, and the ambiguous pairs are kept where step 2 left them in (see [Strand-ambiguous rows](#strand-ambiguous-rows)).
Expand Down Expand Up @@ -222,7 +222,7 @@ The check says the score was reported on the forward strand; it does not prove e

### Hom-ref sites come from the gVCF

The VCF from step 3 lists only sites where you differ from the reference. A scoring variant whose effect allele is the reference allele is missing from that VCF when you carry two copies of it, and adds nothing to your sum. Step 3 also writes a gVCF, which records where you match the reference, and step 25 reads those sites from it as 0/0. With the gVCF, a site is missing only when it was not covered.
The VCF from step 3 lists only sites where you differ from the reference. A scoring variant whose effect allele is the reference allele is missing from that VCF when you carry two copies of it, and adds nothing to your sum. Step 3 also writes a gVCF, which records where you match the reference, and step 25 reads those sites from it as 0/0. With the gVCF, a site is missing when it was not covered or its record is filtered (FILTER other than `PASS` or `.`, such as LowQual or RefCall).

Without a gVCF (`Input` is `vcf`, a sample called by an older version), the sum misses the weight of every reference-allele effect allele you carry on two copies, and few scores reach pgsc_calc's 75% match rate. The step then prints:

Expand Down
2 changes: 1 addition & 1 deletion docs/26b-ancestry-fine.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ The Allen Ancient DNA Resource (AADR, release v66.p1, CC0 1.0 on the [Harvard Da

## What the Script Does Internally

1. Genotypes the panel's SNVs from the gVCF (`bcftools convert --gvcf2vcf`, as step 25 does) and keeps a record only with the panel's two alleles.
1. Genotypes the panel's SNVs from the gVCF (`bcftools convert --gvcf2vcf`, as step 25 does) and keeps a record only with the panel's two alleles. Both gVCF and VCF inputs keep only `FILTER=PASS` or `.`. LowQual, RefCall and other filtered records stay missing. Unfiltered reference blocks still count as reference. This can lower the genotype and reference counts. The VCF rule requiring one reference call in five uses those filtered counts.
2. Writes a one-person ADMIXTURE input on the SNVs ADMIXTURE uses that you have a genotype for (ADMIXTURE stops on a SNV with no genotype, so a missing one is left out), projects it on each level's frequencies, then runs the bootstrap replicates.
3. Chooses the chromosomes to paint: 100 genotyped SNVs or more spanning 5 cM or more. RFMix stops with a segmentation fault on a span of a few cM; on a whole genome every autosome spans more than 40 cM.
4. Phases those chromosomes with Beagle against the panel (`impute=false`, `seed=1`).
Expand Down
4 changes: 2 additions & 2 deletions docs/44-ancient.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ Most people who look at their ancestry want to know where they fall among ancien

## Input

- `${SAMPLE}/vcf/${SAMPLE}.g.vcf.gz` with its `.tbi`: the gVCF of [step 3](03-variant-calling.md). A site where you match the reference reads as a real genotype and a site without reads stays missing; a variant-only VCF cannot tell those apart. The gVCF must hold one sample.
- `${SAMPLE}/vcf/${SAMPLE}.g.vcf.gz` with its `.tbi`: the gVCF of [step 3](03-variant-calling.md). A reference match counts only with `FILTER=PASS` or `.` and genotype quality `GQ >= 20`. LowQual, RefCall, no-calls, low GQ and missing GQ stay missing. Reference blocks must meet the same rule. A variant-only VCF cannot distinguish a reference match from a missing site. The gVCF must hold one sample and define FORMAT/GQ.
- The panel `setup.sh --ancient-panel` builds in `reference/ancient/aadr_<release>/` (`ANCIENT_PANEL_DIR` picks another folder inside `GENOME_DIR`, `ANCIENT_PANEL_DIR=none` skips the step). Without it the step prints one line saying so and exits 0.

In the Nextflow pipeline (`ancient` in `--tools`, with `--ancient_panel <folder>`) the processes `ANCIENT_SITES`, `ANCIENT_GENO`, `ANCIENT_PCA` and `ANCIENT_REPORT` run the same commands for every sample with a gVCF; a sample without one is skipped with a warning. `run-all.sh` lists the step only when `TOOLS` names `ancient`, and passes the panel when its `DONE` file exists.
Expand All @@ -40,7 +40,7 @@ In the Nextflow pipeline (`ancient` in `--tools`, with `--ancient_panel <folder>

## How Step 44 Works

1. bcftools reads your genotype at each panel site from the gVCF.
1. bcftools reads your genotype at each panel site from the gVCF. It keeps only `FILTER=PASS` or `.` with `GQ >= 20`, including expanded reference blocks. Other calls stay missing. The filter matches the archaic ancestry step's treatment of RefCall and LowQual. Fewer sites can count toward `sites_used`, which can change the PCA position and nearest groups. Output names and formats stay the same.
2. Your genotypes become one more column of the present-day panel's genotypes, counted as copies of the AADR's first allele; a site with no genotype, or with an allele the AADR does not list, is missing.
3. smartpca builds the axes from the same present-day people and projects you onto them (`lsqproject`). Each axis is then turned to the sign of the panel's run, through the present-day people both runs share, so your position and the ancient people's positions are on the same axes. The step stops if an axis disagrees (correlation below 0.999): the panel and the run do not match.
4. Each ancient group (the AADR's Group ID) gets the mean position of its people. The distance from you to that mean uses the first four PCs, each weighted by the square root of its eigenvalue, so an axis that explains more variation counts more.
Expand Down
4 changes: 2 additions & 2 deletions modules/local/ancestry_fine/main.nf
Original file line number Diff line number Diff line change
Expand Up @@ -45,8 +45,8 @@ process ANCESTRY_FINE_GENOTYPE {
script:
def id = meta.id
def extract = kind == 'gvcf'
? "bcftools convert --gvcf2vcf -f ${reference} -R ${panel}/panel/positions.tsv -Ou ${vcf} | bcftools view -T ${panel}/panel/positions.tsv --trim-alt-alleles -i 'GT!=\"mis\"' -Ov"
: "bcftools view -T ${panel}/panel/positions.tsv -Ov ${vcf}"
? "bcftools convert --gvcf2vcf -f ${reference} -R ${panel}/panel/positions.tsv -Ou ${vcf} | bcftools view -f PASS,. -T ${panel}/panel/positions.tsv --trim-alt-alleles -i 'GT!=\"mis\"' -Ov"
: "bcftools view -f PASS,. -T ${panel}/panel/positions.tsv -Ov ${vcf}"
"""
set -euo pipefail
# As scripts/26b-ancestry-fine.sh: the awk reads the first sample column
Expand Down
5 changes: 3 additions & 2 deletions modules/local/ancient/main.nf
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,8 @@
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ANCIENT_SITES genotypes the panel's GRCh38 sites from the gVCF: each
reference block over a site becomes a record with the
reference base, and a site without reads has no line
reference base. Only PASS or unfiltered calls with
GQ >= 20 count; no-calls and missing GQ have no line
ANCIENT_GENO the sample as one more column of the panel's present-day
EIGENSTRAT genotypes (og ancient person-geno)
ANCIENT_PCA smartpca: axes from the present-day West Eurasians, the
Expand Down Expand Up @@ -47,7 +48,7 @@ process ANCIENT_SITES {
fi
awk -F'\\t' -v OFS='\\t' 'NR > 1 {print \$2, \$3}' ${panel}/sites_grch38.tsv > sites.tsv
bcftools convert --gvcf2vcf -f ${reference} -R sites.tsv -Ou ${gvcf} \\
| bcftools view -T sites.tsv --trim-alt-alleles -i 'GT!="mis"' -Ou \\
| bcftools view -f PASS,. -T sites.tsv --trim-alt-alleles -i 'GT!="mis" && FMT/GQ>=20' -Ou \\
| bcftools query -f '%CHROM\\t%POS\\t%REF\\t%ALT\\t[%TGT]\\n' > ${meta.id}_ancient_genotypes.tsv
rm -f sites.tsv

Expand Down
4 changes: 2 additions & 2 deletions modules/local/prs/main.nf
Original file line number Diff line number Diff line change
Expand Up @@ -105,10 +105,10 @@ process PRS_SCORE_SITES {
fi

if [ "${input_kind}" = vcf ]; then
bcftools view -T sites.tsv -Ov ${gvcf} | gzip -c > ${meta.id}_score_sites.vcf.gz
bcftools view -f PASS,. -T sites.tsv -Ov ${gvcf} | gzip -c > ${meta.id}_score_sites.vcf.gz
else
bcftools convert --gvcf2vcf -f ${reference} -R sites.tsv -Ou ${gvcf} \\
| bcftools view -T sites.tsv --trim-alt-alleles -i 'GT!="mis"' -Ov \\
| bcftools view -f PASS,. -T sites.tsv --trim-alt-alleles -i 'GT!="mis"' -Ov \\
| awk -F'\\t' -v OFS='\\t' -v alleles=alleles.tsv '
BEGIN { while ((getline l < alleles) > 0) { split(l, f, "\\t"); k = f[1] ":" f[2]; ea[k] = ea[k] " " f[3] } }
/^#/ { print; next }
Expand Down
4 changes: 2 additions & 2 deletions scripts/25-prs.sh
Original file line number Diff line number Diff line change
Expand Up @@ -227,7 +227,7 @@ if [ -f "$GVCF" ] && [ -f "${GVCF}.tbi" ]; then
"${BCFTOOLS_IMAGE}" \
bash -euo pipefail -c '
bcftools convert --gvcf2vcf -f "$1" -R "$2" -Ou "$3" \
| bcftools view -T "$2" --trim-alt-alleles -i "GT!=\"mis\"" -Ov' \
| bcftools view -f PASS,. -T "$2" --trim-alt-alleles -i "GT!=\"mis\"" -Ov' \
_ "${REF_FASTA_C}" "$(cpath "$SITES")" "/genome/${SAMPLE}/vcf/${SAMPLE}.g.vcf.gz" \
| awk -F'\t' -v OFS='\t' -v alleles="$ALLELES" '
BEGIN { while ((getline l < alleles) > 0) { split(l, f, "\t"); k = f[1] ":" f[2]; ea[k] = ea[k] " " f[3] } }
Expand All @@ -242,7 +242,7 @@ else
echo "[3/5] No gVCF beside the VCF: scoring the variant sites only (sites where you match the reference are missing)..."
# shellcheck disable=SC2016 # $1 and $2 belong to the inner bash
run_in --cpus 1 --memory 4g "${BCFTOOLS_IMAGE}" \
bash -euo pipefail -c 'bcftools view -T "$1" -Ov "$2"' \
bash -euo pipefail -c 'bcftools view -f PASS,. -T "$1" -Ov "$2"' \
_ "$(cpath "$SITES")" "/genome/${SAMPLE}/vcf/${SAMPLE}.vcf.gz" | gzip -c > "${TARGET}.tmp"
fi
mv -f "${TARGET}.tmp" "$TARGET"
Expand Down
Loading
Loading