Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
115 commits
Select commit Hold shift + click to select a range
029dedd
refactor(ancestry): og archaic and og ancient in Rust, with a chain l…
GeiserX Nov 13, 2026
f1d1aae
refactor(ancestry): steps 43 and 44 and their setup run og archaic an…
GeiserX Nov 13, 2026
cae6e3e
refactor(ancestry): delete archaic_report.py and ancient_report.py; t…
GeiserX Nov 13, 2026
796f4cd
docs(ancestry): steps 43 and 44 run og archaic and og ancient
GeiserX Nov 13, 2026
7cb54dc
test(ancestry): a mask holds the 1-based bases of its BED intervals only
GeiserX Nov 13, 2026
3b492b9
test(ancestry): RED CONTROL, plant a 1-based mask and a chain block t…
GeiserX Nov 13, 2026
0a23652
Merge remote-tracking branch 'origin/main' into pgp/rust-p6
GeiserX Nov 13, 2026
bb71d56
test(ancestry): revert the red control of 3b492b9a
GeiserX Nov 13, 2026
4ab9d07
fix(ancestry): join genotype calls in one position window
GeiserX Nov 13, 2026
b101631
docs(ancestry): record sorted-input contract and bounded memory
GeiserX Nov 13, 2026
b8aca84
Merge remote-tracking branch 'origin/main' into pgp/rust-p6
GeiserX Nov 13, 2026
c47cd89
feat(summary): whole summary.json and its post pass in og summary col…
GeiserX Nov 13, 2026
5e22114
test(parity): compare the whole summary.json with the Python collector
GeiserX Nov 13, 2026
88dd793
docs(summary): og summary collect writes the whole summary.json
GeiserX Nov 14, 2026
888cd36
test(summary): plant four post-pass faults (red control, reverted next)
GeiserX Nov 14, 2026
0a34ead
Merge remote-tracking branch 'origin/main' into pgp/rust-p6
GeiserX Nov 14, 2026
d3ff258
Merge remote-tracking branch 'origin/main' into HEAD
GeiserX Nov 14, 2026
af005e2
test(summary): remove the planted post-pass faults
GeiserX Nov 14, 2026
7b0a8e7
test(parity): name no image in the whole-summary fixture manifests
GeiserX Nov 14, 2026
8dd88d6
docs(ancestry): give the archaic table's measured cost on the real table
GeiserX Nov 14, 2026
96df5d3
Merge remote-tracking branch 'origin/main' into pgp/rust-p6
GeiserX Nov 14, 2026
8993df9
Merge remote-tracking branch 'origin/main' into pgp/rust-r2d
GeiserX Nov 14, 2026
a3f08e0
fix(summary): claim only error containment in the post pass
GeiserX Nov 14, 2026
fe1542c
fix(ancestry): keep the previous archaic outputs when a run stops, an…
GeiserX Nov 14, 2026
fa7a657
fix(ancestry): remove a stale genotype file only after every new arch…
GeiserX Nov 14, 2026
530eebb
Merge main into pgp/rust-p6: #401 landed
GeiserX Nov 14, 2026
bce28b7
Merge commit '530eebbd' into pgp/rust-r2d: train car before (#402)
GeiserX Nov 14, 2026
c67f440
perf(report): write summary.json with og summary collect
GeiserX Nov 14, 2026
6fd6dc3
Merge remote-tracking branch 'fork/pgp/rust-r2d' into pgp/rust-r3
GeiserX Nov 14, 2026
88d42be
test(summary): check the summary through og summary collect
GeiserX Nov 14, 2026
2545b38
refactor(summary): delete the Python the summary switch left unused
GeiserX Nov 14, 2026
d2bb659
docs: name og summary collect where the docs named the Python collector
GeiserX Nov 14, 2026
17728ec
test(report): check that a run collects the summary once
GeiserX Nov 14, 2026
2d09b4f
perf(summary): read the clinical and other sections one at a time
GeiserX Nov 14, 2026
7c06808
refactor(report): delete run_delta.py, which only the old collector i…
GeiserX Nov 14, 2026
bf525e2
docs(changelog): og writes the summary the reports read, once per run
GeiserX Nov 14, 2026
84027ab
test(report): read the HTML report's generated time up to its separator
GeiserX Nov 14, 2026
8451233
Merge main into pgp/rust-r2d: the car before landed, no file changes
GeiserX Nov 14, 2026
dbba041
Merge the train (pgp/rust-r2d, now with main) into pgp/rust-r3
GeiserX Nov 14, 2026
0aab619
fix(report): collect again after a removed file, and keep the tempora…
GeiserX Nov 14, 2026
c79ccfc
Merge main (#409 landed) into pgp/rust-r3
GeiserX Nov 14, 2026
686a0c6
fix(report): collect again when a section's whole folder is gone
GeiserX Nov 14, 2026
6892ae8
Merge main (#410) into pgp/rust-r3
GeiserX Nov 14, 2026
61dc9c7
Merge main (#414, the Guard time limits) into pgp/rust-r3
GeiserX Nov 14, 2026
eaf6bfc
ci(guard): read the SMN1 marker from the test's own table
GeiserX Nov 14, 2026
7aaf3a9
fix(report): collect again when the run status or a bin or assets tab…
GeiserX Nov 14, 2026
b55093d
test(roh): require chromosome scope and neutral UPD hints
GeiserX Nov 15, 2026
0480d31
Merge main into pgp/i411-roh-scope to clear the PR conflict
GeiserX Nov 15, 2026
ad38e45
Merge remote-tracking branch 'origin/main' into pgp/port-scaffold
GeiserX Nov 15, 2026
2fbbc52
test(setup): require clinical data in a default setup
GeiserX Nov 15, 2026
b81d49a
test(pgx): pin the disease-gene drug warnings for carriers, SCN1A and…
GeiserX Nov 15, 2026
5942701
test(live): check inline reads and same-origin viewer framing
GeiserX Nov 15, 2026
cd57a63
test(freshness): require metadata checks for clinical downloads
GeiserX Nov 15, 2026
7317527
test(bloodtype): pin the record a marker reads when several overlap it
GeiserX Nov 15, 2026
5fc18ba
docs(licence): say GPL-3.0-or-later wherever OwnGenome states its lic…
GeiserX Nov 15, 2026
da97715
docs(acmg): cite ACMG SF v3.3 and say the rule text is a short quotation
GeiserX Nov 15, 2026
7875c95
docs(calling): name the source of the GRCh38 PAR intervals
GeiserX Nov 15, 2026
d58805e
docs(changelog): note the licence and ACMG citation fixes
GeiserX Nov 15, 2026
207283a
fix(clinvar): keep ClinVar's condition on each hit
GeiserX Nov 15, 2026
3c8f081
test(clinvar): fake-docker case for the condition on step 06 hits
GeiserX Nov 15, 2026
cd80172
fix(roh): stream chromosome scope and emit neutral region hints
GeiserX Nov 15, 2026
0de4fd2
fix(archive): verify complete read content before removing BAM
GeiserX Nov 15, 2026
ba002b0
feat(live): open the run reads inline from the genome map
GeiserX Nov 15, 2026
80137d4
test(recheck): red first for a failed lookup read as nothing newer, a…
GeiserX Nov 15, 2026
8c26078
Merge remote-tracking branch 'origin/main' into pgp/rust-r3
GeiserX Nov 15, 2026
ad96065
fix(freshness): verify clinical download links and metadata
GeiserX Nov 15, 2026
cd1d2d8
fix(e2e): collect with og summary collect where two cases still ran t…
GeiserX Nov 15, 2026
baced26
build(rust): give every planned port its own crate, module and verb
GeiserX Nov 15, 2026
abc792a
fix(live): show viewer setup failures inside the reads panel
GeiserX Nov 15, 2026
079b864
fix(pgx): stop warning against valproate for one recessive hit, and d…
GeiserX Nov 15, 2026
4e4945f
ci(guard): run every test with the og-leg marker instead of a hand-ke…
GeiserX Nov 15, 2026
2b43c3c
test(roh): pin scoped summaries and prove the controls fail
GeiserX Nov 15, 2026
c8bd596
docs(rust): name the four new crates, the planned verbs and the og-le…
GeiserX Nov 15, 2026
a4fb1c5
fix(ci): keep ROH caller paths within the supported e2e scope
GeiserX Nov 15, 2026
c78f72a
fix(recheck): say the check failed when every lookup failed, and reru…
GeiserX Nov 15, 2026
c406f62
fix(archive): reject empty Nextflow checksum reports
GeiserX Nov 15, 2026
d70b525
fix(setup): install clinical reference data by default
GeiserX Nov 15, 2026
7be974d
test(ancestry): use a verb no plan adds for the unknown-verb case
GeiserX Nov 15, 2026
4fb1bea
fix(bloodtype): read every record that covers a marker, in any order
GeiserX Nov 15, 2026
f446ec4
docs(bloodtype): say how overlapping records at a marker are read
GeiserX Nov 15, 2026
875196f
test(live): inspect alignment rendering inside the viewer shadow root
GeiserX Nov 15, 2026
67856fd
fix(build): mount and fingerprint the embedded imprinting table
GeiserX Nov 15, 2026
e6fe655
fix(docs): link the imprinting table through the public repository
GeiserX Nov 15, 2026
2bb5acc
fix(archive): regenerate MD/NM when restoring a CRAM without stored tags
GeiserX Nov 15, 2026
beb57c9
fix(recheck): count failed items, not failed lookups, in the last line
GeiserX Nov 15, 2026
def428a
fix(live): clear reads links and labels when the panel closes
GeiserX Nov 15, 2026
604fd42
test(pgx): a split 1/2 site in a recessive gene must warn, not read a…
GeiserX Nov 15, 2026
83d39d6
fix(pgx): count 1/. and ./1 as both copies changed in a recessive gene
GeiserX Nov 15, 2026
c7a4111
Merge the car before (cd1d2d82) into pgp/i411-hits-keep-condition for…
GeiserX Nov 15, 2026
bdea2e9
test(setup): require clinical installs to resume in low space and rep…
GeiserX Nov 15, 2026
91eb772
fix(setup): let an interrupted clinical download resume in tight spac…
GeiserX Nov 15, 2026
b4be807
fix(live): preserve reads history and locus across map navigation
GeiserX Nov 15, 2026
1c83b99
test(live): preserve the targeted reload control guards
GeiserX Nov 15, 2026
4a52aab
fix(ancestry): list the four planned verbs in og ancestry-fine --help
GeiserX Nov 15, 2026
ba3105d
build(live): keep og-live's results module visible to git
GeiserX Nov 15, 2026
7896854
Merge the car before (c7a4111d) into pgp/i411-contraindications for t…
GeiserX Nov 15, 2026
eade22a
test(setup): catch stale extraction space and unverified retained bun…
GeiserX Nov 15, 2026
690e8ae
fix(setup): reclaim staging space and verify retained clinical archives
GeiserX Nov 15, 2026
2ef7dfa
Merge the car before (78968549) into pgp/i411-cram-checksum for the l…
GeiserX Nov 15, 2026
b7e3f47
fix(live): keep the reader's igv place through a resize, and document…
GeiserX Nov 15, 2026
3b22331
test(bloodtype): pin split null allele dosage and contradictions
GeiserX Nov 15, 2026
8f07395
Merge the car before (2ef7dfa0) into pgp/i411-licence-statement for t…
GeiserX Nov 15, 2026
0a0cfc0
fix(roh): write 0.0, not -0.0, when a sample has no long run
GeiserX Nov 15, 2026
9b70dc2
Merge pgp/rust-r3 (#413, cd1d2d82) into pgp/i411-roh-scope
GeiserX Nov 15, 2026
26f7be1
fix(bloodtype): combine distinct null ALTs across split records
GeiserX Nov 15, 2026
6af4386
fix(recheck): replay only the switches run-all.sh records, and refuse…
GeiserX Nov 15, 2026
e6ae4fd
Merge the car before (8f07395c) into pgp/port-scaffold for the landin…
GeiserX Nov 15, 2026
82c723e
Merge the car before (e6ae4fd8) into pgp/map-1g for the landing train
GeiserX Nov 15, 2026
87107a8
Merge the car before (82c723ed) into pgp/i411-bloodtype-record-match …
GeiserX Nov 15, 2026
32d0b87
Merge the car before (87107a8d) into pgp/i411-recheck-honest for the …
GeiserX Nov 15, 2026
85e7455
fix(ci): fail when the PCGR image or bundle moves without its measure…
GeiserX Nov 15, 2026
f59aff6
Merge the car before (32d0b87b) into pgp/i411-roh-scope for the landi…
GeiserX Nov 15, 2026
c26a02a
docs(updating): name the PCGR size, digest and measured-for pins in t…
GeiserX Nov 15, 2026
c22faee
Merge the car before (f59aff63) into pgp/screen-setup-clinical-defaul…
GeiserX Nov 15, 2026
27c4038
Merge main into pgp/screen-setup-clinical-default: the car before lan…
GeiserX Nov 15, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n
- A record that starts upstream and spans a marker with an allele that changes the marker's base (a deletion over it, or a padded `CG CA`, as DeepVariant writes `CG C,CA` when a deletion and an SNV overlap) now makes the site a no call, with a note that names the record. Before, the site had no record at its own position and 8 reads made it a confident reference: a K/K person read k/k at high confidence.
- A VCF with one record per position and no overlapping records, as DeepVariant writes, gives the same output as before: the 34 parity cases still agree with the Python oracle byte for byte.
- `og recheck` no longer says "Nothing here is newer than what the run used." when a lookup failed. With no network it said that line, and a cron log then read as up to date. Now the last line says how many of the looked-up items failed and that the check is incomplete, or "Could not check: all N looked-up items failed" when none could be checked. The exit code was already 1 and stays 1. A failed lookup also no longer hides a newer file or pin that is already installed here: a newer ClinVar file on disk stays `rerun-ready`, with its steps, in `state.json`, the reports and the live page, and the item keeps its `error`. `og recheck --rerun` no longer replays the stored command through `bash -c`. It reads `meta command` of `logs/run_status.tsv` back word by word as the call `run-all.sh` records (`cd DIR && [SWITCH=VALUE ...] .../scripts/run-all.sh ARGS`) and starts that `run-all.sh` with its arguments one by one. A row of any other shape, such as one edited to add `; rm ...`, or a `cd` folder that is gone, is refused with exit 2 and nothing runs. A row that `run-all.sh` wrote replays as before. No option, output file or format changes. [docs/recheck.md](docs/recheck.md)
- Setup now installs VEP 116, the PCGR/CPSR reference bundle and VEP 115 by default so `run-all.sh` can plan the clinical steps. `--no-clinical-data` skips them; `--clinical-data` installs only these data. Each install checks its disk peak plus a 5 GB margin, credits the allocated space of a partial download, and warns without stopping the remaining installs. Stale extraction files are removed before the disk check. Retained PCGR archives are checked against the pinned digest before reuse. `--clinical-data` exits 1 when one item is still missing. The bundle is checked against a pinned SHA-256, resumes interrupted transfers, and is published only after extraction. Validation and setup docs now name the affected steps and the repair command. Freshness checks the three archive URLs with HEAD, both Ensembl checksum entries and the bundle's byte count. No analysis output format changes.

- The genome map shows where you are in the whole genome and lists what it draws. While a chromosome is open, a strip above the map draws every chromosome end to end, with the view as a brass brush and a tick for each finding on the map; a click on the strip opens a chromosome, and a drag brushes a new view or pans (on a phone the strip is a picture only). Under the map, an "In view" table lists every mark the map draws one by one in view: kind, what, where, size, who, copy and source, sorted by place with the findings first, a click on a header to sort by it. It shows 12 rows, then all of them (a scrolled list past 200 rows, at most 5,000; on a phone at most 200 rows, then zoom in), and a row flies the map to its feature. It follows the map once the map stops, announces only its count to a screen reader, and is folded on a phone; T folds and opens it. A closed result is never a row or a tick. One rule decides what the canvas, the strip and the table draw, and `og map build --predicate-table` writes the same rule's answers so a browser check holds the two to one set. A margin update no longer runs while the map is still moving: it waits for the view the map comes to rest on.

Expand Down
54 changes: 27 additions & 27 deletions docs/00-reference-setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,22 @@ One-time downloads required before running the pipeline. Each heading below give

> **Estimated time:** 1-3 hours depending on internet speed. The two VEP caches (26 GB for step 13, 24 GB for step 17) are the largest downloads of a default run.

A plain `./scripts/setup.sh "$GENOME_DIR"` installs both VEP caches and the PCGR/CPSR bundle after AnnotSV.
These clinical downloads add about 57 GB to transfer and 65 GB on disk. Allow 1 to 3 hours.
Setup checks each install separately: VEP 116 needs 56 GB peak, the bundle 15 GB, and VEP 115 needs 52 GB.
Each check adds a 5 GB margin and credits the disk space occupied by a partial download.
Stale extraction files are removed before the check. Retained PCGR archives are checked against the pinned SHA-256 before reuse. An item that does not fit is skipped with a warning. Setup continues.

```bash
./scripts/setup.sh --no-clinical-data "$GENOME_DIR" # skip all three clinical downloads
./scripts/setup.sh --clinical-data "$GENOME_DIR" # install or retry only these three
```

An interrupted transfer resumes. Existing caches and bundles are kept.
`--clinical-data` exits 1 when any of the three is still missing after it runs.
Without these data, `run-all.sh` skips the affected steps and records the reason.
A bare `nextflow run` still needs its tools and data parameters. Setup does not change that selection.

## GRCh38 Reference Genome

The foundation for everything. All tools need this. The pipeline uses NCBI's **GRCh38 no-ALT analysis set**: chr1-22, X, Y and M, the unplaced and unlocalized scaffolds and the EBV genome, 195 sequences, with UCSC names (`chr1`, `chrM`). `setup.sh` downloads it, checks it and stores it as `reference/GRCh38_no_alt_analysis_set.fasta`:
Expand Down Expand Up @@ -143,7 +159,7 @@ Required for step 5. `setup.sh` downloads them (about 5 GB, about 20 GB unpacked

## VEP Cache (~26 GB)

Ensembl Variant Effect Predictor annotation database. Required for step 13 and the pipeline's VEP. Without it, a run skips VEP, vcfanno, the clinical filter and slivar. Install it before the first run:
Ensembl Variant Effect Predictor annotation database. Required for step 13 and the pipeline's VEP. Setup installs it by default. Without it, a run skips VEP, vcfanno, the clinical filter and slivar. To install only this cache:

```bash
./scripts/setup.sh --vep-cache ${GENOME_DIR}
Expand Down Expand Up @@ -178,38 +194,22 @@ Required for step 17 (CPSR cancer predisposition screening). Includes ClinVar, g

> **Important:** PCGR 2.3.2 bundles VEP 115, which requires a **release-115** cache — different from the release-116 cache used by step 13 above. See the next section for the VEP 115 download. The bundle date (`PCGR_DATA_BUNDLE`) and the cache release (`PCGR_VEP_CACHE_RELEASE`) are in `versions.env` and move with `PCGR_IMAGE`.

```bash
mkdir -p ${GENOME_DIR}/pcgr_data
cd ${GENOME_DIR}/pcgr_data

# Download (~7 GB)
wget -c https://insilico.hpc.uio.no/pcgr/pcgr_ref_data.20260620.grch38.tgz
Setup installs it by default. To add the clinical data to an existing setup:

# Extract and organize into version-stamped directory
tar xzf pcgr_ref_data.20260620.grch38.tgz
mkdir -p 20260620 && mv data/ 20260620/
# Creates: ${GENOME_DIR}/pcgr_data/20260620/data/

# Optional: delete the tarball to save 7 GB
# rm pcgr_ref_data.20260620.grch38.tgz
```bash
./scripts/setup.sh --clinical-data "$GENOME_DIR"
```

The installer checks the archive against `PCGR_DATA_SHA256` in `versions.env`, then extracts it into a temporary folder.
The server publishes no checksum, so the size (`PCGR_DATA_BYTES`) and the digest are measured from a full download. `PCGR_DATA_MEASURED_FOR` names the image tag and bundle date they belong to, and `scripts/ci/check-coupled-versions.sh` fails when `PCGR_IMAGE` or `PCGR_DATA_BUNDLE` moves without it.
Only a complete tree is moved to `pcgr_data/20260620/data/`. The archive is deleted after installation.
The [PCGR resource list](https://sigven.github.io/pcgr/articles/annotation_resources.html) names the bundle's third-party databases. Their own terms apply. Setup prints the resource link before downloading.

## VEP 115 Cache for CPSR (~24 GB)

PCGR 2.3.2 (step 17) bundles VEP 115 internally, which needs the **release-115** cache. This is separate from the release-116 cache used by step 13. Both coexist in the same `vep_cache/` directory under different subdirectories (`116_GRCh38/` and `115_GRCh38/`). A `113_GRCh38/` cache left from PCGR 2.2.5 is no longer read and can be deleted.

```bash
mkdir -p ${GENOME_DIR}/vep_cache/tmp
cd ${GENOME_DIR}/vep_cache/tmp

# Download VEP 115 cache (~24 GB)
wget -c https://ftp.ensembl.org/pub/release-115/variation/indexed_vep_cache/homo_sapiens_vep_115_GRCh38.tar.gz

# Extract alongside the existing release-116 cache
cd ${GENOME_DIR}/vep_cache
tar xzf tmp/homo_sapiens_vep_115_GRCh38.tar.gz
# Creates: ${GENOME_DIR}/vep_cache/homo_sapiens/115_GRCh38/
```
Setup installs this cache by default, after the PCGR bundle. `--clinical-data` retries it using Ensembl's `CHECKSUMS` file and a temporary extraction folder.

> If you only run step 13 (VEP annotation) and skip step 17 (CPSR), you only need the release-116 cache. If you only run step 17, you only need release-115.

Expand Down Expand Up @@ -510,7 +510,7 @@ wget -c https://giab.s3.amazonaws.com/release/AshkenazimTrio/HG002_NA24385_son/N

[Hardware and storage requirements](hardware-requirements.md#shared-reference-data-one-time) has the one table of every download above, with the totals.

> **Tip:** If disk space is tight, you can skip the VEP cache (step 13) and PCGR bundle (step 17) initially. The core pipeline (steps 2-3-6-7) only needs the reference FASTA and ClinVar (~3.5 GB unpacked).
> **Tip:** If disk space is tight, use `setup.sh --no-clinical-data` to skip both VEP caches and the PCGR bundle initially. The core pipeline (steps 2-3-6-7) only needs the reference FASTA and ClinVar (~3.5 GB unpacked).

## Verifying Your Setup

Expand Down
4 changes: 2 additions & 2 deletions docs/13-vep-annotation.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ Pinned in `versions.env`; [Image versions](versions.md) lists the current tag.

## Prerequisites
- The VCF from step 3 **and its `.tbi`**: the step refuses a VCF without its index, which may be half written
- Offline VEP cache: step 13 downloads, checks and unpacks it the first time it runs (see step 00-reference-setup)
- Offline VEP cache: setup installs it by default. To retry, run `./scripts/setup.sh --clinical-data "$GENOME_DIR"` ([reference setup](00-reference-setup.md#vep-cache-26-gb)). The pipeline skips VEP when the cache is missing.
- Cache size: see [Hardware and storage requirements](hardware-requirements.md#shared-reference-data-one-time) (about 26 GB to download, 30 GB unpacked)

## Command
Expand Down Expand Up @@ -112,4 +112,4 @@ After annotation, use step 23 (clinical filter) which automatically detects avai
- `--dir_cache /opt/vep/.vep`: the cache is mounted there, not in the home directory VEP looks in by default
- `--fasta`: running `--offline` without a FASTA file disables HGVS notation (`INFO: Disabling --hgvs`), so the script always passes the reference ([`REF_FASTA`](00-reference-setup.md#the-reference-path-on-every-page) is the reference path)
- VEP does NOT assess variant pathogenicity in ClinVar context — combine with step 6 (ClinVar screen) for full picture
- **Upgrading from an older release:** VEP reads the cache directory named after the release passed in `--cache_version` (`homo_sapiens/116_GRCh38/` for the pinned release 116). When that directory is missing, VEP stops with an error instead of annotating from an older cache such as `112_GRCh38/`; step 13 downloads the matching cache on its first run.
- **Upgrading from an older release:** VEP reads the cache directory named after the release passed in `--cache_version` (`homo_sapiens/116_GRCh38/` for the pinned release 116). When that directory is missing, VEP stops with an error instead of annotating from an older cache such as `112_GRCh38/`. Setup installs the matching cache by default. The standalone step script can download a missing cache, but `run-all.sh` skips VEP until the cache is installed.
25 changes: 12 additions & 13 deletions docs/17-cpsr.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,22 +22,21 @@ The image is PCGR 2.3.2 from `ghcr.io/sigven/pcgr` (Docker Hub's `sigven/pcgr` s

### 1. VEP Cache
PCGR 2.3.2 bundles VEP 115, which requires the **release-115** cache (~24 GB). This is different from the release-116 cache used by step 13. Both coexist in the same `vep_cache/` directory under different subdirectories (`116_GRCh38/` and `115_GRCh38/`).
```bash
mkdir -p ${GENOME_DIR}/vep_cache
wget -c -P ${GENOME_DIR}/vep_cache https://ftp.ensembl.org/pub/release-115/variation/indexed_vep_cache/homo_sapiens_vep_115_GRCh38.tar.gz
tar xzf ${GENOME_DIR}/vep_cache/homo_sapiens_vep_115_GRCh38.tar.gz -C ${GENOME_DIR}/vep_cache
```

### 2. PCGR Ref Data Bundle
PCGR 2.x uses a ref data bundle (~7 GB for `20260620`) separate from VEP:
### 2. PCGR ref data bundle
PCGR 2.x uses a separate ref data bundle (`20260620`, about 7 GB).
A plain setup installs the VEP 115 cache and this bundle by default, alongside VEP 116 for step 13.
For an existing setup, install or retry all three with:

```bash
mkdir -p ${GENOME_DIR}/pcgr_data
cd ${GENOME_DIR}/pcgr_data
wget -c https://insilico.hpc.uio.no/pcgr/pcgr_ref_data.20260620.grch38.tgz
tar xzf pcgr_ref_data.20260620.grch38.tgz
mkdir -p 20260620 && mv data/ 20260620/
./scripts/setup.sh --clinical-data "$GENOME_DIR"
```
This creates a `20260620/data/` directory with ClinVar, CancerMine, UniProt, and other databases. VEP cache is now mounted separately.

The bundle goes to `pcgr_data/20260620/data/`; the caches coexist under `vep_cache/homo_sapiens/`.
Setup checks Ensembl's cache checksums and the bundle's pinned SHA-256 before extraction, then removes each archive.
It checks free space for each install and continues with a warning if one fails or does not fit.
`--no-clinical-data` skips all three. Without either CPSR item, `run-all.sh` skips step 17.
See [reference setup](00-reference-setup.md) for disk peaks and the [PCGR resource list](https://sigven.github.io/pcgr/articles/annotation_resources.html) for the bundled databases. Review each source's terms before use.

### Upgrading from PCGR 2.2.5
PCGR 2.2.5 read bundle `20250314` and the release-113 cache. 2.3.2 needs both new ones: about 31 GB to download (the 7 GB bundle and the 24 GB cache). Once step 17 runs on 2.3.2, `pcgr_data/20250314/` and `vep_cache/homo_sapiens/113_GRCh38/` are no longer read and can be deleted.
Expand Down
12 changes: 9 additions & 3 deletions docs/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
| **CPU** | 4 cores | 8-16 cores | `run-all.sh` runs DeepVariant with at most 8 shards, fewer when `--max_cpus` is lower; more cores run other steps beside it ([CPU](hardware-requirements.md#cpu-requirements)) |
| **RAM** | 16 GB | 32 GB | Every container has a hard memory limit; the 8-CPU steps ask for 32 GB, capped at the machine's RAM |
| **Disk** | 500 GB free from a BAM, about 700 GB from FASTQ | 1 TB+ | See [detailed breakdown](hardware-requirements.md#disk-space-breakdown) |
| **Internet** | Broadband | 100+ Mbps | ~70-75 GB core downloads + ~175 GB optional annotation databases |
| **Internet** | Broadband | 100+ Mbps | ~73-78 GB core downloads + ~175 GB optional annotation databases |
| **OS** | Linux (amd64) | Ubuntu 22.04+ | macOS/ARM works but slower (see below) |

> **Disk space is the #1 surprise.** A single 30X WGS sample produces 60-90 GB of FASTQ, 80-120 GB of BAM, plus reference genomes and databases. See [docs/hardware-requirements.md](hardware-requirements.md) for the full breakdown.
Expand Down Expand Up @@ -46,11 +46,17 @@ Every analysis tool runs inside Docker: no conda environments, no Python version
| PCGR/CPSR data bundle + VEP 115 cache | ~31 GB | Step 17 (cancer predisposition) |
| Docker images (all steps) | ~10-15 GB | All steps |
| Annotation databases (CADD, SpliceAI, REVEL, AlphaMissense) | ~175 GB | Steps 30-31 (optional) |
| **Total one-time setup (core)** | **~70-75 GB** | |
| **Total one-time setup (core)** | **~73-78 GB** | |
| **Total with annotation enrichment** | **~250 GB** | |

See [docs/00-reference-setup.md](00-reference-setup.md) for download instructions.

Plain setup includes VEP 116, the PCGR/CPSR bundle and VEP 115: about 57 GB of clinical downloads and 65 GB unpacked.
Allow 1 to 3 hours. Setup checks each install's disk peak plus a 5 GB margin and warns if it skips one.
Use `./scripts/setup.sh --no-clinical-data "$GENOME_DIR"` to opt out.
Use `./scripts/setup.sh --clinical-data "$GENOME_DIR"` to add or retry only these data later.
`run-all.sh` skips steps whose data are still missing; inspect its plan and setup warnings.

## Platform Notes

### Linux (Recommended)
Expand Down Expand Up @@ -120,7 +126,7 @@ Then, once per machine and once per sample:

```bash
export GENOME_DIR=/path/to/your/data
./scripts/setup.sh $GENOME_DIR # reference, ClinVar, small data files, images and og (once)
./scripts/setup.sh $GENOME_DIR # reference, ClinVar, VEP caches, CPSR bundle, images and og (once)
./scripts/validate-setup.sh your_name # what is installed and what is missing
./scripts/run-all.sh your_name male # or female: sets the chrX/chrY ploidy and the sex check
```
Expand Down
6 changes: 3 additions & 3 deletions docs/hardware-requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ A run from FASTQ needs more space than a run from a BAM. Until you delete `<samp

One row per download, in GB as `wget` and `du -h` count them (1 GB = 2^30 bytes); the download sizes were read from each server on 2026-10-02 (the two step 17 rows on 2026-10-03). Each total is the sum of the rows above it; the `setup.sh` line is the sum of the rows it names. [Reference setup](00-reference-setup.md) has the commands, under a heading per database with the same size.

**Default run** (`setup.sh` downloads the first three rows and the Docker images; the VEP caches and the PCGR bundle are for steps 13 and 17, which a default run skips when they are missing):
**Default run** (`setup.sh` downloads all these rows. `--no-clinical-data` skips both VEP caches and the PCGR bundle):

| Resource | Download | On disk | Used by |
|---|---|---|---|
Expand All @@ -55,9 +55,9 @@ One row per download, in GB as `wget` and `du -h` count them (1 GB = 2^30 bytes)
| VEP cache, release 115 | ~24 GB | ~28 GB | step 17 (CPSR's own VEP) |
| Docker images | ~10-15 GB | ~10-15 GB | every step |
| **Total, default run** | **~73-78 GB** | **~98-103 GB** | |
| Of which `setup.sh` downloads (first three rows + Docker images) | ~16-21 GB | ~33-38 GB | |
| With `--no-clinical-data` (first three rows + Docker images) | ~16-21 GB | ~33-38 GB | |

While a VEP cache installs, the tarball and the unpacked tree are on disk together until the tarball is deleted (`install_vep_cache` in `scripts/lib/common.sh`): about 56 GB free for release 116 and 52 GB for release 115.
While a VEP cache installs, the tarball and the unpacked tree are on disk together until the tarball is deleted (`install_vep_cache` in `scripts/lib/common.sh`): 56 GB peak for release 116 and 52 GB for release 115. The bundle needs 15 GB peak. Setup adds a 5 GB margin to each check and warns when it skips an item. The three clinical downloads add about 57 GB to transfer and 65 GB on disk. Allow 1 to 3 hours. To retry only these downloads, run `./scripts/setup.sh --clinical-data "$GENOME_DIR"`.

**Optional** (only for the step named):

Expand Down
Loading
Loading