diff --git a/CHANGELOG.md b/CHANGELOG.md index b3e8e4021..98c03f710 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -33,6 +33,7 @@ Each release's notes. From 1.0.0 on, a major or minor release has hand-written n - A record that starts upstream and spans a marker with an allele that changes the marker's base (a deletion over it, or a padded `CG CA`, as DeepVariant writes `CG C,CA` when a deletion and an SNV overlap) now makes the site a no call, with a note that names the record. Before, the site had no record at its own position and 8 reads made it a confident reference: a K/K person read k/k at high confidence. - A VCF with one record per position and no overlapping records, as DeepVariant writes, gives the same output as before: the 34 parity cases still agree with the Python oracle byte for byte. - `og recheck` no longer says "Nothing here is newer than what the run used." when a lookup failed. With no network it said that line, and a cron log then read as up to date. Now the last line says how many of the looked-up items failed and that the check is incomplete, or "Could not check: all N looked-up items failed" when none could be checked. The exit code was already 1 and stays 1. A failed lookup also no longer hides a newer file or pin that is already installed here: a newer ClinVar file on disk stays `rerun-ready`, with its steps, in `state.json`, the reports and the live page, and the item keeps its `error`. `og recheck --rerun` no longer replays the stored command through `bash -c`. It reads `meta command` of `logs/run_status.tsv` back word by word as the call `run-all.sh` records (`cd DIR && [SWITCH=VALUE ...] .../scripts/run-all.sh ARGS`) and starts that `run-all.sh` with its arguments one by one. A row of any other shape, such as one edited to add `; rm ...`, or a `cd` folder that is gone, is refused with exit 2 and nothing runs. A row that `run-all.sh` wrote replays as before. No option, output file or format changes. [docs/recheck.md](docs/recheck.md) +- Setup now installs VEP 116, the PCGR/CPSR reference bundle and VEP 115 by default so `run-all.sh` can plan the clinical steps. `--no-clinical-data` skips them; `--clinical-data` installs only these data. Each install checks its disk peak plus a 5 GB margin, credits the allocated space of a partial download, and warns without stopping the remaining installs. Stale extraction files are removed before the disk check. Retained PCGR archives are checked against the pinned digest before reuse. `--clinical-data` exits 1 when one item is still missing. The bundle is checked against a pinned SHA-256, resumes interrupted transfers, and is published only after extraction. Validation and setup docs now name the affected steps and the repair command. Freshness checks the three archive URLs with HEAD, both Ensembl checksum entries and the bundle's byte count. No analysis output format changes. - The genome map shows where you are in the whole genome and lists what it draws. While a chromosome is open, a strip above the map draws every chromosome end to end, with the view as a brass brush and a tick for each finding on the map; a click on the strip opens a chromosome, and a drag brushes a new view or pans (on a phone the strip is a picture only). Under the map, an "In view" table lists every mark the map draws one by one in view: kind, what, where, size, who, copy and source, sorted by place with the findings first, a click on a header to sort by it. It shows 12 rows, then all of them (a scrolled list past 200 rows, at most 5,000; on a phone at most 200 rows, then zoom in), and a row flies the map to its feature. It follows the map once the map stops, announces only its count to a screen reader, and is folded on a phone; T folds and opens it. A closed result is never a row or a tick. One rule decides what the canvas, the strip and the table draw, and `og map build --predicate-table` writes the same rule's answers so a browser check holds the two to one set. A margin update no longer runs while the map is still moving: it waits for the view the map comes to rest on. diff --git a/docs/00-reference-setup.md b/docs/00-reference-setup.md index dfa88ad7d..b6baa03d2 100644 --- a/docs/00-reference-setup.md +++ b/docs/00-reference-setup.md @@ -4,6 +4,22 @@ One-time downloads required before running the pipeline. Each heading below give > **Estimated time:** 1-3 hours depending on internet speed. The two VEP caches (26 GB for step 13, 24 GB for step 17) are the largest downloads of a default run. +A plain `./scripts/setup.sh "$GENOME_DIR"` installs both VEP caches and the PCGR/CPSR bundle after AnnotSV. +These clinical downloads add about 57 GB to transfer and 65 GB on disk. Allow 1 to 3 hours. +Setup checks each install separately: VEP 116 needs 56 GB peak, the bundle 15 GB, and VEP 115 needs 52 GB. +Each check adds a 5 GB margin and credits the disk space occupied by a partial download. +Stale extraction files are removed before the check. Retained PCGR archives are checked against the pinned SHA-256 before reuse. An item that does not fit is skipped with a warning. Setup continues. + +```bash +./scripts/setup.sh --no-clinical-data "$GENOME_DIR" # skip all three clinical downloads +./scripts/setup.sh --clinical-data "$GENOME_DIR" # install or retry only these three +``` + +An interrupted transfer resumes. Existing caches and bundles are kept. +`--clinical-data` exits 1 when any of the three is still missing after it runs. +Without these data, `run-all.sh` skips the affected steps and records the reason. +A bare `nextflow run` still needs its tools and data parameters. Setup does not change that selection. + ## GRCh38 Reference Genome The foundation for everything. All tools need this. The pipeline uses NCBI's **GRCh38 no-ALT analysis set**: chr1-22, X, Y and M, the unplaced and unlocalized scaffolds and the EBV genome, 195 sequences, with UCSC names (`chr1`, `chrM`). `setup.sh` downloads it, checks it and stores it as `reference/GRCh38_no_alt_analysis_set.fasta`: @@ -143,7 +159,7 @@ Required for step 5. `setup.sh` downloads them (about 5 GB, about 20 GB unpacked ## VEP Cache (~26 GB) -Ensembl Variant Effect Predictor annotation database. Required for step 13 and the pipeline's VEP. Without it, a run skips VEP, vcfanno, the clinical filter and slivar. Install it before the first run: +Ensembl Variant Effect Predictor annotation database. Required for step 13 and the pipeline's VEP. Setup installs it by default. Without it, a run skips VEP, vcfanno, the clinical filter and slivar. To install only this cache: ```bash ./scripts/setup.sh --vep-cache ${GENOME_DIR} @@ -178,38 +194,22 @@ Required for step 17 (CPSR cancer predisposition screening). Includes ClinVar, g > **Important:** PCGR 2.3.2 bundles VEP 115, which requires a **release-115** cache — different from the release-116 cache used by step 13 above. See the next section for the VEP 115 download. The bundle date (`PCGR_DATA_BUNDLE`) and the cache release (`PCGR_VEP_CACHE_RELEASE`) are in `versions.env` and move with `PCGR_IMAGE`. -```bash -mkdir -p ${GENOME_DIR}/pcgr_data -cd ${GENOME_DIR}/pcgr_data - -# Download (~7 GB) -wget -c https://insilico.hpc.uio.no/pcgr/pcgr_ref_data.20260620.grch38.tgz +Setup installs it by default. To add the clinical data to an existing setup: -# Extract and organize into version-stamped directory -tar xzf pcgr_ref_data.20260620.grch38.tgz -mkdir -p 20260620 && mv data/ 20260620/ -# Creates: ${GENOME_DIR}/pcgr_data/20260620/data/ - -# Optional: delete the tarball to save 7 GB -# rm pcgr_ref_data.20260620.grch38.tgz +```bash +./scripts/setup.sh --clinical-data "$GENOME_DIR" ``` +The installer checks the archive against `PCGR_DATA_SHA256` in `versions.env`, then extracts it into a temporary folder. +The server publishes no checksum, so the size (`PCGR_DATA_BYTES`) and the digest are measured from a full download. `PCGR_DATA_MEASURED_FOR` names the image tag and bundle date they belong to, and `scripts/ci/check-coupled-versions.sh` fails when `PCGR_IMAGE` or `PCGR_DATA_BUNDLE` moves without it. +Only a complete tree is moved to `pcgr_data/20260620/data/`. The archive is deleted after installation. +The [PCGR resource list](https://sigven.github.io/pcgr/articles/annotation_resources.html) names the bundle's third-party databases. Their own terms apply. Setup prints the resource link before downloading. + ## VEP 115 Cache for CPSR (~24 GB) PCGR 2.3.2 (step 17) bundles VEP 115 internally, which needs the **release-115** cache. This is separate from the release-116 cache used by step 13. Both coexist in the same `vep_cache/` directory under different subdirectories (`116_GRCh38/` and `115_GRCh38/`). A `113_GRCh38/` cache left from PCGR 2.2.5 is no longer read and can be deleted. -```bash -mkdir -p ${GENOME_DIR}/vep_cache/tmp -cd ${GENOME_DIR}/vep_cache/tmp - -# Download VEP 115 cache (~24 GB) -wget -c https://ftp.ensembl.org/pub/release-115/variation/indexed_vep_cache/homo_sapiens_vep_115_GRCh38.tar.gz - -# Extract alongside the existing release-116 cache -cd ${GENOME_DIR}/vep_cache -tar xzf tmp/homo_sapiens_vep_115_GRCh38.tar.gz -# Creates: ${GENOME_DIR}/vep_cache/homo_sapiens/115_GRCh38/ -``` +Setup installs this cache by default, after the PCGR bundle. `--clinical-data` retries it using Ensembl's `CHECKSUMS` file and a temporary extraction folder. > If you only run step 13 (VEP annotation) and skip step 17 (CPSR), you only need the release-116 cache. If you only run step 17, you only need release-115. @@ -510,7 +510,7 @@ wget -c https://giab.s3.amazonaws.com/release/AshkenazimTrio/HG002_NA24385_son/N [Hardware and storage requirements](hardware-requirements.md#shared-reference-data-one-time) has the one table of every download above, with the totals. -> **Tip:** If disk space is tight, you can skip the VEP cache (step 13) and PCGR bundle (step 17) initially. The core pipeline (steps 2-3-6-7) only needs the reference FASTA and ClinVar (~3.5 GB unpacked). +> **Tip:** If disk space is tight, use `setup.sh --no-clinical-data` to skip both VEP caches and the PCGR bundle initially. The core pipeline (steps 2-3-6-7) only needs the reference FASTA and ClinVar (~3.5 GB unpacked). ## Verifying Your Setup diff --git a/docs/13-vep-annotation.md b/docs/13-vep-annotation.md index c1e859f6e..b3c0ca43d 100644 --- a/docs/13-vep-annotation.md +++ b/docs/13-vep-annotation.md @@ -16,7 +16,7 @@ Pinned in `versions.env`; [Image versions](versions.md) lists the current tag. ## Prerequisites - The VCF from step 3 **and its `.tbi`**: the step refuses a VCF without its index, which may be half written -- Offline VEP cache: step 13 downloads, checks and unpacks it the first time it runs (see step 00-reference-setup) +- Offline VEP cache: setup installs it by default. To retry, run `./scripts/setup.sh --clinical-data "$GENOME_DIR"` ([reference setup](00-reference-setup.md#vep-cache-26-gb)). The pipeline skips VEP when the cache is missing. - Cache size: see [Hardware and storage requirements](hardware-requirements.md#shared-reference-data-one-time) (about 26 GB to download, 30 GB unpacked) ## Command @@ -112,4 +112,4 @@ After annotation, use step 23 (clinical filter) which automatically detects avai - `--dir_cache /opt/vep/.vep`: the cache is mounted there, not in the home directory VEP looks in by default - `--fasta`: running `--offline` without a FASTA file disables HGVS notation (`INFO: Disabling --hgvs`), so the script always passes the reference ([`REF_FASTA`](00-reference-setup.md#the-reference-path-on-every-page) is the reference path) - VEP does NOT assess variant pathogenicity in ClinVar context — combine with step 6 (ClinVar screen) for full picture -- **Upgrading from an older release:** VEP reads the cache directory named after the release passed in `--cache_version` (`homo_sapiens/116_GRCh38/` for the pinned release 116). When that directory is missing, VEP stops with an error instead of annotating from an older cache such as `112_GRCh38/`; step 13 downloads the matching cache on its first run. +- **Upgrading from an older release:** VEP reads the cache directory named after the release passed in `--cache_version` (`homo_sapiens/116_GRCh38/` for the pinned release 116). When that directory is missing, VEP stops with an error instead of annotating from an older cache such as `112_GRCh38/`. Setup installs the matching cache by default. The standalone step script can download a missing cache, but `run-all.sh` skips VEP until the cache is installed. diff --git a/docs/17-cpsr.md b/docs/17-cpsr.md index abf28e100..b652e9bbf 100644 --- a/docs/17-cpsr.md +++ b/docs/17-cpsr.md @@ -22,22 +22,21 @@ The image is PCGR 2.3.2 from `ghcr.io/sigven/pcgr` (Docker Hub's `sigven/pcgr` s ### 1. VEP Cache PCGR 2.3.2 bundles VEP 115, which requires the **release-115** cache (~24 GB). This is different from the release-116 cache used by step 13. Both coexist in the same `vep_cache/` directory under different subdirectories (`116_GRCh38/` and `115_GRCh38/`). -```bash -mkdir -p ${GENOME_DIR}/vep_cache -wget -c -P ${GENOME_DIR}/vep_cache https://ftp.ensembl.org/pub/release-115/variation/indexed_vep_cache/homo_sapiens_vep_115_GRCh38.tar.gz -tar xzf ${GENOME_DIR}/vep_cache/homo_sapiens_vep_115_GRCh38.tar.gz -C ${GENOME_DIR}/vep_cache -``` -### 2. PCGR Ref Data Bundle -PCGR 2.x uses a ref data bundle (~7 GB for `20260620`) separate from VEP: +### 2. PCGR ref data bundle +PCGR 2.x uses a separate ref data bundle (`20260620`, about 7 GB). +A plain setup installs the VEP 115 cache and this bundle by default, alongside VEP 116 for step 13. +For an existing setup, install or retry all three with: + ```bash -mkdir -p ${GENOME_DIR}/pcgr_data -cd ${GENOME_DIR}/pcgr_data -wget -c https://insilico.hpc.uio.no/pcgr/pcgr_ref_data.20260620.grch38.tgz -tar xzf pcgr_ref_data.20260620.grch38.tgz -mkdir -p 20260620 && mv data/ 20260620/ +./scripts/setup.sh --clinical-data "$GENOME_DIR" ``` -This creates a `20260620/data/` directory with ClinVar, CancerMine, UniProt, and other databases. VEP cache is now mounted separately. + +The bundle goes to `pcgr_data/20260620/data/`; the caches coexist under `vep_cache/homo_sapiens/`. +Setup checks Ensembl's cache checksums and the bundle's pinned SHA-256 before extraction, then removes each archive. +It checks free space for each install and continues with a warning if one fails or does not fit. +`--no-clinical-data` skips all three. Without either CPSR item, `run-all.sh` skips step 17. +See [reference setup](00-reference-setup.md) for disk peaks and the [PCGR resource list](https://sigven.github.io/pcgr/articles/annotation_resources.html) for the bundled databases. Review each source's terms before use. ### Upgrading from PCGR 2.2.5 PCGR 2.2.5 read bundle `20250314` and the release-113 cache. 2.3.2 needs both new ones: about 31 GB to download (the 7 GB bundle and the 24 GB cache). Once step 17 runs on 2.3.2, `pcgr_data/20250314/` and `vep_cache/homo_sapiens/113_GRCh38/` are no longer read and can be deleted. diff --git a/docs/getting-started.md b/docs/getting-started.md index 6067bc415..e72239787 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -18,7 +18,7 @@ | **CPU** | 4 cores | 8-16 cores | `run-all.sh` runs DeepVariant with at most 8 shards, fewer when `--max_cpus` is lower; more cores run other steps beside it ([CPU](hardware-requirements.md#cpu-requirements)) | | **RAM** | 16 GB | 32 GB | Every container has a hard memory limit; the 8-CPU steps ask for 32 GB, capped at the machine's RAM | | **Disk** | 500 GB free from a BAM, about 700 GB from FASTQ | 1 TB+ | See [detailed breakdown](hardware-requirements.md#disk-space-breakdown) | -| **Internet** | Broadband | 100+ Mbps | ~70-75 GB core downloads + ~175 GB optional annotation databases | +| **Internet** | Broadband | 100+ Mbps | ~73-78 GB core downloads + ~175 GB optional annotation databases | | **OS** | Linux (amd64) | Ubuntu 22.04+ | macOS/ARM works but slower (see below) | > **Disk space is the #1 surprise.** A single 30X WGS sample produces 60-90 GB of FASTQ, 80-120 GB of BAM, plus reference genomes and databases. See [docs/hardware-requirements.md](hardware-requirements.md) for the full breakdown. @@ -46,11 +46,17 @@ Every analysis tool runs inside Docker: no conda environments, no Python version | PCGR/CPSR data bundle + VEP 115 cache | ~31 GB | Step 17 (cancer predisposition) | | Docker images (all steps) | ~10-15 GB | All steps | | Annotation databases (CADD, SpliceAI, REVEL, AlphaMissense) | ~175 GB | Steps 30-31 (optional) | -| **Total one-time setup (core)** | **~70-75 GB** | | +| **Total one-time setup (core)** | **~73-78 GB** | | | **Total with annotation enrichment** | **~250 GB** | | See [docs/00-reference-setup.md](00-reference-setup.md) for download instructions. +Plain setup includes VEP 116, the PCGR/CPSR bundle and VEP 115: about 57 GB of clinical downloads and 65 GB unpacked. +Allow 1 to 3 hours. Setup checks each install's disk peak plus a 5 GB margin and warns if it skips one. +Use `./scripts/setup.sh --no-clinical-data "$GENOME_DIR"` to opt out. +Use `./scripts/setup.sh --clinical-data "$GENOME_DIR"` to add or retry only these data later. +`run-all.sh` skips steps whose data are still missing; inspect its plan and setup warnings. + ## Platform Notes ### Linux (Recommended) @@ -120,7 +126,7 @@ Then, once per machine and once per sample: ```bash export GENOME_DIR=/path/to/your/data -./scripts/setup.sh $GENOME_DIR # reference, ClinVar, small data files, images and og (once) +./scripts/setup.sh $GENOME_DIR # reference, ClinVar, VEP caches, CPSR bundle, images and og (once) ./scripts/validate-setup.sh your_name # what is installed and what is missing ./scripts/run-all.sh your_name male # or female: sets the chrX/chrY ploidy and the sex check ``` diff --git a/docs/hardware-requirements.md b/docs/hardware-requirements.md index 67a7b09ee..dacc9aba5 100644 --- a/docs/hardware-requirements.md +++ b/docs/hardware-requirements.md @@ -43,7 +43,7 @@ A run from FASTQ needs more space than a run from a BAM. Until you delete ` `) in the same PR: the server publishes no checksum, and `scripts/ci/check-coupled-versions.sh` fails until all of them move together. Then rerun step 17 on a sample with an earlier result. CI does not run CPSR on a sample. The image test checks that `cpsr --help` still lists every flag step 17 passes, that `cpsr --version` names the release line of the image tag and the `pcgrr` R package its full release, and runs the bundled VEP with `--database` on 50 variants. - **Python**: resolve `scripts/cyrius-constraints.txt` again in the new image, as its header says, and rerun step 21 on a test BAM. - **Nextflow**: the PR runs the stub and E2E jobs on the new version. The prose names `NEXTFLOW_VERSION` rather than a number, and the fake nextflow of the fake-docker suite reads it from `versions.env`, so a patch needs no hand edit. A new release line (26.10 to 27.04, say) also needs the line named in `nextflow.config`, `nextflow.yml` and the prose of [Nextflow Execution](nextflow.md) and [Lessons Learned](lessons-learned.md) moved by hand. - **PGS scoring files and old PRS summaries**: after a scoring file update, rerun step 25 and treat its output as a new baseline ([PRS](25-prs.md#maintenance)). A `_prs_summary.tsv` whose header has `Variants_Used` comes from the scoring path used before 2026-10-01: its `Score` is plink2's average, its `Variants_Used` a dosage sum, and its PGS000738 row (a vitiligo score) is labelled schizophrenia. Rerun step 25 rather than reuse or compare it ([Lessons Learned](lessons-learned.md#a-prs-summary-with-a-variants_used-column-comes-from-the-old-scoring-path)). diff --git a/renovate.json b/renovate.json index 14f063c4e..09ad6ba29 100644 --- a/renovate.json +++ b/renovate.json @@ -187,7 +187,7 @@ "versioning": "regex:^release_(?\\d+)\\.(?\\d+)$" }, { - "description": "PCGR/CPSR: every update waits for approval on the dependency dashboard, because each release reads its own reference bundle and VEP cache (PCGR_DATA_BUNDLE and PCGR_VEP_CACHE_RELEASE in versions.env). The image comes from ghcr.io/sigven/pcgr (Docker Hub's sigven/pcgr stops at 2.2.5); the image matcher reads ghcr.io like any other registry. allowedVersions keeps the release tags (x.y.z): ghcr also carries development builds (2.3.2.9006) and .singularity tags.", + "description": "PCGR/CPSR: every update waits for approval on the dependency dashboard, because each release reads its own reference bundle and VEP cache (PCGR_DATA_BUNDLE and PCGR_VEP_CACHE_RELEASE in versions.env). Renovate never moves the bundle: no manager matches PCGR_DATA_BUNDLE, PCGR_DATA_BYTES or PCGR_DATA_SHA256, because the server publishes no checksum and the digest has to be measured from a full download. So an approved image PR fails scripts/ci/check-coupled-versions.sh until a person moves the bundle date, the VEP cache release, the size, the digest and PCGR_DATA_MEASURED_FOR in the same PR. The image comes from ghcr.io/sigven/pcgr (Docker Hub's sigven/pcgr stops at 2.2.5); the image matcher reads ghcr.io like any other registry. allowedVersions keeps the release tags (x.y.z): ghcr also carries development builds (2.3.2.9006) and .singularity tags.", "matchDatasources": ["docker"], "matchPackageNames": ["ghcr.io/sigven/pcgr"], "allowedVersions": "/^\\d+\\.\\d+\\.\\d+$/", diff --git a/rust/crates/og-fresh/src/lookups.rs b/rust/crates/og-fresh/src/lookups.rs index 70a6e37bb..14511e2a8 100644 --- a/rust/crates/og-fresh/src/lookups.rs +++ b/rust/crates/og-fresh/src/lookups.rs @@ -223,6 +223,77 @@ pub fn pcgr(c: &Ctx<'_>) -> Lookup { }) } +/// Verify a pinned cache URL and its entry in Ensembl's small CHECKSUMS file. +/// Large archives are only read with HEAD, never downloaded by freshness. +fn cache_download(c: &Ctx<'_>, pin: &str, label: &str) -> Lookup { + let release = c.pins.need(pin)?; + let url = vep_cache_template(c.common)?.replace("%s", release); + c.fetch.head(&url)?; + let (base, name) = url.rsplit_once('/').expect("cache URL has a basename"); + let checksums = c.fetch.text(&format!("{base}/CHECKSUMS"))?; + let listed = checksums.lines().any(|line| { + let fields: Vec<_> = line.split_whitespace().collect(); + fields.len() == 3 + && fields[2] == name + && fields[0].parse::().is_ok() + && fields[1].parse::().is_ok_and(|n| n > 0) + }); + if !listed { + return failed(format!("{name} has no usable entry in {base}/CHECKSUMS")); + } + Ok(Found { + row: row( + label, + release.into(), + "URL live; CHECKSUMS entry present".into(), + false, + String::new, + ), + upstream: Upstream::ReportOnly, + }) +} + +/// The release used by steps 13, 30, 23 and 31. +pub fn vep_download(c: &Ctx<'_>) -> Lookup { + cache_download(c, "VEP_CACHE_RELEASE", "VEP cache download") +} + +/// The separate release used inside CPSR's image. +pub fn cpsr_download(c: &Ctx<'_>) -> Lookup { + cache_download(c, "PCGR_VEP_CACHE_RELEASE", "CPSR cache download") +} + +/// The pinned PCGR bundle URL and size. SHA-256 is checked by setup, which +/// downloads the file; freshness deliberately reads only its headers. +pub fn pcgr_download(c: &Ctx<'_>) -> Lookup { + let bundle = c.pins.need("PCGR_DATA_BUNDLE")?; + let expected = c + .pins + .need("PCGR_DATA_BYTES")? + .parse::() + .map_err(|_| LookupError::Shape("PCGR_DATA_BYTES is not a byte count".into()))?; + let url = format!("https://insilico.hpc.uio.no/pcgr/pcgr_ref_data.{bundle}.grch38.tgz"); + let headers = c.fetch.head(&url)?; + let size = headers + .get("Content-Length") + .and_then(|s| s.parse::().ok()); + if expected == 0 || size != Some(expected) { + return failed(format!( + "{url}: Content-Length {size:?}, expected {expected}" + )); + } + Ok(Found { + row: row( + "PCGR bundle download", + bundle.into(), + format!("URL live; {expected} bytes"), + false, + String::new, + ), + upstream: Upstream::ReportOnly, + }) +} + pub fn pypgx(c: &Ctx<'_>) -> Lookup { let image_tag = tag_of(c.pins, "PYPGX_IMAGE")?; let held = image_tag.split("--").next().unwrap_or("").to_owned(); @@ -667,6 +738,9 @@ pub type LookupFn = fn(&Ctx<'_>) -> Lookup; pub const DATABASES: &[(&str, LookupFn)] = &[ ("Ensembl / VEP cache", ensembl), ("PCGR / CPSR", pcgr), + ("VEP cache download", vep_download), + ("CPSR cache download", cpsr_download), + ("PCGR bundle download", pcgr_download), ("pypgx", pypgx), ("PharmCAT", pharmcat), ("IPD-IMGT/HLA", imgt), diff --git a/rust/crates/og-fresh/tests/screen_setup_clinical_default.rs b/rust/crates/og-fresh/tests/screen_setup_clinical_default.rs new file mode 100644 index 000000000..0564a0ace --- /dev/null +++ b/rust/crates/og-fresh/tests/screen_setup_clinical_default.rs @@ -0,0 +1,156 @@ +//! Clinical links must be live and metadata must agree, without archive GETs. +use og_fresh::http::{Body, Fetch, GitHub, Headers, Request, Want}; +use og_fresh::lookups::{Ctx, DATABASES}; +use og_fresh::pins::Pins; +use og_fresh::selftest::FakeRegistry; +use og_fresh::{Lookup, failed}; +use std::path::Path; +use std::sync::Mutex; + +struct Downloads { + github: GitHub, + missing: bool, + bad_length: bool, + bad_checksum: bool, + calls: Mutex>, +} +impl Fetch for Downloads { + fn github_api(&self) -> &GitHub { + &self.github + } + fn request(&self, r: &Request<'_>) -> Lookup<(Body, Headers)> { + self.calls + .lock() + .unwrap() + .push((r.method.into(), r.url.into())); + if self.missing { + return failed("HTTP 404: release does not exist"); + } + if r.url.ends_with("CHECKSUMS") { + assert_eq!(r.want, Want::Text); + let body = if self.bad_checksum { + "123 5 wrong.tar.gz\n".into() + } else { + "123 5 homo_sapiens_vep_116_GRCh38.tar.gz\n456 6 homo_sapiens_vep_115_GRCh38.tar.gz\n".into() + }; + Ok((Body::Text(body), Headers::default())) + } else { + assert_eq!(r.method, "HEAD", "large archive body was requested"); + Ok(( + Body::None, + Headers::from_pairs(vec![( + "Content-Length".into(), + if self.bad_length { "1" } else { "7664082467" }.into(), + )]), + )) + } + } +} +fn check(missing: bool, bad_length: bool, bad_checksum: bool) { + let fetch = Downloads { + github: GitHub { + api: "https://invalid".into(), + token: None, + }, + missing, + bad_length, + bad_checksum, + calls: Mutex::new(vec![]), + }; + let registry = FakeRegistry { + tags: |_| panic!("registry is not needed"), + digests: |_| panic!("registry is not needed"), + asked: Mutex::new(vec![]), + }; + let pins = Pins::parse( + "VEP_CACHE_RELEASE=\"116\"\nPCGR_VEP_CACHE_RELEASE=\"115\"\nPCGR_DATA_BUNDLE=\"20260620\"\nPCGR_DATA_BYTES=\"7664082467\"\n", + ); + let root = Path::new(env!("CARGO_MANIFEST_DIR")).join("../../.."); + let common = root.join("scripts/lib/common.sh"); + let c = Ctx { + fetch: &fetch, + registry: ®istry, + pins: &pins, + versions: Path::new("unused"), + setup: Path::new("unused"), + common: &common, + now: "2026-11-15T00:00:00Z".parse().unwrap(), + }; + for name in [ + "VEP cache download", + "CPSR cache download", + "PCGR bundle download", + ] { + let (_, lookup) = DATABASES + .iter() + .find(|(n, _)| *n == name) + .unwrap_or_else(|| panic!("missing liveness row: {name}")); + let got = lookup(&c); + let should_fail = missing + || (bad_length && name == "PCGR bundle download") + || (bad_checksum && name != "PCGR bundle download"); + assert_eq!(got.is_err(), should_fail, "{name}: {got:?}"); + } + let calls = fetch.calls.lock().unwrap(); + assert!(calls.iter().any(|(_, url)| url.contains("release-115/"))); +} +#[test] +fn clinical_links_head_only() { + check(false, false, false); +} +#[test] +fn missing_release_fails() { + check(true, false, false); +} +#[test] +fn changed_bundle_length_fails() { + check(false, true, false); +} +#[test] +fn missing_cache_checksum_fails() { + check(false, false, true); +} + +#[test] +#[ignore = "public network: archive HEADs and small Ensembl CHECKSUMS only"] +fn clinical_downloads_live() { + use og_fresh::http::Client; + use og_fresh::registry::Online; + let root = Path::new(env!("CARGO_MANIFEST_DIR")).join("../../.."); + let versions = root.join("versions.env"); + let setup = root.join("scripts/setup.sh"); + let common = root.join("scripts/lib/common.sh"); + let client = Client::default(); + let registry = Online(&client); + let pins = Pins::read(&versions).unwrap(); + let c = Ctx { + fetch: &client, + registry: ®istry, + pins: &pins, + versions: &versions, + setup: &setup, + common: &common, + now: "2026-11-15T00:00:00Z".parse().unwrap(), + }; + for name in [ + "VEP cache download", + "CPSR cache download", + "PCGR bundle download", + ] { + let (_, lookup) = DATABASES.iter().find(|(n, _)| *n == name).unwrap(); + let found = lookup(&c).unwrap(); + println!("{}: {}", found.row.item, found.row.upstream); + } + let mut bad_pins = pins.clone(); + bad_pins + .values + .insert("VEP_CACHE_RELEASE".into(), "99999999".into()); + let bad = Ctx { + pins: &bad_pins, + ..c + }; + assert!( + og_fresh::lookups::vep_download(&bad).is_err(), + "nonexistent release accepted" + ); +} diff --git a/scripts/ci/check-coupled-versions.sh b/scripts/ci/check-coupled-versions.sh index a00a76208..a50f1a8d9 100755 --- a/scripts/ci/check-coupled-versions.sh +++ b/scripts/ci/check-coupled-versions.sh @@ -7,7 +7,12 @@ # VEP_CACHE_RELEASE must equal the major of VEP_IMAGE (release_116.0 -> 116) # PYPGX_BUNDLE_VERSION must equal the version of PYPGX_IMAGE (0.26.0--pyh... -> 0.26.0) # ANNOTSV_ANNOTATIONS_VERSION must equal major.minor of ANNOTSV_IMAGE (3.5.10--h... -> 3.5) -# PCGR_IMAGE, PCGR_VEP_CACHE_RELEASE and PCGR_DATA_BUNDLE must all be set +# PCGR_IMAGE, PCGR_VEP_CACHE_RELEASE, PCGR_DATA_BUNDLE, PCGR_DATA_BYTES and +# PCGR_DATA_SHA256 must all be set, the size a byte +# count and the digest 64 hex digits +# PCGR_DATA_MEASURED_FOR must be " " +# (2.3.2 20260620): the size and digest were measured +# for that image and bundle, so either moving alone fails # YLEAF_DATA_VERSION must equal the version of YLEAF_IMAGE (3.2.1--pyh... -> 3.2.1) # GENCODE_RELEASE (scripts/lib/common.sh) must be the GENCODE release of the # Ensembl release of VEP_IMAGE, its major minus 66 (116 -> 50) @@ -31,7 +36,7 @@ check() { # shellcheck disable=SC2016 # $1 and ${!v} expand in the inner shell vals=$(env -i bash --noprofile --norc -c '. "$1"; for v in VEP_IMAGE VEP_CACHE_RELEASE PYPGX_IMAGE PYPGX_BUNDLE_VERSION \ ANNOTSV_IMAGE ANNOTSV_ANNOTATIONS_VERSION PCGR_IMAGE PCGR_VEP_CACHE_RELEASE PCGR_DATA_BUNDLE \ - YLEAF_IMAGE YLEAF_DATA_VERSION VERIFYBAMID2_IMAGE; do + PCGR_DATA_BYTES PCGR_DATA_SHA256 PCGR_DATA_MEASURED_FOR YLEAF_IMAGE YLEAF_DATA_VERSION VERIFYBAMID2_IMAGE; do printf "%s=%s\n" "$v" "${!v-}"; done' _ "$1") || { echo "FAIL: cannot read $1"; return 1; } get() { sed -n "s/^$1=//p" <<<"$vals"; } tag() { local i; i=$(get "$1"); i=${i##*/}; [[ "$i" == *:* ]] && echo "${i#*:}"; } @@ -64,15 +69,24 @@ check() { t=$(tag VERIFYBAMID2_IMAGE || true); t=${t%%--*} rule "VB2_PANEL_FILES (scripts/setup.sh) come from the VerifyBamID tag of VERIFYBAMID2_IMAGE" "$tags" "${t:+v$t}" local v missing="" - for v in PCGR_IMAGE PCGR_VEP_CACHE_RELEASE PCGR_DATA_BUNDLE; do + for v in PCGR_IMAGE PCGR_VEP_CACHE_RELEASE PCGR_DATA_BUNDLE PCGR_DATA_BYTES PCGR_DATA_SHA256; do [ -n "$(get "$v")" ] || missing="${missing} ${v}" done if [ -z "$missing" ]; then - echo "OK PCGR_IMAGE, PCGR_VEP_CACHE_RELEASE and PCGR_DATA_BUNDLE are set together" + echo "OK PCGR_IMAGE, PCGR_VEP_CACHE_RELEASE, PCGR_DATA_BUNDLE, PCGR_DATA_BYTES and PCGR_DATA_SHA256 are set together" else echo "FAIL PCGR variables must be set together; unset:${missing}" fail=1 fi + if [[ "$(get PCGR_DATA_BYTES)" =~ ^[1-9][0-9]*$ ]] && [[ "$(get PCGR_DATA_SHA256)" =~ ^[0-9a-f]{64}$ ]]; then + echo "OK PCGR_DATA_BYTES is a byte count and PCGR_DATA_SHA256 a SHA-256 digest" + else + echo "FAIL PCGR_DATA_BYTES must be a byte count and PCGR_DATA_SHA256 64 lowercase hex digits: $(get PCGR_DATA_BYTES), $(get PCGR_DATA_SHA256)" + fail=1 + fi + t=$(tag PCGR_IMAGE || true) + rule "PCGR_DATA_MEASURED_FOR names the PCGR_IMAGE tag and PCGR_DATA_BUNDLE of PCGR_DATA_BYTES and PCGR_DATA_SHA256" \ + "$(get PCGR_DATA_MEASURED_FOR)" "$( [ -n "$t" ] && [ -n "$(get PCGR_DATA_BUNDLE)" ] && echo "$t $(get PCGR_DATA_BUNDLE)")" return "$fail" } @@ -103,20 +117,29 @@ self_test() { [ "$rc" -eq 0 ] || { echo "self-test: versions.env itself failed:"; printf '%s\n' "$out"; fail=1; } # The values the unchanged file holds, so a legitimate bump keeps this # test green; each planted value differs from them. - local vep pypgx annotsv yleaf vb2 + local vep pypgx annotsv yleaf vb2 pcgr vep=$(sed -nE 's/^VEP_IMAGE="[^"]*:release_([0-9]+).*/\1/p' "${ROOT}/versions.env") pypgx=$(sed -nE 's/^PYPGX_IMAGE="[^"]*:([^"]*)--.*/\1/p' "${ROOT}/versions.env") annotsv=$(sed -nE 's/^ANNOTSV_IMAGE="[^"]*:([0-9]+\.[0-9]+).*/\1/p' "${ROOT}/versions.env") yleaf=$(sed -nE 's/^YLEAF_IMAGE="[^"]*:([^"]*)--.*/\1/p' "${ROOT}/versions.env") vb2=$(sed -nE 's/^VERIFYBAMID2_IMAGE="[^"]*:([^"]*)--.*/\1/p' "${ROOT}/versions.env") - if [ -z "$vep" ] || [ -z "$pypgx" ] || [ -z "$annotsv" ] || [ -z "$yleaf" ] || [ -z "$vb2" ]; then - echo "self-test: cannot read VEP_IMAGE, PYPGX_IMAGE, ANNOTSV_IMAGE, YLEAF_IMAGE or VERIFYBAMID2_IMAGE from versions.env"; return 1 + pcgr=$(sed -nE 's/^PCGR_IMAGE="[^"]*:([^"]*)".*/\1/p' "${ROOT}/versions.env") + if [ -z "$vep" ] || [ -z "$pypgx" ] || [ -z "$annotsv" ] || [ -z "$yleaf" ] || [ -z "$vb2" ] || [ -z "$pcgr" ]; then + echo "self-test: cannot read VEP_IMAGE, PYPGX_IMAGE, ANNOTSV_IMAGE, YLEAF_IMAGE, VERIFYBAMID2_IMAGE or PCGR_IMAGE from versions.env"; return 1 fi expect vep-cache "s/^VEP_CACHE_RELEASE=.*/VEP_CACHE_RELEASE=\"$((vep - 1))\"/" "^FAIL VEP_CACHE_RELEASE .*: $((vep - 1)), want ${vep}\$" expect vep-image "s/^(VEP_IMAGE=\"[^\"]*:release_)[0-9]+/\\1$((vep + 1))/" "^FAIL VEP_CACHE_RELEASE .*: ${vep}, want $((vep + 1))\$" expect pypgx 's/^PYPGX_BUNDLE_VERSION=.*/PYPGX_BUNDLE_VERSION="9.9.9"/' "^FAIL PYPGX_BUNDLE_VERSION .*: 9\\.9\\.9, want ${pypgx//./\\.}\$" expect annotsv 's/^ANNOTSV_ANNOTATIONS_VERSION=.*/ANNOTSV_ANNOTATIONS_VERSION="0.1"/' "^FAIL ANNOTSV_ANNOTATIONS_VERSION .*: 0\\.1, want ${annotsv//./\\.}\$" expect pcgr '/^PCGR_DATA_BUNDLE=/d' '^FAIL PCGR variables must be set together; unset: PCGR_DATA_BUNDLE' + expect pcgr-sha256 '/^PCGR_DATA_SHA256=/d' '^FAIL PCGR variables must be set together; unset: PCGR_DATA_SHA256' + expect pcgr-bytes 's/^PCGR_DATA_BYTES=.*/PCGR_DATA_BYTES="7.6 GB"/' '^FAIL PCGR_DATA_BYTES must be a byte count' + expect pcgr-digest 's/^PCGR_DATA_SHA256=.*/PCGR_DATA_SHA256="0123"/' '^FAIL PCGR_DATA_BYTES must be a byte count and PCGR_DATA_SHA256 64' + # A new bundle date, or a new image, without a new size and digest. + expect pcgr-bundle-alone 's/^PCGR_DATA_BUNDLE=.*/PCGR_DATA_BUNDLE="29991231"/' \ + "^FAIL PCGR_DATA_MEASURED_FOR .*: ${pcgr//./\\.} [0-9]+, want ${pcgr//./\\.} 29991231\$" + expect pcgr-image-alone 's/^(PCGR_IMAGE="[^"]*:)[^"]*/\19.9.9/' \ + "^FAIL PCGR_DATA_MEASURED_FOR .*: ${pcgr//./\\.} [0-9]+, want 9\\.9\\.9 [0-9]+\$" expect yleaf 's/^YLEAF_DATA_VERSION=.*/YLEAF_DATA_VERSION="9.9.9"/' "^FAIL YLEAF_DATA_VERSION .*: 9\\.9\\.9, want ${yleaf//./\\.}\$" expect gencode "s/^(GENCODE_RELEASE=)[^ ]*/\\1$((vep - 67))/" "^FAIL GENCODE_RELEASE .*: $((vep - 67)), want $((vep - 66))\$" common expect gencode-vep "s/^(VEP_IMAGE=\"[^\"]*:release_)[0-9]+/\\1$((vep + 1))/" "^FAIL GENCODE_RELEASE .*: $((vep - 66)), want $((vep - 65))\$" diff --git a/scripts/ci/fake-docker/lib.sh b/scripts/ci/fake-docker/lib.sh index d8a28d811..b2e447442 100644 --- a/scripts/ci/fake-docker/lib.sh +++ b/scripts/ci/fake-docker/lib.sh @@ -28,6 +28,32 @@ REF_FASTA_MD5=$(fake_body "$REF_FASTA_URL" | fake_md5) REF_FAI_MD5=$(fake_body "${REF_FASTA_URL}.fai" | fake_md5) export REF_FASTA_MD5 REF_FAI_MD5 +# Clinical setup uses real archive layouts, served by the existing fake curl +# and wget. CHECKSUMS is shared by basename, so it lists both VEP archives. +seed_clinical_downloads() { + local rel name + export FAKE_DOWNLOAD_DIR="${CASE_WORK}/downloads" + mkdir -p "$FAKE_DOWNLOAD_DIR" + for rel in 116 115; do + mkdir -p "${CASE_WORK}/cache/homo_sapiens/${rel}_GRCh38" + printf 'fake VEP cache\n' > "${CASE_WORK}/cache/homo_sapiens/${rel}_GRCh38/info.txt" + tar -czf "${FAKE_DOWNLOAD_DIR}/homo_sapiens_vep_${rel}_GRCh38.tar.gz" -C "${CASE_WORK}/cache" "homo_sapiens/${rel}_GRCh38" + done + for rel in 116 115; do + name="homo_sapiens_vep_${rel}_GRCh38.tar.gz" + printf '%s %s\n' "$(sum -r "${FAKE_DOWNLOAD_DIR}/${name}" | awk '{print $1, $2}')" "$name" + done > "${FAKE_DOWNLOAD_DIR}/CHECKSUMS" + mkdir -p "${CASE_WORK}/bundle/data/grch38" + printf 'fake PCGR data\n' > "${CASE_WORK}/bundle/data/grch38/README" + tar -czf "${FAKE_DOWNLOAD_DIR}/pcgr_ref_data.20260620.grch38.tgz" -C "${CASE_WORK}/bundle" data + if command -v sha256sum >/dev/null 2>&1; then + PCGR_DATA_SHA256=$(sha256sum "${FAKE_DOWNLOAD_DIR}/pcgr_ref_data.20260620.grch38.tgz" | awk '{print $1}') + else + PCGR_DATA_SHA256=$(shasum -a 256 "${FAKE_DOWNLOAD_DIR}/pcgr_ref_data.20260620.grch38.tgz" | awk '{print $1}') + fi + export PCGR_DATA_SHA256 +} + fail() { echo "ASSERT FAIL: $*" >&2 exit 1 diff --git a/scripts/lib/common.sh b/scripts/lib/common.sh index 74f3cfa2e..17374e740 100644 --- a/scripts/lib/common.sh +++ b/scripts/lib/common.sh @@ -632,24 +632,112 @@ install_vep_cache() { url=$(vep_cache_url "$rel") tarball="${dir}/$(basename "$url")" final="${dir}/homo_sapiens/${rel}_GRCh38" + if [ -f "${final}/info.txt" ]; then + echo "[OK] VEP ${rel} cache already present." + return 0 + fi if [ -e "$final" ]; then echo "ERROR: ${final} exists but has no info.txt (an interrupted extraction?)." >&2 echo " Remove it and run again." >&2 return 1 fi mkdir -p "${dir}/homo_sapiens" - fetch "$url" "$tarball" sum "$(dirname "$url")/CHECKSUMS" || return 1 + # fetch publishes the tarball only after its check, so one already here is + # whole: an install stopped during extraction unpacks it again. + [ -f "$tarball" ] || fetch "$url" "$tarball" sum "$(dirname "$url")/CHECKSUMS" || return 1 echo " Extracting $(basename "$tarball")..." + rm -rf "${dir}"/.extract.* tmp=$(mktemp -d "${dir}/.extract.XXXXXX") if ! tar xzf "$tarball" -C "$tmp" || [ ! -f "${tmp}/homo_sapiens/${rel}_GRCh38/info.txt" ]; then rm -rf "$tmp" echo "ERROR: could not unpack ${tarball} into a ${rel}_GRCh38 cache." >&2 return 1 fi - mv "${tmp}/homo_sapiens/${rel}_GRCh38" "$final" + if ! mv "${tmp}/homo_sapiens/${rel}_GRCh38" "$final"; then + rm -rf "$tmp" + return 1 + fi rm -rf "$tmp" "$tarball" } +# clinical_disk_ok DIR PEAK_GB LABEL [ARCHIVE]: GB as du counts them +# (2^30 bytes), plus a 5 GB margin. +# Use the destination's filesystem, creating the directory when it is new. +# Allocated blocks of ARCHIVE, or of its interrupted ARCHIVE.part, are already +# part of the peak, so a resumed download needs only the rest. +# An unreadable df is a failed preflight, never permission to fill the disk. +clinical_disk_ok() { + local dir=$1 peak=$2 label=$3 archive=${4:-} free have=0 f + mkdir -p "$dir" || return 1 + # A stopped extraction is disposable. Reclaim it before measuring space, + # otherwise its staging tree can prevent every retry from reaching cleanup. + rm -rf "${dir}"/.extract.* || return 1 + free=$(df -Pk "$dir" 2>/dev/null | awk 'NR == 2 {print $4}') || free="" + case "$free" in + ''|*[!0-9]*) echo "[WARN] Cannot read free space for ${label}; run: ./scripts/setup.sh --clinical-data \"${GENOME_DIR}\""; return 1 ;; + esac + if [ -n "$archive" ]; then + for f in "$archive" "${archive}.part"; do + if [ -f "$f" ]; then + have=$(du -k "$f" 2>/dev/null | awk 'NR == 1 {print $1}') || have="" + case "$have" in + ''|*[!0-9]*) echo "[WARN] Cannot read allocated space for ${label}; retry the clinical install."; return 1 ;; + esac + break + fi + done + fi + if [ "$(( free + have ))" -lt "$(( (peak + 5) * 1048576 ))" ]; then + echo "[WARN] Skipping ${label}: needs ${peak} GB peak plus a 5 GB margin; $(awk -v k="$free" 'BEGIN {printf "%.1f", k / 1048576}') GB free." + echo " Install later: ./scripts/setup.sh --clinical-data \"${GENOME_DIR}\"" + return 1 + fi +} + +# install_pcgr_bundle DIR: check the pinned archive before extracting into a +# staging tree. Upstream ships data/ at the root; CPSR reads BUNDLE/data/. +# The shared fetch helper resumes interrupted transfers and deletes bad ones. +# shellcheck disable=SC2154 # PCGR_DATA_BUNDLE and PCGR_DATA_SHA256 are pins in versions.env +install_pcgr_bundle() { + local dir=$1 url tarball tmp + if [ -d "${dir}/${PCGR_DATA_BUNDLE}/data" ]; then + echo "[OK] PCGR/CPSR ${PCGR_DATA_BUNDLE} bundle already present." + return 0 + fi + echo "PCGR/CPSR reference data includes third-party databases with their own terms:" + echo " https://sigven.github.io/pcgr/articles/annotation_resources.html" + echo " Review those terms before use; --no-clinical-data skips this download." + url="https://insilico.hpc.uio.no/pcgr/pcgr_ref_data.${PCGR_DATA_BUNDLE}.grch38.tgz" + tarball="${dir}/${url##*/}" + mkdir -p "$dir" || return 1 + # A retained archive may have changed since fetch checked it. Verify it + # against the pin again before reuse, replacing a mismatch through fetch. + if [ -f "$tarball" ] && ! _fetch_check sha256 "$PCGR_DATA_SHA256" "$tarball" "$tarball" "${url##*/}"; then + echo "[WARN] Removing PCGR/CPSR archive that failed its checksum; downloading it again." + rm -f "$tarball" || return 1 + fi + [ -f "$tarball" ] || fetch "$url" "$tarball" sha256 "$PCGR_DATA_SHA256" || return 1 + rm -rf "${dir}"/.extract.* + tmp=$(mktemp -d "${dir}/.extract.XXXXXX") || return 1 + if ! tar xzf "$tarball" -C "$tmp" || [ ! -d "${tmp}/data/grch38" ]; then + rm -rf "$tmp" + echo "ERROR: could not unpack ${tarball} into PCGR data/grch38." >&2 + return 1 + fi + # An incomplete destination is never mistaken for a completed bundle. + if [ -e "${dir}/${PCGR_DATA_BUNDLE}" ]; then + rm -rf "$tmp" + echo "ERROR: ${dir}/${PCGR_DATA_BUNDLE} exists without data; remove it and retry." >&2 + return 1 + fi + if ! mv "$tmp" "${dir}/${PCGR_DATA_BUNDLE}"; then + rm -rf "$tmp" + return 1 + fi + rm -f "$tarball" + echo "[OK] PCGR/CPSR ${PCGR_DATA_BUNDLE} bundle installed." +} + # lock_acquire DIR / lock_release DIR: a lock shared by concurrent runs. The # lock is a directory holding the owner's PID. It is stale when that PID is # gone (a killed run), or when it has no PID a minute after it was made (a run diff --git a/scripts/setup.sh b/scripts/setup.sh index 326dfeddd..20d0fb79e 100755 --- a/scripts/setup.sh +++ b/scripts/setup.sh @@ -1,6 +1,8 @@ #!/usr/bin/env bash # setup.sh — One-stop setup: download references, pull Docker images, validate # Usage: ./scripts/setup.sh +# ./scripts/setup.sh --no-clinical-data +# ./scripts/setup.sh --clinical-data # ./scripts/setup.sh --pull-only pull the Docker images and exit # ./scripts/setup.sh --og put OwnGenome's og binaries in bin/.og # (Linux, and macOS for a live-site host): @@ -82,8 +84,8 @@ # checkout of pgsc_calc and the Nextflow plugin it needs, so step 25 and # the PRS process run without the network # -# The VEP cache (~26 GB, --vep-cache) and PCGR ref data (~7 GB) are -# downloaded separately because they take a long time. +# 7. VEP 116, PCGR/CPSR data and VEP 115 (~57 GB download, ~65 GB unpacked). +# --no-clinical-data skips them; --clinical-data installs only these three. # # Every download goes through fetch (scripts/lib/common.sh): it is written to # .part, checked, and only then renamed, so a file that exists is whole. @@ -99,6 +101,9 @@ # existing samples. set -euo pipefail +# Keep an explicit checksum override for fixture tests and verified mirrors. +PCGR_SHA_OVERRIDE=${PCGR_DATA_SHA256:-} + # The usage text below names data releases, so read them before it prints. # shellcheck source=../versions.env . "$(dirname "$0")/../versions.env" @@ -107,12 +112,19 @@ PULL_ONLY=false REFRESH="" SAMPLE_QC_ONLY=false OPT_IN="" +CLINICAL_DATA=true case "${1:-}" in --og) exec "$(dirname "$0")/../rust/og.sh" install ;; --cyrius|--parascopy-data|--kir-data|--ancestry-panel|--ancestry-fine-panel|--yleaf-data|--consumer-sites|--ancient-panel|--viral-data|--masked-reference|--exomiser-data|--chip-data|--strchive-data|--megane-kmers|--mocha-data|--locityper-data|--vep-cache|--annotsv-data|--archaic) OPT_IN=${1#--} shift ;; + --clinical-data) + OPT_IN=clinical-data + shift ;; + --no-clinical-data) + CLINICAL_DATA=false + shift ;; --pull-only) PULL_ONLY=true shift ;; @@ -132,6 +144,8 @@ esac GENOME_DIR=${1:-${GENOME_DIR:-""}} if [ -z "$GENOME_DIR" ] && ! $PULL_ONLY; then echo "Usage: $0 " + echo " $0 --no-clinical-data " + echo " $0 --clinical-data " echo " $0 --pull-only" echo " $0 --og" echo " $0 --refresh clinvar " @@ -140,6 +154,8 @@ if [ -z "$GENOME_DIR" ] && ! $PULL_ONLY; then echo "" echo " Where to store reference data and sample outputs." echo " Needs at least 500 GB free space per sample." + echo " --no-clinical-data Skip both VEP caches and the PCGR/CPSR bundle." + echo " --clinical-data Install only those three downloads for an existing setup." echo " --pull-only Pull the Docker images listed in versions.env and exit." echo " --og Put OwnGenome's og binaries in bin/.og, for Linux and for a macOS live-site host" echo " (built from rust/, or a release's) and exit." @@ -210,6 +226,7 @@ export GENOME_DIR # Image tags, data versions and the helpers (run_in, fetch, pipeline_images) # shellcheck source=lib/common.sh . "$(dirname "$0")/lib/common.sh" +if [ -n "$PCGR_SHA_OVERRIDE" ]; then PCGR_DATA_SHA256=$PCGR_SHA_OVERRIDE; fi # NCBI's GRCh38 analysis set without ALT contigs: chr1-22, X, Y, M, the # unplaced and unlocalized scaffolds and chrEBV, 195 sequences. NCBI lists the @@ -2065,6 +2082,7 @@ install_vep_cache_opt_in() { echo "[OK] VEP ${VEP_CACHE_RELEASE} cache already present: ${dir}/homo_sapiens/${VEP_CACHE_RELEASE}_GRCh38" return 0 fi + clinical_disk_ok "$dir" 56 "VEP ${VEP_CACHE_RELEASE} cache" "${dir}/$(basename "$(vep_cache_url "$VEP_CACHE_RELEASE")")" || return 1 echo "Installing the VEP ${VEP_CACHE_RELEASE} cache: ~26 GB download, ~30 GB unpacked, both on disk until it ends." if ! install_vep_cache "$dir" "$VEP_CACHE_RELEASE"; then echo "[WARN] Could not install the VEP ${VEP_CACHE_RELEASE} cache. Run this command again: the download resumes." @@ -2073,6 +2091,40 @@ install_vep_cache_opt_in() { echo "[OK] VEP ${VEP_CACHE_RELEASE} cache: ${dir}/homo_sapiens/${VEP_CACHE_RELEASE}_GRCh38" } +# Each missing item gets its own disk check. A skipped or failed item must +# not prevent the smaller bundle or the other cache from installing; it makes +# the return status 1. +install_clinical_data() { + local dir="${GENOME_DIR}/vep_cache" rel peak missing=0 + echo "Clinical data: about 57 GB to download, 65 GB unpacked; allow 1 to 3 hours." + for rel in "$VEP_CACHE_RELEASE" bundle "$PCGR_VEP_CACHE_RELEASE"; do + if [ "$rel" = bundle ]; then + [ ! -d "${GENOME_DIR}/pcgr_data/${PCGR_DATA_BUNDLE}/data" ] || { echo "[OK] PCGR/CPSR bundle already present."; continue; } + if clinical_disk_ok "${GENOME_DIR}/pcgr_data" 15 'PCGR/CPSR bundle' "${GENOME_DIR}/pcgr_data/pcgr_ref_data.${PCGR_DATA_BUNDLE}.grch38.tgz"; then + install_pcgr_bundle "${GENOME_DIR}/pcgr_data" || { missing=1; echo "[WARN] Could not install PCGR/CPSR. Step 17 needs it; run: ./scripts/setup.sh --clinical-data \"${GENOME_DIR}\""; } + else + missing=1 + fi + else + [ ! -f "${dir}/homo_sapiens/${rel}_GRCh38/info.txt" ] || { echo "[OK] VEP ${rel} cache already present."; continue; } + peak=56 + [ "$rel" != "$PCGR_VEP_CACHE_RELEASE" ] || peak=52 + if clinical_disk_ok "$dir" "$peak" "VEP ${rel} cache" "${dir}/$(basename "$(vep_cache_url "$rel")")"; then + if ! install_vep_cache "$dir" "$rel"; then + missing=1 + echo "[WARN] Could not install VEP ${rel}. Run: ./scripts/setup.sh --clinical-data \"${GENOME_DIR}\" (interrupted downloads resume)." + if [ "$rel" = "$VEP_CACHE_RELEASE" ]; then + echo " To retry only this cache: ./scripts/setup.sh --vep-cache \"${GENOME_DIR}\"" + fi + fi + else + missing=1 + fi + fi + done + return "$missing" +} + case "$OPT_IN" in ancestry-panel) install_ancestry_panel; exit $? ;; ancestry-fine-panel) install_ancestry_fine_panel; exit $? ;; @@ -2090,6 +2142,7 @@ case "$OPT_IN" in megane-kmers) ensure_image MEGANE_IMAGE 50 && install_megane_kmers; exit $? ;; mocha-data) install_mocha_data; exit $? ;; locityper-data) install_locityper_data; exit $? ;; + clinical-data) install_clinical_data; exit $? ;; vep-cache) install_vep_cache_opt_in; exit $? ;; annotsv-data) install_annotsv_data; exit $? ;; archaic) install_archaic; exit $? ;; @@ -2459,47 +2512,15 @@ echo "=== Phase 5: AnnotSV annotation data (~5.3 GB + ~1.7 GB download, ~20 GB u install_annotsv_data || true ############################################################################### -# Phase 6: Optional Downloads (instructions only) +# Phase 6: Clinical reference data (default) ############################################################################### echo "" -echo "=== Phase 6: Optional Downloads (manual) ===" -echo "" -echo "The following are only needed for specific steps and are large downloads:" -echo "" - -VEPDIR="${GENOME_DIR}/vep_cache" -if [ -f "${VEPDIR}/homo_sapiens/${VEP_CACHE_RELEASE}_GRCh38/info.txt" ]; then - echo "[OK] VEP ${VEP_CACHE_RELEASE} cache already present." -else - echo "[SKIP] VEP ${VEP_CACHE_RELEASE} cache (~26 GB) — without it a run skips VEP, vcfanno, the clinical filter and slivar" - echo " Install it before the first run (downloads, checks and unpacks it; ~30 GB unpacked):" - echo " ./scripts/setup.sh --vep-cache ${GENOME_DIR}" -fi -echo "" - -PCGRDIR="${GENOME_DIR}/pcgr_data" -if [ -d "${PCGRDIR}/${PCGR_DATA_BUNDLE}/data" ]; then - echo "[OK] PCGR/CPSR ref data bundle already present." -else - echo "[SKIP] PCGR/CPSR ref data (~7 GB download) — needed for step 17 (cancer predisposition)" - echo " Download:" - echo " mkdir -p ${PCGRDIR} && cd ${PCGRDIR}" - echo " curl -fL -C - -O https://insilico.hpc.uio.no/pcgr/pcgr_ref_data.${PCGR_DATA_BUNDLE}.grch38.tgz" - echo " tar xzf pcgr_ref_data.${PCGR_DATA_BUNDLE}.grch38.tgz" - echo " mkdir -p ${PCGR_DATA_BUNDLE} && mv data/ ${PCGR_DATA_BUNDLE}/" -fi -echo "" - -# CPSR runs the VEP release inside the PCGR image, not the one step 13 uses. -if [ -f "${VEPDIR}/homo_sapiens/${PCGR_VEP_CACHE_RELEASE}_GRCh38/info.txt" ]; then - echo "[OK] VEP ${PCGR_VEP_CACHE_RELEASE} cache (for CPSR) already present." +echo "=== Phase 6: Clinical reference data ===" +if $CLINICAL_DATA; then + install_clinical_data || true else - echo "[SKIP] VEP ${PCGR_VEP_CACHE_RELEASE} cache (~24 GB) — needed for step 17 (CPSR runs VEP ${PCGR_VEP_CACHE_RELEASE} inside the PCGR image)" - echo " This is separate from the release-${VEP_CACHE_RELEASE} cache used by step 13. Both coexist in vep_cache/." - echo " Download:" - echo " mkdir -p ${VEPDIR}" - echo " curl -fL -C - -o ${VEPDIR}/$(basename "$(vep_cache_url "$PCGR_VEP_CACHE_RELEASE")") $(vep_cache_url "$PCGR_VEP_CACHE_RELEASE")" - echo " tar xzf ${VEPDIR}/$(basename "$(vep_cache_url "$PCGR_VEP_CACHE_RELEASE")") -C ${VEPDIR}" + echo "[SKIP] Clinical data (--no-clinical-data): steps 13, 30, 23, 31 and 17 need the caches or bundle." + echo " Install later: ./scripts/setup.sh --clinical-data \"${GENOME_DIR}\"" fi ############################################################################### @@ -2520,4 +2541,4 @@ echo " 2. Run: export GENOME_DIR=${GENOME_DIR}" echo " 3. Run: ./scripts/validate-setup.sh " echo " 4. Run: ./scripts/run-all.sh " echo "" -echo "For optional VEP and CPSR setup, see the instructions above." +echo "Review any warnings above; use --clinical-data to retry missing clinical downloads." diff --git a/scripts/validate-setup.sh b/scripts/validate-setup.sh index e8c19b6e4..60beff37c 100755 --- a/scripts/validate-setup.sh +++ b/scripts/validate-setup.sh @@ -588,13 +588,14 @@ else echo " Install it: ./scripts/setup.sh --sample-qc-data ${GENOME_DIR}" fi - # --- VEP cache of the VEP image's release, for step 13 (optional) --- + # --- Clinical data installed by default by setup.sh --- VEP_DIR="${GENOME_DIR}/vep_cache/homo_sapiens" if [ -f "${VEP_DIR}/${VEP_CACHE_RELEASE}_GRCh38/info.txt" ]; then pass "VEP ${VEP_CACHE_RELEASE} cache (step 13): present" else warn "VEP ${VEP_CACHE_RELEASE} cache not found at: ${VEP_DIR}/${VEP_CACHE_RELEASE}_GRCh38" - echo " Required for step 13 (VEP annotation), which downloads it (~26 GB) the first time it runs." + echo " Missing cache skips steps 13 (VEP), 30 (vcfanno), 23 (clinical filter) and 31 (slivar)." + echo " Install it: ./scripts/setup.sh --clinical-data \"${GENOME_DIR}\"" echo " See docs/00-reference-setup.md for full instructions." fi @@ -604,7 +605,7 @@ else else warn "VEP ${PCGR_VEP_CACHE_RELEASE} cache not found at: ${VEP_DIR}/${PCGR_VEP_CACHE_RELEASE}_GRCh38" echo " Required for step 17 (CPSR). ${PCGR_IMAGE} needs VEP ${PCGR_VEP_CACHE_RELEASE}, separate from step 13's VEP ${VEP_CACHE_RELEASE}." - echo " curl -fL -C - -O $(vep_cache_url "$PCGR_VEP_CACHE_RELEASE")" + echo " Missing cache skips step 17. Install it: ./scripts/setup.sh --clinical-data \"${GENOME_DIR}\"" echo " See docs/17-cpsr.md for full instructions." fi @@ -614,10 +615,8 @@ else pass "PCGR/CPSR ref data bundle: present" else warn "PCGR ref data bundle not found at: ${PCGR_DIR}" - echo " Required for step 17 (CPSR cancer predisposition). Download ~7 GB:" - echo " cd ${GENOME_DIR}/pcgr_data" - echo " curl -fL -C - -O https://insilico.hpc.uio.no/pcgr/pcgr_ref_data.${PCGR_DATA_BUNDLE}.grch38.tgz" - echo " tar xzf pcgr_ref_data.${PCGR_DATA_BUNDLE}.grch38.tgz && mkdir -p ${PCGR_DATA_BUNDLE} && mv data/ ${PCGR_DATA_BUNDLE}/" + echo " Missing bundle skips step 17 (CPSR cancer predisposition)." + echo " Install it: ./scripts/setup.sh --clinical-data \"${GENOME_DIR}\"" echo " See docs/17-cpsr.md for full instructions." fi diff --git a/tests/fake-docker/base-setup.sh b/tests/fake-docker/base-setup.sh index d373d9df9..82c7ee9dd 100755 --- a/tests/fake-docker/base-setup.sh +++ b/tests/fake-docker/base-setup.sh @@ -4,6 +4,7 @@ # finish with exit 0. # shellcheck source=../../scripts/ci/fake-docker/lib.sh . "${REPO_ROOT:?}/scripts/ci/fake-docker/lib.sh" +seed_clinical_downloads G="${CASE_WORK}/genome" mkdir -p "$G" @@ -13,7 +14,9 @@ output_lacks setup 'unbound variable' output_has setup 'Setup complete!' for f in reference/GRCh38_no_alt_analysis_set.fasta reference/GRCh38_no_alt_analysis_set.fasta.fai \ - clinvar/clinvar.vcf.gz clinvar/clinvar.vcf.gz.tbi; do + clinvar/clinvar.vcf.gz clinvar/clinvar.vcf.gz.tbi \ + vep_cache/homo_sapiens/116_GRCh38/info.txt vep_cache/homo_sapiens/115_GRCh38/info.txt \ + pcgr_data/20260620/data/grch38/README; do [ -s "${G}/${f}" ] || fail "setup.sh did not download ${f}" done grep -q '^image inspect' "$FAKE_DOCKER_LOG" || fail "setup.sh never checked a docker image" diff --git a/tests/fake-docker/setup-clinical-default.sh b/tests/fake-docker/setup-clinical-default.sh new file mode 100644 index 000000000..4c6b49e46 --- /dev/null +++ b/tests/fake-docker/setup-clinical-default.sh @@ -0,0 +1,123 @@ +#!/usr/bin/env bash +# Default clinical downloads, explicit opt-out, disk gates and corrupt input. +# shellcheck source=../../scripts/ci/fake-docker/lib.sh +. "${REPO_ROOT:?}/scripts/ci/fake-docker/lib.sh" +seed_clinical_downloads +export FETCH_TRIES=1 FETCH_WAIT=0 +G="${CASE_WORK}/without" +run_expect 0 without "${SCRIPTS}/setup.sh" --no-clinical-data "$G" +output_has without 'Setup complete!' +if grep -E '^(wget|curl).*homo_sapiens_vep_|^(wget|curl).*pcgr_ref_data' "$FAKE_DOCKER_LOG"; then + fail '--no-clinical-data downloaded clinical data' +fi + +G="${CASE_WORK}/clinical" +run_expect 0 clinical "${SCRIPTS}/setup.sh" --clinical-data "$G" +for rel in 116 115; do + [ -s "${G}/vep_cache/homo_sapiens/${rel}_GRCh38/info.txt" ] || fail "VEP ${rel} missing" +done +[ -s "${G}/pcgr_data/20260620/data/grch38/README" ] || fail 'PCGR bundle missing' +output_lacks clinical 'Phase 1:|Docker Images' +# Installed clinical data turns the three default launch-plan rows on. +seed_reference "$G" +seed_clinvar "$G" +seed_sample "$G" sample1 +use_output_hook +GENOME_DIR="$G" SKIP_VALIDATION=true run_expect 0 plan "${SCRIPTS}/run-all.sh" sample1 male +output_has plan '^ 13 VEP +runs$' +output_has plan '^ 23 Clinical filter +runs$' +output_has plan '^ 17 CPSR +runs$' + +# A fake df that reports 40 GiB free. +mkdir -p "${CASE_WORK}/low-bin" +cat > "${CASE_WORK}/low-bin/df" <<'DF' +#!/usr/bin/env bash +echo 'Filesystem 1024-blocks Used Available Capacity Mounted on' +echo 'fakefs 41943040 0 41943040 0% /' +DF +chmod +x "${CASE_WORK}/low-bin/df" + +# A second install does no network work, even with too little space. +cp "$FAKE_DOCKER_LOG" "${CASE_WORK}/installed.log" +PATH="${CASE_WORK}/low-bin:${PATH}" run_expect 0 again "${SCRIPTS}/setup.sh" --clinical-data "$G" +cmp "$FAKE_DOCKER_LOG" "${CASE_WORK}/installed.log" || fail 'installed clinical data downloaded again' + +# A skipped item makes --clinical-data exit 1; the other items still install. +G="${CASE_WORK}/low" +PATH="${CASE_WORK}/low-bin:${PATH}" run_expect 1 low "${SCRIPTS}/setup.sh" --clinical-data "$G" +output_has low '56 GB.*5 GB margin' +output_has low '52 GB.*5 GB margin' +[ ! -e "${G}/vep_cache/homo_sapiens/116_GRCh38" ] || fail 'low-space VEP installed' +[ -s "${G}/pcgr_data/20260620/data/grch38/README" ] || fail 'small bundle skipped with 40 GB free' + +# Model an allocated 22 GiB partial download without writing 22 GiB in CI. +cat > "${CASE_WORK}/low-bin/du" <<'DU' +#!/usr/bin/env bash +case "$2" in + */resume/vep_cache/*.part) printf '23068672\t%s\n' "$2" ;; + *) /usr/bin/du "$@" ;; +esac +DU +chmod +x "${CASE_WORK}/low-bin/du" + +# A partial download counts as space already taken: 40 GiB free plus a +# 22 GiB .part covers VEP 116's 56 GB peak and margin, so it resumes. +G="${CASE_WORK}/resume" +mkdir -p "${G}/vep_cache" +dd if=/dev/null of="${G}/vep_cache/homo_sapiens_vep_116_GRCh38.tar.gz.part" bs=1048576 seek=22528 2>/dev/null +PATH="${CASE_WORK}/low-bin:${PATH}" run_expect 1 resume "${SCRIPTS}/setup.sh" --clinical-data "$G" +output_lacks resume 'Skipping VEP 116' +[ -s "${G}/vep_cache/homo_sapiens/116_GRCh38/info.txt" ] || fail 'partial VEP 116 download not resumed with low space' +[ ! -e "${G}/vep_cache/homo_sapiens_vep_116_GRCh38.tar.gz.part" ] || fail 'resumed VEP 116 left its .part' + +# An install stopped during extraction keeps its checked archive: the next +# run unpacks it without downloading again and removes the old staging tree. +G="${CASE_WORK}/extract" +mkdir -p "${G}/vep_cache/.extract.old" "${G}/pcgr_data/.extract.old" +cp "${FAKE_DOWNLOAD_DIR}/homo_sapiens_vep_116_GRCh38.tar.gz" "${G}/vep_cache/" +cp "${FAKE_DOWNLOAD_DIR}/pcgr_ref_data.20260620.grch38.tgz" "${G}/pcgr_data/" +# Model space held by each stale tree. Only cleanup can make its preflight pass. +mkdir -p "${CASE_WORK}/staging-bin" +cat > "${CASE_WORK}/staging-bin/df" <<'DF' +#!/usr/bin/env bash +echo 'Filesystem 1024-blocks Used Available Capacity Mounted on' +if [ -d "${@: -1}/.extract.old" ]; then + echo 'fakefs 2147483648 2141192192 6291456 99% /' +else + echo 'fakefs 2147483648 0 2147483648 0% /' +fi +DF +chmod +x "${CASE_WORK}/staging-bin/df" +PATH="${CASE_WORK}/staging-bin:${PATH}" FAKE_DOWNLOAD_FAIL='vep_116|pcgr_ref_data' run_expect 0 extract "${SCRIPTS}/setup.sh" --clinical-data "$G" +[ -s "${G}/vep_cache/homo_sapiens/116_GRCh38/info.txt" ] || fail 'checked VEP 116 archive downloaded again' +[ -s "${G}/pcgr_data/20260620/data/grch38/README" ] || fail 'checked PCGR archive downloaded again' +[ ! -e "${G}/vep_cache/.extract.old" ] || fail 'stale VEP staging tree kept' +[ ! -e "${G}/pcgr_data/.extract.old" ] || fail 'stale PCGR staging tree kept' + +# A retained archive with a valid layout but wrong data must be replaced. +G="${CASE_WORK}/retained-bad" +mkdir -p "${G}/pcgr_data" "${CASE_WORK}/altered/data/grch38" +printf 'modified reference\n' > "${CASE_WORK}/altered/data/grch38/README" +tar -czf "${G}/pcgr_data/pcgr_ref_data.20260620.grch38.tgz" -C "${CASE_WORK}/altered" data +run_expect 0 retained_bad "${SCRIPTS}/setup.sh" --clinical-data "$G" +cmp "${G}/pcgr_data/20260620/data/grch38/README" "${CASE_WORK}/bundle/data/grch38/README" || fail 'unverified retained PCGR archive published' +[ ! -e "${G}/pcgr_data/pcgr_ref_data.20260620.grch38.tgz" ] || fail 'replaced archive kept' + +# Sparse holes occupy no disk: they must not grant space for an installation. +G="${CASE_WORK}/sparse" +mkdir -p "${G}/vep_cache" +dd if=/dev/null of="${G}/vep_cache/homo_sapiens_vep_116_GRCh38.tar.gz.part" bs=1048576 seek=22528 2>/dev/null +PATH="${CASE_WORK}/low-bin:${PATH}" run_expect 1 sparse "${SCRIPTS}/setup.sh" --clinical-data "$G" +output_has sparse 'Skipping VEP 116' +[ ! -e "${G}/vep_cache/homo_sapiens/116_GRCh38" ] || fail 'sparse holes counted as allocated space' + +# The digest belongs to the original fixture. Serve a different valid tarball. +printf 'corrupt\n' >> "${FAKE_DOWNLOAD_DIR}/pcgr_ref_data.20260620.grch38.tgz" +G="${CASE_WORK}/corrupt" +run_expect 1 corrupt "${SCRIPTS}/setup.sh" --clinical-data "$G" +output_has corrupt 'sha256 checksum.*expected' +output_has corrupt '\[WARN\].*PCGR' +[ ! -e "${G}/pcgr_data/20260620" ] || fail 'corrupt bundle published' +[ ! -e "${G}/pcgr_data/pcgr_ref_data.20260620.grch38.tgz.part" ] || fail 'bad bundle kept' +[ -s "${G}/vep_cache/homo_sapiens/115_GRCh38/info.txt" ] || fail 'bundle failure stopped remaining install' + diff --git a/versions.env b/versions.env index 304024130..c305cb838 100644 --- a/versions.env +++ b/versions.env @@ -190,6 +190,14 @@ VEP_CACHE_RELEASE="116" # with PCGR_IMAGE. PCGR_VEP_CACHE_RELEASE="115" PCGR_DATA_BUNDLE="20260620" +# Exact archive size from the server; SHA-256 is measured from the full download +# (the server publishes no checksum). PCGR_DATA_MEASURED_FOR names the PCGR_IMAGE +# tag and PCGR_DATA_BUNDLE they were measured for: scripts/ci/check-coupled-versions.sh +# fails when the image or the bundle moves without it, so a bump downloads the +# new bundle and records its size and digest in the same change. +PCGR_DATA_MEASURED_FOR="2.3.2 20260620" +PCGR_DATA_BYTES="7664082467" +PCGR_DATA_SHA256="f9b8149e37c7e63bd7d877713cdb11a1a609c492b3f1a055fb7e90cd80273e54" # pypgx-bundle git tag (step 32): must equal the version of PYPGX_IMAGE. PYPGX_BUNDLE_VERSION="0.26.0" # Cyrius (step 21, opt-in; legacy: no automated bumps) is installed by