Skip to content

fix(ancestry): keep failed calls out of the ancestry steps - #448

Open
geiresx wants to merge 4 commits into
GeiserX:mainfrom
geiresx:pgp/i411-gvcf-pass-gate
Open

geiresx wants to merge 4 commits into
GeiserX:mainfrom
geiresx:pgp/i411-gvcf-pass-gate

Conversation

@geiresx

@geiresx geiresx commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

Failed calls could reach fine ancestry, ancient ancestry and PRS as usable genotypes. A LowQual or RefCall record counted the same as a passing call. Fine ancestry then rewrote every record with FILTER ., so ADMIXTURE, Beagle and RFMix could not see the failure.

The fix

Steps 25, 26b and 44 now keep only FILTER=PASS or . records, in both bash and Nextflow. LowQual and RefCall records stay missing. Unfiltered reference blocks still count as reference.

Step 44 also requires GQ >= 20, reference blocks included. Low or missing GQ stays missing. Decision and reason: step 44 projects one person with smartpca lsqproject, so a doubtful 0/0 pulls the position, while a missing site only drops out. GQ 20 is the same floor step 7 already uses for reference blocks. Steps 25 and 26b keep only the FILTER rule, as the plan asked. A gVCF with no FORMAT/GQ header line now stops step 44 with bcftools' own error. Step 3's DeepVariant gVCF always defines GQ.

What changes in the outputs: fine ancestry's genotype and reference counts can fall, so its VCF rule of one reference call in five can refuse an input it used to accept. Ancient ancestry's sites_used, PCA position and nearest groups can change. PRS can match fewer sites. Output names and formats stay the same.

Files beyond the step scripts, modules and their docs: docs/25-prs.md (it said a gVCF site is missing only when not covered), and scripts/ci/e2e-case-paths.tsv and scripts/ci/e2e-case-times.tsv (one row each to register the new case).

How it was checked

  • Red first: a1cb411 adds tests/e2e/gvcf-pass-gate.sh against the old callers: https://github.com/geiresx/OwnGenome/blob/a1cb4117/tests/e2e/gvcf-pass-gate.sh . With pinned bcftools 1.24 and synthetic data, all ten extraction branches fail there. Removing a branch in a scratch copy also fails the branch-count check. This red state was run on the box, not in CI.
  • Green on the box: all 11 checks pass. LowQual and RefCall disappear. PASS calls, unfiltered calls and passing reference blocks stay. Step 44 drops GQ 19, zero-GQ blocks and missing GQ, and keeps GQ 20.
  • Step 44 headroom: on the last green fork E2E run, ancient-similarity-1 used 53,324 of 53,790 panel sites (99.1%). The case needs 80%, so the GQ gate has wide margin on the 30x fixture: https://github.com/geiresx/OwnGenome/actions/runs/43712029449
  • ShellCheck, the E2E scope and shard checks and their planted-fault self-tests pass.
  • Local oracle: all 427 items in 90 ancestry and PRS cases agree. The parity self-test passes all 15 checks.
  • No Rust or Python changes. No real-genome run is needed for this fix.
  • Upstream fast checks run on the head of this PR (Guard with ci-ok, Rust, Lint, Nextflow, Docs, Container Test, Personal data).
  • E2E: the earlier runs on 10cd071 were cancelled, on GeiserX on purpose and on the fork by the queue. No test failed in them, and no CI run has run gvcf-pass-gate yet. The landing E2E run on the final head covers gvcf-pass-gate, ancient-similarity-1, the prs cases and ancestry-fine-grained-1.

Part of #411. Claims settled: lin-50, lin-51.

Bead: lin-50
Bead: lin-51

The synthetic bcftools case fails all ten branch assertions before the filter fix.
Bead: lin-50
Bead: lin-51

Keep PASS and unfiltered records in both callers of steps 25, 26b and 44.
Require GQ >= 20 in ancient ancestry, including reference blocks.
Document changes to counts and keep both output contracts unchanged.
@coderabbitai

coderabbitai Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 30 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 9078a1d and 8ac02ca.

Files ignored due to path filters: 2

  • scripts/ci/e2e-case-paths.tsv is excluded by !**/*.tsv
  • scripts/ci/e2e-case-times.tsv is excluded by !**/*.tsv
Files selected for processing: 11

  • CHANGELOG.md
  • docs/25-prs.md
  • docs/26b-ancestry-fine.md
  • docs/44-ancient.md
  • modules/local/ancestry_fine/main.nf
  • modules/local/ancient/main.nf
  • modules/local/prs/main.nf
  • scripts/25-prs.sh
  • scripts/26b-ancestry-fine.sh
  • scripts/44-ancient.sh
  • tests/e2e/gvcf-pass-gate.sh
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants