Skip to content

test(family,y,setup): check family, Y and setup tools in cargo test and the chip step in shell - #453

Open
geiresx wants to merge 335 commits into
GeiserX:mainfrom
geiresx:pgp/port-tests-family-y-dev
Open

geiresx wants to merge 335 commits into
GeiserX:mainfrom
geiresx:pgp/port-tests-family-y-dev

Conversation

@geiresx

@geiresx geiresx commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

Part of #411. Settles owner decision D1 for these nine files: the Python goes away for 1.0.0.

The problem

Nine Python test files still guarded code that is no longer Python. Seven ran Rust tools (og family, og y, og setup, og run delta) as subprocesses. One checked the parameters page against nextflow_schema.json. One drove scripts/chip-to-vcf.sh with the real bcftools and Picard images. To ship 1.0.0 without owned Python, each check needs a home outside Python, with every case kept.

The fix

Each file moves, with the same cases and the same expectations:

Removed Now
tests/test_family_segments.py rust/crates/og-steps/tests/family_segments.rs
tests/test_ibd_report.py rust/crates/og-steps/tests/family_ibd.rs
tests/test_y_private_snps.py rust/crates/og-reports/tests/y_private_snps.rs
tests/test_y_tree_place.py rust/crates/og-reports/tests/y_tree_place.rs
tests/test_y_private_mask_install.py rust/crates/og-setup/tests/port_tests_family_y_dev_mask_install.rs (runs the real setup.sh with a fake curl)
tests/test_og_setup_zip.py rust/crates/og-setup/tests/port_tests_family_y_dev_zip.rs
tests/test_run_delta.py rust/crates/og-runlog/tests/delta_runs.rs
tests/test_parameters_doc.py og dev params-doc (rust/crates/og-dev/src/params_doc.rs); its unit tests run it on the checkout, so Guard runs it in cargo test
tests/test_chip_to_vcf_mt.py tests/test_chip_to_vcf_mt.sh (it drives docker images, so it stays a shell test; the unit-tests loop finds it by glob)

og dev params-doc [--repo DIR] replaces the not ported yet stub. It exits 0 when every schema parameter has exactly one row in docs/parameters.md and every row names a parameter (or pedigree, while main.nf reads it). It exits 1 and names the parameters otherwise, and 2 on an unreadable file. An unknown option such as --og-scaffold-probe is refused before any file is read.

No tool code changes, and no output, option or file name changes. docs/rust.md drops params-doc from the list of stubs and says that cargo test runs it on the checkout.

Decisions, each the smallest change that keeps main true:

  • The zips of the setup test are written by the test itself (a small writer of the layout Python's zipfile writes, ZIP64 included), not committed as fixtures. A fixture would need a path outside this package's files, and the 2.3 MB member is not small. The trees that extractall and the old inline map program wrote are spelled out by content. I checked them once against Python's zipfile locally.
  • The cases that read a tool's stderr (og family ibd, og setup unzip|genetic-map) run the tool in a child process: the test binary runs itself again with one environment variable set. Those tools print to stderr directly, and this keeps the tests free of new crates.
  • The mask install test sits in og-setup, beside the other setup.sh test, since it runs setup.sh.
  • No og-pedigree test file: none of the nine checks tests og-pedigree on its own.
  • No CHANGELOG line: nothing a user sees changes.
  • docs/rust.md is outside this package's file list. I changed its two lines anyway: without the change it lists params-doc as a stub, which is no longer true. Other open ports edit the same stub list, so whichever lands second resolves that line by keeping both removals.
  • The Y private SNP test writes its gzipped VCF and its callable mask as one plain gzip member, as Python's gzip.open did and as the 1000 Genomes mask that setup.sh installs is. It uses stored deflate blocks and a CRC-32 written in the test, so og-reports needs no new crate.

How it was checked

  • The eight cargo test targets pass: cargo test -p og-dev --lib params_doc, and --test family_segments, family_ibd, y_private_snps, y_tree_place, port_tests_family_y_dev_mask_install, port_tests_family_y_dev_zip, delta_runs. cargo fmt --check, cargo clippy --all-targets --all-features -D warnings on the five crates and cargo doc -D warnings on og-dev are clean.
  • Red controls. Commit fff6c47 plants one wrong expectation in each ported test, and 1edcadd takes them out again. With cargo test --no-fail-fast on fff6c47 every target went red: trio_duo_no_par_and_report, one_and_two_chromosomes, private_snps_gates_tables_and_mask, tree_place_and_tree_positions, setup_keeps_the_checked_chry_mask, unzip_writes_the_tree_extractall_writes, deltas_of_two_runs, and the_repository_page_lists_every_schema_parameter_once. A row planted in docs/parameters.md turned og dev params-doc red as well. Inside the tests, the dropped-row and planted-row controls of og dev params-doc, the one-byte-changed tree of the zip test and the SVG well-formedness check each show their check can fail. The CI run on fff6c47 was cancelled before it started: about 100 runs were queued, and a full Guard run only for the red control would have held the queue for hours. So that red run is the local one described here.
  • tests/test_chip_to_vcf_mt.sh reads every genotype with one awk, not cut piped into grep -q: under pipefail a SIGPIPE in that pipe could turn a failing check into a pass. The awk check exits 1 on a genotype with a trailing carriage return and on AA, and 0 on 0/1, 1, ./. and ., with both BSD awk and Ubuntu's mawk.
  • tests/test_chip_to_vcf_mt.sh with docker: 21 checks pass, the same 21 the Python made. Two broken copies of chip-to-vcf.sh turned it red: one that keeps the carriage return (1 check fails) and one that skips the hg19 to rCRS move of 23andMe's MT sites (4 checks fail).
  • A plain gzip writer that cannot fail would prove nothing, so I broke it once: with a wrong CRC-32 in the gzip trailer, cargo test -p og-reports --test y_private_snps goes red (og refuses the mask). With the right CRC it passes.
  • The personal-data scan passes on every changed file.
  • Upstream fast checks on head e33b9d4 are green: every job of Guard https://github.com/GeiserX/OwnGenome/actions/runs/43769657643 (its rust job runs the eight new cargo test targets and all nine params_doc unit tests; unit-tests runs tests/test_chip_to_vcf_mt.sh), Rust https://github.com/GeiserX/OwnGenome/actions/runs/43769657590, Lint https://github.com/GeiserX/OwnGenome/actions/runs/43769657678, Nextflow https://github.com/GeiserX/OwnGenome/actions/runs/43769657575, Docs https://github.com/GeiserX/OwnGenome/actions/runs/43769657635, Container Test https://github.com/GeiserX/OwnGenome/actions/runs/43769657638, Personal data https://github.com/GeiserX/OwnGenome/actions/runs/43769657682.
  • This PR adds and changes no e2e case and no code an e2e case runs, so it has no own e2e case to prove.

Out of scope, stale mentions of the removed files in files outside this package (comments and one doc line): docs/37-y-haplogroup.md:98, tests/fixtures/chip/myheritage.csv:2, rust/crates/og-reports/src/ytree.rs:975, tests/parity/cases.tsv:981, tests/parity/gen/family-3gen.sh:3, tests/parity/gen/ibd-cases.sh:3, tests/parity/gen/y-inputs.sh:4.

Beads: D1 (this package's share).

…aic output is in place

spans.tsv is now staged with the other outputs, and a chromosome's old
genotype file is removed last. A failed publication keeps it.
render_report.py (step 24, HTML_REPORT and generate-report.sh) now runs
og summary collect instead of the Python collector. generate-report.sh
renders the text report from the summary step 24 wrote when nothing it
reads has changed since, so a run collects each sample once. The folder
walk leaves out the Nextflow launch folder run-all.sh keeps in the sample
folder. collect_summary.py keeps a collect() that runs og, and only the
names other Python still imports.

Bead: pgp-bs9u.45
The tests that called the Python section readers now read the section og
summary collect writes for the same folder. Checks of a Python internal
(the G2 reader, the probable CYP2D6 rule, the poly-C tracts, the
imprinted regions, the repeat ranges table) move to og-summary's own
cargo tests. The demo maker writes step 26b, 49 and 05 tables without the
Python remnants. collect_summary.py imports re for half_call.

Bead: pgp-bs9u.45
With summary.json written by og summary collect, nothing imports
pgx_outside_calls.py, ancestry_fine_report.py, clip_check.py,
annotsv_postfilter.py, chip_report.py, traits_report.py or
genotype_rules_indels.py any more. They go, with test_clip_check.py and
test_genotype_rules_indels.py (their checks are og's cargo tests now).
The e2e cases that read a pinned record from them read og-summary's and
og-cn's constants. collect_summary.py is listed as a remnant for the
report, couple, index and live code until their own ports.

Bead: pgp-bs9u.45
The fake-docker run-all case now checks that generate-report.sh renders
from the summary step 24 wrote, that both reports carry one generated
time, and that a step output newer than summary.json makes it collect
again.

Bead: pgp-bs9u.45
og summary collect read groups b and c at once, and group c on a pool of
up to eight threads, so a run's peak memory was the sum of the sections
in flight. The sample VCF, the one input that takes time, stays on its
own thread; every other section is read one at a time meanwhile. On a
synthetic folder with every side table 50 times its demo size the peak
footprint fell from 119-130 MB to 105-112 MB, CPU time unchanged.

Bead: pgp-bs9u.45
…mported

og run delta makes the comparison and og summary collect writes New
since. The two run-delta checks of test_collect_summary.py 21b now run
og run delta; test_run_eta.py no longer checks the old command line.

Bead: pgp-bs9u.45
The HTML header puts the latest run after the generated time on the same
line, so the comparison with the text report took both.

Bead: pgp-bs9u.45
…n and count its acts as its page does

A parent's page listed a gated child's flagged-card count, which changes with
the cards the unopened results flag. family.json now gives attention null for a
child whose own page keeps the report closed (the family plugin and the demo
builder alike). /runs.json counts a child's results to act on from the summary
its results.json reads, also while summary.json is older than a finished step.
The server, the home page and the person page decide who is a child by one rule
(child_of). Adults and no-act children are unchanged.
…nd search they have without the gate

The event stream opens when live.js asks for it and its messages wait for the
gate's decision. A file the gate reads first keeps its answer whatever the
status, read with the options of the view that reads it (a replay's
frames.json with the default cache). person.js answers the charts and the
search as before the gate, so a child's open page says what search leaves out.
A guessed finding address releases to the run's page. The unused veil CSS goes.
…ead, on search and on the ticket

Live-mode request order on the real server, the parent's page whichever card
holds a child's results, the stale-summary home count, a 404 family.json, the
fetch cache mode, a search on a no-act child, a guessed finding address and a
mutant that keeps the opening ticket.
…ry summary private

generate-report.sh rendered from step 24's summary.json when no file was
newer, so a step output removed after the collect went unnoticed. A
subfolder newer than the summary (other than logs/) now means collect
again, and the fake-docker run-all case checks it.

collect_summary.collect() without OUT gave og a file in the shared
temporary folder, and og's rename over it dropped mkstemp's owner-only
mode. It now uses a private temporary folder.

Bead: pgp-bs9u.45
Main's tree is the train's, so the merge keeps this branch's tree.

Bead: pgp-bs9u.45
Removing a step folder after the collect changes only the sample folder's
own time, which the reports change too, so generate-report.sh rendered the
removed results from summary.json. It now also collects again when a file
a section names as its source is gone. The fake-docker run-all case checks
an unchanged folder, a newer output, a removed file and a removed folder,
each from a summary.json that is the newest file.

Bead: pgp-bs9u.45
Guard's rust job ran past its old 20-minute limit on a busy runner twice;
main now allows 30.

Bead: pgp-bs9u.45
The traits control reached the marker through traits_report, which this
branch deletes; test_traits_report.py holds MARKERS itself now.

Bead: pgp-bs9u.45
…le is gone

The summary records that it read logs/run_status.tsv, which the subfolder
test leaves out with logs/, and the tables in bin/ and assets/ are no
section's source. generate-report.sh now collects again when that run
status is gone, or when bin/, bin/.og/ or assets/ is newer than the
summary. The fake-docker run-all case checks both. The CHANGELOG says
the shared generated time holds only when the text report renders from
step 24's summary.

Bead: pgp-bs9u.45
Bead: lin-10
Bead: lin-13
Bead: lin-15
Bead: lin-16
Bead: lin-17
Bead: lic-41
…hey missed

A planted wrong line survived in cigar (no X op), x-ref-sites (spacing),
truth-genotypes (a 0/0 truth record), segments-check (the 19-site row) and
ystr-periodic (a half-periodic region). Each test now has the input that
catches it.

Bead: D1
The G2 reader beside the APOL1 rule applied no GQ floor, so a G2 call
or a reference call over the span at GQ 0 to 19 gave a count, and two
weak G2 copies read G2/G2. It now takes the D10 floor of the trait
sites: GQ below 20, or a reference call with a FILTER other than PASS,
. or RefCall, leaves G2 not read. GQ 20 and a missing GQ are read.

The rule-line status list in docs/42-traits.md now names low_gq.

Bead: trait-01
The case's own expectation took a GQ of . as below 20, while step 42
reads it. The header comments now name . as a readable FILTER.

Bead: D10
@geiresx

geiresx commented Nov 15, 2026

Copy link
Copy Markdown
Collaborator Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

… the landing train

# Conflicts:
#	CHANGELOG.md
#	docs/rust.md
An unknown or mistyped kind printed only the comphet and trim-pharmcat
lines, so the eight e2e kinds were missing from the error a user sees.
The usage error now prints the same kinds as --help, and a unit test
checks every kind is named.
…sults closed

An adult whose family.json had not answered or failed got a page with results
marked opened and a close button that did nothing. person.js now asks only the
gate (state.json's child rule). A person closed as another family's child gets
their buttons under that family's members, not in its chapter of children.
…ed answer, and where a partner's buttons sit
- append-n refuses an OUT that is SRC (by device and inode), which it
  would truncate before copying, and ends a SRC whose last line has no
  newline before it writes the new contig header.
- hba-tiles takes the read group from the ID: field of the first @rg
  line, wherever it sits, not from the second field.
- rfc1 --samd12 writes a CIGAR of the read's own length, so a unit that
  is not five bases long still gives a valid record.

The bytes the e2e cases use do not change: their pins still hold.
…landing train

# Conflicts:
#	CHANGELOG.md
#	tests/parity/golden/summary-collect/all-history/all-history.json
#	tests/parity/golden/summary-collect/all-nf/all-nf.json
#	tests/parity/golden/summary-collect/all/all.json
#	tests/parity/golden/summary-collect/c-full/full.json
#	tests/parity/golden/summary-collect/c-nf/nf.json
…otypes without a pipe

The callable mask setup.sh installs is a plain one-member gzip, not BGZF,
so the test now writes that shape (stored deflate blocks and a CRC-32 in
the test, no new crate). The genotype check of the chip shell test reads
the file with awk instead of cut piped into grep -q, so a SIGPIPE can no
longer turn a failing check into a pass.
… landing train

# Conflicts:
#	rust/crates/og-dev/src/fixture.rs
#	scripts/ci/e2e-case-paths.tsv
…he landing train

# Conflicts:
#	docs/rust.md
@geiresx
geiresx requested a review from GeiserX as a code owner November 16, 2026 00:39

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants