Repository navigation
Conversation
…aic output is in place spans.tsv is now staged with the other outputs, and a chromosome's old genotype file is removed last. A failed publication keeps it.
render_report.py (step 24, HTML_REPORT and generate-report.sh) now runs og summary collect instead of the Python collector. generate-report.sh renders the text report from the summary step 24 wrote when nothing it reads has changed since, so a run collects each sample once. The folder walk leaves out the Nextflow launch folder run-all.sh keeps in the sample folder. collect_summary.py keeps a collect() that runs og, and only the names other Python still imports. Bead: pgp-bs9u.45
# Conflicts: # bin/collect_summary.py
The tests that called the Python section readers now read the section og summary collect writes for the same folder. Checks of a Python internal (the G2 reader, the probable CYP2D6 rule, the poly-C tracts, the imprinted regions, the repeat ranges table) move to og-summary's own cargo tests. The demo maker writes step 26b, 49 and 05 tables without the Python remnants. collect_summary.py imports re for half_call. Bead: pgp-bs9u.45
With summary.json written by og summary collect, nothing imports pgx_outside_calls.py, ancestry_fine_report.py, clip_check.py, annotsv_postfilter.py, chip_report.py, traits_report.py or genotype_rules_indels.py any more. They go, with test_clip_check.py and test_genotype_rules_indels.py (their checks are og's cargo tests now). The e2e cases that read a pinned record from them read og-summary's and og-cn's constants. collect_summary.py is listed as a remnant for the report, couple, index and live code until their own ports. Bead: pgp-bs9u.45
The fake-docker run-all case now checks that generate-report.sh renders from the summary step 24 wrote, that both reports carry one generated time, and that a step output newer than summary.json makes it collect again. Bead: pgp-bs9u.45
og summary collect read groups b and c at once, and group c on a pool of up to eight threads, so a run's peak memory was the sum of the sections in flight. The sample VCF, the one input that takes time, stays on its own thread; every other section is read one at a time meanwhile. On a synthetic folder with every side table 50 times its demo size the peak footprint fell from 119-130 MB to 105-112 MB, CPU time unchanged. Bead: pgp-bs9u.45
…mported og run delta makes the comparison and og summary collect writes New since. The two run-delta checks of test_collect_summary.py 21b now run og run delta; test_run_eta.py no longer checks the old command line. Bead: pgp-bs9u.45
Bead: pgp-bs9u.45
The HTML header puts the latest run after the generated time on the same line, so the comparison with the text report took both. Bead: pgp-bs9u.45
Bead: pgp-bs9u.45
…n and count its acts as its page does A parent's page listed a gated child's flagged-card count, which changes with the cards the unopened results flag. family.json now gives attention null for a child whose own page keeps the report closed (the family plugin and the demo builder alike). /runs.json counts a child's results to act on from the summary its results.json reads, also while summary.json is older than a finished step. The server, the home page and the person page decide who is a child by one rule (child_of). Adults and no-act children are unchanged.
…nd search they have without the gate The event stream opens when live.js asks for it and its messages wait for the gate's decision. A file the gate reads first keeps its answer whatever the status, read with the options of the view that reads it (a replay's frames.json with the default cache). person.js answers the charts and the search as before the gate, so a child's open page says what search leaves out. A guessed finding address releases to the run's page. The unused veil CSS goes.
…ead, on search and on the ticket Live-mode request order on the real server, the parent's page whichever card holds a child's results, the stale-summary home count, a 404 family.json, the fetch cache mode, a search on a no-act child, a guessed finding address and a mutant that keeps the opening ticket.
…d what the page keeps as before
…ry summary private generate-report.sh rendered from step 24's summary.json when no file was newer, so a step output removed after the collect went unnoticed. A subfolder newer than the summary (other than logs/) now means collect again, and the fake-docker run-all case checks it. collect_summary.collect() without OUT gave og a file in the shared temporary folder, and og's rename over it dropped mkstemp's owner-only mode. It now uses a private temporary folder. Bead: pgp-bs9u.45
Main's tree is the train's, so the merge keeps this branch's tree. Bead: pgp-bs9u.45
Removing a step folder after the collect changes only the sample folder's own time, which the reports change too, so generate-report.sh rendered the removed results from summary.json. It now also collects again when a file a section names as its source is gone. The fake-docker run-all case checks an unchanged folder, a newer output, a removed file and a removed folder, each from a summary.json that is the newest file. Bead: pgp-bs9u.45
Bead: pgp-bs9u.45
Guard's rust job ran past its old 20-minute limit on a busy runner twice; main now allows 30. Bead: pgp-bs9u.45
The traits control reached the marker through traits_report, which this branch deletes; test_traits_report.py holds MARKERS itself now. Bead: pgp-bs9u.45
…le is gone The summary records that it read logs/run_status.tsv, which the subfolder test leaves out with logs/, and the tables in bin/ and assets/ are no section's source. generate-report.sh now collects again when that run status is gone, or when bin/, bin/.og/ or assets/ is newer than the summary. The fake-docker run-all case checks both. The CHANGELOG says the shared generated time holds only when the text report renders from step 24's summary. Bead: pgp-bs9u.45
Bead: lin-10 Bead: lin-13 Bead: lin-15 Bead: lin-16 Bead: lin-17 Bead: lic-41
# Conflicts: # CHANGELOG.md
…g dev fixture Bead: D1
…e Python Bead: D1
…hey missed A planted wrong line survived in cigar (no X op), x-ref-sites (spacing), truth-genotypes (a 0/0 truth record), segments-check (the 19-site row) and ystr-periodic (a half-periodic region). Each test now has the input that catches it. Bead: D1
The G2 reader beside the APOL1 rule applied no GQ floor, so a G2 call or a reference call over the span at GQ 0 to 19 gave a count, and two weak G2 copies read G2/G2. It now takes the D10 floor of the trait sites: GQ below 20, or a reference call with a FILTER other than PASS, . or RefCall, leaves G2 not read. GQ 20 and a missing GQ are read. The rule-line status list in docs/42-traits.md now names low_gq. Bead: trait-01
The case's own expectation took a GQ of . as below 20, while step 42 reads it. The header comments now name . as a readable FILTER. Bead: D10
Collaborator
Author
|
@coderabbitai review |
Action not completed
|
… the landing train # Conflicts: # CHANGELOG.md # docs/rust.md
An unknown or mistyped kind printed only the comphet and trim-pharmcat lines, so the eight e2e kinds were missing from the error a user sees. The usage error now prints the same kinds as --help, and a unit test checks every kind is named.
…sults closed An adult whose family.json had not answered or failed got a page with results marked opened and a close button that did nothing. person.js now asks only the gate (state.json's child rule). A person closed as another family's child gets their buttons under that family's members, not in its chapter of children.
…ed answer, and where a partner's buttons sit
… iframe, and fix the producer comments
# Conflicts: # CHANGELOG.md # docs/rust.md
- append-n refuses an OUT that is SRC (by device and inode), which it would truncate before copying, and ends a SRC whose last line has no newline before it writes the new contig header. - hba-tiles takes the read group from the ID: field of the first @rg line, wherever it sits, not from the second field. - rfc1 --samd12 writes a CIGAR of the read's own length, so a unit that is not five bases long still gives a valid record. The bytes the e2e cases use do not change: their pins still hold.
…landing train # Conflicts: # CHANGELOG.md # tests/parity/golden/summary-collect/all-history/all-history.json # tests/parity/golden/summary-collect/all-nf/all-nf.json # tests/parity/golden/summary-collect/all/all.json # tests/parity/golden/summary-collect/c-full/full.json # tests/parity/golden/summary-collect/c-nf/nf.json
…e landing train # Conflicts: # CHANGELOG.md
…otypes without a pipe The callable mask setup.sh installs is a plain one-member gzip, not BGZF, so the test now writes that shape (stored deflate blocks and a CRC-32 in the test, no new crate). The genotype check of the chip shell test reads the file with awk instead of cut piped into grep -q, so a SIGPIPE can no longer turn a failing check into a pass.
… and its usage error
…g train # Conflicts: # docs/41-bloodtype.md
… landing train # Conflicts: # rust/crates/og-dev/src/fixture.rs # scripts/ci/e2e-case-paths.tsv
…he landing train # Conflicts: # docs/rust.md
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #411. Settles owner decision D1 for these nine files: the Python goes away for 1.0.0.
The problem
Nine Python test files still guarded code that is no longer Python. Seven ran Rust tools (
og family,og y,og setup,og run delta) as subprocesses. One checked the parameters page againstnextflow_schema.json. One drovescripts/chip-to-vcf.shwith the real bcftools and Picard images. To ship 1.0.0 without owned Python, each check needs a home outside Python, with every case kept.The fix
Each file moves, with the same cases and the same expectations:
tests/test_family_segments.pyrust/crates/og-steps/tests/family_segments.rstests/test_ibd_report.pyrust/crates/og-steps/tests/family_ibd.rstests/test_y_private_snps.pyrust/crates/og-reports/tests/y_private_snps.rstests/test_y_tree_place.pyrust/crates/og-reports/tests/y_tree_place.rstests/test_y_private_mask_install.pyrust/crates/og-setup/tests/port_tests_family_y_dev_mask_install.rs(runs the realsetup.shwith a fake curl)tests/test_og_setup_zip.pyrust/crates/og-setup/tests/port_tests_family_y_dev_zip.rstests/test_run_delta.pyrust/crates/og-runlog/tests/delta_runs.rstests/test_parameters_doc.pyog dev params-doc(rust/crates/og-dev/src/params_doc.rs); its unit tests run it on the checkout, so Guard runs it incargo testtests/test_chip_to_vcf_mt.pytests/test_chip_to_vcf_mt.sh(it drives docker images, so it stays a shell test; the unit-tests loop finds it by glob)og dev params-doc [--repo DIR]replaces thenot ported yetstub. It exits 0 when every schema parameter has exactly one row indocs/parameters.mdand every row names a parameter (orpedigree, whilemain.nfreads it). It exits 1 and names the parameters otherwise, and 2 on an unreadable file. An unknown option such as--og-scaffold-probeis refused before any file is read.No tool code changes, and no output, option or file name changes.
docs/rust.mddropsparams-docfrom the list of stubs and says thatcargo testruns it on the checkout.Decisions, each the smallest change that keeps main true:
zipfilewrites, ZIP64 included), not committed as fixtures. A fixture would need a path outside this package's files, and the 2.3 MB member is not small. The trees thatextractalland the old inline map program wrote are spelled out by content. I checked them once against Python'szipfilelocally.og family ibd,og setup unzip|genetic-map) run the tool in a child process: the test binary runs itself again with one environment variable set. Those tools print to stderr directly, and this keeps the tests free of new crates.setup.shtest, since it runssetup.sh.docs/rust.mdis outside this package's file list. I changed its two lines anyway: without the change it listsparams-docas a stub, which is no longer true. Other open ports edit the same stub list, so whichever lands second resolves that line by keeping both removals.gzip.opendid and as the 1000 Genomes mask thatsetup.shinstalls is. It uses stored deflate blocks and a CRC-32 written in the test, so og-reports needs no new crate.How it was checked
cargo test -p og-dev --lib params_doc, and--test family_segments,family_ibd,y_private_snps,y_tree_place,port_tests_family_y_dev_mask_install,port_tests_family_y_dev_zip,delta_runs.cargo fmt --check,cargo clippy --all-targets --all-features -D warningson the five crates andcargo doc -D warningson og-dev are clean.cargo test --no-fail-faston fff6c47 every target went red:trio_duo_no_par_and_report,one_and_two_chromosomes,private_snps_gates_tables_and_mask,tree_place_and_tree_positions,setup_keeps_the_checked_chry_mask,unzip_writes_the_tree_extractall_writes,deltas_of_two_runs, andthe_repository_page_lists_every_schema_parameter_once. A row planted indocs/parameters.mdturnedog dev params-docred as well. Inside the tests, the dropped-row and planted-row controls ofog dev params-doc, the one-byte-changed tree of the zip test and the SVG well-formedness check each show their check can fail. The CI run on fff6c47 was cancelled before it started: about 100 runs were queued, and a full Guard run only for the red control would have held the queue for hours. So that red run is the local one described here.tests/test_chip_to_vcf_mt.shreads every genotype with oneawk, notcutpiped intogrep -q: underpipefaila SIGPIPE in that pipe could turn a failing check into a pass. Theawkcheck exits 1 on a genotype with a trailing carriage return and onAA, and 0 on0/1,1,./.and., with both BSD awk and Ubuntu's mawk.tests/test_chip_to_vcf_mt.shwith docker: 21 checks pass, the same 21 the Python made. Two broken copies ofchip-to-vcf.shturned it red: one that keeps the carriage return (1 check fails) and one that skips the hg19 to rCRS move of 23andMe's MT sites (4 checks fail).cargo test -p og-reports --test y_private_snpsgoes red (og refuses the mask). With the right CRC it passes.params_docunit tests; unit-tests runstests/test_chip_to_vcf_mt.sh), Rust https://github.com/GeiserX/OwnGenome/actions/runs/43769657590, Lint https://github.com/GeiserX/OwnGenome/actions/runs/43769657678, Nextflow https://github.com/GeiserX/OwnGenome/actions/runs/43769657575, Docs https://github.com/GeiserX/OwnGenome/actions/runs/43769657635, Container Test https://github.com/GeiserX/OwnGenome/actions/runs/43769657638, Personal data https://github.com/GeiserX/OwnGenome/actions/runs/43769657682.Out of scope, stale mentions of the removed files in files outside this package (comments and one doc line):
docs/37-y-haplogroup.md:98,tests/fixtures/chip/myheritage.csv:2,rust/crates/og-reports/src/ytree.rs:975,tests/parity/cases.tsv:981,tests/parity/gen/family-3gen.sh:3,tests/parity/gen/ibd-cases.sh:3,tests/parity/gen/y-inputs.sh:4.Beads: D1 (this package's share).