[haodongchen]

Three changes to dnascope-pangenome that shorten its runtime on large hosts. Each is its own commit with its own tests, so they can be reviewed or reverted separately. A fourth commit makes the review's changes to the first.

1. One alignment lane per NUMA node

DNAscope already splits bwa into one job per NUMA node on large hosts. dnascope-pangenome now does the same with FASTQ input, and does it for minimap2 too. It uses DNAscope's rule, split_alignment(): NUMA nodes, halved until each piece has at most 64 CPUs.

• bwa-extract: one job per lane. fqidx extract -F j/N feeds bwa mem -p, and taskset pins bwa and pgutil extract to the lane's CPUs.
• mm2-lift: one job per lane, pinned to the same CPUs. It reads that lane's extracted reads and waits only for its own bwa lane.
• Dedup: both dedup stages read every lane's output.
• Job names: each lane's job has its own name, bwa-extract-<j> and mm2-lift-<j>. Unsplit runs keep bwa-extract and mm2-lift.
• Opting out: a hidden --no_split_alignment keeps the single jobs.
• Unaffected: BAM/CRAM input, hosts with 32 or fewer cores, and the hybrid pipelines.

-K: fqidx extract rejects -K below 10,000,000, so dnascope-pangenome now aligns with -K 10000000 whether or not it splits, and its results don't depend on the host's NUMA layout. This changes the alignments slightly from the previous default of 2,000,000. hybrid-pangenome shares the stage and keeps 2,000,000.

2. Give indel2cnv's workers one variant at a time

indel2cnv sorts the candidate variants longest first. It then grouped them into batches of 5 and passed them to starmap with the default chunksize, which is about len(batches) / (4 * workers). So the first chunk held the most expensive variants of the whole run, and they all went to a single worker. That worker kept running long after the others went idle.

Each task is now one variant, with chunksize=1. Results come back in the same order, so the output is unchanged.

3. Run vg haplotypes alongside bwa when both fit in memory

vg-haplotypes needs only the k-mer counts. The current code expects it to overlap bwa when memory allows, but it never did: the job reserved every core, so the scheduler held it until bwa finished, whatever its dependencies said.

Now, if the host has room for vg-haplotypes (100 GB) plus 32 GB per bwa lane, the job reserves no threads, as kmc already does, and runs next to bwa. Otherwise it depends on every bwa lane, as before. This replaces the fixed 70 GB check.

The 100 GB budget covers measured peaks on the HPRC v2.0 and v2.1 graphs: about 60 GB with vg 1.73.0 and 1.78.0, and about 92 GB with a vg build that samples in parallel. The 32 GB per lane leaves headroom over bwa-extract's measured 19 GB peak.

Results

HG002 Illumina 35x, HPRC v2.0, FASTQ to small-variant, SV and CNV calls, on AWS on-demand instances.

	Before	After	Notes
c8a.48xlarge (192 vCPU, 2 NUMA nodes)	52:42	39:16	lanes; bwa-extract 1,152 → 555 s, mm2-lift 539 → 283 s
c8g.48xlarge (192 vCPU, 2 NUMA nodes)	68:09	53:17	lanes; bwa-extract 1,162 → 667 s, mm2-lift 688 → 271 s
c8a.48xlarge, with lanes	30:10	26:24	indel2cnv 156 → 49 s, vg-haplotypes alongside bwa
c7a.16xlarge (64 vCPU, 128 GB)	1:23:50	1:17:55	indel2cnv 420 → 68 s; not enough memory to run vg-haplotypes alongside bwa

The first two rows add lanes only. In the last two rows, before and after used the same vg build, so only changes 2 and 3 differ.

Accuracy against GIAB v4.2.1 (Aardvark, genotype match):

• Lanes changed total errors from 5,626 to 5,621 on c8a and from 5,623 to 5,619 on c8g. SNV and Indel F1 were unchanged to two decimal places. These are split runs at -K 10000000, the setting every run now uses.
• Changes 2 and 3 left the output VCF and CNV VCF bodies byte-identical on both instance types.

Tests

New unit tests:

• test_pangenome_numa_lanes.py, run on a mocked 2-node, 192-CPU host
• test_pangenome_vg_scheduling.py

flake8, mypy and the full test suite pass at every commit, and black passes on every file this PR touches. sentieon_cli/archive.py fails black and test_job_log_capture.py::test_output_proc_sub_stderr_is_captured fails about 1% of the time, both already on dev (2 failures in 200 runs on dev, 3 in 200 on this branch). This PR touches neither.

Not run end-to-end:

• 96- and 128-vCPU hosts, which would get 2 lanes
• BAM/CRAM input, where the code path is unchanged apart from the KMC cap
• the hybrid pipelines

🤖 Generated with Claude Code

[DonFreed]

@haodongchen back to you for some minor changes. Let's keep job names unique and use the same -K for split and unsplit jobs.

[DonFreed]

Thanks!