Skip to content

UPDATE the scan to drop a match before it builds one - #22

Merged
emattei merged 2 commits into
mainfrom
perf/core-pam-filter
Nov 9, 2026
Merged

emattei merged 2 commits into
mainfrom
perf/core-pam-filter

Conversation

@emattei

@emattei emattei commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator

#18 taught scan_fasta to filter, but scan() had already built both
strings of every Hit and pushed it into a per-contig Vec, so memory
still rose with every match rather than with the ones worth keeping. The
PAM is now read off the raw bytes, in guide orientation, before anything
is allocated.

It is not faster. 1,000 9-mers against hg38 take 33.9 s against 33.3.
The rolling lookup that finds the match runs whatever the PAM says, and
with a library of every 9-mer it succeeds nearly everywhere, so the
allocations were never the marginal cost. I expected otherwise and
measured it.

It halves the peak, 2.91 GB to 1.45, and that is what pays: batches of
5,000 now fit in 5.5 GB, which spreads the 21-second genome pass over 53
runs instead of 263 and takes the full 262,144-guide library from 2.5
hours to 1.4. Output is byte-identical, counters included.

The per-base rule moves into code_ok, which matches_pam was the only
caller of, so that goes; pam_ok_at reads it twice per match without
allocating. fits is now shared with plain_slice rather than restated,
which is what the minus-strand bounds depend on: start - pam_length
wraps to usize::MAX for a match at the very start of a contig.

Tests: a parametrised check over random genomes that the filtered scan
equals the full scan filtered and that the tallies match the oracle's own
count, plus that contig-start case. The oracle itself only ever exercises
the unfiltered path, so it would not have caught a tally drifting into its
new home.

Ran the full 9-mer library against hg38 on this build: 53 batches, 85
minutes, no failures.

🤖 Generated with Claude Code

emattei and others added 2 commits November 9, 2026 12:07
scan_fasta filtered, but scan() had already built both strings of every
Hit and pushed it into a per-contig Vec, so the memory still rose with
every match rather than with the ones worth keeping. The PAM is now read
off the raw bytes, in guide orientation, before anything is allocated.

It is not faster: 1,000 9-mers against hg38 take 33.9 s against 33.3,
because the rolling lookup that finds the match runs whatever the PAM
says, and with a library of every 9-mer it succeeds nearly everywhere.
It halves the peak, 2.91 GB to 1.45, and that is what pays: batches of
5,000 now fit in 5.5 GB, which spreads the 19-second genome pass over 53
runs rather than 263 and takes the full library from 2.5 hours to 1.3.

The per-base rule moves into code_ok, which matches_pam was the only
caller of, so it goes. Output is byte-identical, counters included.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@emattei
emattei merged commit e2ff8de into main Nov 9, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant