Repository navigation
UPDATE the scan to drop a match before it builds one - #22
Merged
Merged
Conversation
scan_fasta filtered, but scan() had already built both strings of every Hit and pushed it into a per-contig Vec, so the memory still rose with every match rather than with the ones worth keeping. The PAM is now read off the raw bytes, in guide orientation, before anything is allocated. It is not faster: 1,000 9-mers against hg38 take 33.9 s against 33.3, because the rolling lookup that finds the match runs whatever the PAM says, and with a library of every 9-mer it succeeds nearly everywhere. It halves the peak, 2.91 GB to 1.45, and that is what pays: batches of 5,000 now fit in 5.5 GB, which spreads the 19-second genome pass over 53 runs rather than 263 and takes the full library from 2.5 hours to 1.3. The per-base rule moves into code_ok, which matches_pam was the only caller of, so it goes. Output is byte-identical, counters included. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#18 taught
scan_fastato filter, butscan()had already built bothstrings of every
Hitand pushed it into a per-contigVec, so memorystill rose with every match rather than with the ones worth keeping. The
PAM is now read off the raw bytes, in guide orientation, before anything
is allocated.
It is not faster. 1,000 9-mers against hg38 take 33.9 s against 33.3.
The rolling lookup that finds the match runs whatever the PAM says, and
with a library of every 9-mer it succeeds nearly everywhere, so the
allocations were never the marginal cost. I expected otherwise and
measured it.
It halves the peak, 2.91 GB to 1.45, and that is what pays: batches of
5,000 now fit in 5.5 GB, which spreads the 21-second genome pass over 53
runs instead of 263 and takes the full 262,144-guide library from 2.5
hours to 1.4. Output is byte-identical, counters included.
The per-base rule moves into
code_ok, whichmatches_pamwas the onlycaller of, so that goes;
pam_ok_atreads it twice per match withoutallocating.
fitsis now shared withplain_slicerather than restated,which is what the minus-strand bounds depend on:
start - pam_lengthwraps to
usize::MAXfor a match at the very start of a contig.Tests: a parametrised check over random genomes that the filtered scan
equals the full scan filtered and that the tallies match the oracle's own
count, plus that contig-start case. The oracle itself only ever exercises
the unfiltered path, so it would not have caught a tally drifting into its
new home.
Ran the full 9-mer library against hg38 on this build: 53 batches, 85
minutes, no failures.
🤖 Generated with Claude Code