Repository navigation
Fix quadratic MIME boundary matching in the parser - #524
Merged
Merged
Conversation
parse_line() called is_substr_in_hash() on every line, which walks all hash buckets and strstr()s the line against every boundary seen so far. A deeply nested message with a distinct boundary per level therefore took O(N^2) time to parse (~24s for a 4 MiB message, minutes for 16 MiB). - Add is_boundary_line(): a line is a boundary only if it is "--" + boundary (optionally followed by "--" and whitespace), checked with a hash lookup instead of a substring scan over all boundaries. A boundary string appearing mid-line no longer matches, as per RFC 2046. - Cap the number of stored boundaries per message (MAX_BOUNDARIES=128) so hash chains stay short. - Trim trailing whitespace from extracted boundaries. The same nested test message now parses in 0.11s (4 MiB) and 0.42s (16 MiB), scaling linearly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
parse_line()callsis_substr_in_hash()on every line. That walks all 277 hash buckets and runsstrstr()against every boundary seen so far.extract_boundary()has no limit on how many boundaries it stores, so a deeply nested message with a distinct boundary per level parses in O(N²). Any message accepted on the SMTP path reaches this code.Fix
is_boundary_line(): a line counts as a boundary only if it is--+ boundary, optionally followed by--(closing delimiter) and trailing whitespace. It checks with a hash lookup instead of a substring scan.MAX_BOUNDARIES(128): a cap on boundaries per message, so hash chains stay short. Without it, lookups still grew superlinearly.extract_boundary()now strips trailing whitespace from boundaries.test_is_boundary_line().Benchmark
Nested message with a distinct boundary per level,
parse_message()only:Behaviour changes
Testing
Unit tests pass in
sutoj/builder:resolute, including the DB-backedcheck_parseragainst the .eml fixtures.🤖 Generated with Claude Code