Skip to content

Fix quadratic MIME boundary matching in the parser - #524

Merged
jsuto merged 1 commit into
masterfrom
fix-boundary-lookup-quadratic
Nov 6, 2026
Merged

jsuto merged 1 commit into
masterfrom
fix-boundary-lookup-quadratic

Conversation

@jsuto

@jsuto jsuto commented Nov 6, 2026

Copy link
Copy Markdown
Owner

Problem

parse_line() calls is_substr_in_hash() on every line. That walks all 277 hash buckets and runs strstr() against every boundary seen so far. extract_boundary() has no limit on how many boundaries it stores, so a deeply nested message with a distinct boundary per level parses in O(N²). Any message accepted on the SMTP path reaches this code.

Fix

  • is_boundary_line(): a line counts as a boundary only if it is -- + boundary, optionally followed by -- (closing delimiter) and trailing whitespace. It checks with a hash lookup instead of a substring scan.
  • MAX_BOUNDARIES (128): a cap on boundaries per message, so hash chains stay short. Without it, lookups still grew superlinearly.
  • Trimming: extract_boundary() now strips trailing whitespace from boundaries.
  • Test: new unit test test_is_boundary_line().

Benchmark

Nested message with a distinct boundary per level, parse_message() only:

Size master this PR
1 MiB 1.46 s 0.027 s
2 MiB 5.73 s 0.055 s
4 MiB 23.7 s 0.109 s
16 MiB — 0.42 s

Behaviour changes

  • Mid-line boundaries: a boundary string in the middle of a line, or with leading whitespace, is no longer treated as a delimiter (as RFC 2046 requires).
  • Nesting limit: boundaries after the 128th in a message are ignored.
  • Long boundaries: delimiter lines longer than about 509 chars are not recognised. RFC 2046 limits boundaries to 70 chars.

Testing

Unit tests pass in sutoj/builder:resolute, including the DB-backed check_parser against the .eml fixtures.

🤖 Generated with Claude Code

parse_line() called is_substr_in_hash() on every line, which walks all
hash buckets and strstr()s the line against every boundary seen so far.
A deeply nested message with a distinct boundary per level therefore
took O(N^2) time to parse (~24s for a 4 MiB message, minutes for 16 MiB).

- Add is_boundary_line(): a line is a boundary only if it is "--" +
  boundary (optionally followed by "--" and whitespace), checked with a
  hash lookup instead of a substring scan over all boundaries. A
  boundary string appearing mid-line no longer matches, as per RFC 2046.
- Cap the number of stored boundaries per message (MAX_BOUNDARIES=128)
  so hash chains stay short.
- Trim trailing whitespace from extracted boundaries.

The same nested test message now parses in 0.11s (4 MiB) and 0.42s
(16 MiB), scaling linearly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jsuto
jsuto merged commit 857299f into master Nov 6, 2026
4 checks passed
@jsuto
jsuto deleted the fix-boundary-lookup-quadratic branch November 6, 2026 07:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant