Skip to content

[mypyc] Add primitives for int and bytes containment in bytes - #22409

Merged
JukkaL merged 4 commits into
python:masterfrom
Jahongir-Qurbonov:mypyc-bytes-contains
Nov 19, 2026
Merged

JukkaL merged 4 commits into
python:masterfrom
Jahongir-Qurbonov:mypyc-bytes-contains

Conversation

@Jahongir-Qurbonov

Copy link
Copy Markdown
Contributor

Add primitives for x in b / x not in b where b is bytes and x is int or bytes.

Previously these used PySequence_Contains. For a bytes item that is particularly slow, since bytes.__contains__ first tries to convert the item to an integer, which raises and clears a TypeError, and then acquires a buffer.

  • int in bytes: for an exact bytes object, a range check followed by memchr. Out-of-range values raise the same ValueError as CPython.
  • bytes in bytes: if both are exact bytes objects and the needle is at most 32 bytes long, search directly using memchr + memcmp. Longer needles fall back to the generic implementation, which has better worst-case behavior.
  • Subclasses (which may override __contains__) use the generic path.

Tests compare the results against the generic implementation (through Any) for all byte values, out-of-range and big ints, and all substrings (plus near-miss variants) of several haystacks.

Micro-benchmark (compiled code, before -> after, Python 3.13):

  • int in bytes (3.2M checks): 0.025s -> 0.0086s
  • 1-byte bytes in bytes (3.2M checks): 0.49s -> 0.026s
  • multi-byte bytes in bytes (3.2M checks): 0.49s -> 0.039s

Part of mypyc/mypyc#644.

@Jahongir-Qurbonov Jahongir-Qurbonov mentioned this pull request Nov 10, 2026
13 of 14 tasks
@JukkaL

JukkaL commented Nov 12, 2026

Copy link
Copy Markdown
Collaborator

This PR has merge conflicts, please rebase.

# Conflicts:
#	mypyc/test-data/irbuild-bytes.test
#	mypyc/test-data/run-bytes.test

@JukkaL JukkaL left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! I have an idea about how to further improve performance in some cases -- I'll prepare a follow-up PR.

@JukkaL
JukkaL merged commit 068e8b7 into python:master Nov 19, 2026
19 checks passed
JukkaL added a commit that referenced this pull request Nov 20, 2026
…22528)

The memchr + memcmp search that was added in #22409 is fast on typical
data, but when the first byte to search for is frequent, each iteration
makes little progress. This seems fairly common, but in the worst case
this was ~9x slower than CPython.

Fall back to CPython search algorithm if there are many first byte hits
that don't match the whole search string. On Python 3.10 we need to fall
back to PySequence_Contains, but on newer versions there is a private
CPython API we can use.

Benchmark results on Linux x86-64 (time relative to master, lower is
better):

 * 1 MB Python source, `b"    raise NotImplemented!"` (miss): 0.12x
 * `b"a" * 100_000`, `b"ab"` (miss): 0.11x
 * Short inputs (8-80 bytes): 0.86-0.99x

The worst case is a dense region followed by a long sparse one, e.g.
`b"a" * 200 + b"x" * 1_000_000` and `b"ab"`, which is now about as slow
as the generic CPython search (~35x slower than master).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants