Skip to content

[mypyc] Fix OverflowError when a native __hash__ returns a large int - #22453

Merged
JukkaL merged 1 commit into
python:masterfrom
rheard:fix-mypyc-1253
Nov 13, 2026
Merged

JukkaL merged 1 commit into
python:masterfrom
rheard:fix-mypyc-1253

Conversation

@rheard

@rheard rheard commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

Fixes mypyc/mypyc#1253

The tp_hash slot of a native class converted the value returned by __hash__ with CPyTagged_AsSsize_t(), so an int that doesn't fit in Py_hash_t made hash() raise OverflowError. CPython's slot_tp_hash, which a class defined in Python uses, accepts any int and hashes one that doesn't fit again with int.__hash__.

generate_hash_wrapper() now does the same:

  • A short int is still handled inline, with -1 changed to -2 as before.
  • A long int goes through the new CPy_HashFromResult() in generic_ops.c, which does what slot_tp_hash
    does with the result: an int that fits in Py_hash_t is used as is, a larger one is hashed again, and
    anything that isn't an int raises TypeError: __hash__ method should return an integer.
  • Other return types are boxed if needed and passed to CPy_HashFromResult(). With -> Any, a result that
    isn't an int used to raise TypeError: an integer is required. A __hash__ declared to return a native int
    type such as i64 segfaulted (or raised SystemError if it returned 0), because the slot passed the C value
    to PyLong_AsSsize_t() as an object pointer.

The new run test testDundersHashResult covers -> int and -> Any with values that fit in Py_hash_t, -1, values that don't fit, a long int that fits only on 64-bit platforms (2**62), True and results that aren't ints, plus an -> i64 method that returns -1. The expected results are the same on 32-bit platforms.

@JukkaL JukkaL left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, looks good! This fixes a few different bugs.

@JukkaL
JukkaL merged commit 0952029 into python:master Nov 13, 2026
34 of 35 checks passed
p-sawicki pushed a commit that referenced this pull request Nov 20, 2026
…n__ methods (#22485)

The `sq_length`/`mp_length` slot of a native class didn't check the
value returned by `__len__` like CPython's `slot_sq_length` does, and
the compiled `len()` of a native class instance only partly did:

- A negative result from `__len__` declared to return `int` or `Any` was
returned from the slot as is, without an exception set. `len(x)` then
raised `SystemError: <built-in function len> returned NULL without
setting an exception` when called from interpreted code, and crashed the
interpreter (segfault) in some other cases, such as `len(x)` in compiled
code where `x` has type `Any`.
- `__len__` declared to return `bool` or a native int type (`i64`,
`i32`, `i16`, `i8`, `u8`, ...) generated `PyLong_AsSsize_t(retval)` with
a C integer `retval`, which is a C compile error (`-Wint-conversion`)
with current GCC and clang.
- With `-> Any`, a result that isn't an int raised `TypeError: an
integer is required`, while Python accepts any object with `__index__`.
- The compiled `len(x)`, where `x` is a native class instance, checked
that the length isn't negative, but returned a value that doesn't fit in
`Py_ssize_t` (such as `2**63`) instead of raising `OverflowError`.

```python
class C:
    def __init__(self, n: int) -> None:
        self.n = n

    def __len__(self) -> int:
        return self.n

class B:
    def __len__(self) -> bool:  # C compile error
        return True
```

```python
>>> len(C(-1))
SystemError: <built-in function len> returned NULL without setting an exception
```

CPython's `slot_sq_length` converts the result with `__index__`, raises
`ValueError: __len__() should return >= 0` if it's negative, and
`OverflowError: cannot fit 'int' into an index-sized integer` if it
doesn't fit in `Py_ssize_t`.

Now:

- `generate_len_wrapper()` handles a short int inline. Anything else
goes through the new `CPy_LenFromResult()` in `generic_ops.c`, which
does what `slot_sq_length` does, after boxing the value if needed (like
`generate_hash_wrapper()` after #22453).
- In the compiled `len()` of a native class instance, the length check
is the new `CPyTagged_CheckLen()` in `CPy.h`. It's inline for short
ints, and for long ints it also raises `OverflowError` if the value
doesn't fit in `Py_ssize_t`. If `__len__` returns something other than
an int type (for example `Any`), the result is converted by
`CPyObject_LenFromResult()` like in the slot.

Tests: the new run test `testDundersLenResult` covers `-> int`, `->
Any`, `-> bool`, `-> i64`, `-> i8` and `-> u8`, through both the
compiled `len()` and the slot (`len()` of an `Any` value), with negative
values, values that don't fit in `Py_ssize_t`, `True`, an object with
`__index__` and results that aren't ints. I updated `testDundersLen` in
`irbuild-dunders.test` and added `testDundersLenAny`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

hash() raises OverflowError when a native __hash__ returns a large int

2 participants