Repository navigation
Speed up narrowing of large nominal unions - #22492
Open
soceanainn wants to merge 1 commit into
Open
soceanainn wants to merge 1 commit into
soceanainn wants to merge 1 commit into
Conversation
Reuse unchanged unrestricted instances when narrowing a union only removes existing members. Preserve recursive narrowing for introduced members, literal values, and restricted truthiness. Add regression coverage and a 300-member type-checking benchmark in check-unions.test.
soceanainn
force-pushed
the
codex/speed-up-large-unions
branch
from
November 16, 2026 02:01
3b87fdf to
5e190aa
Compare
Contributor
|
Diff from mypy_primer, showing the effect of this PR on open source code: colour (https://github.com/colour-science/colour)
- colour/utilities/array.py:3810: error: Incompatible return value type (got "list[Any] | Any | Buffer | _SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]] | complex | str | _NestedSequence[complex | bytes | str]", expected "ndarray[tuple[Any, ...], dtype[Any]]") [return-value]
+ colour/utilities/array.py:3810: error: Incompatible return value type (got "list[Any] | Buffer | _SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]] | complex | str | _NestedSequence[complex | bytes | str] | Any", expected "ndarray[tuple[Any, ...], dtype[Any]]") [return-value]
pandera (https://github.com/pandera-dev/pandera)
- pandera/engines/geopandas_engine.py:151: error: Invalid index type "Any | Series[bool] | DataFrame" for "Series[Any]"; expected type "int | signedinteger[_32Bit | _64Bit] | integer[Any] | unsignedinteger[Any] | signedinteger[Any] | signedinteger[_8Bit]" [index]
+ pandera/engines/geopandas_engine.py:151: error: Invalid index type "Series[bool] | DataFrame | Any" for "Series[Any]"; expected type "int | signedinteger[_32Bit | _64Bit] | integer[Any] | unsignedinteger[Any] | signedinteger[Any] | signedinteger[_8Bit]" [index]
jax (https://github.com/google/jax)
- jax/_src/state/indexing.py:377: error: Argument 1 to "pp_var" has incompatible type "int | Any | Array"; expected "Var | Literal" [arg-type]
+ jax/_src/state/indexing.py:377: error: Argument 1 to "pp_var" has incompatible type "int | Array | Any"; expected "Var | Literal" [arg-type]
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Large unions become expensive to narrow when successive
isinstancechecks leave most members unchanged.narrow_declared_typerepeatedly compares those members against the remaining union, which is O(N**2). This change reuses unchanged, unrestricted instances when the narrowed union contains only existing declared members - giving us an O(N) 'fastpath'.Other members retain recursive narrowing, preserving behavior for subclasses, generics, literals, truthiness restrictions, and numeric promotions. Eligibility depends on type membership.
Related to #12526 but optimizes the narrowing path ahead of calling
make_simplified_unionwhich is incredibly slow when measured.Testing
Adds
testUnionOfManyNominalClassesNarrowing, with 300 classes (T0–T299), ten returningisinstancebranches, and checks on the remaining union.testUnionOfManyNominalClassesNarrowingTimes are median pytest call durations across three fresh processes per version, using Python 3.12.3 and uncompiled mypy. This saves 15.326s (84.3%) on this case.
Run the benchmark with timing:
Run the relevant regression suites with timing:
Validation: 248 passed, 2 skipped. Mypy’s self-check, Black, and Ruff also pass.