Skip to content

Speed up narrowing of large nominal unions - #22492

Open
soceanainn wants to merge 1 commit into
python:masterfrom
soceanainn:codex/speed-up-large-unions
Open

soceanainn wants to merge 1 commit into
python:masterfrom
soceanainn:codex/speed-up-large-unions

Conversation

@soceanainn

@soceanainn soceanainn commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Large unions become expensive to narrow when successive isinstance checks leave most members unchanged. narrow_declared_type repeatedly compares those members against the remaining union, which is O(N**2). This change reuses unchanged, unrestricted instances when the narrowed union contains only existing declared members - giving us an O(N) 'fastpath'.

Other members retain recursive narrowing, preserving behavior for subclasses, generics, literals, truthiness restrictions, and numeric promotions. Eligibility depends on type membership.

Related to #12526 but optimizes the narrowing path ahead of calling make_simplified_union which is incredibly slow when measured.

Testing

Adds testUnionOfManyNominalClassesNarrowing, with 300 classes (T0–T299), ten returning isinstance branches, and checks on the remaining union.

Benchmark Before After Speedup
testUnionOfManyNominalClassesNarrowing 18.177s 2.851s 6.38×

Times are median pytest call durations across three fresh processes per version, using Python 3.12.3 and uncompiled mypy. This saves 15.326s (84.3%) on this case.

Run the benchmark with timing:

python -m pytest -n0 \
  mypy/test/testcheck.py::TypeCheckSuite::check-unions.test::testUnionOfManyNominalClassesNarrowing \
  -q --durations=1 --durations-min=0

Run the relevant regression suites with timing:

python -m pytest -n0 \
  mypy/test/testcheck.py::TypeCheckSuite::check-unions.test \
  mypy/test/testcheck.py::TypeCheckSuite::check-type-promotion.test \
  mypy/test/testtypes.py \
  mypy/test/testsubtypes.py \
  -q --durations=5 --durations-min=0

Validation: 248 passed, 2 skipped. Mypy’s self-check, Black, and Ruff also pass.

Reuse unchanged unrestricted instances when narrowing a union only removes existing members. Preserve recursive narrowing for introduced members, literal values, and restricted truthiness.

Add regression coverage and a 300-member type-checking benchmark in check-unions.test.
@soceanainn
soceanainn force-pushed the codex/speed-up-large-unions branch from 3b87fdf to 5e190aa Compare November 16, 2026 02:01
@github-actions

Copy link
Copy Markdown
Contributor

Diff from mypy_primer, showing the effect of this PR on open source code:

colour (https://github.com/colour-science/colour)
- colour/utilities/array.py:3810: error: Incompatible return value type (got "list[Any] | Any | Buffer | _SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]] | complex | str | _NestedSequence[complex | bytes | str]", expected "ndarray[tuple[Any, ...], dtype[Any]]")  [return-value]
+ colour/utilities/array.py:3810: error: Incompatible return value type (got "list[Any] | Buffer | _SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]] | complex | str | _NestedSequence[complex | bytes | str] | Any", expected "ndarray[tuple[Any, ...], dtype[Any]]")  [return-value]

pandera (https://github.com/pandera-dev/pandera)
- pandera/engines/geopandas_engine.py:151: error: Invalid index type "Any | Series[bool] | DataFrame" for "Series[Any]"; expected type "int | signedinteger[_32Bit | _64Bit] | integer[Any] | unsignedinteger[Any] | signedinteger[Any] | signedinteger[_8Bit]"  [index]
+ pandera/engines/geopandas_engine.py:151: error: Invalid index type "Series[bool] | DataFrame | Any" for "Series[Any]"; expected type "int | signedinteger[_32Bit | _64Bit] | integer[Any] | unsignedinteger[Any] | signedinteger[Any] | signedinteger[_8Bit]"  [index]

jax (https://github.com/google/jax)
- jax/_src/state/indexing.py:377: error: Argument 1 to "pp_var" has incompatible type "int | Any | Array"; expected "Var | Literal"  [arg-type]
+ jax/_src/state/indexing.py:377: error: Argument 1 to "pp_var" has incompatible type "int | Array | Any"; expected "Var | Literal"  [arg-type]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant