Skip to content

Do not expand Flag enums into a union of member literals - #21942

Open
istoolsfox wants to merge 1 commit into
python:masterfrom
istoolsfox:fix/flag-enum-equality-narrowing
Open

istoolsfox wants to merge 1 commit into
python:masterfrom
istoolsfox:fix/flag-enum-equality-narrowing

Conversation

@istoolsfox

@istoolsfox istoolsfox commented Sep 5, 2026 •

Copy link
Copy Markdown

Fixes #21937

Description

try_expanding_sum_type_to_union() no longer expands enum.Flag subclasses into a union of
member literals.

A Flag value can hold any combination of members, so the declared members are not an exhaustive
set of values, and turning the enum into a union of member literals is unsound. In the reported
case:

def f(programs: Programs) -> None:
    if programs == Programs.NONE:
        return
    if programs == Programs.P2:
        return
    if Programs.P2 in programs:  # error: Unsupported operand types for in
        ...

the guards narrowed programs to Literal[Programs.P1, Programs.P3, Programs.ALL] -- a type
that a value such as Programs.P1 | Programs.P2 cannot inhabit. The union then surfaced as an
error on in, because Flag.__contains__(self, other: Self) has no member in that union that
accepts Programs.P2.

Regular enums keep expanding, and narrowing a union that was written out explicitly still
removes members:

def f(p: Literal[P.A] | Literal[P.B]) -> None:
    if p is P.A:
        reveal_type(p)  # Literal[P.A]
    else:
        reveal_type(p)  # Literal[P.B]

Positive narrowing for Flag is preserved (if programs is Programs.P1 -> Literal[Programs.P1]);
only the negative branch stays at the enum class, since mypy cannot spell Programs & ~Literal[Programs.P1].

Checklist

  • Two new cases in check-enum.test; both fail without the change (testFlagEnumNotExpandedIntoUnion)
    or guard against overreach (testFlagEnumExplicitLiteralUnionStillNarrows)
  • testcheck: 8314 passed, 34 skipped, 7 xfailed
  • testfinegrained: 756 passed, 27 skipped
  • mypy --config-file mypy_self_check.ini -p mypy -p mypyc on the touched tree
  • ruff check / ruff format --check on the touched files

Not in this PR

__contains__ needs a separate fix: when the receiver is a member literal
(p: Literal[Programs.P2]; Programs.P1 in p), Self in Flag.__contains__(self, other: Self)
is still bound to the member literal. That is tracked separately from #21937.

@github-actions

This comment has been minimized.

Comment thread mypy/typeops.py Outdated
return isinstance(typ, LiteralType) or is_singleton_identity_type(typ)


FLAG_ENUM_BASES: Final = ("enum.Flag", "enum.IntFlag")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

enum.IntFlag is subclass of enum.Flag

@hauntsaninja hauntsaninja left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This isn't an issue with narrowing, it's an issue with handling of __contains__

def f(p: Literal[Programs.ALL]) -> None:
    if Programs.P2 in p:
        pass

@zedzhen

zedzhen commented Nov 17, 2026

Copy link
Copy Markdown

This isn't an issue with narrowing, it's an issue with handling of __contains__

def f(p: Literal[Programs.ALL]) -> None:
    if Programs.P2 in p:
        pass

The problem is with narrowing after is/== (it throws an error where there shouldn’t be any). The absence of narrowing after in only prevents some errors from being raised.
pyright, ty, and pyrefly do not narrow the type after in (pyright and pyrefly also do not narrow after is; ty narrows to F & ~Literal[F.A]). Thus, the problem lies precisely in the narrowing when using is/==.

from enum import Flag, auto
from typing import reveal_type


class F(Flag):
    A = auto()
    B = auto()

def f(x: F):
    reveal_type(x)  # all: F
    if x is F.A:
        reveal_type(x) # all: Literal[F.A]
        return
    reveal_type(x) # pyright, pyrefly, mypy (with this PR): F, mypy(now): Literal[F.B], ty: F & ~Literal[F.A]

def g(x: F):
    reveal_type(x)  # all: F
    if F.A in x:
        reveal_type(x)  # all: F
        return
    reveal_type(x)  # all: F

@hauntsaninja

Copy link
Copy Markdown
Collaborator

We should just not expand a flag enum to a Union (it would still be valid to narrow a Union of explicitly listed flag literals). And separately, the __contains__ handling also needs a fix.

@hauntsaninja

Copy link
Copy Markdown
Collaborator

Should be in try_expanding_sum_type_to_union

A Flag value can hold any combination of members, so the declared members
are not an exhaustive set of values. Expanding a Flag enum into a union of
member literals is therefore unsound: after `x is not F.A`, values such as
`F.A | F.B` are still possible and must not be pruned.

Skip the expansion for Flag subclasses in try_expanding_sum_type_to_union().
Enums whose members form a closed set still expand, and narrowing a union
that was written out explicitly still removes members.

Fixes python#21937
@istoolsfox
istoolsfox force-pushed the fix/flag-enum-equality-narrowing branch from 42c9149 to 302bab1 Compare November 17, 2026 07:42
@istoolsfox istoolsfox changed the title Do not narrow Flag enums by member equality Do not expand Flag enums into a union of member literals Nov 17, 2026
@istoolsfox

Copy link
Copy Markdown
Author

You're right -- I was fixing a symptom. Reworked along the lines you suggested:

  • the expansion is now skipped for Flag subclasses in try_expanding_sum_type_to_union(), so
    ==/is narrowing can no longer prune the declared members into a bogus union;
  • regular enums are untouched, and an explicitly written union of flag literals still narrows
    (Literal[P.A] | Literal[P.B] + is P.A -> Literal[P.A] / Literal[P.B]);
  • compared to the previous version, positive narrowing survives (if p is P.A ->
    Literal[P.A]); only the negative branch stays at the enum class.

testcheck 8314 passed, testfinegrained 756 passed. testFlagEnumNotExpandedIntoUnion fails on
master and passes with the change.

On the __contains__ half: I left it out of this PR since you said it is a separate fix. For the
record, what remains after this PR is the case where the receiver is already a member literal:

p: Literal[Programs.P2]
Programs.P1 in p  # still an error: Self is bound to Literal[Programs.P2]

The same binding also makes IntFlag.__or__ return Literal[I.A] for i | 2, which is unsound.
I can file an issue and send a follow-up PR for it -- or fold it into this one, whichever you
prefer.

@github-actions

Copy link
Copy Markdown
Contributor

According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Incorrect narrowing of Enum.Flag

3 participants