Skip to content

[mypyc] Fix comprehensions over lists that are modified while iterating - #22414

Merged
JukkaL merged 5 commits into
python:masterfrom
Jahongir-Qurbonov:mypyc-comprehension-prealloc
Nov 13, 2026
Merged

JukkaL merged 5 commits into
python:masterfrom
Jahongir-Qurbonov:mypyc-comprehension-prealloc

Conversation

@Jahongir-Qurbonov

Copy link
Copy Markdown
Contributor

Comprehensions over a list preallocate the result using the list's initial length, but the loop re-reads the length on every iteration, as Python semantics require. If the list is modified during iteration, the result was corrupted. This can happen in the element expression, or from another thread (also on GIL builds):

  • If the list grew, items were written past the end of the preallocated list/tuple/vec. That is a heap overflow, typically free(): invalid size or a segfault.
  • If the list shrank, the result contained NULL items, which caused crashes later.

This affected:

  • [f(x) for x in a]
  • list(...), tuple(...), sorted(...) and str.join(...) with a generator argument
  • vec[T]([... for x in a])

Repro:

def shrink(a: list[str], x: str) -> str:
    a.pop()
    return x

def f(a: list[str]) -> str:
    return ",".join(shrink(a, x) for x in a)

print(f(["a", "b", "c", "d"]))  # CPython: 'a,b'; compiled: segfault

Fix: keep the preallocation, which is significantly faster than appending, but handle sources whose length can change:

  • When the source is a list whose length isn't known at compile time, items that don't fit in the preallocated result are appended.
  • After the loop, the length of the result is set to the number of items. Lists use a new CPyList_SetSizeUnsafe helper; vecs update the len field.
  • Tuples can't be resized in place. Instead, items that don't fit are moved to a new tuple with double the capacity (CPySequenceTuple_SetItemGrow), and after the loop the items are moved to a tuple of the correct size if the length changed (CPySequenceTuple_SetSize).

This matches CPython: the result contains every item the loop sees. Sources whose length can't change (tuple, str, bytes, vec, list displays) generate the same IR as before.

If the list isn't modified, the only extra cost is one comparison per item. The performance of tuple(x for x in a) stays the same as on master (n=10000: 18.2µs on master vs 17.7µs with this PR).

A comprehension over a list, such as [f(x) for x in a], preallocates the
result using the initial length of the list, but the loop checks the
current length of the list on each iteration. If the list grew during
the loop (e.g. in f, or in another thread), items were written past the
end of the result (heap overflow). If it shrank, the result had NULL
items at the end, which caused crashes later. This affected list
comprehensions, list(), tuple(), sorted() and str.join() with a
generator expression argument, and vec comprehensions.

Keep the preallocation, but append items that don't fit in the
preallocated result, and reduce the length of the result after the loop
if the list shrank. This matches CPython, which includes all items it
sees. A tuple can't be resized, so tuple(<generator>) over a list now
builds a list first, like other generator sources.
The previous commit made tuple(<generator over a list>) build a list
first and then convert it to a tuple, since a preallocated tuple can't
be resized if the list is modified during iteration. This made it up to
about 2x slower.

Instead, preallocate the tuple as before, but if an item doesn't fit,
move the items to a larger tuple (the capacity is doubled). After the
loop, move the items to a tuple of the correct size if the length
changed. If the list isn't modified, the only extra cost is one
comparison per item.
@JukkaL

JukkaL commented Nov 11, 2026

Copy link
Copy Markdown
Collaborator

There's not a merge conflict, can you rebase? I had a quick look and the approach looks good.

…on-prealloc

# Conflicts:
#	mypyc/irbuild/for_helpers.py
@Jahongir-Qurbonov

Copy link
Copy Markdown
Contributor Author

Merged master.

@JukkaL JukkaL left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found this regression -- this works on master but crashes the compiler in the branch with KeyError: <Register '' at 0x...>:

import asyncio

from librt.vecs import vec


async def ident(x: str) -> str:
    await asyncio.sleep(0)
    return x

async def build(a: list[str]) -> vec[str]:
    return vec[str]([await ident(x) for x in a])

print(asyncio.run(build(["a", "b"])))

…on-prealloc

# Conflicts:
#	mypyc/irbuild/specialize.py
#	mypyc/lib-rt/CPy.h
SetElement didn't include the item in its sources, so the generator spill
transform didn't see the use of the loop index register, which is used as
the new vec length after the loop, and didn't move it to the generator
frame. This caused a KeyError in codegen.
@Jahongir-Qurbonov

Copy link
Copy Markdown
Contributor Author

Thanks for finding this! The crash was caused by SetElement not including item in sources(), so the generator spill transform didn't see the use of the loop index register (used as the new vec length after the loop) and didn't move it to the generator frame. I fixed SetElement.sources()/set_sources() (and the corresponding IRTransform visitor), and added a run test with await in list and vec comprehensions over lists, including lists modified while iterating.

@JukkaL JukkaL left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix, looks good now!

@JukkaL
JukkaL merged commit 91643be into python:master Nov 13, 2026
18 checks passed
@Jahongir-Qurbonov
Jahongir-Qurbonov deleted the mypyc-comprehension-prealloc branch November 14, 2026 22:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants