Repository navigation
[mypyc] Fix comprehensions over lists that are modified while iterating - #22414
Conversation
A comprehension over a list, such as [f(x) for x in a], preallocates the result using the initial length of the list, but the loop checks the current length of the list on each iteration. If the list grew during the loop (e.g. in f, or in another thread), items were written past the end of the result (heap overflow). If it shrank, the result had NULL items at the end, which caused crashes later. This affected list comprehensions, list(), tuple(), sorted() and str.join() with a generator expression argument, and vec comprehensions. Keep the preallocation, but append items that don't fit in the preallocated result, and reduce the length of the result after the loop if the list shrank. This matches CPython, which includes all items it sees. A tuple can't be resized, so tuple(<generator>) over a list now builds a list first, like other generator sources.
The previous commit made tuple(<generator over a list>) build a list first and then convert it to a tuple, since a preallocated tuple can't be resized if the list is modified during iteration. This made it up to about 2x slower. Instead, preallocate the tuple as before, but if an item doesn't fit, move the items to a larger tuple (the capacity is doubled). After the loop, move the items to a tuple of the correct size if the length changed. If the list isn't modified, the only extra cost is one comparison per item.
|
There's not a merge conflict, can you rebase? I had a quick look and the approach looks good. |
…on-prealloc # Conflicts: # mypyc/irbuild/for_helpers.py
|
Merged master. |
JukkaL
left a comment
There was a problem hiding this comment.
I found this regression -- this works on master but crashes the compiler in the branch with KeyError: <Register '' at 0x...>:
import asyncio
from librt.vecs import vec
async def ident(x: str) -> str:
await asyncio.sleep(0)
return x
async def build(a: list[str]) -> vec[str]:
return vec[str]([await ident(x) for x in a])
print(asyncio.run(build(["a", "b"])))…on-prealloc # Conflicts: # mypyc/irbuild/specialize.py # mypyc/lib-rt/CPy.h
SetElement didn't include the item in its sources, so the generator spill transform didn't see the use of the loop index register, which is used as the new vec length after the loop, and didn't move it to the generator frame. This caused a KeyError in codegen.
|
Thanks for finding this! The crash was caused by |
JukkaL
left a comment
There was a problem hiding this comment.
Thanks for the fix, looks good now!
Comprehensions over a list preallocate the result using the list's initial length, but the loop re-reads the length on every iteration, as Python semantics require. If the list is modified during iteration, the result was corrupted. This can happen in the element expression, or from another thread (also on GIL builds):
free(): invalid sizeor a segfault.This affected:
[f(x) for x in a]list(...),tuple(...),sorted(...)andstr.join(...)with a generator argumentvec[T]([... for x in a])Repro:
Fix: keep the preallocation, which is significantly faster than appending, but handle sources whose length can change:
CPyList_SetSizeUnsafehelper; vecs update thelenfield.CPySequenceTuple_SetItemGrow), and after the loop the items are moved to a tuple of the correct size if the length changed (CPySequenceTuple_SetSize).This matches CPython: the result contains every item the loop sees. Sources whose length can't change (tuple, str, bytes, vec, list displays) generate the same IR as before.
If the list isn't modified, the only extra cost is one comparison per item. The performance of
tuple(x for x in a)stays the same as on master (n=10000: 18.2µs on master vs 17.7µs with this PR).