runtime: the deadline guard's teardown cannot be interrupted into leaking a timer - #121
Merged
Merged
Conversation
…king a timer The last unverified finding from the 2026-08 sweep, recorded as "traced, never demonstrated". It is real, and it is demonstrated now: without this change the regression test records **six** further interrupts queued after the guard was released. `fire` queues its async exception while holding `lock`, so a guard already blocked on that same lock inside `disarm` is handed the exception the moment it acquires it -- at the next bytecode, which is before `armed` is cleared and before the re-armed timer is cancelled. `disarm` then propagated and the 50ms timer it was meant to cancel stayed alive, still reading `armed` as true, re-raising `NodeDeadlineExceeded` into that thread every 50ms for the life of the thread. On a pooled thread that is an unattributable crash in whatever ran next -- precisely what `test_no_interrupt_survives_the_node_that_earned_it` exists to rule out, reached by a path it did not cover. The lock was not the flaw. Both sides do take it, as the old comment said; what the lock cannot do is stop an asynchronous exception arriving between two bytecodes inside the critical section it protects. So the teardown is retried rather than abandoned, and clears `armed` outside the lock as a last resort -- that single store is what stops `fire` re-arming. Swallowing the interrupt there costs nothing: the guard decides the outcome from `state["fired"]` once `disarm` returns, and still raises on it. Mechanism 1 (SIGALRM) is not affected and is unchanged. CPython runs the Python-level handler at a bytecode boundary, so `setitimer(ITIMER_REAL, 0)` has already completed when the handler raises, and the existing `finally` restores the handler and releases the slot. Two notes on getting the test to say something true, since the first two attempts did not. Raising from `Timer.cancel` proves nothing -- by then `armed` is already false, so `fire` returns early and no timer leaks; that version passed without the fix. And raising from the lock's `__enter__` holds the lock forever, which deadlocks the very timer under test instead of letting it spin, reporting a clean zero for the wrong reason. The interrupt has to be delivered the way CPython delivers it: inside the `with` body, with the block's exit releasing the lock. Verified: 120 tests across the budget, async-kernel and lease files, ruff clean, figure refreshed to 2,190. Red without the fix, green with it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The last unverified item from the 2026-08 sweep, recorded as "traced, never demonstrated". It is real. Without this change the regression test records six further interrupts queued after the guard was released.
The race
firequeues its async exception while holdinglock. A guard already blocked on that same lock insidedisarmis therefore handed the exception the moment it acquires it — at the next bytecode, which is beforearmedis cleared and before the re-armed timer is cancelled.disarmthen propagated, and the 50 ms timer it was meant to cancel stayed alive. Itsfirestill readarmedas true, so it re-raisedNodeDeadlineExceededinto that thread every 50 ms for the life of the thread, long after the run that armed it had finished. On a pooled thread that is an unattributable crash in whatever ran next — exactly whattest_no_interrupt_survives_the_node_that_earned_itexists to rule out, arriving by a path it did not cover.The lock was not the flaw. Both sides do take it, as the old comment said. What a lock cannot do is stop an asynchronous exception arriving between two bytecodes inside the critical section it protects.
The fix
The teardown is retried rather than abandoned, and clears
armedoutside the lock as a last resort — that single store is what stopsfirere-arming. Swallowing the interrupt there costs nothing: the guard decides the outcome fromstate["fired"]oncedisarmreturns, and still raises on it, so a node that blew its deadline still fails.Mechanism 1 (SIGALRM) is unaffected and unchanged. CPython runs the Python-level handler at a bytecode boundary, so
setitimer(ITIMER_REAL, 0)has already completed by the time the handler raises, and the existingfinallyrestores the handler and releases the slot.Two wrong tests before the right one
Worth recording, because both failed in the direction that looks like success:
Timer.cancelproves nothing — by thenarmedis already false, sofirereturns early and nothing leaks. That version passed without the fix, which is how I caught that it was measuring nothing.__enter__holds the lock forever, which deadlocks the very timer under test instead of letting it spin. It reported a clean zero for entirely the wrong reason.The interrupt has to be delivered the way CPython delivers it: inside the
withbody, with the block's exit releasing the lock. The test wraps the lock and raises on the guard thread's second entry — the first being the initial arm, the seconddisarm— and gates on the timer having actually fired, so the teardown it interrupts really does have a re-armed timer to lose.The harm is then measured the way a caller feels it: whether anything is still interrupting the thread once the guard has been released.
Verification
AssertionError: 6 interrupt(s) queued after the guard was released. Green with it.ruff checkclean; figure refreshed to 2,190.With this, every finding from the 2026-08 sweep is either fixed or filed with a decision pending (#110).
🤖 Generated with Claude Code