Skip to content

runtime: the deadline guard's teardown cannot be interrupted into leaking a timer - #121

Merged
Shashankss1205 merged 1 commit into
mainfrom
fix/deadline-guard-teardown-interrupt
Sep 26, 2026
Merged

Shashankss1205 merged 1 commit into
mainfrom
fix/deadline-guard-teardown-interrupt

Conversation

@Shashankss1205

Copy link
Copy Markdown
Collaborator

The last unverified item from the 2026-08 sweep, recorded as "traced, never demonstrated". It is real. Without this change the regression test records six further interrupts queued after the guard was released.

The race

fire queues its async exception while holding lock. A guard already blocked on that same lock inside disarm is therefore handed the exception the moment it acquires it — at the next bytecode, which is before armed is cleared and before the re-armed timer is cancelled.

disarm then propagated, and the 50 ms timer it was meant to cancel stayed alive. Its fire still read armed as true, so it re-raised NodeDeadlineExceeded into that thread every 50 ms for the life of the thread, long after the run that armed it had finished. On a pooled thread that is an unattributable crash in whatever ran next — exactly what test_no_interrupt_survives_the_node_that_earned_it exists to rule out, arriving by a path it did not cover.

The lock was not the flaw. Both sides do take it, as the old comment said. What a lock cannot do is stop an asynchronous exception arriving between two bytecodes inside the critical section it protects.

The fix

The teardown is retried rather than abandoned, and clears armed outside the lock as a last resort — that single store is what stops fire re-arming. Swallowing the interrupt there costs nothing: the guard decides the outcome from state["fired"] once disarm returns, and still raises on it, so a node that blew its deadline still fails.

Mechanism 1 (SIGALRM) is unaffected and unchanged. CPython runs the Python-level handler at a bytecode boundary, so setitimer(ITIMER_REAL, 0) has already completed by the time the handler raises, and the existing finally restores the handler and releases the slot.

Two wrong tests before the right one

Worth recording, because both failed in the direction that looks like success:

  1. Raising from Timer.cancel proves nothing — by then armed is already false, so fire returns early and nothing leaks. That version passed without the fix, which is how I caught that it was measuring nothing.
  2. Raising from the lock's __enter__ holds the lock forever, which deadlocks the very timer under test instead of letting it spin. It reported a clean zero for entirely the wrong reason.

The interrupt has to be delivered the way CPython delivers it: inside the with body, with the block's exit releasing the lock. The test wraps the lock and raises on the guard thread's second entry — the first being the initial arm, the second disarm — and gates on the timer having actually fired, so the teardown it interrupts really does have a re-armed timer to lose.

The harm is then measured the way a caller feels it: whether anything is still interrupting the thread once the guard has been released.

Verification

  • Red without the fix: AssertionError: 6 interrupt(s) queued after the guard was released. Green with it.
  • 120 tests across the budget, async-kernel and lease files pass; ruff check clean; figure refreshed to 2,190.
  • Full matrix left to CI.

With this, every finding from the 2026-08 sweep is either fixed or filed with a decision pending (#110).

🤖 Generated with Claude Code

…king a timer

The last unverified finding from the 2026-08 sweep, recorded as "traced,
never demonstrated". It is real, and it is demonstrated now: without this
change the regression test records **six** further interrupts queued after
the guard was released.

`fire` queues its async exception while holding `lock`, so a guard already
blocked on that same lock inside `disarm` is handed the exception the moment
it acquires it -- at the next bytecode, which is before `armed` is cleared
and before the re-armed timer is cancelled. `disarm` then propagated and the
50ms timer it was meant to cancel stayed alive, still reading `armed` as
true, re-raising `NodeDeadlineExceeded` into that thread every 50ms for the
life of the thread. On a pooled thread that is an unattributable crash in
whatever ran next -- precisely what
`test_no_interrupt_survives_the_node_that_earned_it` exists to rule out,
reached by a path it did not cover.

The lock was not the flaw. Both sides do take it, as the old comment said;
what the lock cannot do is stop an asynchronous exception arriving between
two bytecodes inside the critical section it protects. So the teardown is
retried rather than abandoned, and clears `armed` outside the lock as a last
resort -- that single store is what stops `fire` re-arming. Swallowing the
interrupt there costs nothing: the guard decides the outcome from
`state["fired"]` once `disarm` returns, and still raises on it.

Mechanism 1 (SIGALRM) is not affected and is unchanged. CPython runs the
Python-level handler at a bytecode boundary, so `setitimer(ITIMER_REAL, 0)`
has already completed when the handler raises, and the existing `finally`
restores the handler and releases the slot.

Two notes on getting the test to say something true, since the first two
attempts did not. Raising from `Timer.cancel` proves nothing -- by then
`armed` is already false, so `fire` returns early and no timer leaks; that
version passed without the fix. And raising from the lock's `__enter__`
holds the lock forever, which deadlocks the very timer under test instead of
letting it spin, reporting a clean zero for the wrong reason. The interrupt
has to be delivered the way CPython delivers it: inside the `with` body,
with the block's exit releasing the lock.

Verified: 120 tests across the budget, async-kernel and lease files, ruff
clean, figure refreshed to 2,190. Red without the fix, green with it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@Shashankss1205
Shashankss1205 merged commit 3476104 into main Sep 26, 2026
7 checks passed
@Shashankss1205
Shashankss1205 deleted the fix/deadline-guard-teardown-interrupt branch September 26, 2026 09:29
@Shashankss1205 Shashankss1205 mentioned this pull request Sep 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant