Skip to content

session: a session whose runner died can be reclaimed deliberately - #127

Merged
Shashankss1205 merged 1 commit into
mainfrom
feat/issue-19-reclaim-dead-runner
Sep 26, 2026
Merged

Shashankss1205 merged 1 commit into
mainfrom
feat/issue-19-reclaim-dead-runner

Conversation

@Shashankss1205

Copy link
Copy Markdown
Collaborator

Closes #19. Design decisions were posted as a comment on the issue first, as it asked; this is the implementation of them.

The defect, reproduced on main

status: running  runner_pid: 58011  alive: False
WEDGED: session 'crashme' is running; expected one of ['idle', 'interrupted', ...]
release methods available: NONE
manager methods: NONE

The row keeps the dead runner_pid, so the store knows who died holding it and had no way to act on that. Every later run() raised SessionBusy forever, and the only remedy was sqlite3 ... UPDATE, which skips the lifecycle check and writes no transition row — corrupting the audit trail the store exists to keep.

The decisions

failed, not interrupted. interrupted says a turn stopped somewhere it can be picked up from; a runner that died left no such point. failed says the turn did not settle, which is true, and is already in RESUMABLE.

Open holds survive. pending_approval is untouched, so a session waiting on a human is still waiting afterwards — reclaiming a wedged session must not be a way past an approval gate. Asserted.

The liveness check is inside BEGIN IMMEDIATE, not before it, so two operators reclaiming at once serialise on it. A check outside the write lock is the same deferred-read race _transaction's docstring already describes for claiming.

Nothing calls it automatically. Pid liveness is host-local and pids are recycled; an automatic sweep would be a way for two live runners to fight over one session, which is exactly what the claim exists to prevent.

Four refusals, each tested

Condition Why
not running there is no claim to release
pid alive (incl. PermissionError → read as alive) a live pid does not prove the runner lives, but proves it might
pid is this process a caller bug, not a crash
running with no runner_pid the row is inconsistent in its own right; papering over it hides a different bug

Pid reuse is not solved and is not pretended to be: a recycled pid reads as alive, which produces a refusal — the safe direction.

The tests are real, not mocked

  • The crash is a child that claims the session and os._exits, skipping every cleanup path — what a SIGKILL looks like from the store's side. (stdout is flushed by hand, because _exit will not.)
  • The alive-refusal test parks a real child on a file barrier rather than inventing a pid, and asserts the refusal left no transition row and no status move.
  • The dead pid used in the hold test is spawned and reaped rather than guessed: a made-up number can belong to something, and the check under test refuses a live pid, so an unlucky guess would pass or fail for the wrong reason.

All five are red without the source change.

Docs

The three honesty paragraphs this obsoletes are updated: store.transition's docstring, session/runtime.py's limitations list, and the deep dive's. The issue pointed at README.md:499, but that sentence has since moved to the deep dive — checked rather than assumed.

Out of scope, as the issue set it

A cross-host heartbeat/expiry lease, and any automatic background sweeper.

Verified

2,210 selected, 13 deselected, ruff check . clean; the session suite is 48 green.

🤖 Generated with Claude Code

Closes #19. A runner's claim is a compare-and-set, not a lease: it stops a
second runner from claiming a session and cannot notice one that died holding
it. The row stayed `running` with the dead `runner_pid` still on it -- so the
store knew who died and offered no way to act on it. Every later `run()` raised
`SessionBusy` forever, and the only remedy was a hand-written UPDATE, which
skips the lifecycle check and writes no transition row, corrupting the audit
trail the store exists to keep.

`SessionStore.release_dead_runner`, surfaced as `SessionManager.reclaim`.

**`failed`, not `interrupted`.** `interrupted` says a turn stopped somewhere it
can be picked up from; a runner that died left no such point. `failed` says the
turn did not settle, which is true, and is already resumable.

**Open holds survive.** `pending_approval` is left alone, so a session waiting
on a human is still waiting afterwards. Reclaiming a wedged session must not be
a way past an approval gate, and a test asserts the hold is still there.

**The liveness check runs inside `BEGIN IMMEDIATE`**, not before it, so two
operators reclaiming at once serialise on it. A check outside the write lock is
the same deferred-read race `_transaction`'s docstring already describes for
claiming.

Four refusals, each with a test: the session is not `running`; the pid is alive
(or belongs to another user, `PermissionError`, read as alive on purpose); the
pid is this process; a `running` row carries no `runner_pid` at all, which is
inconsistent in its own right and not something to paper over.

Pid reuse is not solved and is not pretended to be: a recycled pid reads as
alive, which is a refusal -- the safe direction. That is why nothing calls this
automatically. An automatic sweep would be a way for two live runners to fight
over one session, which is what the claim exists to prevent.

The crash is a real one in the tests: a child claims the session and `os._exit`s,
skipping every cleanup path, which is what a SIGKILL looks like from the store's
side. The alive-refusal test parks a real child on a file barrier rather than
inventing a pid, and the dead pid used elsewhere is spawned and reaped rather
than guessed -- a made-up number can belong to something, and the check under
test refuses a live pid.

The three honesty paragraphs this obsoletes are updated: `store.transition`'s
docstring, `session/runtime.py`'s limitations list, and the deep dive's. The
README no longer states it -- that sentence moved to the deep dive since the
issue was filed.

Verified: 2210 selected, 13 deselected, ruff clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@Shashankss1205
Shashankss1205 merged commit 0ade7ca into main Sep 26, 2026
7 checks passed
@Shashankss1205
Shashankss1205 deleted the feat/issue-19-reclaim-dead-runner branch September 26, 2026 13:02
Shashankss1205 added a commit that referenced this pull request Sep 26, 2026
#127 and #126 merged after the release commit and neither is in the notes.
0.1.8 is still unpublished and untagged, so folding them in is correct for the
same reason #124 did it for #123.

The reclaim gets a full entry: it changes shipped behaviour and adds a public
method, and the entry is explicit that it closes a documented *limitation*
rather than a defect -- which comes to the same thing for whoever hit it. What
it refuses, and why nothing calls it automatically, are the load-bearing parts
and are stated.

The styling gate goes in the tooling paragraph, where it belongs: it changes no
shipped behaviour, it changes what a regression in shipped behaviour would be
caught by.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

session: nothing reclaims a session whose runner died holding it

1 participant