Stop empty-session GC from deleting restored Agent Host sessions - #328641
Draft
roblourens wants to merge 2 commits into
Draft
Stop empty-session GC from deleting restored Agent Host sessions#328641roblourens wants to merge 2 commits into
roblourens wants to merge 2 commits into
Conversation
A transient network failure during the Copilot SDK's `session.resume` could
permanently delete a session, its data directory, and its worktree.
Two independent mechanisms combined:
1. `shouldCreateEmptySessionAfterResumeError` treated any `-32603` as "the SDK
cannot resume an empty session" unless the message matched a corruption
keyword. `network fetch failed: ... api.github.com/copilot_internal/user` is
also a `-32603` and matches none of those words, so the launcher fell back to
`createSession({ sessionId })` and the session came back with zero turns even
though its event log was intact on disk.
2. `AgentService._maybeScheduleSessionGc` arms a destructive GC whenever a
session loses its last subscriber with `turns.length === 0`. Thirty seconds
later `_runSessionGc` called `disposeSession`, which deletes the SDK session,
`agentSessionData/<id>`, and the isolated worktree.
"0 turns" was standing in for "never-used draft", but it is also what a failed
history load looks like, so a recoverable error was being turned into
irreversible data loss.
Fixes, at both ends:
- The resume fallback is now an allowlist. It fires only when the SDK positively
reports no events on disk (`Session not found`, `no events`, `empty session`),
which still covers the "Start Over" truncate case. Everything else propagates
so the user sees a recoverable error with their history untouched.
- `AgentHostStateManager` records host-local `SessionProvenance` (`Created` vs
`Restored`) per session entry, exposed via `getSessionProvenance()`. GC is
armed only for sessions this process minted, and `_runSessionGc` re-checks at
fire time so a session rehydrated during the grace window is spared. Only an
explicit `Restored` aborts, so genuinely evicted drafts are still collected.
No protocol change: provenance is host-local bookkeeping on `ISessionEntry`.
The existing test asserting the old behaviour ("falls back to createSession for
an unknown -32603") encoded the unsafe contract and now asserts the error
propagates instead. Both new GC tests were mutation-checked by disabling each
guard in turn and confirming the corresponding test fails.
(Written by Copilot)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Prevents transient resume failures from causing destructive cleanup of durable Agent Host sessions.
Changes:
- Restricts empty-session fallback to known missing-history errors.
- Tracks created versus restored session provenance to gate GC.
- Adds regression tests for resume failures and GC behavior.
Show a summary per file
| File | Description |
|---|---|
copilotSessionLauncher.ts |
Adds an allowlist for safe resume fallback. |
agentService.ts |
Gates empty-session GC using provenance. |
agentHostStateManager.ts |
Records and exposes session provenance. |
copilotSessionLauncher.test.ts |
Tests safe and unsafe resume errors. |
agentService.test.ts |
Tests restored-session GC protection. |
agentHostStateManager.test.ts |
Tests provenance tracking and lookup. |
Review details
- Files reviewed: 6/6 changed files
- Comments generated: 2
- Review effort level: Balanced
Contributor
Addresses review feedback: `Created` provenance recorded where a cache entry originated, which is not the same as "never used". A session created in this process can complete a turn and later receive `ChatTruncated` with no `turnId` — a checkpoint restore or first-message edit — which clears `turns` while the provenance stays `Created`. GC would then still arm and delete the session and its worktree, which is the same data-loss class this PR set out to fix. Replaces the provenance enum with an explicit, monotonic unused-draft flag: set when this process mints a session, and latched off permanently on restore, on any turn activity, and on fork/import turn seeding. Once a session has been used it can never look collectable again, however empty its turns become. Also trims the resume-fallback JSDoc to the allowlist contract. (Written by Copilot) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A transient network failure during the Copilot SDK's
session.resumecould permanently delete an Agent Host session, its data directory, and its isolated worktree. This was found by investigating a real lost session; two sessions on one machine were destroyed this way within two days.What happened
Reconstructed from
agenthost.log:Session idleReleasing idle session from memory (durable state preserved)— normal eviction_resumeSession called — session not in memory, resuming...SDK resumeSession failed: code=-32603, message=... network fetch failed: request failed: error sending request for url (https://api.github.com/copilot_internal/user)Resume failed (code=-32603), falling back to createSession with same IDFallback createSession succeeded→Restored session ... with 0 turnsGC: disposing empty unsubscribed sessionWatcher shutdown because watched path got deleted)Root cause
Two individually-reasonable mechanisms that are jointly destructive.
1. The resume fallback was too permissive.
shouldCreateEmptySessionAfterResumeErrortreated any-32603as "the SDK cannot resume an empty session" unless the message matched a corruption keyword (corrupt|invalid|validation|schema|must be|parse|malformed|unexpected token). A network failure is also-32603and matches none of those, so it fell through tocreateSession({ sessionId })— starting a fresh session over an intact event log and presenting it as having no history.2. The empty-session GC then reaped it.
_maybeScheduleSessionGcarms a 30s timer when a session loses its last subscriber withturns.length === 0;_runSessionGcthen callsdisposeSession, which doesclient.deleteSession(),deleteSessionData(), andremoveCreatedWorktree()(agit worktree remove --force).The underlying design flaw: "0 turns" was being used as a proxy for "never-used draft", but it is also exactly what a failed history load looks like — and what a truncate leaves behind. A destructive action must not be conditioned on a value that a failure path can produce.
The fix
Both ends of the sequence, because neither alone is sufficient — fixing only the predicate leaves GC trusting a fabricable value, and fixing only GC still shows the user a blank session.
Resume fallback is now an allowlist (
copilotSessionLauncher.ts). It fires only when the SDK positively reports no events on disk (Session not found,no events,empty session), which still covers the "Start Over" truncate case that the fallback exists for. Everything else propagates, so a recoverable error surfaces with the on-disk history untouched.GC is gated on an explicit unused-draft flag (
agentHostStateManager.ts,agentService.ts). The flag is set when this process mints a session and latches off permanently on restore, on any turn activity, and on fork/import turn seeding. GC arms only for sessions that are still unused drafts, and_runSessionGcre-checks when the timer fires. Once a session has been used it can never look collectable again, however empty its turns become.This is host-local bookkeeping on
ISessionEntry; there is no AHP/protocol change.Why a use flag rather than just widening the error regex
The allowlist fixes the one trigger we observed. The draft flag is categorical: it protects against any future cause of "empty-looking session with real durable state", not just this error string. Enumerating known-bad messages means every new SDK error text is a fresh data-loss bug.
It also doesn't weaken what GC is for. Of 39 GC-disposed sessions in local logs, 37 were genuine never-used drafts (correct) and 2 had real content — both being the ones that hit the network-caused resume fallback.
An earlier revision of this PR gated on creation origin (
CreatedvsRestored). Code review correctly pointed out that origin is not the same as "never used":ChatTruncatedwith noturnIdclearsturns, andagentHostSessionHandlerdispatches exactly that on a checkpoint restore or first-message edit — so a session created here, used for real work, then truncated would still have been collected. Tracking use monotonically is strictly stronger and is what landed.Validation
npm run typecheck-clientclean;eslintclean on changed filesfalls back to createSession for an unknown -32603) encoded the unsafe contract and now asserts the error propagates.Not included
Follow-ups worth doing separately: refusing to force-remove a worktree with uncommitted changes or unpushed commits, and telemetry on both the fallback and the GC-dispose paths — this surfaced only because a user noticed one missing session by hand.
(Written by Copilot)