Skip to content

Stop empty-session GC from deleting restored Agent Host sessions - #328641

Draft
roblourens wants to merge 2 commits into
mainfrom
roblou/agents/investigate-lost-agent-host-sessions-175fbd7a
Draft

Stop empty-session GC from deleting restored Agent Host sessions#328641
roblourens wants to merge 2 commits into
mainfrom
roblou/agents/investigate-lost-agent-host-sessions-175fbd7a

Conversation

@roblourens

@roblourens roblourens commented Aug 2, 2026

Copy link
Copy Markdown
Member

A transient network failure during the Copilot SDK's session.resume could permanently delete an Agent Host session, its data directory, and its isolated worktree. This was found by investigating a real lost session; two sessions on one machine were destroyed this way within two days.

What happened

Reconstructed from agenthost.log:

Time Event
17:50 Last turn finishes, Session idle
17:51 Releasing idle session from memory (durable state preserved) — normal eviction
18:03 Re-subscribe → _resumeSession called — session not in memory, resuming...
18:03 ⚠️ SDK resumeSession failed: code=-32603, message=... network fetch failed: request failed: error sending request for url (https://api.github.com/copilot_internal/user)
18:03 ⚠️ Resume failed (code=-32603), falling back to createSession with same ID
18:04 Fallback createSession succeededRestored session ... with 0 turns
18:42 💀 GC: disposing empty unsubscribed session
18:42 Worktree deleted (Watcher shutdown because watched path got deleted)

Root cause

Two individually-reasonable mechanisms that are jointly destructive.

1. The resume fallback was too permissive. shouldCreateEmptySessionAfterResumeError treated any -32603 as "the SDK cannot resume an empty session" unless the message matched a corruption keyword (corrupt|invalid|validation|schema|must be|parse|malformed|unexpected token). A network failure is also -32603 and matches none of those, so it fell through to createSession({ sessionId }) — starting a fresh session over an intact event log and presenting it as having no history.

2. The empty-session GC then reaped it. _maybeScheduleSessionGc arms a 30s timer when a session loses its last subscriber with turns.length === 0; _runSessionGc then calls disposeSession, which does client.deleteSession(), deleteSessionData(), and removeCreatedWorktree() (a git worktree remove --force).

The underlying design flaw: "0 turns" was being used as a proxy for "never-used draft", but it is also exactly what a failed history load looks like — and what a truncate leaves behind. A destructive action must not be conditioned on a value that a failure path can produce.

The fix

Both ends of the sequence, because neither alone is sufficient — fixing only the predicate leaves GC trusting a fabricable value, and fixing only GC still shows the user a blank session.

Resume fallback is now an allowlist (copilotSessionLauncher.ts). It fires only when the SDK positively reports no events on disk (Session not found, no events, empty session), which still covers the "Start Over" truncate case that the fallback exists for. Everything else propagates, so a recoverable error surfaces with the on-disk history untouched.

GC is gated on an explicit unused-draft flag (agentHostStateManager.ts, agentService.ts). The flag is set when this process mints a session and latches off permanently on restore, on any turn activity, and on fork/import turn seeding. GC arms only for sessions that are still unused drafts, and _runSessionGc re-checks when the timer fires. Once a session has been used it can never look collectable again, however empty its turns become.

This is host-local bookkeeping on ISessionEntry; there is no AHP/protocol change.

Why a use flag rather than just widening the error regex

The allowlist fixes the one trigger we observed. The draft flag is categorical: it protects against any future cause of "empty-looking session with real durable state", not just this error string. Enumerating known-bad messages means every new SDK error text is a fresh data-loss bug.

It also doesn't weaken what GC is for. Of 39 GC-disposed sessions in local logs, 37 were genuine never-used drafts (correct) and 2 had real content — both being the ones that hit the network-caused resume fallback.

An earlier revision of this PR gated on creation origin (Created vs Restored). Code review correctly pointed out that origin is not the same as "never used": ChatTruncated with no turnId clears turns, and agentHostSessionHandler dispatches exactly that on a checkpoint restore or first-message edit — so a session created here, used for real work, then truncated would still have been collected. Tracking use monotonically is strictly stronger and is what landed.

Validation

  • Full agent host unit suite: 3853 passing, 0 failing
  • npm run typecheck-client clean; eslint clean on changed files
  • The GC guards were mutation-checked — each guard disabled in turn, confirming the corresponding test fails. This mattered: the first two versions of the mid-grace test passed with the guard disabled (no GC was ever armed, then the turns check was doing the work) and only became meaningful once the rehydrated session was made to return zero turns.
  • One existing test (falls back to createSession for an unknown -32603) encoded the unsafe contract and now asserts the error propagates.

Not included

Follow-ups worth doing separately: refusing to force-remove a worktree with uncommitted changes or unpushed commits, and telemetry on both the fallback and the GC-dispose paths — this surfaced only because a user noticed one missing session by hand.

(Written by Copilot)

A transient network failure during the Copilot SDK's `session.resume` could
permanently delete a session, its data directory, and its worktree.

Two independent mechanisms combined:

1. `shouldCreateEmptySessionAfterResumeError` treated any `-32603` as "the SDK
   cannot resume an empty session" unless the message matched a corruption
   keyword. `network fetch failed: ... api.github.com/copilot_internal/user` is
   also a `-32603` and matches none of those words, so the launcher fell back to
   `createSession({ sessionId })` and the session came back with zero turns even
   though its event log was intact on disk.

2. `AgentService._maybeScheduleSessionGc` arms a destructive GC whenever a
   session loses its last subscriber with `turns.length === 0`. Thirty seconds
   later `_runSessionGc` called `disposeSession`, which deletes the SDK session,
   `agentSessionData/<id>`, and the isolated worktree.

"0 turns" was standing in for "never-used draft", but it is also what a failed
history load looks like, so a recoverable error was being turned into
irreversible data loss.

Fixes, at both ends:

- The resume fallback is now an allowlist. It fires only when the SDK positively
  reports no events on disk (`Session not found`, `no events`, `empty session`),
  which still covers the "Start Over" truncate case. Everything else propagates
  so the user sees a recoverable error with their history untouched.

- `AgentHostStateManager` records host-local `SessionProvenance` (`Created` vs
  `Restored`) per session entry, exposed via `getSessionProvenance()`. GC is
  armed only for sessions this process minted, and `_runSessionGc` re-checks at
  fire time so a session rehydrated during the grace window is spared. Only an
  explicit `Restored` aborts, so genuinely evicted drafts are still collected.

No protocol change: provenance is host-local bookkeeping on `ISessionEntry`.

The existing test asserting the old behaviour ("falls back to createSession for
an unknown -32603") encoded the unsafe contract and now asserts the error
propagates instead. Both new GC tests were mutation-checked by disabling each
guard in turn and confirming the corresponding test fails.

(Written by Copilot)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings August 2, 2026 23:13

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Prevents transient resume failures from causing destructive cleanup of durable Agent Host sessions.

Changes:

  • Restricts empty-session fallback to known missing-history errors.
  • Tracks created versus restored session provenance to gate GC.
  • Adds regression tests for resume failures and GC behavior.
Show a summary per file
File Description
copilotSessionLauncher.ts Adds an allowlist for safe resume fallback.
agentService.ts Gates empty-session GC using provenance.
agentHostStateManager.ts Records and exposes session provenance.
copilotSessionLauncher.test.ts Tests safe and unsafe resume errors.
agentService.test.ts Tests restored-session GC protection.
agentHostStateManager.test.ts Tests provenance tracking and lookup.

Review details

  • Files reviewed: 6/6 changed files
  • Comments generated: 2
  • Review effort level: Balanced

Comment thread src/vs/platform/agentHost/node/agentService.ts Outdated
Comment thread src/vs/platform/agentHost/node/copilot/copilotSessionLauncher.ts Outdated
@github-actions

github-actions Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Screenshot Changes

Base: b8fc27c4 Current: ffa7835c

Changed (1)

chat/input/chatInput/VoiceModeConnecting/Dark
Before After
before after

Addresses review feedback: `Created` provenance recorded where a cache entry
originated, which is not the same as "never used". A session created in this
process can complete a turn and later receive `ChatTruncated` with no `turnId`
— a checkpoint restore or first-message edit — which clears `turns` while the
provenance stays `Created`. GC would then still arm and delete the session and
its worktree, which is the same data-loss class this PR set out to fix.

Replaces the provenance enum with an explicit, monotonic unused-draft flag:
set when this process mints a session, and latched off permanently on restore,
on any turn activity, and on fork/import turn seeding. Once a session has been
used it can never look collectable again, however empty its turns become.

Also trims the resume-fallback JSDoc to the allowlist contract.

(Written by Copilot)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants