Skip to content

fix(upkeep): one background helper per window, not one per process (#833) - #859

Draft
hr1juldey wants to merge 1 commit into
trailhq:mainfrom
hr1juldey:fix/673-telemetry-flush-spawn-guard
Draft

hr1juldey wants to merge 1 commit into
trailhq:mainfrom
hr1juldey:fix/673-telemetry-flush-spawn-guard

Conversation

@hr1juldey

Copy link
Copy Markdown

Fixes #833.

Root cause (what is left on current main)

#346 already made both helpers verify their timestamp write before spawning, which closes the sequential hole completely. What remains is the concurrent one: needsRefresh / needsFlush read the stale window, then write checkedAt / flushedAt, then re-read. N processes starting in the same millisecond all read stale, all write, all pass the re-read, and all spawn — a burst of session starts becomes a burst of helpers, which is exactly the storm #833 shows.

Fix

claimWindow(claimPath, staleMs, now) (util/state.ts): the winner creates the claim file exclusively (wx — an OS-level race, not a read-modify-write), every concurrent caller loses.

  • A crashed winner's claim expires after 10 minutes (UPKEEP_CLAIM_STALE_MS) — a dead process cannot block a window forever.
  • A corrupt claim file counts as expired — garbage never wedges the guarded action.
  • Any other error (unwritable home, sandbox) reports "not claimed", and the caller spawns nothing: no claim, no storm.
  • Wired into maybeRefreshInBackground and maybeFlushInBackground ahead of their existing write-reread belts, which stay.

Distinct claim files per helper (telemetry.json.claim vs update-check.json.claim), so a flush claim never blocks an update check.

Tests (RED→GREEN, adversarial)

  • the claim primitive: first caller wins; an immediate second caller loses; a stale claim is reclaimable; a corrupt claim is reclaimable;
  • a fresh foreign claim planted by a "concurrent winner": 8 flush callers, 0 spawns; claim removed: exactly 1 spawn for the window;
  • the same storm shape for the update-check path;
  • full suite: fail-6 (the six pre-existing main failures), 4551 tests.

…railhq#833)

A burst of sessions starting together spawned dozens of concurrent
_update-check and _telemetry-flush helpers. The write-then-reread
guards added in trailhq#346 exclude sequential callers only: N processes that
read the stale window in the same millisecond all pass, all write,
all spawn.

claimWindow (util/state.ts) is the cross-process mutex: the winner
creates the claim file exclusively (wx), every rival loses. A crashed
winner's claim expires after 10 minutes so a dead process cannot block
a window forever, a corrupt claim file counts as expired, and an
unwritable home claims nothing — no claim, no spawn. Wired into
maybeRefreshInBackground and maybeFlushInBackground ahead of their
existing write-reread belts, which stay.
@github-actions

Copy link
Copy Markdown
Contributor

🌱 graft blast radius

3 areas changed → 3 areas can be affected. 4 dependent symbols, depth 2.
Tests: 3 areas updated their tests.
Tag: @anirudhkumar-nanonets — 5 of 6 areas · @shhdwi — 4 of 6 areas · @miroslavkiev — Telemetry Flushing

flowchart TB
  A0(("CLI Upkeep Runner<br/>2 symbols"))
  A1(("Claude Hooks<br/>1 symbol"))
  A2(("MCP Server<br/>1 symbol"))
  classDef reached fill:#D9EDF3,stroke:#3AA7C9,stroke-width:1.5px,color:#0E313C;
  class A0,A1,A2 reached;
Loading
Can be affected Symbols Nearest hop Reached from
CLI Upkeep Runner 2 src/cli.ts:L1-L3499 cli.ts — calls, depth 1 Telemetry Flushing, Background Upkeep +1
Claude Hooks 1 src/claude/hooks.ts:L537-L627 main — calls, depth 1 Telemetry Flushing, Background Upkeep +1
MCP Server 1 src/mcp/server.ts:L101-L256 startMcpServer — calls, depth 1 Telemetry Flushing, Background Upkeep +1
Who knows this code — 4 people across 6 areas
Area Who knows it
State Window Management · changed @anirudhkumar-nanonets — 13 commits, last 16d ago
Telemetry Flushing · changed @miroslavkiev — 1 commit, last 9d ago · Marcelo Caporale — 1 commit, last 3mo ago
Background Upkeep · changed @anirudhkumar-nanonets — 6 commits, last 1mo ago · @shhdwi — 3 commits, last 23d ago
CLI Upkeep Runner · affected @anirudhkumar-nanonets — 75 commits, last yesterday · @shhdwi — 51 commits, last 22d ago
Claude Hooks · affected @anirudhkumar-nanonets — 23 commits, last 10d ago · @shhdwi — 14 commits, last 22d ago
MCP Server · affected @anirudhkumar-nanonets — 7 commits, last 22d ago · @shhdwi — 5 commits, last 22d ago

Ownership is git history over each area's own files, weighted towards recent work (120-day half-life). Merge commits and bots are dropped, and you are dropped from your own PR. A name with no @ has no GitHub handle in its commit email — tag them by hand, or add a .mailmap entry. A suggestion from history, not a CODEOWNERS rule.

All 4 dependent symbols, grouped by area

CLI Upkeep Runner — 2 symbols in 2 files

  • src/cli.ts:L1-L3499 — cli.ts (calls, depth 1)
    111: import { formatUpdateNudge, maybeRefreshInBackground, readStamp, readUpdateCache, refreshUpdateCache, runningVersion, wiredHostIds, wiringOpts, writeStamp } from "./upkeep.js";
  • src/upkeep-run.ts:L85-L102 — runUpkeep (calls, depth 1)
    97: if (opts.background !== false) maybeRefreshInBackground(opts.home);

Claude Hooks — 1 symbol in 1 file

  • src/claude/hooks.ts:L537-L627 — main (calls, depth 1)
    575: maybeFlushInBackground();

MCP Server — 1 symbol in 1 file

  • src/mcp/server.ts:L101-L256 — startMcpServer (calls, depth 1)
    123: if (!brainsOnly) maybeFlushInBackground();
Test signal per changed area — 3 ✓

Reached = a node under a test path has a resolved edge into the changed symbol. It undercounts anything called indirectly — through a CLI, a spawned process or a dynamic import — so read a low ratio as “look here”, never as a coverage gate.

  • ✓ State Window Management — 1 of 1 reached · 1 test file changed here: test/upkeep-claim.test.ts
  • ✓ Telemetry Flushing — 1 of 1 reached · 1 test file changed here: test/upkeep-claim.test.ts
  • ✓ Background Upkeep — 1 of 1 reached · 1 test file changed here: test/upkeep-claim.test.ts
21 test suites also reference this code

23 symbols, kept out of the diagram and the table so they cannot crowd out the areas a reviewer has to look at.

  • test/claude-hooks-perf.test.ts
  • test/claude-hooks.test.ts
  • test/claude-host-standdown.test.ts
  • test/claude-prompt-learnings.test.ts
  • test/claude-prompt-private.test.ts
  • test/claude-session-start-learnings.test.ts
  • test/claude-skills-journey-fixtures.ts
  • test/claude-stop-trace.test.ts
  • test/claude-stop.test.ts
  • test/claude-tally.test.ts
  • test/graft-to-trail.test.ts
  • test/hook-savings.test.ts
  • test/hosts-trail.test.ts
  • test/init-agents-auto.test.ts
  • test/telemetry-flush.test.ts
  • test/trail-init-upgrade.test.ts
  • test/trail-learning-on-touch.test.ts
  • test/trail-note-ask.test.ts
  • test/trail-text-safety.test.ts
  • test/upkeep-hooks.test.ts
  • …1 more

graft blast · origin/main...HEAD · depth 2 · 4 changed files

Open the interactive graph → — click an area to see its dependent symbols at file:line.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(critical): Multiple Trail background processes causing excessive CPU usage

1 participant