Skip to content

Drain native self-host requests before stopping workerd - #109

Merged
shuv1337 merged 8 commits into
v2from
fix/98-graceful-shutdown
Nov 19, 2026
Merged

shuv1337 merged 8 commits into
v2from
fix/98-graceful-shutdown

Conversation

@shuv1337

@shuv1337 shuv1337 commented Nov 19, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • Drain the native public HTTP edge before stopping workerd. SIGTERM and SIGINT take the same path; a second signal cuts the drain without shortening the child graces. Requests waiting on response headers get 503 with Retry-After. Log one count by method and media type, without request URLs.
  • Hold the apps and collector supervisors, cancel watchdogs and the readiness probe, and reap product and apps concurrently. Only then cancel/close the native bridge and stop the collector. The open-ended GET/101 policy is a separate commit.
  • Document shutdown, add process-backed Go lifecycle coverage and a GitHub-hosted Go vet/race job, and add a real-workerd regression scenario.

Invariants

Enforcement
I1 — no new child after shutdown takes ownership shutdown holds both supervisors at drain entry. supervisor.run checks running under the same lock as cmd.Start and hold; the hold wakes backoff.
I2 — no watchdog recovery during shutdown Probe contexts are cancelled at drain entry. Only the serving select reads stalled; app watchdog restarts require both a live context and a running supervisor. Probes are joined after workerd is reaped.
I3 — workerd is signalled only after the public listener closes and requests finish or are cut shutdown closes the listener, drains with http.Server.Shutdown, cancels the edge BaseContext (including hijacked requests), and closes the server and proxy idle pool before child stop.
I4 — collector is signalled after both workerd processes are reaped stopChildren collects both shared-deadline results before the collector phase.
I5 — only a child past its grace is killed during shutdown stopProcess sends TERM, checks for exit before escalation, and reaps after KILL.
I6 — all waits are bounded by phase budgets shutdown bounds the drain/cut, children share their phase timer, and cancelled probes are joined after workerd exits. As with the existing host, reaping assumes the OS delivers SIGKILL and the child is not stuck in uninterruptible kernel I/O.
I7 — zero only for a clean product exit without forced termination shutdown returns the product wait error and any named child kill error to main.
I8 — the bridge stays open until both workerd processes are reaped closeBridge runs only after both workerd results; cancelling its BaseContext also cancels active Git subprocesses.

Deadline budget: 10 s drain + up to 1 s for cut handlers to flush a 503 and release upstreams + 10 s shared product/apps grace + 3 s collector grace = about 24 s including the cut, plus OS reap overhead. Neither second signal nor forced edge cut borrows time from a child grace. Export mode retains its existing shutdown policy.

Verification

All 13 initial Go tests failed on the corresponding deliberate regression below, with the intended source restored afterward. The model was strengthened in round 1; see the verification below.

go vet ./..., go test -race -count=1 -p 2 ./... (9.845 s), bun run check, and actionlint .github/workflows/fork-ci.yml passed. e2e:check-description stopped on the existing fork scheduling issue (cloudOAuthProxyRefusal on cloud has no CI job).

New Go test Planted error that must fail
TestStopClosesTheEdgeBeforeAskingWorkerdToExit Signal workerd before closing the listener/cutting its connection
TestShutdownKeepsItsOrderForEveryEventOrder Close the bridge before workerd is reaped
TestStopLetsAShortRequestFinish Skip the request drain
TestStopCutsAStreamThatNeverEndsAtTheDrainDeadline Drop the edge cut
TestStopEndsOpenEndedStreamsAtOnce Do not end open-ended streams at drain entry
TestStopCutsAHijackedTunnelAtTheDeadline Drop the edge cut with a live tunnel through the real proxy and workerdConn
TestSecondSignalEndsTheDrain Ignore the second signal
TestARequestCutWhileWaitingForHeadersIsAnswered503 Skip the shutdown-specific proxy error response
TestSupervisorDoesNotRestartAfterHold Let a watchdog restart a held child
TestStopKillsOnlyAChildThatIgnoresSIGTERM Kill immediately after sending TERM
TestStopAfterTheProductExitedSkipsTheDrain Keep waiting for the drain after seeing product exit
TestStopFindsChildrenAlreadyGone Wait out the grace for an already-reaped child
TestAStalledProductAfterTheStopBeganIsIgnored Accept stale watchdog reports while draining

Native E2E

The named tests use the packaged workerd, Go host and checked collector. No production access, containers or broader E2E runs.

  • PASS: Native self-host restarts its telemetry collector and stops it with the product — about 9.7 s for the scenario.
  • PASS: Native self-host exits cleanly on SIGTERM with open streams — 16.526 s for the scenario; 10.043 s from SIGTERM to exit 0. Both streams ended, with no killed or ReverseProxy read error. The earlier run measured 10.140 s and logged map[POST text/event-stream:2]. The pinned workerd releases both streams when the proxy cuts them.
  • Timing decision: the original 5 s target assumed a GET stream, but GET /mcp returns 405. Both real subscriptions (POST /mcp with subscriptions/listen and POST /_executor/api/subscribe) deliberately get the full 10 s drain, so the observed exit is about 10.1 s. Keep the shutdown policy unchanged: GET event-stream and NDJSON responses and 101 upgrades end at drain start; POST streams get the drain. The E2E now requires 9–15 s inclusive: drainGrace − 1 s catches an early cut, and drainGrace + 5 s allows a margin for a clean exit. Only the named shutdown scenario was rerun for this assertion change; no Go code changed.
  • Product log: .local/native-auth/2bf1c38ccb0eae55/server.log; runner output: /tmp/shuvcode/executor-98-native-graceful-final.log on the dev worktree.

Round 1 fixes

  • Restore the original Executor watchdog: … restarting it stderr line at product-stall detection, as docker-release-watchdog.spec.ts expects. A watchdog stall skips the HTTP drain, aborts the edge, then gives product and apps 5 seconds together after SIGTERM before escalating. The normal signal path keeps its 10-second grace. Update the watchdog documentation and set --stop-timeout 30 in the Docker quick-start.
  • Unexpected product exit and watchdog stall do not cleanly end GET streams; they abort the edge. A body that reaches genuine EOF at the deadline still completes normally, and a GET stream closed at drain start delivers any bytes returned by its final read.
  • Hold both supervisors before startup-failure cleanup can wait on a child. Remove redundant normal-stop cleanup and unused supervisor states.
  • The shutdown model drives 36 phase-valid histories (testing/synctest.Wait after each event): three drain outcomes, six workerd exit/timeout orders, and two collector exit/timeout orders. It requires live TERM observations, kills only children whose timers fire first, keeps killed children unreaped until their exit arrives, and checks that the bridge and collector wait for both workerd reaps.

Round 1 bug plants: starting the collector early and firing the workerd timer early both fail the synchronized model. Removing close(s.held) strands the supervisor in backoff and fails the hold test. Applying early clean EOF on product failure, rewriting genuine EOF, dropping the last chunk, removing the watchdog detection log, and using the normal grace after a watchdog stall each fail their targeted regression tests.

Round 1 local checks: Go vet, the full Go race suite (9.817 s), 10 iterations of the shutdown race tests (2.673 s), and bun run check passed. Both named native E2Es passed once on the rebuilt Go host: collector restart/shutdown (9.611 s), and open-stream shutdown (14.766 s total, 10.556 s from SIGTERM to exit 0). Both streams ended without a kill or a proxy read error. Runner output: /tmp/shuvcode/executor-109-r1-native-e2e.log; product log: .local/native-auth/649406cab03f58ad/server.log. The release-image watchdog E2E was not run locally; the Go regression checks its exact stderr line.

Round 2 follow-ups

  • Add --stop-timeout 30 to the docs site's Docker example and update the release watchdog spec's header comment to describe SIGTERM and the five-second grace. No code or test assertions changed in round 2.
  • Deviation from the judged plan: Round 1 removed the old supervise and superviseProcess wrappers rather than retaining them for the existing tests. The tests now call startSupervisor, stopChildren, and restartUnlessStopping directly, with their assertions unchanged. This removes test-only production entry points and tests the same APIs the host uses.

Rat-stack pages followed

Language-neutral guidance: lore/lifecycles-are-machines, lore/implicit-state-machines, lore/every-task-needs-an-owner, lore/concurrent-work-needs-an-owner, lore/scopes-own-resources, lore/timeouts-bound-waiting, lore/supervision-is-a-recovery-policy, skills/test-behaviour, lore/model-based-testing.

Closes #98

self-host-native-telemetry.spec.ts

@shuv1337
shuv1337 marked this pull request as ready for review November 19, 2026 21:23

@shuv1337 shuv1337 left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review at e3ac794: LOW only. Two LOW findings, both inline. Neither blocks.

Shutdown rehearsal. Run locally with the real executor-host binary, a fake workerd and collector, and real signals sent with kill. 12 scenarios, no orphaned processes in any of them.

Scenario Host exit Result
TERM 2.16 s exit 0; in-flight 3 s request got 200, SSE ended cleanly, new connections refused
TERM with two 20 s requests 10.08 s both got 503 with Retry-After: 5; exit 0
TERM, then INT 1 s later 1.05 s drain ended early; exit 0
Children ignore TERM 10.04 s killed and named in the log; exit 1
Worst case (all children ignore TERM, 30 s request) 23.54 s exit 1; 6.5 s of margin under stop_grace_period: 30s
HUP 0.05 s 129; unhandled, same as before
Product dies early ~1 s exit 1
TERM during startup 0.05 s exit 0
Watchdog stall 8.2 s cut without drain; exit 1
KILL on the host 0.05 s 137; Pdeathsig reaps the children
Ctrl-C to the whole process group 0.05 s 502 mid-request (LOW 1)
Same with Setpgid: true after the request 200; exit 0

What held

  • Every wait is bounded, os.Exit skips no defers, and the serverError and signal channels are buffered.
  • init: true and tini forward SIGTERM to the host.
  • go vet ./... is clean, and go test -race -count=10 ./... passes.

Workflow (fork-ci.yml)

  • Triggers on pull_request (not pull_request_target) and push.
  • contents: read, persist-credentials: false, no secrets.
  • It really runs go vet and go test -race.
  • Actions are tag-pinned, the same as the existing jobs.

@shuv1337 shuv1337 left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Inline findings for the review above, both LOW.

docker compose -f apps/hosted/self-host/compose.yaml down
```

On SIGTERM or SIGINT the native host stops accepting new requests and gives

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LOW 1: a terminal Ctrl-C skips this drain.

configureChild in apps/hosted/self-host/native/process_linux.go:10 sets only Pdeathsig, so the product, apps and collector share the host's process group. A Ctrl-C in a terminal (native dev, or docker run -it) sends SIGINT to every child at once. workerd exits, and in-flight requests are cut with no drain. On this head, a 5 s request got 502 ("Executor did not complete the request") and the host exited in 0.05 s.

Fix:

command.SysProcAttr = &syscall.SysProcAttr{Pdeathsig: syscall.SIGKILL, Setpgid: true}

With that change, the same test returned 200 after 3.0 s and exited 0, and SIGKILL on the host still left no orphans. docker stop, compose and Railway signal PID 1 only, so they're unaffected.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in bc5ea05. Linux children now use their own process groups while retaining Pdeathsig; macOS is unchanged. The new test sends SIGINT to an isolated host group and verifies its child survives to exit cleanly on SIGTERM.

```

Docker's default 10-second stop timeout is shorter than the host's request drain
plus child process grace periods; keep `--stop-timeout 30` when recreating the container.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LOW 2: Railway gets no drain by default.

Railway's RAILWAY_DEPLOYMENT_DRAINING_SECONDS defaults to 0: per Railway's docs, the deployment "is given 0 seconds to gracefully shutdown before being forcefully stopped with a SIGKILL". So on Railway the new drain never runs, and Pdeathsig kills the children mid-request.

Fix: next to this --stop-timeout 30 note, and in the Railway section (## Deploy on Railway, and ## Railway in apps/hosted/self-host/README.md), say to set RAILWAY_DEPLOYMENT_DRAINING_SECONDS=30 (or deploy.drainingSeconds).

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 3502bda. The Docker timeout note and both Railway sections now document setting RAILWAY_DEPLOYMENT_DRAINING_SECONDS=30 (or deploy.drainingSeconds to 30). No Railway config files were added.

Copy link
Copy Markdown
Owner Author

Delta review at 3502bda (bc5ea05 and 3502bda only): clean. Both earlier LOWs are fixed.

  • LOW 1, Setpgid (bc5ea05): fixed. configureChild now sets Setpgid: true alongside Pdeathsig, so a group-wide SIGINT reaches only the host and it drains. Supervised children don't set stdin, so they read /dev/null and a background group can't get SIGTTIN.
    • The new TestConfigureChildIsolatesTerminalSIGINT runs in its own process group, so it never signals the test runner. It passes 20 of 20 runs under -race, and it fails when Setpgid is removed, so it really guards the fix.
    • go vet is clean, and go test -race -count=3 ./... passes.
    • The e2e telemetry change now signals the collector separately, which matches the new behaviour.
  • LOW 2, Railway drain (3502bda): fixed. RAILWAY_DEPLOYMENT_DRAINING_SECONDS=30 / deploy.drainingSeconds is now in both Railway sections and next to the Docker --stop-timeout note.

Not a finding: Ctrl-Z now suspends only the host, and the children keep running until it resumes.

@shuv1337
shuv1337 merged commit 7193e6a into v2 Nov 19, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Self-host server is killed instead of exiting cleanly on SIGTERM

1 participant