Repository navigation
Drain native self-host requests before stopping workerd - #109
Conversation
shuv1337
left a comment
There was a problem hiding this comment.
Review at e3ac794: LOW only. Two LOW findings, both inline. Neither blocks.
Shutdown rehearsal. Run locally with the real executor-host binary, a fake workerd and collector, and real signals sent with kill. 12 scenarios, no orphaned processes in any of them.
| Scenario | Host exit | Result |
|---|---|---|
| TERM | 2.16 s | exit 0; in-flight 3 s request got 200, SSE ended cleanly, new connections refused |
| TERM with two 20 s requests | 10.08 s | both got 503 with Retry-After: 5; exit 0 |
| TERM, then INT 1 s later | 1.05 s | drain ended early; exit 0 |
| Children ignore TERM | 10.04 s | killed and named in the log; exit 1 |
| Worst case (all children ignore TERM, 30 s request) | 23.54 s | exit 1; 6.5 s of margin under stop_grace_period: 30s |
| HUP | 0.05 s | 129; unhandled, same as before |
| Product dies early | ~1 s | exit 1 |
| TERM during startup | 0.05 s | exit 0 |
| Watchdog stall | 8.2 s | cut without drain; exit 1 |
| KILL on the host | 0.05 s | 137; Pdeathsig reaps the children |
| Ctrl-C to the whole process group | 0.05 s | 502 mid-request (LOW 1) |
Same with Setpgid: true |
after the request | 200; exit 0 |
What held
- Every wait is bounded,
os.Exitskips no defers, and theserverErrorand signal channels are buffered. init: trueand tini forward SIGTERM to the host.go vet ./...is clean, andgo test -race -count=10 ./...passes.
Workflow (fork-ci.yml)
- Triggers on
pull_request(notpull_request_target) and push. contents: read,persist-credentials: false, no secrets.- It really runs
go vetandgo test -race. - Actions are tag-pinned, the same as the existing jobs.
shuv1337
left a comment
There was a problem hiding this comment.
Inline findings for the review above, both LOW.
| docker compose -f apps/hosted/self-host/compose.yaml down | ||
| ``` | ||
|
|
||
| On SIGTERM or SIGINT the native host stops accepting new requests and gives |
There was a problem hiding this comment.
LOW 1: a terminal Ctrl-C skips this drain.
configureChild in apps/hosted/self-host/native/process_linux.go:10 sets only Pdeathsig, so the product, apps and collector share the host's process group. A Ctrl-C in a terminal (native dev, or docker run -it) sends SIGINT to every child at once. workerd exits, and in-flight requests are cut with no drain. On this head, a 5 s request got 502 ("Executor did not complete the request") and the host exited in 0.05 s.
Fix:
command.SysProcAttr = &syscall.SysProcAttr{Pdeathsig: syscall.SIGKILL, Setpgid: true}With that change, the same test returned 200 after 3.0 s and exited 0, and SIGKILL on the host still left no orphans. docker stop, compose and Railway signal PID 1 only, so they're unaffected.
There was a problem hiding this comment.
Fixed in bc5ea05. Linux children now use their own process groups while retaining Pdeathsig; macOS is unchanged. The new test sends SIGINT to an isolated host group and verifies its child survives to exit cleanly on SIGTERM.
| ``` | ||
|
|
||
| Docker's default 10-second stop timeout is shorter than the host's request drain | ||
| plus child process grace periods; keep `--stop-timeout 30` when recreating the container. |
There was a problem hiding this comment.
LOW 2: Railway gets no drain by default.
Railway's RAILWAY_DEPLOYMENT_DRAINING_SECONDS defaults to 0: per Railway's docs, the deployment "is given 0 seconds to gracefully shutdown before being forcefully stopped with a SIGKILL". So on Railway the new drain never runs, and Pdeathsig kills the children mid-request.
Fix: next to this --stop-timeout 30 note, and in the Railway section (## Deploy on Railway, and ## Railway in apps/hosted/self-host/README.md), say to set RAILWAY_DEPLOYMENT_DRAINING_SECONDS=30 (or deploy.drainingSeconds).
There was a problem hiding this comment.
Fixed in 3502bda. The Docker timeout note and both Railway sections now document setting RAILWAY_DEPLOYMENT_DRAINING_SECONDS=30 (or deploy.drainingSeconds to 30). No Railway config files were added.
|
Delta review at 3502bda (bc5ea05 and 3502bda only): clean. Both earlier LOWs are fixed.
Not a finding: Ctrl-Z now suspends only the host, and the children keep running until it resumes. |
Summary
Retry-After. Log one count by method and media type, without request URLs.Invariants
shutdownholds both supervisors at drain entry.supervisor.runchecksrunningunder the same lock ascmd.Startandhold; the hold wakes backoff.stalled; app watchdog restarts require both a live context and a running supervisor. Probes are joined after workerd is reaped.shutdowncloses the listener, drains withhttp.Server.Shutdown, cancels the edgeBaseContext(including hijacked requests), and closes the server and proxy idle pool before child stop.stopChildrencollects both shared-deadline results before the collector phase.stopProcesssends TERM, checks for exit before escalation, and reaps after KILL.shutdownbounds the drain/cut, children share their phase timer, and cancelled probes are joined after workerd exits. As with the existing host, reaping assumes the OS delivers SIGKILL and the child is not stuck in uninterruptible kernel I/O.shutdownreturns the product wait error and any named child kill error tomain.closeBridgeruns only after both workerd results; cancelling itsBaseContextalso cancels active Git subprocesses.Deadline budget: 10 s drain + up to 1 s for cut handlers to flush a 503 and release upstreams + 10 s shared product/apps grace + 3 s collector grace = about 24 s including the cut, plus OS reap overhead. Neither second signal nor forced edge cut borrows time from a child grace. Export mode retains its existing shutdown policy.
Verification
All 13 initial Go tests failed on the corresponding deliberate regression below, with the intended source restored afterward. The model was strengthened in round 1; see the verification below.
go vet ./...,go test -race -count=1 -p 2 ./...(9.845 s),bun run check, andactionlint .github/workflows/fork-ci.ymlpassed.e2e:check-descriptionstopped on the existing fork scheduling issue (cloudOAuthProxyRefusalon cloud has no CI job).TestStopClosesTheEdgeBeforeAskingWorkerdToExitTestShutdownKeepsItsOrderForEveryEventOrderTestStopLetsAShortRequestFinishTestStopCutsAStreamThatNeverEndsAtTheDrainDeadlineTestStopEndsOpenEndedStreamsAtOnceTestStopCutsAHijackedTunnelAtTheDeadlineworkerdConnTestSecondSignalEndsTheDrainTestARequestCutWhileWaitingForHeadersIsAnswered503TestSupervisorDoesNotRestartAfterHoldTestStopKillsOnlyAChildThatIgnoresSIGTERMTestStopAfterTheProductExitedSkipsTheDrainTestStopFindsChildrenAlreadyGoneTestAStalledProductAfterTheStopBeganIsIgnoredNative E2E
The named tests use the packaged workerd, Go host and checked collector. No production access, containers or broader E2E runs.
Native self-host restarts its telemetry collector and stops it with the product— about 9.7 s for the scenario.Native self-host exits cleanly on SIGTERM with open streams— 16.526 s for the scenario; 10.043 s from SIGTERM to exit 0. Both streams ended, with nokilledorReverseProxy read error. The earlier run measured 10.140 s and loggedmap[POST text/event-stream:2]. The pinned workerd releases both streams when the proxy cuts them.GET /mcpreturns 405. Both real subscriptions (POST /mcpwithsubscriptions/listenandPOST /_executor/api/subscribe) deliberately get the full 10 s drain, so the observed exit is about 10.1 s. Keep the shutdown policy unchanged: GET event-stream and NDJSON responses and 101 upgrades end at drain start; POST streams get the drain. The E2E now requires 9–15 s inclusive:drainGrace − 1 scatches an early cut, anddrainGrace + 5 sallows a margin for a clean exit. Only the named shutdown scenario was rerun for this assertion change; no Go code changed..local/native-auth/2bf1c38ccb0eae55/server.log; runner output:/tmp/shuvcode/executor-98-native-graceful-final.logon the dev worktree.Round 1 fixes
Executor watchdog: … restarting itstderr line at product-stall detection, asdocker-release-watchdog.spec.tsexpects. A watchdog stall skips the HTTP drain, aborts the edge, then gives product and apps 5 seconds together after SIGTERM before escalating. The normal signal path keeps its 10-second grace. Update the watchdog documentation and set--stop-timeout 30in the Docker quick-start.testing/synctest.Waitafter each event): three drain outcomes, six workerd exit/timeout orders, and two collector exit/timeout orders. It requires live TERM observations, kills only children whose timers fire first, keeps killed children unreaped until their exit arrives, and checks that the bridge and collector wait for both workerd reaps.Round 1 bug plants: starting the collector early and firing the workerd timer early both fail the synchronized model. Removing
close(s.held)strands the supervisor in backoff and fails the hold test. Applying early clean EOF on product failure, rewriting genuine EOF, dropping the last chunk, removing the watchdog detection log, and using the normal grace after a watchdog stall each fail their targeted regression tests.Round 1 local checks: Go vet, the full Go race suite (9.817 s), 10 iterations of the shutdown race tests (2.673 s), and
bun run checkpassed. Both named native E2Es passed once on the rebuilt Go host: collector restart/shutdown (9.611 s), and open-stream shutdown (14.766 s total, 10.556 s from SIGTERM to exit 0). Both streams ended without a kill or a proxy read error. Runner output:/tmp/shuvcode/executor-109-r1-native-e2e.log; product log:.local/native-auth/649406cab03f58ad/server.log. The release-image watchdog E2E was not run locally; the Go regression checks its exact stderr line.Round 2 follow-ups
--stop-timeout 30to the docs site's Docker example and update the release watchdog spec's header comment to describe SIGTERM and the five-second grace. No code or test assertions changed in round 2.superviseandsuperviseProcesswrappers rather than retaining them for the existing tests. The tests now callstartSupervisor,stopChildren, andrestartUnlessStoppingdirectly, with their assertions unchanged. This removes test-only production entry points and tests the same APIs the host uses.Rat-stack pages followed
Language-neutral guidance:
lore/lifecycles-are-machines,lore/implicit-state-machines,lore/every-task-needs-an-owner,lore/concurrent-work-needs-an-owner,lore/scopes-own-resources,lore/timeouts-bound-waiting,lore/supervision-is-a-recovery-policy,skills/test-behaviour,lore/model-based-testing.Closes #98