Skip to content

Self-host server is killed instead of exiting cleanly on SIGTERM #98

Closed

Description

@shuv1337

What happens

On self-host production (ltc-executor, image built from 98d35986), the server does not exit on its own when stopped. The nightly backup (/home/exedev/executor/backup.sh, executor-v2-backup.timer, 09:00 UTC / 01:00 PST) runs docker stop -t 30 executor-v2-server-1:

  • Nov 17: the stop was logged at 09:00:26Z. About 15 s later (09:00:41Z) the host's reverse proxy logged ReverseProxy read error during body copy: unexpected EOF and Executor request failed: GET read unix @->/tmp/executor-host-…/product.sock: read: connection reset by peer, then Executor: signal: killed. The container restarted at 09:01:13Z and was healthy at 09:01:31Z.
  • Nov 16: the same pattern (stop at 09:00:36Z, restarted at 09:01:17Z).
  • Docker reports exit 0, OOMKilled=false. The container runs executor-entrypoint executor-host serve with StopTimeout 30.

So on SIGTERM the Go host (apps/hosted/self-host/native) ends up SIGKILLing the product (workerd) child instead of draining it. In-flight requests are reset rather than finished or refused cleanly.

Why it matters

  • Every nightly backup and every deploy cuts in-flight requests and kills the product without a clean shutdown. The snapshot is taken after a kill, not after a clean stop.
  • Rollback rehearsals and cutovers depend on a predictable stop.

Expected

On SIGTERM the host stops accepting new requests, lets in-flight ones finish (or answers 503), asks workerd and the collector to exit, and exits 0 well inside the stop timeout, with no signal: killed.

Notes

  • Found while tracing an unexplained 01:01 PST restart. It was the backup, not a crash or OOM.
  • Not fixed in the upstream sync (Sync upstream v2 through 56d893c24 while preserving fork contracts #96). Check whether upstream's shutdown changes in that sync already alter this before writing a fix: re-read the stop logs after the next backup on the new image.

Activity

  1. shuv1337 commented on Nov 18, 2026

    @shuv1337
    OwnerAuthor

    Nov 18 backup check (09:00 UTC / 01:00 PST, image 6fcb0fd0, build 64a2d477)

    Result: Not fixed by the upstream sync (#96). workerd was still SIGKILLed after the Go host's 15 s shutdown deadline; the stop took 15.13 s (09:00:26.671Z → 09:00:41.797Z). The Rust telemetry collector stopped cleanly in 1.4 ms, and the container came back healthy after the backup copy.

    Exact timeline (ltc-executor)

    • 09:00:26.671Z (backup.sh): docker stop -t 30 executor-v2-server-1 begins.
    • 09:00:26.7039Z → 09:00:26.7053Z (docker logs): collector logs {"level":"info","message":"stopping"} then {"level":"info","message":"collector stopped"} (1.4 ms).
    • 09:00:41.7534Z: httputil: ReverseProxy read error during body copy: unexpected EOF
    • 09:00:41.7537Z: Executor request failed: GET read unix @->/tmp/executor-host-75845027/product.sock: read: connection reset by peer
    • 09:00:41.7857Z: Executor: signal: killed
    • 09:00:41.7965Z: container FinishedAt (ExitCode: 0, OOMKilled: false).
    • 09:01:02.4406Z: container restarted after the stopped volume copy; ready in 4 s at 09:01:07.135Z, Health: healthy at 09:01:08.843Z, RestartCount: 0, Image: sha256:6fcb0fd08c758e101e693187c05dfe13af9e7642ab6227781631aa9c3f0e9c88.

    Root-cause pointer in apps/hosted/self-host/native/main.go

    On <-signals (main.go:855-876), serve signals workerd (command.Process.Signal(syscall.SIGTERM) at :868) and waits 15 s (:871-872) before shutting down or closing public (&http.Server{...} at :809-810, which only closes via defer public.Close() when serve returns). Any long-lived streaming GET (e.g. MCP/SSE) through public → product.sock stays open while workerd waits for active connections to finish, exhausting the 15 s timer until command.Process.Kill() fires.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions