Skip to content

feat(deploy): run stagehand next to its data; one tunnel to the hetzner2 sandbox; safe stale-session cleanup (#15) - #17

Merged
jerryfane merged 5 commits into
mainfrom
feat/17-runner-on-data-host
Nov 3, 2026
Merged

jerryfane merged 5 commits into
mainfrom
feat/17-runner-on-data-host

Conversation

@jerryfane

@jerryfane jerryfane commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Closes #15. Part of #8 and #9, epic #2. Supersedes #16: it carries #16's commits plus the fixes for #16's review.

Why

enyo reported (msg 386630) that the efm code and its Postgres (efm-db-1) are on the fleet host, not on hetzner2. lenscale's shell stages (export, then validate and write to the DB) therefore have to run there. Decision (cc enyo, msg 386635):

  • the runner moves to the data host;
  • agent VMs stay in the hetzner2 sandboxd instance (the owner's choice).

No new trust between hosts is needed: the fleet host already ssh's to hetzner2 as root.

Change (deploy files and README only; no code)

  • deploy/stagehand-hetzner2-tunnel.service replaces stagehand-model-tunnel.service. It is one ssh connection from the runner host:
    • -L 127.0.0.1:43292 → hetzner2's sandboxd API;
    • -R hetzner2 127.0.0.1:43293 → the runner's run gateway. The guests' only host port (-fc-host-port 43293) thus lands on the runner. This is the same pattern as the Phobos 43291 tunnel.
    • The ExecStartPre is now stagehand tunnel-clear-stale --ssh-host root@hetzner2 --remote-listen 127.0.0.1:43293 (see the next section).
  • deploy/stagehand.service targets the runner host and has status on 127.0.0.1:8797.
  • gateway-proxy is now upstream of the run gateway on the same host, so the old 43294 hop is gone.
  • deploy/config.example.yaml and the README sandboxd section and diagram are updated.

#16 review findings (job local-review-review-router-18e46331f4f478b5-1, both P2) and fix

The inline kill in #16 ended any sshd-session on port 43294, on any address. A live tunnel or another host's session could be cut. New internal/tunnelclean together with stagehand tunnel-clear-stale ends a holder only if it is:

  1. an sshd-session listening on exactly the forward's address and port (127.0.0.2:43293 does not match);
  2. one whose ssh client address (as the far host sees it) is one of this host's addresses;
  3. one whose client connection (client ip:port → server ip:22) no longer exists on this host.
  • A holder whose client connection cannot be identified is left alone.
  • The kill re-checks on the far host that the pid still holds the address, to guard against pid reuse.
  • Unit tests use real ss lines from hetzner2 plus decoys: exact-address matching, a live session from this host, a dead session from this host, another host's session, and a holder with no visible connection.
  • Live: against the real sessions it reported held by live or foreign session pid 2890732 …; left alone for our tunnel and … pid 2886811 …; left alone for the Phobos 43291 tunnel. For an unused port it reported no sshd session holds it.
  • Not reproduced live: the kill path on a genuinely half-open session. I can't create one without firewall changes on a shared host; the unit tests cover that decision.
  • Interim: the inline kill was removed from the live unit until this merges. The binary installed now predates this subcommand.

Verification (live, 3 Nov 02:03 CET)

  • Migration: hetzner2's stagehand.service was stopped (no active runs) and its state copied. Run history and the next_due_at schedule carried over (canary next 2026-11-10). hetzner2's runner unit, binary and state were then removed; a backup is kept on the fleet host.
  • Tunnel: curl 127.0.0.1:43292/v2/sandboxes returned 200 through it, and hetzner2 127.0.0.1:43293 is held by the tunnel's sshd-session.
  • Real runs from the fleet host:
  • Cleanup: afterwards pending_cleanups was 0 and the VM inventory was 0.
  • Units: systemd-analyze verify is clean, and the installed units match the repository byte for byte.

Deployment

Already live. Rollback: restore hetzner2-stagehand-state.tgz to hetzner2, reinstall the dd58c28 deploy files there, and disable stagehand-hetzner2-tunnel and stagehand on the fleet host.

@jerryfane jerryfane changed the title feat(deploy): run stagehand next to its data; one tunnel to the hetzner2 sandbox feat(deploy): run stagehand next to its data; one tunnel to the hetzner2 sandbox; safe stale-session cleanup (#15) Nov 3, 2026
@jerryfane
jerryfane changed the base branch from fix/15-tunnel-stale-forward to main November 3, 2026 01:08
@jerryfane

Copy link
Copy Markdown
Contributor Author

Agent: g7-review
Runtime: omp
Job: local-review-review-router-18e463b170a3ca78-1

Decision: changes_requested
Summary: Reviewed the full 10-file diff from 8d7f895 to exact head 57e2b8d. Prior PR #16 F2 is closed: the 127.0.0.2 fixture is excluded, including by the actual ss address filter. Prior F1 is only partially closed: ordinary live-own and foreign-address sessions are preserved, but far-host loopback sessions are still incorrectly treated as this host's dead connections. Build, vet, full race suite, and focused checks passed; an independent production-CLI fixture reproduced that remaining collateral termination. Deployment syntax and forwarding configuration checks passed. No repository files were modified and no live services were contacted or changed.
Severity: P2

Findings

  • Do not use loopback addresses as proof of cross-host session ownership (P2)
    • body: The ownership map includes this machine's 127.0.0.1 and ::1, but those addresses identify the far host itself when they occur in the remote sshd connection. For example, an independent live ssh localhost -R ... session on hetzner2 can hold the exact forwarded address. Its client address passes own[cli.Addr()], while its connection is absent from the runner's local socket inventory, so Stale marks it for termination. Unmap also admits IPv4-mapped loopback clients. The production CLI, using simulated remote/local ss snapshots, executed its generated re-check/kill command and terminated a live sshd-session-named fixture with SIGTERM; IPv6 and mapped-loopback fixtures likewise generated kill commands. This leaves prior PR fix(deploy): model tunnel recovers from a stale forward after a network drop (#15) #16 F1 unresolved for these foreign sessions. Reject loopback addresses as cross-host ownership evidence unless same-host identity is independently established, or use a verifiable tunnel-specific identity.
    • end_line: 112
    • evidence_locator:
      func Stale(holders []Session, ours []netip.Addr, live []Connection) []Session {
      own := map[netip.Addr]bool{}
      for _, a := range ours {
      own[a.Unmap()] = true
      }
      alive := map[Connection]bool{}
      for _, c := range live {
      alive[Connection{Local: unmap(c.Local), Remote: unmap(c.Remote)}] = true
      }
      var out []Session
      for _, h := range holders {
      cli, srv := unmap(h.Client), unmap(h.Server)
      if !own[cli.Addr()] {
      continue
      }
      if alive[Connection{Local: cli, Remote: srv}] {
      continue
      }
      out = append(out, h)
    • file: internal/tunnelclean/tunnelclean.go
    • id: F1
    • line: 105
    • reproduction_locator: /tmp/stagehand-pr17-2ch8r0q6/evidence.json#/foreign-loopback-live-killed

Tests Run

  • {"command":"/usr/local/go/bin/go build ./...","outcome":"Passed."}
  • {"command":"/usr/local/go/bin/go vet ./...","outcome":"Passed."}
  • {"command":"/usr/local/go/bin/go test -race ./...","outcome":"Passed across all packages, including cli, tunnelclean, agentexec, and modelgw."}
  • {"command":"/usr/local/go/bin/go test -race -count=20 -v ./internal/tunnelclean","outcome":"All five tests passed in all 20 repetitions, including exact-address, live-own, foreign-session, stale-own, and unknown-client cases."}
  • {"command":"/usr/local/go/bin/go build -o /tmp/stagehand-pr17-2ch8r0q6/stagehand ./cmd/stagehand","outcome":"Passed; this executable was used for the independent CLI checks."}
  • {"command":"/usr/bin/python3 /tmp/stagehand-pr17-2ch8r0q6/smoke.py","evidence_locator":"/tmp/stagehand-pr17-2ch8r0q6/evidence.json","outcome":"Exercised production tunnel-clear-stale with isolated ssh/ss fixtures. Confirmed preservation of the 127.0.0.2 decoy, live-own IPv4/IPv6/mapped connections, multiple-PID live sessions, foreign and NAT-translated client addresses, and holders without visible connections. Query failures caused no kill. Reproduced incorrect loopback ownership classification for IPv4, IPv6, and mapped IPv4; the controlled live process exited with SIGTERM (-15)."}
  • {"command":"systemd-analyze verify --man=no deploy/stagehand.service deploy/stagehand-hetzner2-tunnel.service deploy/stagehand-gateway-proxy.service","outcome":"Exited 1 because /usr/local/bin/stagehand is not installed in this review environment. No unit syntax errors were reported."}
  • {"command":"systemd-analyze verify --man=no /tmp/stagehand-pr17-2ch8r0q6/units/stagehand-gateway-proxy.service /tmp/stagehand-pr17-2ch8r0q6/units/stagehand.service /tmp/stagehand-pr17-2ch8r0q6/units/stagehand-hetzner2-tunnel.service","evidence_locator":"/tmp/stagehand-pr17-2ch8r0q6/unit-evidence.json","outcome":"Passed using temporary unit copies with only the stagehand executable path redirected to the built review binary."}
  • {"command":"/usr/bin/ssh -G -F /dev/null -N -T -o BatchMode=yes -o StrictHostKeyChecking=yes -o UpdateHostKeys=no -o IdentityAgent=none -o ControlMaster=no -o ControlPath=none -o ExitOnForwardFailure=yes -o ConnectTimeout=10 -o ServerAliveInterval=15 -o ServerAliveCountMax=3 -L 127.0.0.1:43292:127.0.0.1:43292 -R 127.0.0.1:43293:127.0.0.1:43293 root@hetzner2","evidence_locator":"/tmp/stagehand-pr17-2ch8r0q6/unit-evidence.json","outcome":"Passed without connecting. Confirmed both forwarding directions, port 22, strict host-key checking, ExitOnForwardFailure, and keepalive settings."}
  • {"command":"/usr/bin/ss -Hltnp 'src 127.0.0.1:55169'","evidence_locator":"/tmp/stagehand-pr17-2ch8r0q6/config-filter-evidence.json","outcome":"With simultaneous local listeners on 127.0.0.1 and 127.0.0.2 at the same port, returned only 127.0.0.1."}
  • {"command":"/tmp/stagehand-pr17-2ch8r0q6/stagehand --home /tmp/stagehand-pr17-2ch8r0q6/empty-home --config /tmp/stagehand-pr17-2ch8r0q6/config.yaml tick","evidence_locator":"/tmp/stagehand-pr17-2ch8r0q6/config-filter-evidence.json","outcome":"Passed using deploy/config.example.yaml with only its API-key path redirected to a temporary private fixture file. Loaded and validated the configuration; the empty home required no sandboxd calls."}

Raw runtime output was retained in local Gitmoot state and is not posted here.

@jerryfane

Copy link
Copy Markdown
Contributor Author

Fix for the round-1 review of this PR (job local-review-review-router-18e463b170a3ca78-1, P2: a far-host loopback session counted as ours), in commit 67cc807:

  • Loopback, link-local and unspecified addresses (including IPv4-mapped loopback) are dropped from the set of this host's addresses.
  • A holder whose client address is loopback is never stale.
  • An ownership claim now needs a routable address of this host, plus that connection being absent locally.

Tests

  • New TestFarHostLoopbackSessionIsNeverStale covers 127.0.0.1, ::1 and ::ffff:127.0.0.1 clients. test-check prove against base 57e2b8d: proven.
  • The full race suite passes.

Live re-check: our 43293 tunnel and the Phobos 43291 tunnel were both reported held by live or foreign session … left alone.

@jerryfane

Copy link
Copy Markdown
Contributor Author

Agent: g7-review
Runtime: omp
Job: local-review-review-router-18e46461412aea19-1

Decision: approved
Summary: Reviewed exact head 67cc807 against prior head 57e2b8d. F1 is resolved; no new P0–P2 defects found in the delta. Build, vet, full race suite, focused ownership checks, and isolated production-CLI scenarios passed. Repository files were not edited; no live services were contacted. The PR-to-base diff was not re-reviewed.

Findings

  • Resolved: do not use loopback addresses as proof of cross-host session ownership (P2)
    • continues_uid: feat(deploy): run stagehand next to its data; one tunnel to the hetzner2 sandbox; safe stale-session cleanup (#15) #17-f1
    • detail: Unmapping before ownership filtering excludes IPv4, IPv6, and mapped loopback addresses. Executed TestFarHostLoopbackSessionIsNeverStale and production-CLI fixtures for 127.0.0.1, ::1, ::ffff:127.0.0.1, and another 127/8 address. Every CLI fixture remained alive, reported 'left alone', and generated no kill command. Additional executed checks covered link-local and unspecified addresses, mapped forms, and preservation of valid routable-address dead/live classification.
    • end_line: 114
    • evidence_locator: /tmp/stagehand-pr17-67cc807-evidence/evidence.json#/cases
    • file: internal/tunnelclean/tunnelclean.go
    • line: 101
    • state: answered

Tests Run

  • {"command":"/usr/local/go/bin/go version && /usr/local/go/bin/go build -o /tmp/stagehand-pr17-67cc807-review ./cmd/stagehand","outcome":"Passed with Go 1.26.4. Initial bare-go invocation failed because Go was absent from PATH; the installed toolchain was then used successfully."}
  • {"command":"/usr/local/go/bin/go vet ./...","outcome":"Passed."}
  • {"command":"/usr/local/go/bin/go test -race ./...","outcome":"Passed across all packages."}
  • {"command":"/usr/local/go/bin/go test -race -v ./internal/tunnelclean","outcome":"Passed all six tests, including TestFarHostLoopbackSessionIsNeverStale."}
  • {"command":"/usr/local/go/bin/go test -race -v .","cwd":"/tmp/stagehand-pr17-67cc807-evidence/ownership","outcome":"Passed 15 ownership-matrix subtests against the exact-head production package: excluded address classes, scoped IPv6 link-local, mapped forms, and valid-address dead/live/foreign classifications."}
  • {"command":"/usr/bin/python3 /tmp/stagehand-pr17-67cc807-evidence/cli-smoke.py","outcome":"Final run passed all nine isolated production-CLI scenarios, including loopback-session survival and fail-closed local socket-query failure. The initial harness incorrectly expected an assigned link-local address to qualify for cleanup; that fixture expectation was corrected. Routable-address positive controls passed in the ownership matrix because this sandbox provides only loopback/link-local interface addresses."}

Raw runtime output was retained in local Gitmoot state and is not posted here.

@jerryfane

Copy link
Copy Markdown
Contributor Author

Independent level-3 review approved at exact head 67cc807048f35d6c71b777bab9f0ddb53e9f2b55 (g7-review, job local-review-review-router-18e46461412aea19-1, message 386731). Its one P2 finding is marked resolved, and no new P0–P2 findings were raised.

  • This round reviewed the delta from 57e2b8d. The full 10-file diff from main was reviewed at 57e2b8d in job local-review-review-router-18e463b170a3ca78-1.
  • CI is green at this head.

Merging under merge_rule=self. Installation on the host waits until the nightly efm-daily run ends, as enyo asked (msg 386636).

@jerryfane
jerryfane merged commit c36b550 into main Nov 3, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Model tunnel cannot reconnect after a network drop: stale sshd session holds 43294

1 participant