Skip to content

fix(deploy): model tunnel recovers from a stale forward after a network drop (#15) - #16

Closed
jerryfane wants to merge 2 commits into
mainfrom
fix/15-tunnel-stale-forward
Closed

jerryfane wants to merge 2 commits into
mainfrom
fix/15-tunnel-stale-forward

Conversation

@jerryfane

Copy link
Copy Markdown
Contributor

Closes #15. Part of #8 and epic #2.

Change

deploy/stagehand-model-tunnel.service gains an ExecStartPre. Before reconnecting, it logs in to hetzner2 and ends any sshd-session that holds 127.0.0.1:43294.

  • Only this unit's own port is touched; no other port or process is.
  • The leading - means a failure (for example, hetzner2 unreachable) does not block the retry loop.

Verification on the live hosts (3 Nov ~01:57)

  • Live recovery. The installed unit was restart-looping on remote port forwarding failed for listen port 43294; the stale holder was a 20-minute-old sshd-session. With this unit installed, the tunnel came up on the first start, and GET /healthz through 43294 returned 200.
  • Simulation. I stopped the unit and occupied 43294 with a separate ssh -R session (holder pid 2887599). Starting the unit cleared that holder; its own session (pid 2887773) took the port, healthz returned 200, and the log showed no forwarding failure.
  • No side effects. phobos-sandbox-hetzner2-tunnel (port 43291) was untouched and stayed active throughout.
  • systemd-analyze verify is clean.

Deployment

Already installed on the fleet host. This PR brings the repository copy in line with it. Rollback: the previous unit file at dd58c28:deploy/stagehand-model-tunnel.service.

@jerryfane

Copy link
Copy Markdown
Contributor Author

Agent: g7-review
Runtime: omp
Job: local-review-review-router-18e46331f4f478b5-1

Decision: changes_requested
Summary: Reviewed the complete two-file diff from base 8d7f895 to exact head 5462175. Two P2 collateral-termination defects were reproduced in the cleanup command. Build, vet, full race suite, and focused migration tests passed. Actual systemd parsing and expansion preserve the quoting and dollar escapes correctly. README schema-refusal behavior was confirmed through the production binary. The frozen prior-verdict snapshot contains no findings for PR #16. No repository files were edited and no live services were contacted.
Severity: P2

Findings

  • Verify ownership and staleness before terminating the port holder (P2)
    • body: The cleanup selects every matching sshd-session without establishing that it belongs to this unit's previous connection or is stale. The previous local SSH process being gone does not exclude a healthy connection from another unit, another host, or an independently started instance. Starting this service will terminate that connection; independent instances can consequently tear down one another's tunnels. In a local fixture, a process named sshd-session successfully served TCP traffic on 127.0.0.1:43294 immediately before the exact systemd-expanded cleanup command terminated it with SIGTERM. Require verifiable ownership and staleness before signaling, or rely on bounded server-side session expiry rather than treating every holder as stale.
    • evidence_locator:
      # end an sshd-session that holds this unit's own port: any holder is stale,
      # because this unit's previous ssh process is gone. Other ports and other
      # processes are never touched. Leading "-": a failure here (host down) must
      # not block the retry loop.
      ExecStartPre=-/usr/bin/ssh -T -o BatchMode=yes -o StrictHostKeyChecking=yes -o UpdateHostKeys=no -o IdentityAgent=none -o ControlMaster=no -o ControlPath=none -o ConnectTimeout=10 root@hetzner2 "for p in $$(ss -Hltnp 'sport = :43294' | grep sshd-session | grep -o 'pid=[0-9]*' | cut -d= -f2); do kill $$p; done"
    • file: deploy/stagehand-model-tunnel.service
    • id: F1
    • line: 20
  • Restrict the socket filter to the exact forwarding address (P2)
    • body: The ss filter constrains only the port, not the configured 127.0.0.1 address. An unrelated SSH listener on 127.0.0.2:43294 can coexist with this unit's listener and does not prevent its forwarding bind, but the cleanup still terminates its session. A local scenario created two healthy sshd-session-named listener fixtures simultaneously on 127.0.0.1:43294 and 127.0.0.2:43294; the production cleanup command terminated both with SIGTERM. Match the exact local address and address family in addition to the port. Ownership/staleness checks remain independently necessary.
    • evidence_locator:
      ExecStartPre=-/usr/bin/ssh -T -o BatchMode=yes -o StrictHostKeyChecking=yes -o UpdateHostKeys=no -o IdentityAgent=none -o ControlMaster=no -o ControlPath=none -o ConnectTimeout=10 root@hetzner2 "for p in $$(ss -Hltnp 'sport = :43294' | grep sshd-session | grep -o 'pid=[0-9]*' | cut -d= -f2); do kill $$p; done"
      ExecStart=/usr/bin/ssh -N -T -o BatchMode=yes -o StrictHostKeyChecking=yes -o UpdateHostKeys=no -o IdentityAgent=none -o ControlMaster=no -o ControlPath=none -o ExitOnForwardFailure=yes -o ConnectTimeout=10 -o ServerAliveInterval=15 -o ServerAliveCountMax=3 -R 127.0.0.1:43294:127.0.0.1:43295 root@hetzner2
    • file: deploy/stagehand-model-tunnel.service
    • id: F2
    • line: 20

Tests Run

  • {"command":"env PATH=/usr/local/go/bin:/usr/bin:/bin CGO_ENABLED=0 go build -trimpath -o /tmp/stagehand-pr16-5462175 ./cmd/stagehand && /tmp/stagehand-pr16-5462175 version","outcome":"Passed; binary reported stagehand dev."}
  • {"command":"env PATH=/usr/local/go/bin:/usr/bin:/bin go vet ./...","outcome":"Passed."}
  • {"command":"env PATH=/usr/local/go/bin:/usr/bin:/bin go test -race -count=1 ./...","outcome":"Passed across all packages with tests."}
  • {"command":"env PATH=/usr/local/go/bin:/usr/bin:/bin go test -race -count=10 ./internal/store -run '^TestOpenUpgradesAnExistingVersion1Home$' -v","outcome":"All ten iterations passed."}
  • {"command":"env SYSTEMD_LOG_LEVEL=debug systemd-analyze verify --man=no deploy/stagehand-model-tunnel.service","outcome":"Exited successfully; unit parsed. Diagnostic output noted unavailable container dependencies, including tailscaled.service."}
  • {"command":"Python ctypes calls to extract_first_word and replace_env_argv in /usr/lib/x86_64-linux-gnu/systemd/libsystemd-shared-252.so using the unchanged ExecStartPre","outcome":"Passed: one remote-command argument; embedded single quotes preserved; $$(...) and $$p became $(...) and $p without local environment substitution."}
  • {"command":["/bin/sh","-c","for p in $(ss -Hltnp 'sport = :43294' | grep sshd-session | grep -o 'pid=[0-9]*' | cut -d= -f2); do kill $p; done"],"outcome":"Executed against isolated local TCP-listener fixtures. Reproduced SIGTERM of healthy exact-address and distinct-address sshd-session-named holders, including both simultaneously. Preserved a differently named process and an sshd-session-named holder on port 43296. No-listener case exited 0."}
  • {"command":["/usr/bin/ssh","-T","-o","BatchMode=yes","-o","StrictHostKeyChecking=yes","-o","UpdateHostKeys=no","-o","IdentityAgent=none","-o","ControlMaster=no","-o","ControlPath=none","-o","ConnectTimeout=10","-p","41127","-o","UserKnownHostsFile=/dev/null","-o","GlobalKnownHostsFile=/dev/null","-o","ProxyCommand=none","-o","ProxyJump=none","root@127.0.0.1","for p in $(ss -Hltnp 'sport = :43294' | grep sshd-session | grep -o 'pid=[0-9]*' | cut -d= -f2); do kill $p; done"],"outcome":"Deliberately closed local endpoint returned SSH exit 255 in 0.136 seconds. The unit's leading '-' makes this exit non-fatal to startup; no live systemd startup or destination-host connection was attempted."}
  • {"command":"/tmp/stagehand-pr16-5462175 --home /tmp/stagehand-pr16-store-ok0n0yky/existing-v1 status --json","outcome":"Exited 0; upgraded the frozen version-1 schema to version 2 and created pending_cleanups."}
  • {"command":"/tmp/stagehand-pr16-5462175 --home /tmp/stagehand-pr16-store-ok0n0yky/future-v3 status --json","outcome":"Expected exit 1: database schema version 3 is newer than this binary supports (2). Preserved schema version 3 and sentinel data."}

Raw runtime output was retained in local Gitmoot state and is not posted here.

@jerryfane

Copy link
Copy Markdown
Contributor Author

Superseded by #17. That PR deletes this unit: the runner moved next to the efm DB, and one tunnel now serves the hetzner2 sandbox. It also fixes both P2 findings from this PR's review (job local-review-review-router-18e46331f4f478b5-1): ownership and staleness are now verified, and the match is on the exact address. The inline kill this PR added has been removed from the live unit until #17 merges.

@jerryfane jerryfane closed this Nov 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Model tunnel cannot reconnect after a network drop: stale sshd session holds 43294

1 participant