Repository navigation
feat(deploy): run stagehand next to its data; one tunnel to the hetzner2 sandbox; safe stale-session cleanup (#15) - #17
Conversation
… tunnel reconnects (#15 review)
Decision: Findings
Tests Run
Raw runtime output was retained in local Gitmoot state and is not posted here. |
…sion is ours (#17 review)
|
Fix for the round-1 review of this PR (job
Tests
Live re-check: our 43293 tunnel and the Phobos 43291 tunnel were both reported |
Decision: Findings
Tests Run
Raw runtime output was retained in local Gitmoot state and is not posted here. |
|
Independent level-3 review approved at exact head
Merging under |
Closes #15. Part of #8 and #9, epic #2. Supersedes #16: it carries #16's commits plus the fixes for #16's review.
Why
enyo reported (msg 386630) that the efm code and its Postgres (
efm-db-1) are on the fleet host, not on hetzner2. lenscale's shell stages (export, then validate and write to the DB) therefore have to run there. Decision (cc enyo, msg 386635):No new trust between hosts is needed: the fleet host already ssh's to hetzner2 as root.
Change (deploy files and README only; no code)
deploy/stagehand-hetzner2-tunnel.servicereplacesstagehand-model-tunnel.service. It is one ssh connection from the runner host:-L 127.0.0.1:43292→ hetzner2's sandboxd API;-R hetzner2 127.0.0.1:43293→ the runner's run gateway. The guests' only host port (-fc-host-port 43293) thus lands on the runner. This is the same pattern as the Phobos 43291 tunnel.ExecStartPreis nowstagehand tunnel-clear-stale --ssh-host root@hetzner2 --remote-listen 127.0.0.1:43293(see the next section).deploy/stagehand.servicetargets the runner host and has status on127.0.0.1:8797.gateway-proxyis now upstream of the run gateway on the same host, so the old 43294 hop is gone.deploy/config.example.yamland the README sandboxd section and diagram are updated.#16 review findings (job
local-review-review-router-18e46331f4f478b5-1, both P2) and fixThe inline
killin #16 ended anysshd-sessionon port 43294, on any address. A live tunnel or another host's session could be cut. Newinternal/tunnelcleantogether withstagehand tunnel-clear-staleends a holder only if it is:sshd-sessionlistening on exactly the forward's address and port (127.0.0.2:43293does not match);client ip:port→server ip:22) no longer exists on this host.sslines from hetzner2 plus decoys: exact-address matching, a live session from this host, a dead session from this host, another host's session, and a holder with no visible connection.held by live or foreign session pid 2890732 …; left alonefor our tunnel and… pid 2886811 …; left alonefor the Phobos 43291 tunnel. For an unused port it reportedno sshd session holds it.Verification (live, 3 Nov 02:03 CET)
stagehand.servicewas stopped (no active runs) and its state copied. Run history and thenext_due_atschedule carried over (canarynext 2026-11-10). hetzner2's runner unit, binary and state were then removed; a backup is kept on the fleet host.curl 127.0.0.1:43292/v2/sandboxesreturned 200 through it, and hetzner2 127.0.0.1:43293 is held by the tunnel's sshd-session.canaryModel path: per-run gateway, gateway-proxy and the hetzner2 tunnel #6 succeeded: VM on hetzner2, 3 model requests, cost 0.0149, answerRomewith a europa.eu source;canary-overcapsandboxd instancestagehandand omp image on hetzner2 #7, started at the same time, waited for the sandbox slot, then failedover budget: spent 0.0046with onlyrows.jsonin its work dir.pending_cleanupswas 0 and the VM inventory was 0.systemd-analyze verifyis clean, and the installed units match the repository byte for byte.Deployment
Already live. Rollback: restore
hetzner2-stagehand-state.tgzto hetzner2, reinstall thedd58c28deploy files there, and disablestagehand-hetzner2-tunnelandstagehandon the fleet host.