diff --git a/README.md b/README.md index ea8baa1..9342c10 100644 --- a/README.md +++ b/README.md @@ -296,6 +296,21 @@ WantedBy=multi-user.target - **Crashes:** `KillMode=control-group` makes systemd kill any stage processes left behind if the runner crashes. +### Upgrading and rolling back + +1. Check that nothing is running: `stagehand status --json` shows `active_run_id: null` for every pipeline. +2. Put the new binary next to the old one, then `mv` it over `/usr/local/bin/stagehand`. Renaming + on the same filesystem is atomic and avoids "text file busy". +3. `systemctl restart stagehand`, then check `stagehand version` and `GET /status`. + +To roll back, do the same with the previous binary. The database only gets new migrations: an older +binary refuses a newer schema rather than misreading it (`database schema version N is newer than +this binary supports`). Rolling back across a schema change therefore needs the database file +backed up before the upgrade. + +The model tunnel (`deploy/stagehand-model-tunnel.service`) reconnects on its own after a network +drop. Before each connection, it ends a stale session that still holds its port on the far side. + ## Development ```sh diff --git a/deploy/stagehand-model-tunnel.service b/deploy/stagehand-model-tunnel.service index 6a26052..b871634 100644 --- a/deploy/stagehand-model-tunnel.service +++ b/deploy/stagehand-model-tunnel.service @@ -10,6 +10,14 @@ After=network-online.target tailscaled.service stagehand-gateway-proxy.service Type=simple User=root UMask=0077 +# After a network drop, hetzner2's sshd keeps the dead session (its +# ClientAliveInterval is 0), and that session still holds 43294, so every +# reconnect fails with "remote port forwarding failed". Before connecting, +# end an sshd-session that holds this unit's own port: any holder is stale, +# because this unit's previous ssh process is gone. Other ports and other +# processes are never touched. Leading "-": a failure here (host down) must +# not block the retry loop. +ExecStartPre=-/usr/bin/ssh -T -o BatchMode=yes -o StrictHostKeyChecking=yes -o UpdateHostKeys=no -o IdentityAgent=none -o ControlMaster=no -o ControlPath=none -o ConnectTimeout=10 root@hetzner2 "for p in $$(ss -Hltnp 'sport = :43294' | grep sshd-session | grep -o 'pid=[0-9]*' | cut -d= -f2); do kill $$p; done" ExecStart=/usr/bin/ssh -N -T -o BatchMode=yes -o StrictHostKeyChecking=yes -o UpdateHostKeys=no -o IdentityAgent=none -o ControlMaster=no -o ControlPath=none -o ExitOnForwardFailure=yes -o ConnectTimeout=10 -o ServerAliveInterval=15 -o ServerAliveCountMax=3 -R 127.0.0.1:43294:127.0.0.1:43295 root@hetzner2 Restart=always RestartSec=5