Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -296,6 +296,21 @@ WantedBy=multi-user.target
- **Crashes:** `KillMode=control-group` makes systemd kill any stage processes left behind if the
runner crashes.

### Upgrading and rolling back

1. Check that nothing is running: `stagehand status --json` shows `active_run_id: null` for every pipeline.
2. Put the new binary next to the old one, then `mv` it over `/usr/local/bin/stagehand`. Renaming
on the same filesystem is atomic and avoids "text file busy".
3. `systemctl restart stagehand`, then check `stagehand version` and `GET /status`.

To roll back, do the same with the previous binary. The database only gets new migrations: an older
binary refuses a newer schema rather than misreading it (`database schema version N is newer than
this binary supports`). Rolling back across a schema change therefore needs the database file
backed up before the upgrade.

The model tunnel (`deploy/stagehand-model-tunnel.service`) reconnects on its own after a network
drop. Before each connection, it ends a stale session that still holds its port on the far side.

## Development

```sh
Expand Down
8 changes: 8 additions & 0 deletions deploy/stagehand-model-tunnel.service
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,14 @@ After=network-online.target tailscaled.service stagehand-gateway-proxy.service
Type=simple
User=root
UMask=0077
# After a network drop, hetzner2's sshd keeps the dead session (its
# ClientAliveInterval is 0), and that session still holds 43294, so every
# reconnect fails with "remote port forwarding failed". Before connecting,
# end an sshd-session that holds this unit's own port: any holder is stale,
# because this unit's previous ssh process is gone. Other ports and other
# processes are never touched. Leading "-": a failure here (host down) must
# not block the retry loop.
ExecStartPre=-/usr/bin/ssh -T -o BatchMode=yes -o StrictHostKeyChecking=yes -o UpdateHostKeys=no -o IdentityAgent=none -o ControlMaster=no -o ControlPath=none -o ConnectTimeout=10 root@hetzner2 "for p in $$(ss -Hltnp 'sport = :43294' | grep sshd-session | grep -o 'pid=[0-9]*' | cut -d= -f2); do kill $$p; done"
ExecStart=/usr/bin/ssh -N -T -o BatchMode=yes -o StrictHostKeyChecking=yes -o UpdateHostKeys=no -o IdentityAgent=none -o ControlMaster=no -o ControlPath=none -o ExitOnForwardFailure=yes -o ConnectTimeout=10 -o ServerAliveInterval=15 -o ServerAliveCountMax=3 -R 127.0.0.1:43294:127.0.0.1:43295 root@hetzner2
Restart=always
RestartSec=5
Expand Down
Loading