Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 29 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -138,26 +138,30 @@ uid 1000), and the VM is destroyed afterwards.
attempt. It accepts only that attempt's random token, removes it, and forwards the call.
- **Credentials:** the real model credential is added further along, by
`stagehand gateway-proxy`, on the host that holds the omp auth-gateway token.
- **Where things run:** the runner (and so the shell stages) runs next to the data its shell
stages need. The sandboxd instance can be on another host: one ssh tunnel forwards its API to
the runner (`-L`) and its guest host port back to the runner's run gateway (`-R`).

```
guest omp ── token for this attempt ──▶ run gateway (stagehand serve, 127.0.0.1:43293)
──▶ 127.0.0.1:43294 (ssh -R tunnel) ──▶ stagehand gateway-proxy (127.0.0.1:43295, adds bearer)
guest omp ── token for this attempt ──▶ 10.0.2.2:43293 (sandbox host's -fc-host-port)
──▶ ssh -R ──▶ run gateway (stagehand serve on the runner host, 127.0.0.1:43293)
──▶ stagehand gateway-proxy (127.0.0.1:43295, adds bearer)
──▶ omp auth-gateway (127.0.0.1:4011) ──▶ provider
```

```yaml
agent:
executor: sandboxd
sandboxd:
url: http://127.0.0.1:43292 # the instance's -listen
api_key_file: /var/lib/sandboxd-stagehand/api-key
url: http://127.0.0.1:43292 # the instance's -listen (via the tunnel's -L)
api_key_file: /etc/stagehand/sandboxd-api-key
template: stagehand-omp-strict
guest_omp: /usr/local/bin/omp # default
max_ttl: 1h # the instance's -max-ttl
model_gateway:
listen: 127.0.0.1:43293 # = the instance's -fc-host-port
listen: 127.0.0.1:43293 # reached from the instance's -fc-host-port (tunnel -R)
guest_url: http://10.0.2.2:43293
upstream: http://127.0.0.1:43294 # tunnel to gateway-proxy
upstream: http://127.0.0.1:43295 # stagehand gateway-proxy
```

- **Which models work:** the VM's omp gets only `ANTHROPIC_BASE_URL` and `ANTHROPIC_API_KEY`
Expand Down Expand Up @@ -296,6 +300,25 @@ WantedBy=multi-user.target
- **Crashes:** `KillMode=control-group` makes systemd kill any stage processes left behind if the
runner crashes.

### Upgrading and rolling back

1. Check that nothing is running: `stagehand status --json` shows `active_run_id: null` for every pipeline.
2. Put the new binary next to the old one, then `mv` it over `/usr/local/bin/stagehand`. Renaming
on the same filesystem is atomic and avoids "text file busy".
3. `systemctl restart stagehand`, then check `stagehand version` and `GET /status`.

To roll back, do the same with the previous binary. The database only gets new migrations: an older
binary refuses a newer schema rather than misreading it (`database schema version N is newer than
this binary supports`). Rolling back across a schema change therefore needs the database file
backed up before the upgrade.

The sandbox tunnel (`deploy/stagehand-hetzner2-tunnel.service`) reconnects on its own after a
network drop. Before each connection, `stagehand tunnel-clear-stale` ends a session on the far side
that still holds the forwarded port, but only one it can prove is this host's own dead
connection. That means an sshd-session on exactly that address and port, whose client is this host
and whose TCP connection no longer exists here. Live sessions and other hosts' sessions are left
alone.

## Development

```sh
Expand Down
10 changes: 5 additions & 5 deletions deploy/config.example.yaml
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
# /etc/stagehand/config.yaml on hetzner2
# /etc/stagehand/config.yaml on the runner host
agent:
executor: sandboxd
sandboxd:
url: http://127.0.0.1:43292
api_key_file: /var/lib/sandboxd-stagehand/api-key
url: http://127.0.0.1:43292 # hetzner2's instance, via stagehand-hetzner2-tunnel (-L)
api_key_file: /etc/stagehand/sandboxd-api-key # copy of the instance's api-key, 0600
template: stagehand-omp-strict
guest_omp: /usr/local/bin/omp
max_ttl: 1h
model_gateway:
listen: 127.0.0.1:43293 # = sandboxd -fc-host-port; the guest sees 10.0.2.2:43293
listen: 127.0.0.1:43293 # reached from hetzner2's -fc-host-port via the tunnel (-R)
guest_url: http://10.0.2.2:43293
upstream: http://127.0.0.1:43294 # stagehand-model-tunnel -> gateway-proxy on the fleet host
upstream: http://127.0.0.1:43295 # stagehand gateway-proxy on this host
4 changes: 2 additions & 2 deletions deploy/stagehand-gateway-proxy.service
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Fleet host (holds the omp auth-gateway token): adds the gateway bearer to
# model calls arriving from the hetzner2 tunnel. Loopback only.
# Runner host (holds the omp auth-gateway token): adds the gateway bearer to
# agents' model calls coming from the per-attempt run gateway. Loopback only.
[Unit]
Description=stagehand model credential proxy
After=network-online.target
Expand Down
36 changes: 36 additions & 0 deletions deploy/stagehand-hetzner2-tunnel.service
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# Runner host: links the stagehand runner to the sandboxd instance "stagehand"
# on hetzner2, where agent VMs run.
# -L 127.0.0.1:43292 -> hetzner2 127.0.0.1:43292 sandboxd control and guest API
# -R hetzner2 127.0.0.1:43293 -> 127.0.0.1:43293 the guests' only host port
# (= sandboxd -fc-host-port), served
# by the runner's per-attempt run gateway
# Modeled on phobos-sandbox-hetzner2-tunnel.service.
[Unit]
Description=stagehand tunnel to the hetzner2 sandboxd instance (API 43292, guest gateway 43293)
Wants=network-online.target tailscaled.service
After=network-online.target tailscaled.service

[Service]
Type=simple
User=root
UMask=0077
# After a network drop, hetzner2's sshd keeps the dead session (its
# ClientAliveInterval is 0), and it still holds 43293, so every reconnect
# fails with "remote port forwarding failed". Before connecting, end that
# session only if it is provably ours and dead: an sshd-session on exactly
# 127.0.0.1:43293 whose client is this host and whose connection no longer
# exists here. Live or foreign sessions are left alone. Leading "-": a
# failure here (host down) must not block the retry loop.
ExecStartPre=-/usr/local/bin/stagehand tunnel-clear-stale --ssh-host root@hetzner2 --remote-listen 127.0.0.1:43293
ExecStart=/usr/bin/ssh -N -T -o BatchMode=yes -o StrictHostKeyChecking=yes -o UpdateHostKeys=no -o IdentityAgent=none -o ControlMaster=no -o ControlPath=none -o ExitOnForwardFailure=yes -o ConnectTimeout=10 -o ServerAliveInterval=15 -o ServerAliveCountMax=3 -L 127.0.0.1:43292:127.0.0.1:43292 -R 127.0.0.1:43293:127.0.0.1:43293 root@hetzner2
Restart=always
RestartSec=5
KillMode=control-group
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=read-only
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6

[Install]
WantedBy=multi-user.target
24 changes: 0 additions & 24 deletions deploy/stagehand-model-tunnel.service

This file was deleted.

10 changes: 5 additions & 5 deletions deploy/stagehand.service
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
# hetzner2: the runner. Agent stages run in the sandboxd instance "stagehand";
# shell stages run here as root.
# Runner host (next to the data the shell stages use). Agent stages run in the
# sandboxd instance "stagehand" on hetzner2 via stagehand-hetzner2-tunnel.
[Unit]
Description=stagehand pipeline runner
After=network-online.target sandboxd-stagehand.service
Wants=network-online.target
After=network-online.target stagehand-hetzner2-tunnel.service stagehand-gateway-proxy.service
Wants=network-online.target stagehand-hetzner2-tunnel.service stagehand-gateway-proxy.service

[Service]
Type=simple
Environment=STAGEHAND_HOME=/var/lib/stagehand
Environment=STAGEHAND_CONFIG=/etc/stagehand/config.yaml
ExecStart=/usr/local/bin/stagehand serve --status-addr 127.0.0.1:8795
ExecStart=/usr/local/bin/stagehand serve --status-addr 127.0.0.1:8797
Restart=on-failure
RestartSec=10
# Stop sends SIGTERM: running stages are stopped and their runs parked as
Expand Down
10 changes: 10 additions & 0 deletions internal/cli/cli.go
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,8 @@ Service:
tick one pass: fire due schedules, run them to the end
gateway-proxy [--listen ADDR] [--upstream URL] [--token-file FILE]
model credential proxy for sandboxed agent stages
tunnel-clear-stale --ssh-host HOST --remote-listen ADDR:PORT
end this host's dead ssh session still holding a forward
status [--json] last and next run per pipeline
version

Expand Down Expand Up @@ -95,6 +97,14 @@ func Main(args []string, stdout, stderr io.Writer) int {
}
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
if cmd == "tunnel-clear-stale" {
// Runs before a tunnel reconnects; needs no home.
if err := tunnelClearStale(ctx, cmdArgs, stdout, stderr); err != nil {
fmt.Fprintln(stderr, "stagehand:", err)
return 1
}
return 0
}
if cmd == "gateway-proxy" {
// Runs on the host holding the omp auth-gateway token; needs no home.
if err := gatewayProxy(ctx, cmdArgs, stderr); err != nil {
Expand Down
68 changes: 68 additions & 0 deletions internal/cli/gateway.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,15 @@ import (
"fmt"
"io"
"log/slog"
"net"
"net/netip"
"net/url"
"os"
"os/exec"
"strings"

"github.com/gitmoot/stagehand/internal/modelgw"
"github.com/gitmoot/stagehand/internal/tunnelclean"
)

// gatewayProxy runs `stagehand gateway-proxy`: the credential side of the
Expand Down Expand Up @@ -59,3 +63,67 @@ func readSecretFile(path string) (string, error) {
}
return s, nil
}

// tunnelClearStale runs `stagehand tunnel-clear-stale`: before a reverse
// tunnel reconnects, end this host's own dead sshd session that still holds
// the forwarded port on the far host (see package tunnelclean).
func tunnelClearStale(ctx context.Context, args []string, stdout, stderr io.Writer) error {
fs := flag.NewFlagSet("tunnel-clear-stale", flag.ContinueOnError)
host := fs.String("ssh-host", "", "far host, as given to ssh (e.g. root@hetzner2)")
listen := fs.String("remote-listen", "", "the forward's exact address on the far host, e.g. 127.0.0.1:43293")
usage := "tunnel-clear-stale --ssh-host HOST --remote-listen ADDR:PORT"
if _, err := parse(fs, args, 0, usage); err != nil {
return err
}
ap, err := netip.ParseAddrPort(*listen)
if err != nil || *host == "" {
return fmt.Errorf("usage: stagehand %s", usage)
}
sshArgs := []string{"-T", "-o", "BatchMode=yes", "-o", "StrictHostKeyChecking=yes", "-o", "UpdateHostKeys=no",
"-o", "IdentityAgent=none", "-o", "ControlMaster=no", "-o", "ControlPath=none", "-o", "ConnectTimeout=10", *host}
remote := func(cmd string) (string, error) {
out, err := exec.CommandContext(ctx, "ssh", append(sshArgs, cmd)...).Output()
return string(out), err
}
out, err := remote(tunnelclean.QueryCommand(ap))
if err != nil {
return fmt.Errorf("query %s: %w", *host, err)
}
listeners, conns, _ := strings.Cut(out, tunnelclean.Separator)
holders := tunnelclean.Holders(listeners, conns, ap)
if len(holders) == 0 {
fmt.Fprintf(stdout, "%s on %s: no sshd session holds it\n", ap, *host)
return nil
}
local, err := exec.CommandContext(ctx, "ss", "-Htn", "state", "established").Output()
if err != nil {
return fmt.Errorf("local sockets: %w", err)
}
var ours []netip.Addr
ifaddrs, err := net.InterfaceAddrs()
if err != nil {
return err
}
for _, a := range ifaddrs {
if p, err := netip.ParsePrefix(a.String()); err == nil {
ours = append(ours, p.Addr())
}
}
stale := tunnelclean.Stale(holders, ours, tunnelclean.LocalConnections(string(local)))
for _, h := range holders {
isStale := false
for _, s := range stale {
isStale = isStale || s.PID == h.PID
}
if !isStale {
fmt.Fprintf(stdout, "%s on %s: held by live or foreign session pid %d (client %s); left alone\n", ap, *host, h.PID, h.Client)
continue
}
if _, err := remote(tunnelclean.KillCommand(h.PID, ap)); err != nil {
fmt.Fprintf(stderr, "end stale session pid %d: %v\n", h.PID, err)
continue
}
fmt.Fprintf(stdout, "%s on %s: ended stale session pid %d (client %s, gone here)\n", ap, *host, h.PID, h.Client)
}
return nil
}
Loading
Loading