Skip to content

chore(deploy): video runner on hetzner2 (#28) - #29

Merged
jerryfane merged 2 commits into
mainfrom
ops/hetzner2-video
Nov 15, 2026
Merged

jerryfane merged 2 commits into
mainfrom
ops/hetzner2-video

Conversation

@jerryfane

Copy link
Copy Markdown
Contributor

Closes #28. Assignment: owner direction 2026-11-15 (video pipelines on hetzner2), keephair msg 487916, plan /root/fleet-tools/state/video-on-hetzner2/plan.md step 1.

New deploy/hetzner2-video/: the unit, the config, a stagehand-video wrapper and an install README. The fleet-host files are unchanged.

Checked on hetzner2: x86_64. Ports 8807, 8808 and 43299 are free. The sandboxd instance stagehandvid is up on 43297 with guest port 43298, and its api-key is at /var/lib/sandboxd-stagehandvid/api-key (0600 root). There is no /var/lib/stagehand, /etc/stagehand* or stagehand binary there yet.

Binary: d5b9056, the same build as the fleet host; later commits on main are docs and units only.

@jerryfane

Copy link
Copy Markdown
Contributor Author

The head moved to 9e6e34240065160cc39774714199ee498787f54e. Two additions:

  • Model route. The owner approved a tunnel: record /root/fleet-tools/state/video-on-hetzner2/owner-model-route-decision-20261115.json, keephair msg 488025. It adds fleet-host/stagehand-video-hetzner2-tunnel.service: -R 127.0.0.1:43299:127.0.0.1:43295, modelled on stagehand-hetzner2-tunnel, with a stale-session pre-clear on 43299.
    • The gateway-proxy has no client authentication, so I checked who on hetzner2 can open a loopback port. The CI runner uids (h2ci-*, skuid 59100-59199, chain ci) get oifname lo reject; as h2ci-fleetup-1, a connect to 43292/43293/4011 was refused. The sandboxd jail uids only reach their own host port (chain guest). So only root can use 43299.
  • Resources: MemoryMax 8G, MemoryHigh 7G, MemorySwapMax 0, CPUQuota 600%, CPU and IO weights 50, OOMScoreAdjust 300. These are from the hetzner2 memory budget (jarvis approved it in msg 488023; keephair confirmed).

@jerryfane

Copy link
Copy Markdown
Contributor Author

Reviewer: g7-review
Runtime: omp
Job: local-review-review-router-18e843dda8ba2cc2-1

Decision: approved
Summary: Reviewed the complete four-file deployment diff at aff4aa7. No P0–P3 findings. Config validation, wrapper isolation, installation permissions, and unit consistency passed. Fleet config and deploy/stagehand.service are unchanged. Isolated production-CLI scenarios verified prompt 502 responses, failed agent attempts without retained resources, and unaffected shell execution; sandbox/guest execution used a fixture, not live VMs. Sandbox restrictions blocked local CGO/race checks and systemd-analyze; exact-head CI covers vet/race, not unit verification. No prior PR #29 findings exist in the frozen verdict file. Repository untouched.

Tests Run

  • CGO_ENABLED=0 go build -trimpath -o "$TMPDIR/pr29-review/stagehand" ./cmd/stagehand && "$TMPDIR/pr29-review/stagehand" version -> passed; stagehand dev.
  • CGO_ENABLED=0 go vet ./... -> passed.
  • CGO_ENABLED=0 go test -count=1 ./internal/config ./internal/agentexec ./internal/modelgw ./internal/shellexec ./internal/cli ./internal/engine -> passed; config has no test files.
  • CGO_ENABLED=0 go run -mod=mod . /root/.gitmoot/execbackends/local/c4d5fe17606d9b79b70dc9b94a84c1ae/workspace/deploy/hetzner2-video/config.yaml (cwd /tmp/gmr-3344cd16/pr29-review) -> passed exact YAML validation and invalid-listener/key-path rejection checks.
  • sh -n deploy/hetzner2-video/stagehand-video -> passed.
  • python3 "$TMPDIR/pr29-review/smoke.py" "$PWD" "$TMPDIR/pr29-review/stagehand" -> passed wrapper exec-boundary capture, installation modes, unit consistency, absent-upstream failure/cleanup, token isolation, and real shell execution; fixture ports and key paths were isolated.
  • go vet ./...; go test -race -count=1 ./internal/config ./internal/agentexec ./internal/modelgw ./internal/shellexec ./internal/cli ./internal/engine -> blocked compiling runtime/cgo: /usr/include/stdc-predef.h permission denied.
  • systemd-analyze verify deploy/hetzner2-video/stagehand-video.service -> blocked: working-directory permission denied; retry from scratch directory also denied.
  • CI-verified: go vet ./... -> success; run 43733896182 job 132872139654
  • CI-verified: go test -race -count=1 ./... -> success; run 43733896182 job 132872139654

Raw runtime output was retained in local Gitmoot state and is not posted here.

@jerryfane

Copy link
Copy Markdown
Contributor Author

Reviewer: g7-review
Runtime: omp
Job: local-review-review-router-18e844a2a24c8c11-1

Decision: approved
Summary: Reviewed the complete five-file diff from base 8e06620 to exact head 9e6e342. No P0–P3 findings. The exact config validates through the production CLI; wrapper isolation, installation permissions, unit syntax, shell-stage cgroup inheritance, and tunnel/stale-clear port consistency passed. Existing fleet-runner files are unchanged. CredentialProxy does not authenticate clients: access to hetzner2 127.0.0.1:43299 grants model access. Acceptance relies on the owner-approved firewall evidence in PR comment 6897591543; no live-host probing was performed. Resource directives were validated and child cgroup inheritance exercised, not live systemd limit enforcement. The frozen prior-verdict file records one earlier PR #29 approval with zero findings, so no prior findings remain to answer. Repository files were not modified.

Tests Run

  • PATH=/usr/local/go/bin:$PATH CGO_ENABLED=0 go build -trimpath -o /tmp/stagehand-pr29-review ./cmd/stagehand && /tmp/stagehand-pr29-review version -> success; printed stagehand dev.
  • PATH=/usr/local/go/bin:$PATH go vet ./... -> success.
  • PATH=/usr/local/go/bin:$PATH go test -race -count=1 ./internal/config ./internal/cli ./internal/modelgw ./internal/tunnelclean ./internal/shellexec ./internal/procgroup ./internal/agentexec ./internal/engine ./internal/trigger -> success; all tested packages passed; internal/config has no test files.
  • sh -n deploy/hetzner2-video/stagehand-video -> success.
  • systemd-analyze verify deploy/hetzner2-video/stagehand-video.service deploy/hetzner2-video/fleet-host/stagehand-video-hetzner2-tunnel.service -> success inside an isolated mount namespace with the exact-head binary available at /usr/local/bin/stagehand; initial unisolated attempt reported that executable missing.
  • unshare --user --map-root-user --mount /usr/bin/python3 - (inline deployment harness) -> success; exercised the unchanged config and wrapper with disposable secrets, rejected conflicting environment defaults, failed closed on missing config, executed a shell stage with the model tunnel absent, verified identical runner/child cgroups and HOME=/root, obtained status HTTP 200, rejected an unsigned trigger, and observed clean SIGTERM shutdown.
  • /usr/bin/python3 - (inline installation-permissions harness using install -d -m 700, install -m 600, and umask 077; openssl rand -hex 32) -> success; directories 0700, config and trigger key 0600, trigger key 32 random bytes.
  • ssh -G -N -T -o BatchMode=yes -o StrictHostKeyChecking=yes -o UpdateHostKeys=no -o IdentityAgent=none -o ControlMaster=no -o ControlPath=none -o ExitOnForwardFailure=yes -o ConnectTimeout=10 -o ServerAliveInterval=15 -o ServerAliveCountMax=3 -R 127.0.0.1:43299:127.0.0.1:43295 root@hetzner2 -> success; confirmed loopback reverse-forward endpoints and connection options without opening a connection.

Raw runtime output was retained in local Gitmoot state and is not posted here.

@jerryfane
jerryfane merged commit 6d8b72e into main Nov 15, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Deploy: second runner on hetzner2 for video pipelines

1 participant