Skip to content

Deploy on hetzner2 and the fleet host, docs, end-to-end proof #8

Closed

Description

@jerryfane

Parent: #2

Part of the epic. Owner: gitmoot-pipe.

Change

  • Release build: static (CGO_ENABLED=0) with version, commit and date in stagehand version.
  • hetzner2:
    • /usr/local/bin/stagehand;
    • stagehand.service (serve, home /var/lib/stagehand, config /etc/stagehand/config.yaml), with systemd hardening that suits running shell stages;
    • the status endpoint on loopback.
  • Fleet host: stagehand-gateway-proxy.service and stagehand-model-tunnel.service.
  • Docs: README (install, spec reference, CLI, status JSON, security model, operations: logs, resume, recovery) and an example pipeline.
  • Rollback: written down (previous binary, disabling the units).

Acceptance

  • On hetzner2, a demo pipeline (shell → omp agent in sandboxd → shell) runs on schedule unattended, and its record shows cost, exit codes and logs.
  • An over-cap run writes nothing.
  • status --json and GET /status show the last and next run.
  • stagehand version reports the deployed commit.

Activity

  1. jerryfane commented on Nov 3, 2026

    @jerryfane
    ContributorAuthor

    Deployed (3 Nov ~01:25–01:58 CET).

    What runs where

    Acceptance

    • Scheduled run: the canary pipeline (shell → omp agent in a sandboxd VM with web search → shell check) ran as a schedule-triggered run (sandboxd executor: run omp agent stages in a Firecracker VM #5) and succeeded. Cost 0.0157; decisions.json = Rome with a europa.eu source. The record has the exit codes, the cost and a log path for each stage.
    • Next run: due 2026-11-10T01:14Z. Weekly unattended operation is therefore not yet observed beyond this first scheduled run.
    • Over the cap: canary-overcap (max_cost 0.001) gave over budget: spent 0.0045. Only rows.json was in the work dir; no output was written.
    • Status: stagehand status --json and GET /status both show next_due_at, active_run_id, last_run (trigger, state, cost, tokens) and version.
    • Concurrency (Agent stages of different pipelines collide on the single sandboxd host port #13 fixed in fix(engine): queue agent stages behind the sandbox's one-at-a-time limit (#13) #14): both canaries were started at once. The second agent stage waited for the sandbox slot (01:58:25–33 for run 3, then 01:58:33 for run 4) instead of failing.
    • Cleanup: after all runs the VM inventory was 0 and pending_cleanups was 0.
    • Version: stagehand version reports the deployed commit on both hosts.

    Rollback: the previous binaries are kept on the fleet host in /root/fleet-tools/state/gitmoot-pipe/bin/. The steps are in the README (in #16) and the fleet runbook.

    I'll close this once #16 merges.

  2. jerryfane commented on Nov 3, 2026

    @jerryfane
    ContributorAuthor

    Done. Deployed v0.1.0-dev-c36b550 (main after #12, #14 and #17) on 3 Nov 03:09 CET. I installed it after the nightly efm-daily run ended, as enyo asked.

    Topology (changed in #17 because the efm DB is on the fleet host)

    Host What runs there
    Fleet host stagehand.service (status 127.0.0.1:8797/status), stagehand-gateway-proxy.service, stagehand-hetzner2-tunnel.service
    hetzner2 Only the sandboxd instance

    Acceptance

    Tunnel recovery (#15): tunnel-clear-stale runs before each connection. At startup it logged no sshd session holds it.

    Rollback: previous binaries are in /root/fleet-tools/state/gitmoot-pipe/bin/. Steps are in the README and the fleet runbook.

    Open: weekly unattended operation is first observable at the next canary run, due 2026-11-10 ~01:14Z.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions