Skip to content

Epic: stagehand v1 — standalone pipeline runner with omp agent stages in sandboxd #2

Open
7 of 8 issues completed

Description

@jerryfane

Epic. Owner request: Jerry, 2 Nov 2026, in the gitmoot-pipe seat session. Owner seat: gitmoot-pipe (merge rule self). Each step below is its own sub-issue and PR, merged after one independent exact-head review and green CI.

Why

lenscale.ai needs a scheduled production job with one AI-agent step: weekly "news-research", jerryfane/energy-finance-map#345. Gitmoot's gitmoot pipeline already does this, but the owner does not want Gitmoot (daemon, PR watcher, org, reviews, dashboard) as a production dependency. stagehand is the pipeline runner extracted into a small tool of its own, with omp as the harness for agent stages.

Owner decisions (2 Nov 2026)

Topic Decision
Approach Build a standalone tool, not a timer and script first
Name and repo stagehand, gitmoot/stagehand, public, Apache-2.0
Model access omp subscription through omp's own auth-gateway / auth-broker. No API key, no keyring
Quota Shared with the fleet. No separate account
Cap A usage cap (omp's list-price cost.total) is fine. It is not a dollar bill
Where it runs hetzner2. Agent stages run in a dedicated sandboxd instance stagehand on hetzner2. Later this moves to a lenscale server, so hosts, ports and endpoints are settings, not hard-coded
Gitmoot convergence Decided after v1 has run in production. The core keeps Store / Executor seams so Gitmoot could adopt it
Fleet rule "Automatic agent steps may produce data, not code", added to the fleet guide. The agent cannot write files or run commands. The runner saves its answer. A fixed-rule script checks the answer and writes it. Every change is logged and can be undone. There are time and usage limits

What v1 does

Spec (YAML). A subset of Gitmoot's pipeline spec, so specs stay recognisable, plus a few new keys:

name: news-research
description: Weekly research of rows marked "to research"
schedule: {interval: 168h, jitter: 30m}
env_file: /etc/stagehand/news-research.env   # 0600 KEY=VALUE secrets
env: {EFM_LIMIT: "50"}                         # non-secret defaults
stages:
  - id: export
    cmd: python -m efm.research export --out "$STAGEHAND_WORK/rows.json"
    cwd: /root/efm-daily
    timeout: 10m
    retry: 1
    env_keys: [EFM_DB_URL, EFM_LIMIT]          # deny by default
  - id: research
    needs: [export]
    agent: omp
    model: anthropic/claude-opus-5-5
    effort: high
    prompt_file: prompts/research.md
    inputs: [rows.json]                        # copied from the run work dir into the VM
    output: decisions.json                     # final answer, must be valid JSON
    tools: [read, web_search]
    max_cost: 5.0
    timeout: 50m
  - id: write
    needs: [research]
    cmd: python -m efm.research apply --in "$STAGEHAND_WORK/decisions.json"
    cwd: /root/efm-daily
    env_keys: [EFM_DB_URL]
  - id: summary
    needs: [write]
    cmd: ./scripts/gram-summary.sh

Shell stages

  • Run sh -c <cmd> in cwd, with an exact environment: env_keys plus STAGEHAND_*. The service environment is never inherited.
  • They run in their own process group, which is killed on timeout or cancel.
  • The exit code decides the result.
  • A stage may write $STAGEHAND_RESULT as {"decision":"blocked","summary":"…","needs":["…"]} to park the run instead of failing.
  • stdout and stderr go to the stage log file.

Agent stages (agent: omp)

  • Command: headless omp -p --mode json --no-session --no-skills --no-rules --no-extensions --tools=<tools>, with --model and --thinking, in a fresh sandboxd VM, as guest uid 1000. The agent has no write or exec tools.
  • Output: the runner writes output from the final assistant message and fails the stage if that message is not valid JSON. The agent never writes host files.
  • Caps:
    • The time limit is the stage timeout. omp also gets --max-time with a margin.
    • The usage cap: the runner adds up usage.cost.total from each message_end event and kills the run once the total goes over max_cost.
    • omp exits 0 even when a turn fails, so success is read from the event stream, not the exit code.
  • Model calls never carry a real credential in the guest. The path is:
guest omp (baseUrl http://10.0.2.2:43293, apiKey = per-run random token)
  -> hetzner2 127.0.0.1:43293   stagehand run gateway: listens only while an agent stage runs; checks + strips the per-run token
  -> hetzner2 127.0.0.1:43294   reverse-SSH forward from the fleet host (stagehand-model-tunnel.service)
  -> fleet host 127.0.0.1:43295 `stagehand gateway-proxy`: adds the omp auth-gateway bearer token
  -> fleet host 127.0.0.1:4011  omp auth-gateway -> auth-broker (subscription)
  • Item cap: the item cap per run (for example 50 rows) belongs in the consumer's export stage, not the runner.

Runs

  • A blocked or failed stage parks the run, and nothing downstream starts.
  • resume <run> [--from <stage>] reruns that stage, the stages that depend on it, and any stage that has not succeeded.
  • cancel <run> kills the stages that are running.
  • Only one run per pipeline can be active at a time.
  • Each run stores a snapshot of its spec, so editing the spec never changes a run already in progress.

Recovery

  • On start, the runner parks any stage still marked running from a crash with the reason interrupted. It does not retry it automatically.
  • It destroys any VM recorded for that stage.

Records (SQLite, pure Go modernc.org/sqlite)

  • Per pipeline: spec, enabled, next_due_at.
  • Per run: trigger, state, halt stage and reason, start, end, total cost.
  • Per attempt: state, exit code, cost, tokens, log path, sandbox id, start, end, error.

Schedule: interval plus jitter. Missed intervals collapse into a single run.

CLI

  • add, list, show, runs, run [--wait], resume, cancel, enable, disable, remove, status --json.
  • serve: the scheduler and executor loop, with an optional loopback-only GET /status.
  • tick: one pass for timer mode.
  • gateway-proxy: runs on the fleet host.
  • An exclusive lock makes sure only one process executes stages.

Status for admin.lenscale.ai: status --json and GET /status give, per pipeline:

  • enabled;
  • schedule;
  • next_due_at;
  • last run: id, state, start and end, halt stage and reason, cost.

Packaging: one static Go binary (CGO_ENABLED=0) with no GitHub, PR or org code. It runs as a systemd service.

Not in v1

  • Pipeline chaining (after:).
  • Service exposure and bearer-token run API.
  • Receipts and proofs.
  • Bundle publish and pull.
  • Gitmoot's implement, produce, orchestrate, PR gates and auto-merge.
  • Cron schedules.

sandboxd instance on hetzner2

The sandboxd seat drafted the plan and will build it. It is not created yet.

Setting Value
Instance stagehand
Root /var/lib/sandboxd-stagehand/fc
uid-base 3010000 (Phobos uses 3000000–3001999)
Control API 127.0.0.1:43292
Guest host-port 43293
Size max-vms 1, max-ttl 1h, 2 vCPU, 2048 MiB, 2048 MiB home disk, balloon on
Profile strict (guest uid 1000, no root)
Image built from the Phobos recipe's pinned inputs (omp 18.10.4, SHA-256 pinned), minus the Phobos-only parts

The sandboxd seat confirmed there are no collisions:

  • Host-wide names carry the instance name (sbx_fc_stagehand, sbi-stagehand-<id>).
  • sandboxd refuses an instance that overlaps another one's root, UID range or volume dir.

Limits from phobos-dev:

  • Do not restart sandboxd-phobos or its tunnels.
  • Avoid Phobos's names and ports: 43290, 43291, 41359, 43221, 43222, 43341, 61324.
  • Tell phobos-dev before touching the shared kernel, images, firewall or KVM.
  • Schedule runs outside European daytime.

Known facts and risks

  • omp/18.10.4 --mode json reports usage{…,cost{total}} in each assistant message_end. Measured on 2 Nov: about $0.10 and about 12k tokens of fixed system-prompt overhead per call.
  • omp has no turn, token or cost cap of its own, and no sandbox. That is why the runner enforces the caps and sandboxd isolates the agent. omp's read tool also fetches URLs.
  • The host omp auth-gateway must run a current omp (gitmoot#2475).
  • Any root process on hetzner2 can use port 43294. That is accepted: hetzner2 root is fleet-controlled, and the guest cannot reach that port.
  • The quota is shared with the fleet, and rate limits can stop a run. A failed or over-budget run writes nothing (#345 acceptance).

Plan (sub-issues)

  1. Runner core: spec, store, engine, shell stages, scheduler, CLI, status, CI.
  2. omp agent stage: omp runner, stream parsing, caps, output check, local executor.
  3. sandboxd executor.
  4. Model path: run gateway, gateway-proxy, tunnel unit.
  5. sandboxd instance and image on hetzner2 (built by the sandboxd seat).
  6. Deploy on hetzner2 and the fleet host, docs, end-to-end proof.
  7. lenscale adoption (lenscale seat, via enyo; tracked in jerryfane/energy-finance-map#345).

Analysis and evidence: /root/fleet-tools/state/gitmoot-pipe/analysis.md on the fleet host. Code ported from gitmoot 1b124f60, internal/pipeline.

Activity

  1. jerryfane commented on Nov 2, 2026

    @jerryfane
    ContributorAuthor

    Progress (2 Nov 23:5x): runner core in #10 (closes #3); omp agent stage in #11 (closes #4); sandboxd executor and model path in #12 (closes #5 and #6). The PRs are stacked; each is in its exact-head level-3 review. The sandboxd seat has been asked to build the instance (#7).

  2. jerryfane commented on Nov 3, 2026

    @jerryfane
    ContributorAuthor

    Status, 3 Nov 03:10 CET: the runner is built, reviewed and deployed. #3–#8 are closed; #13 and #15 (found live) are fixed.

    • Running version: v0.1.0-dev-c36b550.
    • Where: the runner is on the fleet host next to the efm DB; agent VMs run in the hetzner2 sandboxd instance.
    • Canaries: the weekly canary passes end to end (cost about 0.016 per run). Next run: 2026-11-10.

    Remaining: #9, lenscale's news-research pipeline, owned by enyo-intern (briefed in msg 386634).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    epicTracking issue with sub-issues

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions