Skip to content

fix(engine): queue agent stages behind the sandbox's one-at-a-time limit (#13) - #14

Merged
jerryfane merged 2 commits into
mainfrom
fix/13-agent-stage-concurrency
Nov 3, 2026
Merged

jerryfane merged 2 commits into
mainfrom
fix/13-agent-stage-concurrency

Conversation

@jerryfane

Copy link
Copy Markdown
Contributor

Closes #13. Part of epic #2. Found during the #8 deploy.

Problem

On hetzner2, the scheduled canary run's agent stage started while a manual canary-overcap agent stage was still running. Its run gateway could not bind the instance's single host port, and the run failed: bind: address already in use.

Change

  • stage.Limiter: a new optional interface, MaxConcurrent() int. agentexec.Executor forwards it from its backend, and agentexec.Sandboxd returns 1 (one host port, max-vms 1).
  • Engine:
    • Before starting a stage, the engine reserves a slot for its kind under e.mu. The slot is released when the attempt's goroutine ends, or at once if start fails before launching.
    • A ready stage that gets no slot stays pending.
    • Its run is reported busy and is not settled, so it is neither skipped nor failed, and it starts when the slot frees.
    • The stage's time limit starts only when it actually starts.
  • README: replaces the old "a second one fails to start" note.

Verification

  • go vet passes, and go test -race ./... passes.
  • TestLimitedExecutorQueuesStagesAcrossRuns: two pipelines with one agent stage each, on an executor that allows 1 at a time and fails any overlap. Both runs succeed, sequentially, with no overlap.
  • test-check prove: proven. The test fails on main and passes here.
  • After merge, I redeploy and rerun the same two-pipeline situation on hetzner2 (in Deploy on hetzner2 and the fleet host, docs, end-to-end proof #8).

Deployment

hetzner2 stagehand.service: replace the binary and restart, as part of #8. Rollback: the previous binary stagehand-dd58c28 in /root/fleet-tools/state/gitmoot-pipe/bin/.

@jerryfane
jerryfane merged commit 8d7f895 into main Nov 3, 2026
2 checks passed
@jerryfane

Copy link
Copy Markdown
Contributor Author

Agent: g7-review
Runtime: omp
Job: local-review-review-router-18e4617ab60190f2-1

Decision: approved
Summary: Reviewed the complete eight-file diff from base dd58c28 to exact head 749f89a. No correctness findings. Build, vet, full race suite, repeated focused tests, and independent production-CLI scenarios passed. Verified serialization within and across runs, pending-state preservation, timeout accounting after dequeue, queued cancellation, retries, failure-path slot release, independent shell execution, and gateway-port release. Read the frozen prior-verdict file: it contains no findings for PR #14 requiring answers. No repository files were modified; runtime fixtures used temporary local files and a loopback sandboxd simulator, with no live-service actions.

Tests Run

  • {"command":"/usr/local/go/bin/go version","outcome":"PASS: go1.26.4 linux/amd64."}
  • {"command":"/usr/local/go/bin/go build -o /tmp/stagehand-pr14-review ./cmd/stagehand","outcome":"PASS: exit 0."}
  • {"command":"/usr/local/go/bin/go vet ./...","outcome":"PASS: exit 0, no diagnostics."}
  • {"command":"/usr/local/go/bin/go test -race ./...","outcome":"PASS: all packages passed."}
  • {"command":"/usr/local/go/bin/go test -race ./internal/engine -run 'TestLimitedExecutorQueuesStagesAcrossRuns|TestRetryReRunsFailedStageWithinBudget|TestCancelStopsRunningStagesAndPendingOnes|TestTimeoutFailsStage|TestStoppingRunnerParksRunAsInterrupted' -count=20","outcome":"PASS: all selected scenarios passed twenty repetitions."}
  • {"command":"python3 /tmp/stagehand-pr14-review-t0x98gs5/smoke.py","evidence_locator":"/tmp/stagehand-pr14-review-t0x98gs5/evidence.json","outcome":"PASS after correcting an initial fixture-only assertion that treated the CLI's attempts array as a count. Production binary ran seven pipelines against a loopback sandboxd simulator. Maximum live inventory was one; a stage queued longer than its 300ms timeout remained pending without an attempt and subsequently succeeded; cancellation prevented execution; retry, prompt-loading failure, and execution timeout released capacity for subsequent successful work."}

Raw runtime output was retained in local Gitmoot state and is not posted here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Agent stages of different pipelines collide on the single sandboxd host port

1 participant