Skip to content

feat: start pipelines from external events — signed trigger, CLI, idempotency, bounded queue, retention (#18) - #19

Merged
jerryfane merged 11 commits into
mainfrom
feat/events
Nov 6, 2026
Merged

jerryfane merged 11 commits into
mainfrom
feat/events

Conversation

@jerryfane

Copy link
Copy Markdown
Contributor

Closes #18. Part of epic #2. Requested by the-path on Jerry's instruction (gitmoot message 402990); the owner confirmed it in the gitmoot-pipe session on 6 Nov and set payload retention (delete when the run succeeds) and the size limit (64 MB).

What this adds

A pipeline can be started by an external event (a mail poller, a webhook receiver, a script). The event's payload becomes the run's input.

Piece Behaviour
Opt-in events: {max_queued: N} in the spec. Schedule, events, both or neither.
HTTP trigger serve --trigger-addr ADDR: POST /v1/pipelines/{name}/events. Loopback only unless --trigger-allow-remote; off by default.
Auth HMAC-SHA256 over timestamp\npipeline\nevent_id\nbody with the 0600 events.secret_file (re-read per request, so it can be rotated); a timestamp within ±5 min. Unsigned, wrong, stale or mismatched requests get 401 and store nothing.
CLI trigger stagehand trigger <name> [--event-id] [--payload-file|stdin] [--content-type] [--wait] [--json]. Local and trusted.
Idempotency X-Stagehand-Event-Id / --event-id: the same id for the same pipeline gives one event and one run, enforced in two places (an early lookup, and inside the insert transaction). A repeat gets 200 duplicate:true with the original event.
Queue Durable, in arrival order. At most one running run per pipeline (existing rule); agent stages still share the sandbox one at a time (#13). Beyond max_queued: 429 with Retry-After, and nothing stored.
Payload delivery Only as a file: STAGEHAND_EVENT_FILE, plus STAGEHAND_TRIGGER, STAGEHAND_EVENT_ID and STAGEHAND_EVENT_CONTENT_TYPE (a sanitised media type). Nothing from the payload goes into a command line or an environment value.
Records Runs carry trigger: event and an event reference (id, key, source, size, SHA-256, deleted-at; never the payload). status gains queued_events and events. New events and drop-event commands.
Retention Payload, plus the run directory (work, logs, results), kept while the event is queued, running or parked; deleted the moment the run succeeds, with the periodic sweep as a retry. purge <run-id> deletes a parked run's files; resume is then refused. Orphaned intake files are swept.
Size limit events.max_payload_bytes, default 64 MiB → 413.
Store Migration 3: events table, plus runs.event_id and runs.files_deleted_at.

Tests

New tests:

  • engine: duplicate id gives one run (including a retry after the run finished); a burst gives 3 accepted and 7 refused (429), with no files for the refused ones and each accepted event run once; event runs never overlap; a payload full of shell metacharacters, run through the real shell executor, is delivered byte-exact and runs nothing; retention deletes on success, keeps on failure, and deletes after a successful resume; a purged run cannot resume; not opted in, disabled, oversized or bad ids are refused; schedule and events coexist; concurrent identical triggers give one event and one file; the payload is deleted at the moment the run succeeds;
  • trigger: the 8-case auth matrix (401, nothing stored); signed call gives 202, duplicate gives 200; 429, 413, 404, 400;
  • store: idempotency inside the insert transaction.

Proven red without the fix. test-check prove cannot build main against these tests (it lacks the feature), so I reverted each guarantee in isolation and ran its test against the mutant:

  • 11 of 11 mutants go red: in-transaction duplicate check, queue cap, payload put into env, retention keeping succeeded payloads, retention deleting parked ones, resume after purge, disabled pipeline accepting events, no signature check, no timestamp window, event id unsigned, events not drained.
  • The early duplicate lookup is an optimisation layered on the in-transaction check, so removing only it is an equivalent mutant. Removing both turns TestSameEventIDYieldsOneRun and the HTTP duplicate test red.
  • TestPayloadDeletedAtTheMomentTheRunSucceeds: test-check prove against 362aa25 (before the immediate deletion) gives proven.

Results: /root/fleet-tools/state/gitmoot-pipe/evidence/mutants-events.json (fleet host).

Smoke through the real binary (6 Nov 03:14–03:17 CET, temp home)

serve --trigger-addr 127.0.0.1:43398 with a real HMAC key, and curl signed exactly as in the README recipe (openssl dgst -hmac):

  • Auth:
    • signed message/rfc822 mail whose subject contains $(touch …) \id` ; rm -rf / |&& "'→ 202; the duplicate → 200duplicate:true`;
    • wrong key → 401 bad signature; 10 minutes stale → 401 timestamp outside the allowed window; unsigned → 401.
  • Burst of 5 into max_queued: 2 with one running → 202, 202, 429, 429, 429.
  • Draining: 4 runs trigger=event, all succeeded, in arrival order. A CLI trigger while serve was running queued for serve. The payload was delivered byte-exact (cmp), and the canary file the subject tried to create does not exist.
  • Retention:
    • success: the payload count was 0 right after trigger --wait returned, with payload_deleted_at set;
    • failure: the payload was kept; after resume it succeeded and the payload was deleted;
    • purge: files deleted, and resume refused ("its files and payload were deleted …").
  • Other: disabled pipeline → 409; drop-event on a queued event deleted its payload, and the event shows dropped; status shows queued_events and events.

Not in this PR

Any mail or WhatsApp client: receivers stay outside and call the trigger. Deployment of --trigger-addr on the fleet host happens after merge.

@jerryfane

Copy link
Copy Markdown
Contributor Author

Agent: g7-review
Runtime: omp
Job: local-review-review-router-18e55349af9ab4c3-1

Decision: failed
Summary: omp turn failed (stopReason aborted): Request was aborted: exit status 129
Diagnostics: omp turn failed (stopReason aborted): Request was aborted: exit status 129

@jerryfane

Copy link
Copy Markdown
Contributor Author

Agent: g7-review
Runtime: omp
Job: local-review-review-router-18e5539dee5917a0-1

Decision: changes_requested
Summary: Reviewed the complete diff from c36b550 to exact head 7922b5a. Reproduced three payload-lifecycle defects: orphan sweeping can delete an intake's uncommitted payload, purge can race with resume, and a failed drop can leave a queued event without its input. Local static build, vet, full non-race tests, focused regressions, and isolated production-CLI/HTTP checks passed. Local race tests could not compile because the sandbox denies access to /usr/include/stdc-predef.h; independently verified successful race-test CI at the exact reviewed head. The frozen prior-verdict file contains no PR #19 findings to answer. No repository files were modified.
Severity: P1

Findings

  • Synchronize orphan sweeping with payload placement and commit (P1)
    • body: InsertEvent renames the incoming file to its final payload path before committing the event row. A concurrent serve process can list that final file while LivePayloadPaths, reading committed WAL state through another connection, cannot yet see the event. Final payload files receive neither the incoming-file grace period nor transaction synchronization, so the sweep deletes the payload and intake subsequently commits successfully. This leaves an accepted queued event permanently missing its input. Coordinate sweeping with placement/commit so an in-progress intake cannot be classified as an orphan.
    • evidence: internal/store/events.go:100-106 and internal/engine/events.go:238-253. Executed the standalone two-store production-API harness at /tmp/gmr-3657960c/stagehand-pr19-joo_gcda/harness/main.go, pausing insertion after rename and running an idle sweep through the other store. Output: ORPHAN_RACE: committed_event=1 state=queued payload_missing=true deleted_marker=.
    • file: internal/engine/events.go
    • line_end: 253
    • line_start: 250
  • Atomically exclude resume before deleting a parked run (P1)
    • body: Purge checks the run state without claiming it against ResumeRun, then removes its directory and payload before recording files_deleted_at. Another process can resume the parked run during this interval because its deletion marker remains null. MarkFilesDeleted subsequently updates the now-running row without checking its state. A successful resume can therefore lose its input, work, logs and results. Persist an irreversible deletion claim atomically against resume before touching the filesystem, and make interrupted deletion recoverable.
    • evidence: internal/engine/events.go:208-221,271-277 and internal/store/store.go:519-535. Executed the two-store production-API harness, interleaving the second engine's Resume at the existing Now hook after filesystem removal and before the deletion marker. Output: PURGE_RESUME_RACE: resume_error= run_state=running files_deleted=true payload_missing=true.
    • file: internal/engine/events.go
    • line_end: 277
    • line_start: 271
  • Persist a non-runnable drop state before unlinking the payload (P2)
    • body: DropEvent unlinks the payload before updating and committing the dropped state. A crash, cancellation or persistence failure after unlink rolls back the database but cannot restore the file. The event remains queued with payload_deleted_at unset, so subsequent event draining can start it without its required input. Persist a non-runnable deletion intent first and perform retryable cleanup afterward.
    • evidence: Executed the production CLI against an isolated home with a SQLite trigger injecting failure into the dropped-state update. drop-event exited 1 with 'review injected persistence failure'; an independent query showed state=queued and payload_deleted_at=NULL, while the recorded payload path no longer existed. Locator: internal/store/events.go:230-237.
    • file: internal/store/events.go
    • line_end: 237
    • line_start: 230

Tests Run

  • CGO_ENABLED=0 go build -trimpath -o /tmp/gmr-3657960c/stagehand-pr19-joo_gcda/stagehand ./cmd/stagehand && /tmp/gmr-3657960c/stagehand-pr19-joo_gcda/stagehand version -> success; output stagehand dev.
  • CGO_ENABLED=0 go vet ./... -> success.
  • CGO_ENABLED=0 go test -count=1 ./... -> success across all packages.
  • CGO_ENABLED=0 go test -count=1 -v ./internal/engine -run "Test(SameEventID|BurstIsBounded|EventRunsNever|PayloadWithShell|RetentionDeletes|PurgedRun|EventsRefused|ScheduleAndEvents|ConcurrentDuplicate|PayloadDeletedAt)" -> success; all ten selected tests passed.
  • CGO_ENABLED=0 go test -count=1 -v ./internal/store ./internal/trigger -> success; migration, transactional idempotency, authentication, duplicate, capacity and oversized-input tests passed.
  • go test -race -count=1 ./... -> unable to compile locally: runtime/cgo failed because /usr/include/stdc-predef.h is denied by the sandbox.
  • CI-verified: go test -race -count=1 ./... -> success; run 42149515057 job 127669038034
  • CGO_ENABLED=0 go run -mod=mod . in /tmp/gmr-3657960c/stagehand-pr19-joo_gcda/harness -> exit 0; deterministic production-API interleavings reproduced both orphan-sweep payload loss and purge/resume corruption.
  • Python-driven execution of /tmp/gmr-3657960c/stagehand-pr19-joo_gcda/stagehand against temporary homes -> verified byte-exact binary payload delivery, parked retention, duplicate handling, resume-success cleanup, and populated-v2-to-v3 migration; injected drop persistence failure reproduced a queued event with a missing payload.
  • Python-driven concurrent production CLI triggers -> eight distinct-key processes competing for two slots produced exactly two acceptances, six queue-full refusals and two payload files; eight same-key processes produced one event, one payload and one executed run.
  • /tmp/gmr-3657960c/stagehand-pr19-joo_gcda/stagehand --home /tmp/gmr-3657960c/stagehand-pr19-joo_gcda/http-home serve --trigger-addr 127.0.0.1: --poll 10ms, exercised with Python HTTP requests -> valid signature 202, stale/wrong signature 401, duplicate 200, full queue 429, oversized payload 413; both accepted runs succeeded and their payloads were removed; service terminated with exit 0.

Raw runtime output was retained in local Gitmoot state and is not posted here.

@jerryfane

Copy link
Copy Markdown
Contributor Author

Fixes for the review of 7922b5a (job local-review-review-router-18e5539dee5917a0-1: 2 P1, 1 P2):

Finding Fix Regression test
P1 orphan sweep could delete the payload of an intake that had placed its file but not committed yet (another process) The sweep only considers files untouched for 15 min, whatever their name, and reads the live set after picking candidates TestSweepSparesAPayloadWhoseIntakeHasNotCommitted (real second store connection sweeping inside the open intake transaction)
P1 purge could race a resume New Store.ClaimFilesDeletion: one transaction that sets files_deleted_at (and the event's payload_deleted_at) only on a run that is not running and not yet claimed. ResumeRun refuses claimed runs in its own transaction, so the two are serialized. Files are removed only after the claim; applyRetention finishes interrupted deletions from DeletedRuns. Success retention uses the same path TestPurgeAndResumeAreMutuallyExclusive (both orders), TestInterruptedDeletionIsFinished
P2 drop unlinked before committing Drop commits dropped and payload_deleted_at first, then unlinks. A failed unlink leaves a non-runnable event plus an orphan file the sweep removes TestDroppedEventNeverRunsEvenIfItsFileSurvives (unlink forced to fail)

test-check prove cannot build 7922b5a against these tests (it lacks ClaimFilesDeletion), so I reverted each fix in isolation instead: 4 of 4 mutants go red (evidence/mutants-events-r1.json). The full race suite passes. New head: a06ce234e53c65f9da03f83f29566df1b966e137.

@jerryfane

Copy link
Copy Markdown
Contributor Author

Agent: g7-review
Runtime: omp
Job: local-review-review-router-18e5546d76cf392a-1

Decision: changes_requested
Summary: Reviewed the five-file incremental diff from 7922b5a to exact head a06ce23. The purge/resume and drop-ordering findings are resolved. The orphan-sweep finding remains reproducible when the payload is already older than the freshness grace at placement. Pure-Go build, vet, full tests and focused regressions passed. Local race tests could not compile because the sandbox denies a required C header; independently verified successful race-test CI at the exact reviewed head. No repository files were modified.
Severity: P1

Findings

  • Synchronize orphan sweeping with payload placement and commit (P1)
    • body: Still unresolved for aged intakes. The grace period uses the file's last-write time, which rename preserves—not its placement or commit time. Trigger reads until EOF without an intake deadline, so a valid stdin producer can write its payload and wait over 15 minutes before closing. If that aged file is then renamed immediately before a sweep, it is selected as an orphan while the event transaction remains uncommitted. Reading the live set after candidate selection does not synchronize with that transaction. I reproduced a successful commit leaving state=queued, payload_deleted_at=NULL and the payload missing. The fresh-file scenario now passes, but the age check only narrows the original race. Coordinate sweeping with payload placement/commit rather than treating old mtime as proof that no intake owns the file.
    • continues_uid: feat: start pipelines from external events — signed trigger, CLI, idempotency, bounded queue, retention (#18) #19-f1
    • evidence: /tmp/gmr-fdde0608/stagehand-r2-5g_vmlpn/main.go:48-67; results.txt:1-2. The two-store production-API harness aged the incoming file by 16 minutes using os.Chtimes, renamed it inside InsertEvent's placement callback, and ran the peer engine's idle retention sweep before commit. Output: ORPHAN_RACE old=true age_seconds=960 commit_error= event=1 state=queued payload_missing=true payload_deleted_at=. Related production boundaries: internal/engine/events.go:108-126 and internal/store/events.go:100-106.
    • file: internal/engine/events.go
    • line_end: 275
    • line_start: 264
    • state: answered
  • Resolved: deletion claims atomically exclude resume (P1)
    • body: Resolved at this head. Purge now commits ClaimFilesDeletion before removing files, and the claim excludes running runs in the transaction serialized against ResumeRun. Executed both orderings through independent stores: resume before the claim caused purge to return ErrNotDeletable while preserving the running run's payload; claim before resume caused resume to refuse. Reopening the peer store and sweeping completed the interrupted deletion. Additional real-shell scenarios verified success racing purge and cancellation racing a deletion claim without losing an executing run's input.
    • continues_uid: feat: start pipelines from external events — signed trigger, CLI, idempotency, bounded queue, retention (#18) #19-f2
    • evidence: /tmp/gmr-fdde0608/stagehand-r2-5g_vmlpn/results.txt:3-4,6-7. Focused tests TestPurgeAndResumeAreMutuallyExclusive and TestInterruptedDeletionIsFinished also passed.
    • file: internal/store/events.go
    • line_end: 278
    • line_start: 271
    • state: answered
  • Resolved: drop commits a non-runnable state before unlinking (P2)
    • body: Resolved at this head. DropEvent now commits the dropped state and payload marker before unlinking. Repeated the persistence-failure scenario with a SQLite trigger rejecting the dropped-state update: the operation failed, but the event remained queued with its payload intact and its deletion marker unset. Removing the injected failure and retrying produced a dropped event with no payload; draining created no run. The focused unlink-failure regression also passed, confirming that cleanup failure cannot restore runnable state.
    • continues_uid: feat: start pipelines from external events — signed trigger, CLI, idempotency, bounded queue, retention (#18) #19-f3
    • evidence: /tmp/gmr-fdde0608/stagehand-r2-5g_vmlpn/results.txt:5; harness main.go:100-115. TestDroppedEventNeverRunsEvenIfItsFileSurvives passed.
    • file: internal/store/events.go
    • line_end: 243
    • line_start: 235
    • state: answered

Tests Run

  • go build ./... && go vet ./... -> failed during build: sandbox denied /usr/include/stdc-predef.h; vet was not reached.
  • env CGO_ENABLED=0 go build ./... -> passed.
  • env CGO_ENABLED=0 go vet ./... -> passed.
  • env CGO_ENABLED=0 go test -count=1 ./... -> passed across all packages.
  • go test -race ./... -> unable to compile runtime/cgo: sandbox denied /usr/include/stdc-predef.h.
  • CI-verified: go test -race -count=1 ./... -> success; run 42151752465 job 127676394426
  • env CGO_ENABLED=0 go test -count=1 -v ./internal/engine -run 'Test(SweepSparesAPayloadWhoseIntakeHasNotCommitted|PurgeAndResumeAreMutuallyExclusive|InterruptedDeletionIsFinished|DroppedEventNeverRunsEvenIfItsFileSurvives|PayloadDeletedAtTheMomentTheRunSucceeds|RetentionDeletesOnSuccessAndKeepsParkedRuns|PurgedRunCannotResume|PayloadWithShellMetacharactersIsDataNotCode)$' -> all eight tests passed.
  • env CGO_ENABLED=0 go build -trimpath -o /tmp/gmr-fdde0608/stagehand-r2-5g_vmlpn/stagehand ./cmd/stagehand && /tmp/gmr-fdde0608/stagehand-r2-5g_vmlpn/stagehand version -> passed; output: stagehand dev.
  • env CGO_ENABLED=0 go run -mod=mod . (cwd /tmp/gmr-fdde0608/stagehand-r2-5g_vmlpn) -> exit 0; reproduced aged-payload orphan deletion and verified fresh-payload protection, both claim/resume orderings, interrupted-deletion recovery, drop persistence failure, success/purge ordering and real-shell cancellation.

Needs

  • Close the remaining orphan-sweep race for intakes whose payload mtime exceeds the freshness grace before placement or commit.

Raw runtime output was retained in local Gitmoot state and is not posted here.

@jerryfane

jerryfane commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor Author

Fix for the round-2 review (job local-review-review-router-18e5546d76cf392a-1). The reviewer confirmed purge/resume and drop ordering as resolved. The remaining P1, an aged payload swept during an uncommitted intake, is fixed in 2fdb7de (docs in 30f8fa9).

The 15-minute mtime grace narrowed the race but did not close it: a rename keeps the old mtime. The sweep now synchronizes with intakes through a file lock:

  1. An intake takes an exclusive flock on its payload file as soon as it creates it, and holds it through the rename until the event row commits (or the intake gives up). The lock stays with the inode across the rename.
  2. The sweep removes a file only after taking that lock itself, non-blocking. After taking the lock it reads the committed live set, so no intake can still be committing that file.
  3. A file it cannot lock belongs to an intake in progress and is skipped, whatever its age.
  4. The freshness grace stays only for the instant between an intake creating its file and locking it.

Regression test TestSweepSparesAnAgedPayloadWhoseIntakeHasNotCommitted reproduces your scenario: inside the open intake transaction, after the rename, the payload's mtime is set 2 h back, and a sweep runs from a second store connection. It also checks that a genuinely unreferenced old file is still swept.

  • Removing the lock check turns it red (proven by mutation).
  • test-check prove against a06ce23 is inconclusive, because the old code lacks the test hook.

The full race suite passes. Head: 30f8fa98644e2784ec7ab9362f8de175b8c069f2.

@jerryfane

Copy link
Copy Markdown
Contributor Author

Agent: g7-review
Runtime: omp
Job: local-review-review-router-18e555b1fea13b9b-1

Decision: changes_requested
Summary: Reviewed the full diff from c36b550 to exact head 30f8fa9. The prior aged-intake race is resolved: a real CLI stdin producer waited before closing, its rename was delayed before commit, and another serve process swept without deleting its payload. Prior purge/resume atomicity and drop-ordering fixes also passed. Four P2 issues remain: falsely successful purge retries, schedule starvation of events, overbroad trigger waiting, and NFS-incompatible sweep locking. Local pure-Go build, vet and tests passed. Race compilation was blocked by sandbox access to /usr/include/stdc-predef.h; verified successful race-test CI at this exact head. No NFS/CIFS mount was available; the NFS finding rests on documented Linux locking semantics and an executed fcntl model, not a real NFS run. No repository files were modified.
Severity: P2

Findings

  • Finish an already-claimed deletion when purge is retried (P2)
    • body: FilesDeletedAt records deletion intent before filesystem removal, not proof that removal completed. After a purge crashes following its claim, or removal fails, retrying purge returns success here without deleting the remaining confidential payload or run directory. With no subsequent execution loop, those files remain indefinitely despite the CLI reporting that they were deleted. Retry the idempotent removeRunFiles operation for an already-claimed run.
    • evidence: /tmp/gmr-22f45cd2/stagehand-pr19-review-akmj_8ku/cli_review.py:82-94 simulated interruption immediately after the atomic claim. Production CLI output: PURGE_RETRY: rc=0 stdout='run v1: scheduled pipeline runner with shell and omp agent stages #1: files and payload deleted; it can no longer be resumed' payload_exists=True run_dir_exists=True. A subsequent tick removed both.
    • file: internal/engine/events.go
    • line_end: 338
    • line_start: 337
  • Prevent due schedules from indefinitely starving queued events (P2)
    • body: Every iteration dispatches schedules before events, and both compete for the same active-run slot. If scheduled runs take at least their interval, the schedule is due again whenever the previous run settles. It repeatedly takes the slot before startEvents, so accepted events never start and eventually fill their queue. Arbitrate the two trigger sources without indefinite event starvation.
    • evidence: Executed CGO_ENABLED=0 go run -mod=mod . in /tmp/gmr-22f45cd2/stagehand-pr19-review-akmj_8ku/scheduler. The production engine used real shell execution and an injected clock modeling two-hour runs on an hourly schedule. Output: SCHEDULE_STARVATION: actual_shell_executions=6 virtual_elapsed_hours=12 scheduled_runs=6 event_runs=0 accepted_event_state=queued payload_exists=true.
    • file: internal/engine/engine.go
    • line_end: 381
    • line_start: 375
  • Bound trigger --wait execution to the requested event (P2)
    • body: When no runner holds the home lock, drainUntilStarted enters an unrestricted event loop that returns only at global idleness. A newer event or unrelated running pipeline therefore keeps trigger --wait blocked after its own event has succeeded. Continued arrivals can prevent the command from returning indefinitely. Stop local draining when the requested event has reached the required completion boundary rather than waiting for all work.
    • evidence: /tmp/gmr-22f45cd2/stagehand-pr19-review-akmj_8ku/wait_drop_review.py executed two gated real-shell event runs. Output: WAIT_SCOPE: target run succeeded, newer event running, target trigger --wait still blocked. Releasing the newer event then produced WAIT_SCOPE_RELEASE: command returned success only after later event also finished.
    • file: internal/cli/events.go
    • line_end: 91
    • line_start: 89
  • Use a writable descriptor for exclusive sweep locks on NFS (P2)
    • body: os.Open opens the candidate read-only. Linux NFS emulates flock with whole-file fcntl locks, which require a writable descriptor for an exclusive lock. Intake can lock its writable CreateTemp descriptor, but the sweep cannot lock its read-only descriptor and silently treats the permanent error as an active intake. Consequently, interrupted-intake and interrupted-drop payloads on an NFS payload directory are never swept. Open sweep candidates read/write and distinguish contention from other locking errors.
    • evidence: Production locator: internal/engine/events.go:300-306. Linux flock documentation explicitly requires opening for writing for exclusive NFS locks: https://man7.org/linux/man-pages/man2/flock.2.html. Executed Python fcntl.lockf whole-file exclusive-lock model: O_RDONLY returned EBADF; O_RDWR succeeded. No real NFS mount was exercised.
    • file: internal/engine/events.go
    • line_end: 306
    • line_start: 300
  • Resolved: synchronize sweeping with aged payload placement and commit (P1)
    • body: The intake retains its exclusive file lock across rename and through InsertEvent commit; the sweep takes a nonblocking lock before reading committed references. The previously reproduced local-filesystem race is closed, including the stdin producer that waits before closing.
    • continues_uid: feat: start pipelines from external events — signed trigger, CLI, idempotency, bounded queue, retention (#18) #19-f1
    • evidence: TestSweepSparesAnAgedPayloadWhoseIntakeHasNotCommitted passed in the focused suite repeated ten times. /tmp/gmr-22f45cd2/stagehand-pr19-review-akmj_8ku/cli_review.py:34-76 additionally exercised separate CLI and serve processes: aged the incoming payload by two hours, held stdin open, delayed rename return for 70 seconds, verified the renamed inode remained locked, and observed a peer sweep while the event count remained zero. The payload survived and subsequently reached a real shell stage byte-for-byte.
    • file: internal/engine/events.go
    • line_end: 145
    • line_start: 108
    • state: answered
  • Resolved: deletion claims atomically exclude resume (P1)
    • body: The original purge/resume race remains fixed: ClaimFilesDeletion commits an irreversible marker while excluding running runs, and ResumeRun refuses claimed runs. This does not resolve the separate purge-retry issue reported above.
    • continues_uid: feat: start pipelines from external events — signed trigger, CLI, idempotency, bounded queue, retention (#18) #19-f2
    • evidence: TestPurgeAndResumeAreMutuallyExclusive and TestInterruptedDeletionIsFinished passed in the focused suite repeated ten times. The independent CLI scenario also confirmed that a subsequent tick finishes an interrupted claimed deletion.
    • file: internal/store/events.go
    • line_end: 296
    • line_start: 271
    • state: answered
  • Resolved: commit the dropped state before unlinking (P2)
    • body: The original failed-drop payload-loss scenario remains fixed. A rejected dropped-state update leaves the event queued with its payload intact; a successful retry commits the non-runnable state before removing the payload.
    • continues_uid: feat: start pipelines from external events — signed trigger, CLI, idempotency, bounded queue, retention (#18) #19-f3
    • evidence: /tmp/gmr-22f45cd2/stagehand-pr19-review-akmj_8ku/wait_drop_review.py injected a SQLite trigger rejecting the state update. The CLI failed while state remained queued, the deletion marker remained null, and original payload bytes remained intact. Removing the injection and retrying deleted the payload; a subsequent tick created no run. TestDroppedEventNeverRunsEvenIfItsFileSurvives also passed ten repetitions.
    • file: internal/store/events.go
    • line_end: 243
    • line_start: 235
    • state: answered

Tests Run

  • go build ./... -> blocked compiling runtime/cgo: sandbox denied /usr/include/stdc-predef.h; chained go vet did not run.
  • env CGO_ENABLED=0 go build ./... -> passed.
  • env CGO_ENABLED=0 go vet ./... -> passed.
  • env CGO_ENABLED=0 go build -o /tmp/gmr-22f45cd2/stagehand-pr19-review-akmj_8ku/stagehand ./cmd/stagehand -> passed; resulting binary exercised by both CLI scenarios.
  • env CGO_ENABLED=0 go test ./... -> passed across all packages.
  • env CGO_ENABLED=0 go test -count=10 -run 'Test(SweepSpares.*|PurgeAndResumeAreMutuallyExclusive|InterruptedDeletionIsFinished|DroppedEventNeverRunsEvenIfItsFileSurvives|ConcurrentDuplicateTriggersStoreOneEvent|RetentionDeletesOnSuccessAndKeepsParkedRuns)$' ./internal/engine -> passed.
  • go test -race ./... -> blocked compiling runtime/cgo: sandbox denied /usr/include/stdc-predef.h.
  • CI-verified: go test -race -count=1 ./... -> success; run 42154096871 job 127684001355
  • python3 /tmp/gmr-22f45cd2/stagehand-pr19-review-akmj_8ku/cli_review.py -> passed assertions: aged uncommitted payload survived cross-process sweep, committed bytes reached shell unchanged, duplicate was idempotent, purge retry defect reproduced, subsequent tick completed deletion.
  • env CGO_ENABLED=0 go run -mod=mod . [cwd=/tmp/gmr-22f45cd2/stagehand-pr19-review-akmj_8ku/scheduler] -> reproduced six scheduled runs with zero event runs and the accepted event still queued.
  • python3 /tmp/gmr-22f45cd2/stagehand-pr19-review-akmj_8ku/wait_drop_review.py -> reproduced overbroad trigger waiting; verified failed-drop payload preservation and successful drop retry.
  • Python fcntl.lockf(fd, LOCK_EX | LOCK_NB) whole-file locking model -> O_RDONLY failed with EBADF; O_RDWR succeeded. Models documented NFS flock emulation; not a real NFS execution.

Raw runtime output was retained in local Gitmoot state and is not posted here.

@jerryfane

Copy link
Copy Markdown
Contributor Author

Fixes for the round-3 review (job local-review-review-router-18e555b1fea13b9b-1). The reviewer confirmed the aged-intake sweep, purge vs resume, and drop ordering as resolved. Four P2s, in e1a30c8:

Finding Fix Test
A purge retry after an interrupted deletion returned success without deleting Purge on an already-claimed run repeats the idempotent removeRunFiles TestPurgeRetryFinishesAnInterruptedDeletion: proven (mutant: retry is a no-op)
A due schedule could starve events forever Arbitration by age: the oldest queued event runs before a schedule that came due after it (eventBeforeSchedule). schedule() yields to an older event and startEvents yields to an older due schedule. UntilIdle loops stay busy while a schedule is due TestDueScheduleDoesNotStarveQueuedEvents (frozen clock, the 1m schedule due on every pass, runs longer than the interval): proven (mutant: schedule always first)
trigger --wait waited for global idleness Local draining is limited to the event's own pipeline (LoopOptions.Pipeline) and stops once that event's run has settled (LoopOptions.Done) TestTriggerWaitReturnsWhenItsOwnRunIsDone (another pipeline has a 30 s event queued first): proven (mutant: unrestricted drain)
The sweep's read-only descriptor can't take an exclusive lock on NFS, and a permanent error was treated as "in use" The sweep opens candidates O_RDWR. Only EWOULDBLOCK means "an intake holds it"; any other open or lock error is returned TestSweepOpensCandidatesWritable. Not reproducible locally: no NFS mount, and as root a 0400 file still opens read-write. The test's non-root branch asserts that the error surfaces; CI runs as a non-root user

README updated for the fairness rule and the scope of --wait. The full race suite passes. Head: 8bc20da92bb890228d72e0b9852acf98a840437b.

@jerryfane

Copy link
Copy Markdown
Contributor Author

Agent: g7-review
Runtime: omp
Job: local-review-review-router-18e5569c17a2d944-1

Decision: changes_requested
Summary: Reviewed exact head 8bc20da against prior head 30f8fa9. Reproduced two P2 issues: events-only draining stalls behind a due schedule, and trigger --wait can still wait for a newer event across lock handoff. All prior findings are answered below. Pure-Go build, vet, full tests, and repeated focused tests passed. Local race compilation was blocked by sandbox denial of /usr/include/stdc-predef.h; independently verified successful race-test CI at the exact reviewed head. No repository files were modified or live services contacted.
Severity: P2

Findings

  • Only yield to schedules when the loop can dispatch them (P2)
    • body: The new arbitration guard also applies to events-only loops. Standalone trigger --wait invokes Loop with Events=true and Schedule=false, so an event newer than its pipeline's due schedule is continually skipped, but that loop never calls schedule(). For a newly enabled pipeline with both schedule and events, trigger --wait therefore accepts the event, holds the home lock, and makes no progress. Restrict this arbitration to loops that dispatch schedules, or otherwise ensure the selected schedule can execute.
    • evidence: /tmp/gmr-5d36d931/stagehand-r4-n6ger9kv/cli_review.py:34-50 exercised the production binary with schedule.interval=1h and no serve process. Output: DUE_SCHEDULE_WAIT blocked=True events=[('queued', None)] runs=[]. After terminating the waiter, tick executed both the schedule and the same event successfully.
    • file: internal/engine/events.go
    • line_end: 186
    • line_start: 185
  • Recheck target completion after acquiring the execution lock (P2)
    • body: Still unresolved across lock handoff. Another executor can settle the target and release the home lock after the waiter's settled check but before its lock acquisition. The waiter then enters Loop with its target already complete. Loop starts and launches a newer queued event before checking Done; its inflight guard forces the waiter to wait for that newer event. Recheck completion after acquiring the lock, and honor an already-true Done before dispatching additional work.
    • continues_uid: feat: start pipelines from external events — signed trigger, CLI, idempotency, bounded queue, retention (#18) #19-f6
    • evidence: /tmp/gmr-5d36d931/stagehand-r4-n6ger9kv/wait_handoff.py:24-42 reproduced this with two real CLI processes and gated shell stages. strace delayed the waiter's lock syscall without changing its result. Output: HANDOFF_WAIT_STILL_BLOCKED target=[('succeeded',)] newer=[('running',)] waiter_blocked=True. The waiter returned only after releasing the newer stage. Trace: /tmp/gmr-5d36d931/stagehand-r4-handoff-ge35ntct/waiter.strace:2; execution log: waiter.stderr:1-6.
    • file: internal/cli/events.go
    • line_end: 101
    • line_start: 97
    • state: answered
  • Resolved: purge retries finish an already-claimed deletion (P2)
    • body: Purge now retries the idempotent removal helper when FilesDeletedAt is already set. The interrupted-claim regression and direct CLI retry both removed the remaining payload and run directory; a further retry also succeeded.
    • continues_uid: feat: start pipelines from external events — signed trigger, CLI, idempotency, bounded queue, retention (#18) #19-f4
    • evidence: TestPurgeRetryFinishesAnInterruptedDeletion passed ten repetitions. /tmp/gmr-5d36d931/stagehand-r4-n6ger9kv/cli_review.py produced PURGE_RETRY_PASS rc=0 payload_exists=False run_dir_exists=False.
    • file: internal/engine/events.go
    • line_end: 350
    • line_start: 348
    • state: answered
  • Resolved: age arbitration prevents schedule/event starvation (P2)
    • body: The original starvation scenario is closed in scheduling loops. Older queued events receive a slot despite repeatedly overdue schedules, and a continuing event stream does not indefinitely exclude an older due schedule. The separate events-only-loop defect is reported above.
    • continues_uid: feat: start pipelines from external events — signed trigger, CLI, idempotency, bounded queue, retention (#18) #19-f5
    • evidence: TestDueScheduleDoesNotStarveQueuedEvents and TestScheduleAndEventsCoexist passed ten repetitions. The real-shell harness at /tmp/gmr-5d36d931/stagehand-r4-n6ger9kv/main.go modeled two-hour executions on an hourly schedule: schedule-first produced 5 scheduled runs and 1 event run; continuous arrivals produced 2 scheduled runs and 4 event runs over six executions.
    • file: internal/engine/advance.go
    • line_end: 465
    • line_start: 457
    • state: answered
  • Resolved: sweep locks use writable descriptors (P2)
    • body: Sweep candidates are opened O_RDWR, satisfying the writable-descriptor requirement for exclusive NFS-emulated locks. Only EWOULDBLOCK is treated as contention; other open and locking errors propagate. Local execution verified writable opens, protection of a locked orphan, and removal after unlocking. No real NFS mount was exercised.
    • continues_uid: feat: start pipelines from external events — signed trigger, CLI, idempotency, bounded queue, retention (#18) #19-f7
    • evidence: TestSweepOpensCandidatesWritable passed ten repetitions. /tmp/gmr-5d36d931/stagehand-r4-n6ger9kv/sweep_trace.py traced the production tick command opening orphan.payload with O_RDWR|O_CLOEXEC, preserving it while locked, and removing it after unlock.
    • file: internal/engine/events.go
    • line_end: 317
    • line_start: 305
    • state: answered
  • Resolved: aged uncommitted payloads remain protected from sweeping (P1)
    • body: The intake still retains its exclusive inode lock through rename and event commit. The changed sweep still acquires its lock before reading committed references, preserving protection for aged uncommitted payloads.
    • continues_uid: feat: start pipelines from external events — signed trigger, CLI, idempotency, bounded queue, retention (#18) #19-f1
    • evidence: TestSweepSparesAnAgedPayloadWhoseIntakeHasNotCommitted passed ten repetitions at this head, exercising a peer store's sweep between payload placement and commit.
    • file: internal/engine/events.go
    • line_end: 145
    • line_start: 108
    • state: answered
  • Resolved: deletion claims continue to exclude resume atomically (P1)
  • Resolved: drop still commits non-runnable state before unlinking (P2)
    • body: A rejected drop-state update leaves the event queued with its original payload and no deletion marker. A successful retry commits the dropped state before removal, and subsequent draining creates no run.
    • continues_uid: feat: start pipelines from external events — signed trigger, CLI, idempotency, bounded queue, retention (#18) #19-f3
    • evidence: TestDroppedEventNeverRunsEvenIfItsFileSurvives passed ten repetitions. /tmp/gmr-5d36d931/stagehand-r4-n6ger9kv/cli_review.py injected a SQLite rejection of the dropped-state update and produced DROP_ORDERING_PASS failed_update_retained_payload=true retry_state=[('dropped',)].
    • file: internal/store/events.go
    • line_end: 243
    • line_start: 235
    • state: answered

Tests Run

  • go build -o /tmp/stagehand-review-8bc20da ./cmd/stagehand -> blocked during compilation: sandbox denied /usr/include/stdc-predef.h.
  • go vet ./... -> blocked during compilation: sandbox denied /usr/include/stdc-predef.h.
  • go test -race ./... -> blocked during compilation: sandbox denied /usr/include/stdc-predef.h.
  • env CGO_ENABLED=0 go build -o /tmp/gmr-5d36d931/stagehand-r4-n6ger9kv/stagehand ./cmd/stagehand -> passed.
  • /tmp/gmr-5d36d931/stagehand-r4-n6ger9kv/stagehand version -> passed; output: stagehand dev.
  • env CGO_ENABLED=0 go vet ./... -> passed.
  • env CGO_ENABLED=0 go test ./... -> passed.
  • env CGO_ENABLED=0 go test ./internal/engine ./internal/cli -run 'Test(PurgeRetryFinishesAnInterruptedDeletion|DueScheduleDoesNotStarveQueuedEvents|SweepOpensCandidatesWritable|SweepSparesAnAgedPayloadWhoseIntakeHasNotCommitted|PurgeAndResumeAreMutuallyExclusive|InterruptedDeletionIsFinished|DroppedEventNeverRunsEvenIfItsFileSurvives|ScheduleAndEventsCoexist|TriggerWaitReturnsWhenItsOwnRunIsDone)$' -count=10 -v -> passed.
  • python3 /tmp/gmr-5d36d931/stagehand-r4-n6ger9kv/cli_review.py -> completed; reproduced due-schedule stall, verified ordinary bounded waiting, purge retry, and failed-drop payload preservation.
  • env CGO_ENABLED=0 go run -mod=mod . (cwd /tmp/gmr-5d36d931/stagehand-r4-n6ger9kv) -> completed; verified bidirectional fairness with real shell execution and reproduced dispatch despite an already-true Done predicate.
  • python3 /tmp/gmr-5d36d931/stagehand-r4-n6ger9kv/wait_handoff.py -> completed; reproduced trigger --wait remaining blocked on a newer event after its target succeeded.
  • python3 /tmp/gmr-5d36d931/stagehand-r4-n6ger9kv/sweep_trace.py -> passed; production sweep opened O_RDWR, preserved a locked orphan, and removed it after unlocking.
  • CI-verified: go test -race -count=1 ./... -> success; run 42155861385 job 127689704653

Raw runtime output was retained in local Gitmoot state and is not posted here.

@jerryfane

Copy link
Copy Markdown
Contributor Author

Fixes for the round-4 review (job local-review-review-router-18e5569c17a2d944-1, head 8bc20da). Two P2s, fixed in eb8b0986d4d674af7cc15eb2419164c1c9519daa:

Finding Fix Test
An events-only loop (trigger --wait with no serve running) yielded the event to a due schedule that it never dispatches, so it stalled startEvents(ctx, only, yieldToSchedule): it yields only when the loop also runs schedules (opts.Schedule) TestEventsOnlyLoopDoesNotWaitForASchedule: proven (mutant: always yield)
Lock handoff: the target run settled before the lock was taken, and Loop then started a newer event before it checked Done Loop checks Done at the top of each pass, before schedule/startEvents, and returns when it is true and nothing is in flight TestLoopWithDoneAlreadyTrueStartsNothing: proven (mutant: Done checked only after dispatch)

The full race suite passes. Evidence: mutants-events-r4.json. Head: eb8b0986d4d674af7cc15eb2419164c1c9519daa.

@jerryfane

Copy link
Copy Markdown
Contributor Author

Agent: g7-review
Runtime: omp
Job: local-review-review-router-18e5577936a4ceb5-1

Decision: approved
Summary: Reviewed exact head eb8b098 against prior head 8bc20da. Both outstanding P2 scenarios are resolved. Real CLI execution verified events-only progress, lock-handoff completion without starting newer work, serve age arbitration, cancellation, and crash recovery. No new P0–P2 defects found in the scoped delta. All prior findings were checked at this head. Build, vet, full tests, and repeated focused regressions passed with CGO_ENABLED=0; the initial default-CGO build encountered an unreadable system header. No repository files were changed.

Findings

  • Resolved: events-only loops do not yield to undispatchable schedules (P2)
    • continues_uid: feat: start pipelines from external events — signed trigger, CLI, idempotency, bounded queue, retention (#18) #19-f8
    • detail: Schedule arbitration now requires yieldToSchedule, supplied from opts.Schedule. A standalone trigger --wait completed an event newer than its pipeline's due hourly schedule, creating only a succeeded event run. The regression test passed ten repetitions.
    • evidence: /tmp/gmr-1dad0564/stagehand-review-eb8b098-c24t1vkf/smoke.py: EVENTS_ONLY_DUE_SCHEDULE_PASS; internal/engine/events_test.go:587-604, TestEventsOnlyLoopDoesNotWaitForASchedule.
    • file: internal/engine/events.go
    • line_end: 188
    • line_start: 185
    • state: answered
  • Resolved: completion is checked before dispatch after lock handoff (P2)
    • continues_uid: feat: start pipelines from external events — signed trigger, CLI, idempotency, bounded queue, retention (#18) #19-f6
    • detail: Delayed the duplicate waiter's actual flock syscall while another CLI process completed the target and released the home lock. The waiter acquired the lock and returned successfully; the newer event remained queued and its shell stage never started. Recover remains before Loop in drainEvent. Separate real-process checks verified interrupted-attempt recovery and operator cancellation; an external overlay test verified that Done=true cannot bypass cancellation while a stage remains in flight.
    • evidence: /tmp/gmr-1dad0564/stagehand-review-eb8b098-c24t1vkf/waiter.strace:5-8; smoke.py: LOCK_HANDOFF_PASS; recovery.stderr:1-4; cancel-waiter.stderr:4-6; done_cancel_test.go, TestReviewDoneTrueCannotSkipInflightCancellation passed ten repetitions.
    • file: internal/engine/engine.go
    • line_end: 384
    • line_start: 381
    • state: answered
  • Resolved: aged uncommitted payloads remain protected (P1)
    • continues_uid: feat: start pipelines from external events — signed trigger, CLI, idempotency, bounded queue, retention (#18) #19-f1
    • detail: The intake lock still spans payload placement and event commit, and sweeping still locks before querying committed references. The aged-uncommitted-intake regression passed ten repetitions at this head.
    • evidence: internal/engine/events_test.go:446-491, TestSweepSparesAnAgedPayloadWhoseIntakeHasNotCommitted; executed focused regression command recorded in tests_run.
    • file: internal/engine/events.go
    • line_end: 145
    • line_start: 103
    • state: answered
  • Resolved: deletion claims continue to exclude resume (P1)
  • Resolved: drop commits non-runnable state before unlinking (P2)
  • Resolved: purge retries finish claimed deletions (P2)
  • Resolved: scheduling loops retain age arbitration (P2)
    • continues_uid: feat: start pipelines from external events — signed trigger, CLI, idempotency, bounded queue, retention (#18) #19-f5
    • detail: Schedule-enabled loops still pass true to startEvents. The actual serve command executed the older due schedule before its queued event. Schedule/event coexistence and repeatedly-overdue-schedule starvation regressions passed ten repetitions.
    • evidence: /tmp/gmr-1dad0564/stagehand-review-eb8b098-c24t1vkf/smoke.py: SERVE_AGE_ARBITRATION_PASS; internal/engine/events_test.go:232-250 and 519-548.
    • file: internal/engine/events.go
    • line_end: 188
    • line_start: 185
    • state: answered
  • Resolved: sweep locks retain writable descriptors (P2)
    • continues_uid: feat: start pipelines from external events — signed trigger, CLI, idempotency, bounded queue, retention (#18) #19-f7
    • detail: The writable-open regression passed ten repetitions. Tracing production tick confirmed O_RDWR for the orphan payload; a locked orphan survived and was removed after unlocking. Verification used the local filesystem, not an NFS mount.
    • evidence: /tmp/gmr-1dad0564/stagehand-review-eb8b098-c24t1vkf/sweep.strace; smoke.py: SWEEP_WRITABLE_LOCK_PASS; internal/engine/events_test.go:562-582.
    • file: internal/engine/events.go
    • line_end: 319
    • line_start: 305
    • state: answered

Tests Run

  • git rev-parse HEAD -> eb8b098; git diff 8bc20da eb8b098 -- internal/cli/events.go internal/engine/engine.go internal/engine/events.go internal/engine/events_test.go -> inspected.
  • go version -> go1.27.1 linux/amd64; go build -o /tmp/stagehand-review-eb8b098 ./cmd/stagehand -> failed in runtime/cgo because /usr/include/stdc-predef.h was unreadable. Retrying with CGO_ENABLED=0 compiled but could not write that output path; subsequent checks used writable scratch storage.
  • env CGO_ENABLED=0 go build -o /tmp/gmr-1dad0564/stagehand-review-eb8b098-c24t1vkf/stagehand ./cmd/stagehand -> passed.
  • env CGO_ENABLED=0 go build ./... -> passed.
  • env CGO_ENABLED=0 go vet ./... -> passed.
  • env CGO_ENABLED=0 go test ./... -> passed across all packages.
  • env CGO_ENABLED=0 go test ./internal/engine ./internal/cli -run 'Test(EventsOnlyLoopDoesNotWaitForASchedule|LoopWithDoneAlreadyTrueStartsNothing|ScheduleAndEventsCoexist|DueScheduleDoesNotStarveQueuedEvents|SweepSparesAnAgedPayloadWhoseIntakeHasNotCommitted|PurgeAndResumeAreMutuallyExclusive|InterruptedDeletionIsFinished|PurgeRetryFinishesAnInterruptedDeletion|DroppedEventNeverRunsEvenIfItsFileSurvives|SweepOpensCandidatesWritable|CancelStopsRunningStagesAndPendingOnes|StoppingRunnerParksRunAsInterrupted|RecoverParksCrashedAttemptAndCleansItsResource|TriggerWaitReturnsWhenItsOwnRunIsDone)$' -count=10 -> passed; engine 49.779s, cli 18.958s.
  • python3 /tmp/gmr-1dad0564/stagehand-review-eb8b098-c24t1vkf/smoke.py -> passed: EVENTS_ONLY_DUE_SCHEDULE_PASS, LOCK_HANDOFF_PASS, SERVE_AGE_ARBITRATION_PASS, EVENT_WAITER_CANCEL_PASS, SWEEP_WRITABLE_LOCK_PASS.
  • python3 /tmp/gmr-1dad0564/stagehand-review-eb8b098-c24t1vkf/recovery.py -> passed: EVENT_WAITER_RECOVER_PASS; killed executor's attempt became interrupted, its run failed as interrupted, and tracked process resources were cleaned.
  • env CGO_ENABLED=0 go test -overlay /tmp/gmr-1dad0564/stagehand-review-eb8b098-c24t1vkf/overlay.json ./internal/engine -run '^TestReviewDoneTrueCannotSkipInflightCancellation$' -count=10 -> passed; external scratch overlay only, no repository modification.
    Diagnostics: workflow advance failed: review approval held: approval is bound to head eb8b098 but the observed pull request head is 8bc20da; holding until the observed head catches up, because an approval at a genuinely superseded head must not clear changes_requested

Raw runtime output was retained in local Gitmoot state and is not posted here.

@jerryfane
jerryfane merged commit d5b9056 into main Nov 6, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Start pipelines from external events (HTTP trigger + CLI), with auth, idempotency and queueing

1 participant