Skip to content

Start pipelines from external events (HTTP trigger + CLI), with auth, idempotency and queueing #18

Closed

Description

@jerryfane

Request: from the-path, on Jerry's instruction (gitmoot message 402990, 6 Nov). Confirmed by the owner in the gitmoot-pipe session on 6 Nov, together with the decisions below. Part of epic #2. Owner: gitmoot-pipe.

Why

Jerry's "AI Reviewer": a pitch deck arrives by email, gets scored, and feedback is sent; a CMO chats on WhatsApp. The planned shape is one stagehand pipeline run per incoming event. A schedule can only poll; the pipeline must start the moment something happens, with that event as its input.

Receivers (mail poller, webhook) live outside stagehand and only call the trigger. No email or WhatsApp code goes into stagehand.

Owner decisions (6 Nov)

  • Build it.
  • Payload retention: a payload (and the files of its run) is kept while the event is queued or its run is running or parked (so it can be resumed), and deleted as soon as the run succeeds.
  • Payload size limit: 64 MB by default; a host setting.

Design

Opt-in per pipeline

events:
  max_queued: 20   # its presence enables event triggers; at most this many events wait

A pipeline may have a schedule, events, both, or neither (manual only).

Triggering

  • HTTP: POST /v1/pipelines/{name}/events on serve --trigger-addr (loopback only unless --trigger-allow-remote; off by default). The body is the payload.
  • CLI: stagehand trigger <pipeline> [--event-id ID] [--payload-file F | stdin] [--content-type T] [--wait] [--json]. Local and trusted: it uses the same queue as HTTP, with no HTTP auth.

Authentication (HTTP): HMAC-SHA256 with a 0600 host secret file (events.secret_file).

  • Headers: X-Stagehand-Timestamp (unix seconds, within ±5 min) and X-Stagehand-Signature: sha256=<hex> over timestamp \n pipeline \n event_id \n body.
  • Anything unsigned or with a bad signature gets 401.

Idempotency: the caller sends X-Stagehand-Event-Id ([A-Za-z0-9._:-]{1,128}). The same id for the same pipeline gives one event and one run. A repeat returns 200 with duplicate: true, plus the original event's state and run. Without an id, every call is a new event.

Queue and concurrency

  • Accepted events are stored durably: a database row plus a 0600 payload file.
  • They start in arrival order. A pipeline still has at most one running run (existing rule), and agent stages still share the sandbox one at a time (Agent stages of different pipelines collide on the single sandboxd host port #13).
  • Over max_queued, the trigger is refused with 429, and nothing is stored. The caller knows to retry, so accepted events are never lost and a burst can't start unbounded runs.
  • Schedule and events share the one-run-at-a-time rule.
  • enable/disable now covers both: a disabled pipeline refuses events (409), and its queued events wait until it is enabled again.

The payload reaches stages as a file, never as text in a command

  • Shell stages get STAGEHAND_TRIGGER (schedule, event, manual), STAGEHAND_EVENT_ID, STAGEHAND_EVENT_FILE (path to the payload) and STAGEHAND_EVENT_CONTENT_TYPE (a sanitized media type).
  • stagehand never interpolates anything into cmd.
  • Agent stages get the payload only if a shell stage copies it into $STAGEHAND_WORK and lists it in inputs.

Records: runs show trigger event, plus the event's id, size and SHA-256, and whether its files were deleted. status gains queued_events per pipeline. Resume works as for other runs, until the files are deleted.

Retention

  • When an event run succeeds, its payload and run directory (work, logs, results) are deleted. The database records stay: states, exit codes, cost, summaries.
  • stagehand purge <run-id> deletes the files of a parked event run that won't be resumed; resume is then refused.
  • Orphaned payload files (from a crash during intake) are swept.

Size limit: events.max_payload_bytes in the host config, default 64 MiB. Larger payloads get 413.

Acceptance

  • Tests that tell old behaviour from new:
    • duplicate event id gives one run;
    • a burst beyond max_queued gives 429, and every accepted event runs exactly once, one at a time;
    • bad, missing or stale signatures get 401;
    • a payload full of shell metacharacters is delivered byte-exact and runs nothing;
    • retention deletes on success and keeps on failure.
  • Docs.
  • A smoke test through the real binary: serve --trigger-addr, then signed curls, a duplicate, a burst, and a CLI trigger.

Activity

  1. jerryfane commented on Nov 6, 2026

    @jerryfane
    ContributorAuthor

    Shipped in #19 (approved at exact head eb8b098 by g7-review, job local-review-review-router-18e5577936a4ceb5-1).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions