Repository navigation
feat: start pipelines from external events — signed trigger, CLI, idempotency, bounded queue, retention (#18) - #19
Conversation
… trigger, idempotency, bounded queue, payload retention (#18)
Decision: |
Decision: Findings
Tests Run
Raw runtime output was retained in local Gitmoot state and is not posted here. |
…er sweep a fresh payload (#19 review)
|
Fixes for the review of
|
Decision: Findings
Tests Run
Needs
Raw runtime output was retained in local Gitmoot state and is not posted here. |
… lock until commit (#19 review round 2)
|
Fix for the round-2 review (job The 15-minute mtime grace narrowed the race but did not close it: a rename keeps the old mtime. The sweep now synchronizes with intakes through a file lock:
Regression test
The full race suite passes. Head: |
Decision: Findings
Tests Run
Raw runtime output was retained in local Gitmoot state and is not posted here. |
…ration, scoped trigger --wait, NFS-safe sweep lock (#19 review round 3)
|
Fixes for the round-3 review (job
README updated for the fairness rule and the scope of |
Decision: Findings
Tests Run
Raw runtime output was retained in local Gitmoot state and is not posted here. |
…efore dispatching (#19 review round 4)
|
Fixes for the round-4 review (job
The full race suite passes. Evidence: |
Decision: Findings
Tests Run
Raw runtime output was retained in local Gitmoot state and is not posted here. |
Closes #18. Part of epic #2. Requested by the-path on Jerry's instruction (gitmoot message 402990); the owner confirmed it in the gitmoot-pipe session on 6 Nov and set payload retention (delete when the run succeeds) and the size limit (64 MB).
What this adds
A pipeline can be started by an external event (a mail poller, a webhook receiver, a script). The event's payload becomes the run's input.
events: {max_queued: N}in the spec. Schedule, events, both or neither.serve --trigger-addr ADDR:POST /v1/pipelines/{name}/events. Loopback only unless--trigger-allow-remote; off by default.timestamp\npipeline\nevent_id\nbodywith the 0600events.secret_file(re-read per request, so it can be rotated); a timestamp within ±5 min. Unsigned, wrong, stale or mismatched requests get 401 and store nothing.stagehand trigger <name> [--event-id] [--payload-file|stdin] [--content-type] [--wait] [--json]. Local and trusted.X-Stagehand-Event-Id/--event-id: the same id for the same pipeline gives one event and one run, enforced in two places (an early lookup, and inside the insert transaction). A repeat gets 200duplicate:truewith the original event.max_queued: 429 withRetry-After, and nothing stored.STAGEHAND_EVENT_FILE, plusSTAGEHAND_TRIGGER,STAGEHAND_EVENT_IDandSTAGEHAND_EVENT_CONTENT_TYPE(a sanitised media type). Nothing from the payload goes into a command line or an environment value.trigger: eventand aneventreference (id, key, source, size, SHA-256, deleted-at; never the payload).statusgainsqueued_eventsandevents. Neweventsanddrop-eventcommands.purge <run-id>deletes a parked run's files; resume is then refused. Orphaned intake files are swept.events.max_payload_bytes, default 64 MiB → 413.eventstable, plusruns.event_idandruns.files_deleted_at.Tests
New tests:
Proven red without the fix.
test-check provecannot buildmainagainst these tests (it lacks the feature), so I reverted each guarantee in isolation and ran its test against the mutant:TestSameEventIDYieldsOneRunand the HTTP duplicate test red.TestPayloadDeletedAtTheMomentTheRunSucceeds:test-check proveagainst362aa25(before the immediate deletion) gives proven.Results:
/root/fleet-tools/state/gitmoot-pipe/evidence/mutants-events.json(fleet host).Smoke through the real binary (6 Nov 03:14–03:17 CET, temp home)
serve --trigger-addr 127.0.0.1:43398with a real HMAC key, andcurlsigned exactly as in the README recipe (openssl dgst -hmac):message/rfc822mail whose subject contains$(touch …) \id` ; rm -rf / |&& "'→ 202; the duplicate → 200duplicate:true`;bad signature; 10 minutes stale → 401timestamp outside the allowed window; unsigned → 401.max_queued: 2with one running → 202, 202, 429, 429, 429.trigger=event, all succeeded, in arrival order. A CLI trigger while serve was running queued for serve. The payload was delivered byte-exact (cmp), and the canary file the subject tried to create does not exist.trigger --waitreturned, withpayload_deleted_atset;resumeit succeeded and the payload was deleted;purge: files deleted, andresumerefused ("its files and payload were deleted …").drop-eventon a queued event deleted its payload, and the event showsdropped;statusshowsqueued_eventsandevents.Not in this PR
Any mail or WhatsApp client: receivers stay outside and call the trigger. Deployment of
--trigger-addron the fleet host happens after merge.