Skip to content

feat(events): deduplicate emits with a dedupe_key - #77

Merged
psteinroe merged 1 commit into
mainfrom
feat/emit-dedupe
Oct 13, 2026
Merged

psteinroe merged 1 commit into
mainfrom
feat/emit-dedupe

Conversation

@psteinroe

@psteinroe psteinroe commented Oct 2, 2026 •

Copy link
Copy Markdown
Owner

Webhook providers redeliver, so ingress may emit the same delivery twice. conductor.emit and ctx.emit now take a dedupe_key option, named like the one on conductor.invoke. Repeating a key for the same event name stores nothing and returns the original event ID. emit() always returns the event ID; the dedupe_key is the caller's own key and is never returned.

const eventId = await conductor.emit("slack.mentioned", payload, { dedupe_key: `slack:${deliveryId}` });
await ctx.emit("x", payload, { dedupe_key });

SQL

pgconductor.emit_event gains a trailing p_dedupe_key text default null, so existing plain-SQL callers are unaffected. The two mechanisms stay separate:

  • the client-generated p_id uuid from fix(events): make retried emits idempotent #63 makes a retry after a lost response idempotent;
  • p_dedupe_key is the caller's key. The dispatch execution stores '<event name>:<dedupe_key>' as its dedupe_key, and the existing unique (task_key, dedupe_key, queue) constraint enforces it.

The insert now uses an untargeted on conflict do nothing, which covers both. It then returns the stored row's ID. This is insert-or-return-existing, not invoke's dedupe semantics: the original payload is kept. Concurrent emits with the same key serialize on the unique index. The loser waits for the winner to commit, then returns the winner's ID.

Retention window

The internal pgconductor.event-dispatch task changes from removeOnComplete: true to { days: 1 }, in both the task definition and the _private_tasks seed row. Completed dispatch rows now stay for a day, and the daily maintenance run removes them. Every worker, including the internal pgconductor.internal one, registers a maintenance cron, so keys are remembered for 1–2 days. The tests check that maintenance is scheduled on the internal queue.

The scope (per event name) and the window are documented in triggers.md ("Deduplicating Emits"), api/conductor.md, a new ctx.emit section in api/task-context.md, and next to invoke in deduplication.md.

Verification

New integration tests in event-pipeline.test.ts:

  • The same key twice returns the first event ID and stores one dispatch. A plain-SQL p_dedupe_key caller gets the same ID. The same key under another event name emits a second event.
  • Concurrent emits from two transactions with the same key: the second blocks, then returns the first ID, and there is one dispatch.
  • A repeated conductor.emit key after dispatch triggers the listener once and resumes the waiter once. The waiter's second wait times out instead of seeing the repeat.
  • ctx.emit with a key across a failed and retried attempt stores one event.
  • The key is still remembered 23h after dispatch. After maintenance cleanup at 25h, the same key emits a new event.

Three existing tests asserted that the dispatch row is deleted on completion. They now assert it is completed. just lint, bun run typecheck and the full bun test pass.

Follow-ups

  • feat(executions): typed, propagated execution metadata #72 (execution metadata) also adds an options object to emit. Whichever lands second merges { dedupe_key, metadata } into one options type.
  • A permanently failed dispatch keeps its key until retention removes it, and the dispatch task has no removeOnFail. Out of scope here.
  • Very long keys (above roughly 2.5 KB with the event name) would exceed the btree entry size of the dedupe_key index. Not bounded here.

@psteinroe psteinroe changed the title feat(events): idempotent emit with a caller-supplied id feat(events): deduplicate emits with a dedupe_key Oct 2, 2026
Comment thread migrations/0000000001_setup.sql
@psteinroe
psteinroe merged commit acafb2f into main Oct 13, 2026
9 checks passed
@psteinroe
psteinroe deleted the feat/emit-dedupe branch October 13, 2026 12:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant