Skip to content

fix(worker): retry failed result flushes and finish in-flight flushes on stop - #58

Merged
psteinroe merged 2 commits into
mainfrom
fix/flush-retry
Oct 2, 2026
Merged

psteinroe merged 2 commits into
mainfrom
fix/flush-retry

Conversation

@psteinroe

@psteinroe psteinroe commented Oct 2, 2026 •

Copy link
Copy Markdown
Owner

What

Two fixes to the result flush loop in Worker.flushResults:

  • A failed flush is retried. The flush timer cleared flushTimer before it ran. When the flush failed, the batch went back into the buffer, but nothing rescheduled it. The completed execution stayed locked and uncompleted until another result arrived or the worker shut down, so on an idle worker it never completed. The catch block now reschedules the flush after restoring the batch. Graphile Worker #636 had the same bug.
  • stop() waits for an in-flight timer flush. A timer flush that was already running had taken the buffer, so the final flush saw an empty buffer and returned. The worker reported itself stopped, the orchestrator cleanup released the execution, and the delayed settle then found nothing to update. The execution ended up unlocked and uncompleted with attempts = 1. The worker now keeps the timer flush's promise and awaits it before the final flush.

Verification

  • Two new integration tests in tests/integration/basic-execution.test.ts. Both failed before the fix and pass now:
    • retries a failed flush without new results: the first returnExecutions call fails and the execution must still complete with no new results arriving.
    • stop() waits for an in-flight flush: stop() is called while a slow flush is running, and the execution must end up completed and unlocked.
  • bun test (337 pass), bun run typecheck, just lint, just format.

Follow-ups

  • If the final flush during shutdown fails, the results are still dropped (unchanged behaviour). The executions get released and retried.
  • A flush that keeps failing is retried every flushIntervalMs with no backoff.
  • Because a failed flush is now retried, a worker that a test leaks after destroying its database keeps logging flush errors and emitting settle spans every flushIntervalMs. On CI this happened after a flaky event-trigger test (fixed 300ms sleep, orchestrator never stopped) and broke the telemetry span assertions in later files. This PR replaces those sleeps in event-triggers.test.ts with waitForCondition. Tests that don't stop their orchestrator in finally can still leak workers this way.

@psteinroe psteinroe added the ready label Oct 2, 2026
@psteinroe
psteinroe force-pushed the fix/flush-retry branch 5 times, most recently from 17b6a70 to 9bc8d9e Compare October 2, 2026 17:20
@psteinroe
psteinroe merged commit becd754 into main Oct 2, 2026
9 checks passed
@psteinroe
psteinroe deleted the fix/flush-retry branch October 2, 2026 17:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant