Repository navigation
fix(worker): retry failed result flushes and finish in-flight flushes on stop - #58
Merged
Merged
Conversation
psteinroe
force-pushed
the
fix/flush-retry
branch
5 times, most recently
from
October 2, 2026 17:20
17b6a70 to
9bc8d9e
Compare
psteinroe
force-pushed
the
fix/flush-retry
branch
from
October 2, 2026 17:26
9bc8d9e to
b53b6a6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two fixes to the result flush loop in
Worker.flushResults:flushTimerbefore it ran. When the flush failed, the batch went back into the buffer, but nothing rescheduled it. The completed execution stayed locked and uncompleted until another result arrived or the worker shut down, so on an idle worker it never completed. The catch block now reschedules the flush after restoring the batch. Graphile Worker #636 had the same bug.stop()waits for an in-flight timer flush. A timer flush that was already running had taken the buffer, so the final flush saw an empty buffer and returned. The worker reported itself stopped, the orchestrator cleanup released the execution, and the delayed settle then found nothing to update. The execution ended up unlocked and uncompleted withattempts = 1. The worker now keeps the timer flush's promise and awaits it before the final flush.Verification
tests/integration/basic-execution.test.ts. Both failed before the fix and pass now:retries a failed flush without new results: the firstreturnExecutionscall fails and the execution must still complete with no new results arriving.stop() waits for an in-flight flush:stop()is called while a slow flush is running, and the execution must end up completed and unlocked.bun test(337 pass),bun run typecheck,just lint,just format.Follow-ups
flushIntervalMswith no backoff.flushIntervalMs. On CI this happened after a flaky event-trigger test (fixed 300ms sleep, orchestrator never stopped) and broke the telemetry span assertions in later files. This PR replaces those sleeps inevent-triggers.test.tswithwaitForCondition. Tests that don't stop their orchestrator infinallycan still leak workers this way.