Worker: don't hang when fetch:plan replies with an error - #1521
Open
stuartc wants to merge 1 commit into
Open
Conversation
joinRunChannel awaited sendEvent(GET_PLAN) inside an unguarded async callback, so a rejection (now possible since Lightning can reply an error to fetch:plan for adaptor resolution failures) became an unhandled rejection instead of settling the outer promise. The worker's local capacity slot for that run leaked until restart. Wrap the await in try/catch and reject, mirroring the existing error/timeout branches in the same file.
elias-ba
approved these changes
Sep 2, 2026
elias-ba
left a comment
elias-ba
left a comment
Collaborator
There was a problem hiding this comment.
@josephjclark I looked at this and it looks good to me. Do you wanna have a look too before we merge ? @stuartc and the connections team will need it in prep for their work on the new adaptor registry.
Collaborator
|
Taking a look |
josephjclark
approved these changes
Sep 2, 2026
josephjclark
left a comment
josephjclark
left a comment
Collaborator
There was a problem hiding this comment.
Yep looks good. Do you want this released now?
Member
Author
Ah thanks for the offer, can go out on the next one - or I'll shout. The code that would hit this is still in a branch. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Short Description
Fixes a bug where the worker would hang forever (and quietly leak a capacity slot) if Lightning rejected a
fetch:planrequest after a run had already been claimed.Implementation Details
Lightning is going to start being able to reply with an error to
fetch:plan(e.g. when it can't resolve the job's adaptor —adaptor_not_ready,adaptor_not_found, etc).Found a loose unresolved promise when the run channel is joined and
sendEventrejects when fetching the plan. Turns out we were leaving it stuck (not catching the rejection).What happens is that
joinRunChanneljust hung forever, the upstream try/catch inserver.ts(which does the clean up) doesn't get the exception, and the worker's slot for that run was occupied forever. The lost runs janitor does mark it as lost, so it's not completely "silent" but it doesn't look like we'd get the slot back.So wrapping the call in a try/catch makes a failed call do the same as a failed or timed out join on the run channel.
QA Notes
Theres a test that forces
fetch:planto error and confirmsjoinRunChannelrejects instead of hanging.AI Usage