Fix chat SSE timeout on long-running Temporal-engine turns - #209
Merged
Conversation
stream() used to `await this.invoke(input)` before returning the iterable, so its own Promise didn't settle until the whole turn (up to 30 min) had already finished. server.ts's streaming handler does `const source = await this.graph.stream(...)` before starting `withHeartbeat(source, HEARTBEAT_MS)`, so that wrapper never got a source to race against until the turn was over -- a long-running turn's SSE connection had no guaranteed keep-alive byte cadence at all, and an idle-connection timeout upstream (ingress/ALB/browser) would cancel it (RST_STREAM) even though the AgentRun Job kept running and completed fine server-side. Moving the `await this.invoke(input)` inside the async generator body defers it until withHeartbeat's first `.next()` call, which is exactly when its race against HEARTBEAT_MS needs to start.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Root-causes the chat-UI timeout ("Timeout or cancellation waiting for update: stream terminated by RST_STREAM with error code: CANCEL") seen on a long-running agent turn, even though the underlying
agentrun-*Kubernetes Job kept running and completed successfully server-side.TemporalEngine.stream()(apps/agent-orchestrator/src/engine/temporal-engine.ts) didawait this.invoke(input)before returning its iterable.server.ts's streaming handler does:withHeartbeatexists specifically to emit an SSE keep-alive comment whenever the source stalls for longer thanHEARTBEAT_MS(15s) — but it can't race against a source it doesn't have yet. Becausestream()'s own Promise didn't settle until the entire turn (up to the 30-minute poll timeout) had already finished, the heartbeat wrapper never engaged for the whole duration of a Temporal-engine turn. The only bytes reaching the client during that window were opportunistic progress-narration writes (added in #208) — not a guaranteed cadence — so any lull (job scheduling/image pull, a long tool call, or simply the seconds before the first narration line) longer than an upstream idle-connection timeout got the stream cancelled.Fix
Move the
await this.invoke(input)inside the returned async generator's body instead of before thereturn. Async generator bodies don't run until the first.next()call, sostream()now resolves immediately, lettingwithHeartbeatstart racing the real turn duration against its keep-alive timer from the start — exactly like it already does for the LangGraph engine.No behavior change for a non-streaming caller (
invoke()is untouched) or for what gets rendered — still a single terminal update once the turn settles.Test plan
temporal-engine.test.ts) assertingstream()resolves promptly even while the underlying turn is still "hanging", and that iterating the returned source still drains through to the terminal result once the turn completes.npx vitest run src/engine/temporal-engine.test.ts— 12/12 passing.apps/agent-orchestratorsuite run for regressions — the 2 failing files (nats-agent-channel*.test.ts) and the pre-existingtsc --noEmiterrors innats-agent-channel.tsreproduce identically onmainwithout this change (unrelated, stale workspace-package typing issue).