fix(plugin): make a Laminar trace red iff the turn failed - #144
Merged
Conversation
Two defects made trace status uninformative for v4 runs. A tool that throws is normal agent flow - the model reads the `tool-error` result and adapts - but the AI SDK stamps ERROR on the `ai.toolCall` span and Laminar reds a whole trace if any of its spans is ERROR. One buggy `browser_execute` snippet therefore reported the entire run as a failure: 1,058 errored `browser_execute` spans in two days on prod. Demote those spans to UNSET, keeping the message on `bcode.tool.error` and the `exception` event intact. Conversely nothing ever marked a genuinely failed turn. The AI SDK ends `ai.streamText` / `ai.streamText.doStream` inside a transform `flush` that never runs when the consumer aborts, so a provider error arriving mid-stream drops those spans instead of marking them - the trace showed a clean, shorter run. Mark the turn span ERROR on `session.error`, which `SessionProcessor.halt` publishes just before the session goes idle. Aborts are excluded: stopping a run is not a failure.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Laminar trace status for v4 runs is uninformative in both directions.
Green when the run failed. Staging run
fb5dad0c(session2cbca552, ENG-5640) failed on its 6th LLM step when the provider stream went silent mid-body and the control plane sent a terminalevent: errorframe. Laminar tracedc9b5851reportsstatus: success, 0 errored spans — and the failingsession.llm/ai.streamText.doStreamspans are absent entirely: theturnspan ends 00:22:34.59 while its last child ends 00:21:02.13.The AI SDK opens both spans with
endWhenDone: falseand ends them inside a transformflush(ai/dist/index.mjs:6834,:7494). An error arriving as a stream chunk becomesEffect.fail(event.error)insession/llm/ai-sdk.ts:264, the consumer stops pulling,flushnever runs, and the spans are never ended or exported. Nothing else in the pipeline sets ERROR on anything, so the trace reads as a clean, shorter run.Red when the run succeeded.
Tool.wrapusesEffect.orDie(tool/tool.ts:145), so a tool that throws rejectsexecute(), and the AI SDK callsrecordErrorOnSpanon theai.toolCallspan (ai/dist/index.mjs:2845). Laminar sets a trace toerrorif any of its spans iserror, so one buggy LLM-generatedbrowser_executesnippet reports the whole run as a failure. That is not an edge case — prod, 2 days:A typical one is the model writing
sizeElementswhere it definedsizeButtons— the agent reads thetool-error, fixes the snippet, and finishes the task successfully.Change
Establish one invariant: a trace is red iff the turn failed.
processor.ts— demote ERROR to UNSET onai.toolCallspans inonEnd, preserving the message on abcode.tool.errorattribute and leaving the recordedexceptionevent untouched. Mutated in place becausesetStatus/setAttributeare no-ops once a span has ended.plugin.ts— mark the turn span ERROR onsession.error.SessionProcessor.halt(session/processor.ts:679,693) publishes it immediately beforestatus.set(idle), so the span is still open; the existingsession.idlecase ends it.MessageAbortedErroris excluded — stopping a run is not a failure. The recoverableContextOverflowErrorpath deliberately publishes nothing, so auto-compaction retries stay green.Verification
Ran the real OTel SDK +
InMemorySpanExporterthrough this processor with anai.toolCallspan carrying an ERROR status and a recorded exception, plus a turn span failed via thesession.errorpath:bun typecheckclean inpackages/bcode-laminar.Not addressed
The dropped LLM spans themselves. When a provider error arrives mid-stream, the failing call's tokens, cost, latency and
msToFirstChunkare lost because the AI SDK never ends the span — that needs an upstream fix or a patch toai. This PR makes the failure visible on the turn span; it does not recover the measurements.Summary by cubic
Make Laminar traces go red only when the turn fails, and stay green when tools error as part of normal recovery. Fixes misleading status in v4 runs and aligns with ENG-5640.
ai.toolCallspans to UNSET; preserve the message inbcode.tool.errorand keep the recordedexceptionevent.session.errorwhile it’s still open; ignoreMessageAbortedError, and keep auto-compaction (ContextOverflowError) retries green.Written for commit 22a7286. Summary will update on new commits.