Skip to content

fix(plugin): make a Laminar trace red iff the turn failed - #144

Merged
Alezander9 merged 1 commit into
mainfrom
laminar-trace-status
Aug 4, 2026
Merged

fix(plugin): make a Laminar trace red iff the turn failed#144
Alezander9 merged 1 commit into
mainfrom
laminar-trace-status

Conversation

@Alezander9

@Alezander9 Alezander9 commented Aug 4, 2026

Copy link
Copy Markdown
Member

Problem

Laminar trace status for v4 runs is uninformative in both directions.

Green when the run failed. Staging run fb5dad0c (session 2cbca552, ENG-5640) failed on its 6th LLM step when the provider stream went silent mid-body and the control plane sent a terminal event: error frame. Laminar trace dc9b5851 reports status: success, 0 errored spans — and the failing session.llm / ai.streamText.doStream spans are absent entirely: the turn span ends 00:22:34.59 while its last child ends 00:21:02.13.

The AI SDK opens both spans with endWhenDone: false and ends them inside a transform flush (ai/dist/index.mjs:6834, :7494). An error arriving as a stream chunk becomes Effect.fail(event.error) in session/llm/ai-sdk.ts:264, the consumer stops pulling, flush never runs, and the spans are never ended or exported. Nothing else in the pipeline sets ERROR on anything, so the trace reads as a clean, shorter run.

Red when the run succeeded. Tool.wrap uses Effect.orDie (tool/tool.ts:145), so a tool that throws rejects execute(), and the AI SDK calls recordErrorOnSpan on the ai.toolCall span (ai/dist/index.mjs:2845). Laminar sets a trace to error if any of its spans is error, so one buggy LLM-generated browser_execute snippet reports the whole run as a failure. That is not an edge case — prod, 2 days:

1058  browser_execute (TOOL)
  59  apply_patch (TOOL)
  50  webfetch (TOOL)  50  grep (TOOL)  45  read (TOOL)

A typical one is the model writing sizeElements where it defined sizeButtons — the agent reads the tool-error, fixes the snippet, and finishes the task successfully.

Change

Establish one invariant: a trace is red iff the turn failed.

  • processor.ts — demote ERROR to UNSET on ai.toolCall spans in onEnd, preserving the message on a bcode.tool.error attribute and leaving the recorded exception event untouched. Mutated in place because setStatus/setAttribute are no-ops once a span has ended.
  • plugin.ts — mark the turn span ERROR on session.error. SessionProcessor.halt (session/processor.ts:679,693) publishes it immediately before status.set(idle), so the span is still open; the existing session.idle case ends it. MessageAbortedError is excluded — stopping a run is not a failure. The recoverable ContextOverflowError path deliberately publishes nothing, so auto-compaction retries stay green.

Verification

Ran the real OTel SDK + InMemorySpanExporter through this processor with an ai.toolCall span carrying an ERROR status and a recorded exception, plus a turn span failed via the session.error path:

ai.toolCall status=UNSET |  | bcode.tool.error="browser_execute snippet threw: TypeError" | events=1
turn         status=ERROR | APIError: stream inactive >90s | bcode.tool.error=undefined       | events=0

bun typecheck clean in packages/bcode-laminar.

Not addressed

The dropped LLM spans themselves. When a provider error arrives mid-stream, the failing call's tokens, cost, latency and msToFirstChunk are lost because the AI SDK never ends the span — that needs an upstream fix or a patch to ai. This PR makes the failure visible on the turn span; it does not recover the measurements.


Summary by cubic

Make Laminar traces go red only when the turn fails, and stay green when tools error as part of normal recovery. Fixes misleading status in v4 runs and aligns with ENG-5640.

  • Bug Fixes
    • Demote ERROR on ai.toolCall spans to UNSET; preserve the message in bcode.tool.error and keep the recorded exception event.
    • Mark the turn span ERROR on session.error while it’s still open; ignore MessageAbortedError, and keep auto-compaction (ContextOverflowError) retries green.

Written for commit 22a7286. Summary will update on new commits.

Review in cubic

Two defects made trace status uninformative for v4 runs.

A tool that throws is normal agent flow - the model reads the `tool-error`
result and adapts - but the AI SDK stamps ERROR on the `ai.toolCall` span and
Laminar reds a whole trace if any of its spans is ERROR. One buggy
`browser_execute` snippet therefore reported the entire run as a failure:
1,058 errored `browser_execute` spans in two days on prod. Demote those spans
to UNSET, keeping the message on `bcode.tool.error` and the `exception` event
intact.

Conversely nothing ever marked a genuinely failed turn. The AI SDK ends
`ai.streamText` / `ai.streamText.doStream` inside a transform `flush` that
never runs when the consumer aborts, so a provider error arriving mid-stream
drops those spans instead of marking them - the trace showed a clean, shorter
run. Mark the turn span ERROR on `session.error`, which `SessionProcessor.halt`
publishes just before the session goes idle. Aborts are excluded: stopping a
run is not a failure.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 2 files

Re-trigger cubic

@Alezander9
Alezander9 merged commit 7ca085c into main Aug 4, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant