Environment: braintrust@3.25.0, Flue adapter (FlueInstrumentation), Cloudflare Workers Durable Object (128 MB isolate limit).
Problem
handleTurnRequest logs the turn's full input — the entire conversation so far — as the LLM span input on every turn start (event: { input: input.messages } plus logOperationInput(...)). Span.log/startSpan run each event through deepCopyEvent and push the copy into the _HTTPBackgroundLogger queue. Masking and size truncation are applied at flush time, after the deep copies already exist.
For an agentic loop of N turns, the queue holds N copies of growing conversation prefixes between flushes — O(N²) retained memory. With long reasoning outputs, our production agent exceeded the Durable Object isolate memory limit and crash-looped. Heap snapshots from a local repro attributed ~79% of retained memory to the background-logger queue; successive runs grew retention by ~80–130 MB each until the isolate died.
Asks (in preference order)
- Apply masking/size bounding at enqueue time (before or during
deepCopyEvent), not only at flush.
- For the Flue turn spans, log per-turn message deltas (or a bounded recent window) instead of the full conversation prefix each turn.
- A configurable cap on background-logger queue bytes with explicit truncation markers.
Workaround we shipped
We bound every observation event before handing it to the adapter — a per-string cap plus a per-event budget that keeps the most recent messages intact and elides older message content. That converges retention (+77 → +31 → +1.5 MB across successive runs) but every adapter consumer on memory-limited runtimes will hit this by default.
Environment:
braintrust@3.25.0, Flue adapter (FlueInstrumentation), Cloudflare Workers Durable Object (128 MB isolate limit).Problem
handleTurnRequestlogs the turn's full input — the entire conversation so far — as the LLM span input on every turn start (event: { input: input.messages }pluslogOperationInput(...)).Span.log/startSpanrun each event throughdeepCopyEventand push the copy into the_HTTPBackgroundLoggerqueue. Masking and size truncation are applied at flush time, after the deep copies already exist.For an agentic loop of N turns, the queue holds N copies of growing conversation prefixes between flushes — O(N²) retained memory. With long reasoning outputs, our production agent exceeded the Durable Object isolate memory limit and crash-looped. Heap snapshots from a local repro attributed ~79% of retained memory to the background-logger queue; successive runs grew retention by ~80–130 MB each until the isolate died.
Asks (in preference order)
deepCopyEvent), not only at flush.Workaround we shipped
We bound every observation event before handing it to the adapter — a per-string cap plus a per-event budget that keeps the most recent messages intact and elides older message content. That converges retention (+77 → +31 → +1.5 MB across successive runs) but every adapter consumer on memory-limited runtimes will hit this by default.