Retain large completed event outputs in sidecars

[ymichael]

Human comments

What was wrong

Completed command and tool outputs were stored inline in events.data, so every ordinary timeline/event-row scan had to read and transform large values even though the normal product surface only renders a bounded preview. The existing delayed truncation sweep eventually reduced old rows, but new large completions still imposed their full materialization and main-event-loop cost throughout the retention window, while there was no separate short-lived source from which explicit raw/detail reads or event copies could recover the complete value.

What changed

• New large item/completed outputs for commandExecution.aggregatedOutput, toolCall.result, and webFetch/webSearch.resultText now store a 2 KiB head + 2 KiB tail preview in the ordinary event row and the full string in a one-row-per-event sidecar for seven days. The existing 32 KiB eligibility threshold and retention constants are shared with the current storage policy.
• A single insertion seam covers direct event inserts, daemon appends, and stored event appends. It only creates a sidecar after the event insert succeeds, so ignored sequence duplicates do not leave orphan data.
• Raw event reads, explicit uncapped timeline reads, and turn-summary details hydrate retained values. Detail hydration keeps the existing 4 MiB response guard. Fork/copy insertion hydrates the source and re-applies the original event timestamp, so copying is lossless while retained without extending expiry.
• Expiry selects by (expires_at, event_id) and deletes one sidecar per periodic retention tick. It never rewrites or deletes the ordinary event preview. Hydration uses bounded batches of 100 IDs and the event-ID primary key.
• RetainedEventOutputTarget and the authoritative four-entry RETAINED_EVENT_OUTPUT_TARGETS are exported from the local data module for the separately scoped legacy-migration work, without expanding the root @bb/db API.
• Drizzle generated 0113_graceful_mojo.sql and its snapshot from current origin/main; the migration only creates the sidecar and expiry index and does not migrate legacy inline values.

No host-daemon wire contract changed. This deliberately excludes legacy inline-output migration, destroyed-environment cleanup, watchers, outlines, timeline projection redesign, and background workers; all measured and production code remains synchronous on the Node main event loop.

How you verified

The focused regressions were run red before their implementation and green afterward: the storage integration initially found the full value in events.data; uncapped timeline, raw-event, and detail reads initially returned the retained preview; copying initially lost the full value; and the periodic sweep initially left the expired sidecar in place.

• pnpm exec turbo run test --filter=@bb/db --force — 32 files, 450 tests passed, including in-memory SQLite storage/copy/expiry integration, the direct migration preservation/FK/index test, historical migration rewinds, and query-plan assertions for hydration selection, expiry selection, and primary-key deletion.
• pnpm exec turbo run test --filter=@bb/server -- --run test/services/threads/timeline-in-turn-window.test.ts test/public/public-thread-timeline-output-preview.test.ts test/services/periodic-sweeps.test.ts — 3 files, 42 tests passed.
• pnpm exec turbo run typecheck --filter=@bb/db --filter=@bb/server — passed.
• pnpm exec turbo run build --filter=@bb/db --filter=@bb/server — passed (@bb/server built; @bb/db has no build task).
• git diff --check and origin/main...HEAD scope inspection — clean, 17 files, only this PR.
• GitHub CI — build/typecheck/lint, server, package, integration, all three app shards, and Linux/macOS package smoke jobs passed.

Controlled same-machine benchmark

The deterministic production-shaped fixture has 95 event rows over 12 turns (11 completed, one running), including 72 completed 65,536-character outputs (4,718,592 output characters). Reads used 12 warmups + 60 measured iterations; writes used eight warmups + 40 measured immediate transactions of four large completions. Both revisions ran the byte-identical harness (SHA-256 16f0062baf584b30dde86e3777289b763bacec01ddd8b105c819c2175414dc07) on the same arm64 macOS host under Node v22.23.1, with separate disposable SQLite databases and the repository's production WAL/cache/mmap settings. Percentiles use nearest rank.

Main-loop workload latency, ms	origin/main p50/p95/max	branch p50/p95/max
Normal bounded event read	12.740 / 42.382 / 166.079	0.556 / 0.902 / 1.273
Normal bounded timeline read	29.161 / 42.752 / 47.202	2.441 / 5.857 / 7.343
Raw event hydration read	4.128 / 7.127 / 28.982	16.035 / 26.562 / 49.142
Detail hydration read	1.074 / 1.794 / 2.516	1.702 / 3.110 / 3.507
Four-row write transaction	1.787 / 10.157 / 11.006	1.423 / 5.929 / 6.215

Zero-delay timer delay, ms	origin/main p50/p95/max	branch p50/p95/max
Normal bounded event read	13.191 / 42.412 / 169.818	0.960 / 1.970 / 4.080
Normal bounded timeline read	29.395 / 44.008 / 48.866	2.720 / 6.738 / 8.088
Raw event hydration read	4.230 / 7.474 / 29.023	16.527 / 26.618 / 49.178
Detail hydration read	1.343 / 2.789 / 3.074	1.912 / 3.524 / 4.890
Four-row write transaction	1.843 / 10.354 / 11.076	1.464 / 5.972 / 6.269

Bounded event/timeline selection held row/work counts at 95/6 while selected/materialized event data fell from 2,327,734 to 331,678 bytes (85.8% less); the timeline response fell from 198,652 to 31,480 bytes. Raw reads materialized 95 rows / 4,755,746 bytes / 4,718,592 output chars on both revisions. Detail reads materialized six work rows / 396,190 bytes / 393,216 output chars on both. Fixture event-row data fell from 4,739,518 to 331,678 bytes; the branch sidecar held 72 rows / 4,718,592 bytes, and the checkpointed DB grew from 5,414,912 to 5,771,264 bytes (6.6%) while both full and preview values coexist.

Raw, detail, and written full-output hashes exactly matched their deterministic expected hashes on both revisions (dc1df...40a0, 4b640e...68cb, e88059...35cf). Bounded timeline identity hashes also matched (7e5f41...fb5a); preview hashes intentionally differ because the branch returns the storage contract's head-and-tail preview instead of the old prefix-only SQL cap.

Raw BB thread-storage artifacts: comparison, origin/main JSON, branch JSON, byte-identical harness, and SHA-256 manifest. They are also browsable from the BB benchmark thread.

Fixes: no linked issue.

AGENT GENERATED

[ymichael]

Superseded by #2920 at the user’s request. PR #2920 now targets main and contains this complete sidecar foundation (b78bf6038f + export seam 259f937f7b) together with the incremental legacy migration (d5e5a814a9 + 1f46348a9b). The combined PR is MERGEABLE/CLEAN and all required CI checks are green. Please review and merge #2920 only.