Skip to content

feat(storage): add bounded ledger archive reconstruction reader - #4972

Merged
likun666661 merged 2 commits into
mainfrom
feat/4071-ledger-archive-reader
Sep 7, 2026
Merged

feat(storage): add bounded ledger archive reconstruction reader#4972
likun666661 merged 2 commits into
mainfrom
feat/4071-ledger-archive-reader

Conversation

@likun666661

Copy link
Copy Markdown
Member

Summary

Refs #4071. Reader-first follow-up after #4963 and #4965.

Tool Result archive bytes are derived from durable model projections. Before changing writers, this PR adds a read-only reconstruction path that proves when an existing v1 archive can be recovered from the ledger alone.

  • Add a lease-bound SQLite evidence reader for one Session/event target. Select sizes and at most 65 transition metadata rows before loading JSON; reject more than 64 transitions or more than 2 MiB total evidence. A read is complete or refused, never a silently truncated history.
  • Add an indexed Session/target lookup and refuse unscoped malformed transition evidence. Use one read transaction for metadata admission and body reads. Keep node:sqlite loading lazy.
  • Reuse the existing reducer to accept only applied archive transitions. Reconstruct the source immediately before the matching replacement, including predecessor chains and concurrent sibling selection.
  • Freeze v1 archive serialization independently of future provider/UI formatting. Verify the stored placeholder identity, hash and byte length; do not expose raw tool results or run the legacy/live tool projector when a durable projection is missing.
  • Publish narrow reader entrypoints, but do not wire them into production ArchiveRead, change writers, delete payloads or activate a legacy fallback in this slice.

Verification

  • Core/storage/runtime builds and typechecks passed.
  • Full storage suite: 1196 passed, 8 skipped, 0 failed.
  • Final focused reader/evidence/reducer/ArchiveRead suites: 52 passed, 0 failed.
  • Real SQLite close/reopen reconstruction with no Artifact payload; cross-Session refusal; predecessor-source recovery; sibling/duplicate transitions; unknown versions; missing projections; hash/size limits.
  • 12k unrelated transition records do not enter the target evidence result. EXPLAIN asserts the target index; oversize evidence returns before any ledger JSON is materialized. v8-to-v9 migration preserves transition bytes.
  • Negative control: bypassing the reducer's applied-transition filter makes the refusal regression fail. Restored before final verification.
  • Root lint, format check, ASF headers and diff check passed.

Full runtime suite is not green locally: 3299 passed, 13 skipped, 2 failed in unchanged Responses reasoning coverage:

  • Alibaba Responses keeps multiple streamed reasoning items distinct through replay.
  • The pinned SDK maps official summary events across raw byte chunks.

The second reproduces when run alone. No Responses/provider implementation is changed here. Full Host/Desktop suites, Windows/Linux execution and large-workspace latency/RSS benchmarks were not run.

Migration and Rollout

Core-execution schema advances from 8 to 9 to create the partial expression index over existing transition records. This is a rebuildable index, not another history authority. JSON-invalid rows are indexed as unscoped instead of crashing index extraction.

Index construction has a one-time schema-upgrade cost. This PR does not claim index creation itself is bounded or that #4027 startup acceptance has passed. Normal reads use indexed target lookup and bounded metadata/body retrieval.

The reader is intentionally conservative: missing durable projections, incomplete/corrupt evidence and evidence exceeding the internal 64-transition/2-MiB limits are refused. The evidence byte limit includes the containing RuntimeEvent, so a small projection inside a very large raw-result event may also be refused. Existing Artifact reads remain unchanged.

The next writer slice must introduce its own new ledger-reference protocol and copy/remap contract using this reader foundation. This PR only reconstructs current v1 archive identities; it does not emit new durable refs or claim all historical archives are reconstructible. No old payload can be reclaimed merely because this reader exists.

AI use

  • No generative tool made a substantive contribution
  • Generative tooling made a substantive contribution

Tool(s) and scope: Codex inspected the ledger and storage contracts, implemented reader/index/test changes, ran local checks and prepared this PR. Commit carries Generated-by: Codex.

Checklist

  • Tests cover the change and fail without it
  • Lint, format, typecheck and the affected suites pass locally (focused and full storage pass; full runtime failures documented above)

Does this PR entail a change in behavior?

  • Yes — adds a schema index and opt-in read capability; production archive routing is unchanged.
  • No

Read complete target-scoped evidence through a lease-bound indexed query, then reconstruct only applied v1 archive sources using the existing projection reducer and frozen encoding. Keep production writers and Artifact fallback unchanged.

Generated-by: Codex
@github-actions github-actions Bot added the effort/L Under 1000 readable lines label Sep 7, 2026

@Astro-Han Astro-Han left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the reader-first implementation. Re-reviewed e6ec11f33d1a6dc559c6483c09a2034c77826847 with independent design, storage and reconstruction reviews. One API error-classification finding is inline, independently reproduced against the existing lease-revocation test. The production ArchiveRead route remains outside this slice.

AI disclosure: this review was performed by Codex agents; the coordinating agent checked the finding and reran the counterexample. This is not an independent human review.

中文摘要

已对当前提交重新进行设计、存储和重建路径交叉审查。发现一项 reader API 的错误分类问题,主审已通过现有 lease 撤销测试独立复现,详情见行内评论。生产 ArchiveRead 切换仍属于后续范围。本次为 Codex 自动化审查,不冒充人工验收。

Comment thread packages/storage/src/tool-result-archive-evidence.ts Outdated
Keep lease and database availability failures separate from JSON and record decoding failures. Assert exact revocation reasons and runtime error mapping.

Generated-by: Codex

@Astro-Han Astro-Han left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@likun666661
likun666661 merged commit f08f12a into main Sep 7, 2026
13 checks passed
@likun666661
likun666661 deleted the feat/4071-ledger-archive-reader branch September 7, 2026 11:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

effort/L Under 1000 readable lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants