feat(ledger): retention and compaction learned from the ledger's own history

[CodeWithJuber]

Summary

Ledger retention and compaction, learned from the ledger's own history. Every cut-off comes from the data; there is no fixed window and no tuned threshold.

What it replaces: pruning archived a tombstoned or dormant claim only after a fixed 2 × 45-day window, the near-duplicate grouper (τ = 0.7) was never called, and nothing recorded which claims were ever served. The per-session summary claims that nothing ever contradicts therefore grew without bound.

The three archive rules

• Never served. A tombstoned or dormant claim is archived at once: retrieve() never serves it, so its chance of being served is zero by construction, not by a threshold.
• Idle past the longest comeback. A live claim is archived once it has been idle longer than the longest gap after which any claim in this ledger was active again, and only once the usage log spans longer than that gap.
• Near-duplicates. Each claim's nearest-neighbour MinHash similarity to its own kind is modelled as one Gaussian or two (Otsu split); BIC picks the model, and only a two-component fit yields a boundary. A member is dropped only against the survivor's own pair, so a chain A–B–C never archives a claim that is no duplicate of the survivor.

A fitted cut-off was tried first and rejected. Fitting the cut-off by F1 over a replay of history learned "idle more than 2 days is dead" when a 3-day claim was present, and would have archived a 10-day claim on the day it fell due. An archived claim is no longer served and so cannot prove itself useful again, which makes that mistake self-confirming. Hence the longest comeback rather than a typical one. test/ledger_retention.test.js pins both properties.

Everything is reversible: the claim bytes move to attic/, its logs stay in place, new evidence un-archives it, and show / blame now read the attic.

The usage log

Retention can only learn from use if use is recorded, and retrieve() is pure: every caller discarded the ids. Each place that serves claims now appends one line to .forge/ledger/.usage.jsonl:

• the session-start lesson block;
• the pre-edit lesson advisory;
• the déjà-vu advisory, only when a hit is actually surfaced;
• forge ledger query;
• the MCP forge_ledger_query.

The log is machine-local (gitignored) and outside ledgerSignature, so appending never invalidates the snapshot cache. Writes go through appendLine, which terminates a line a killed process left torn, and the .gitignore entry is appended rather than rewritten so two processes cannot drop each other's.

Command

forge ledger compact [--dry-run] [--json] prints every learned number and what it would archive. Real output:

$ forge ledger compact --dry-run
Forge ledger — compact (every cut-off learned from this ledger)  [dry run]

  claims: 11 · claims with logged use: 10
  retention: idle cut-off 4 d = the longest idle stretch any claim came back from (199 comebacks, typical gap 4 d; usage log spans 90 d)
  duplicates: boundary 0.28 (two components beat one: BIC -72.6 < 3.5) · 1 group(s)

  archive: 3
    34a49b8d036e  idle 86 d > learned cut-off 4 d
    a5e218fd1814  tombstoned (never served)
    d3a5a1c9941e  near-duplicate of c70ee7d4f505 (similarity 0.55 ≥ learned 0.28)

  dry run: nothing written

The Stop hook applies the first two rules; the pairwise duplicate pass runs only in this command.

Verification

Check	Result
npm test	1,403 tests: 1,395 pass, 0 fail, 8 skipped
npm run lint	0 errors, 14 warnings (same as master)
npm run typecheck	0 errors
forge docs check	passes

Tests that prove the rules adapt rather than hide a constant:

• The same code on two ledgers whose calendars differ by a factor of 10 gives cut-offs that differ by exactly 10.
• A claim on a 10-day rhythm beside one on a 3-day rhythm is never archived while it is due.
• It refuses to act with no usage recorded, with no claim ever active twice, or with a usage log younger than the longest comeback.
• A ledger of distinct facts yields no duplicate boundary at all; a chain A–B–C drops only B.
• One end-to-end test drives the real hook, CLI and MCP entrypoints and asserts each logged what it served.

Independent review

An independent reviewer (fresh context, diff + spec + test results only) returned allow and raised six findings. All are addressed in this branch:

1. Chained duplicate groups could archive a claim that was no duplicate of the survivor, and the reported similarity could be from another pair → each drop is now judged and reported against the survivor's own pair, with a test.
2. Four serving points had no end-to-end test → test/ledger_usage.test.js drives the real entrypoints.
3. The usage append bypassed the torn-line guard → it now uses appendLine.
4. One outlier raises the cut-off for every claim → kept deliberately (see the rejected fitted cut-off above) and now stated in the CHANGELOG.
5. A stale comment in deja.js still described the 2·T window → rewritten.
6. The .gitignore writer could race → append-only now.

🤖 Generated with Claude Code