Records: sealed shards for every store, readers of packed shards, packed writing off

[daniloc]

Builds the record layout #121 designed and the owner approved ("let's do sealed shards and packed format"). Every record store (journal, runs, work) now writes sealed shards. Every reader reads them, packed ones included. Packed writing ships off.

Why: GitHub's merge does not honour merge=union (d-44ee2214). Two branches of one session that append to <session>.jsonl conflict on GitHub, and CI runs none of its jobs.

Shard naming

• Line shard: .coherence/<store>/<session>.<token>.jsonl, one record per line. The token is 8 random lowercase hex characters.
• Packed shard: .coherence/<store>/<session>.<token>.jsonseq. The file starts with U+001E (record separator), each record is followed by U+001E, and the file has no newline. Git and GitHub count it as one line.
• Legacy files: <session>.jsonl files are read as committed and never appended to again.
• The session from a file name: it is everything before the first . for any session id with no dot in it. Claude Code and Codex ids have none. SESSION_TOKEN does allow dots; for such an id, the session is the name minus its last two dot-separated parts (sessionOfStoreFile in src/journal/stores.ts).
• Attribution: no reader takes it from a file name. Each record's own session and agent fields stay the attribution, and vetting is unchanged.

When a shard is sealed, and the writer's state

shardFor in src/journal/shards.ts is the one place every writer gets its file from: appendRecord, appendWork, appendRun and appendRefutation. A writer reuses its last shard in a store only while all of these hold:

• HEAD is on the same commit and the same branch;
• HEAD's reflog is the same size, so HEAD did not move and come back (for example, a commit followed by a mixed reset);
• the shard is the size the writer left it, and under 2 MiB;
• the shard's extension matches the current packed setting;
• no index.lock exists, and git's index does not hold the shard, staged or committed.

If any of these fails, the writer starts a new shard with a new token. So once a stage or a commit holds a shard, nothing appends to it again.

Where the state lives: .coherence/cache/shards/<session>.json, one per checkout. It is ignored by git and written to a temp file, then renamed. It is only a hint (c-e23e2e42):

• A lost, torn or wrongly shaped state starts a new shard.
• A state that names a committed shard, with every other field current, is still caught by git's own index. The index is searched whenever its identity has changed since the shard was last seen outside it, or whenever it was written at or after the shard's last write. Staging or committing the shard writes the index after that last write.
• Two writers racing (hooks and the warm process) cost at most one extra shard.

Cost on the hot path: everything is read from git's files with no spawn. That covers .git, a worktree's gitdir:, the common dir, loose refs and packed-refs, logs/HEAD, and the index. Most appends pay one stat for the index. A full index read on every append made the work meter's 10× budget test fail (4 KB of reads at 1×, 31 KB at 10×), which is why the identity-and-time test exists. Git is spawned only for a reftable repository, or for an index that is version 4 or split.

Packed format, and what 1.7.x does with it

• The write: a byte append of the record and its separator, never a rewrite. Append-only holds at the byte level. A sealed packed shard is never touched again. A torn record is ended by the next separator, the same way a newline ends a torn line. Concurrent writers each append whole, with no rename that could lose one writer's record.
• Why not a rewrite: rewriting the line through a temp file and a rename on every append was rejected for exactly that reason (d-7f00519e).
• Cost: a packed append costs what a line append does.
• 1.7.x readers skip it (bench/records/old-readers.ts, run against origin/main's 1.7.2 source):
  • its loaders, run index, peer feed, vocabulary check and records summary only read *.jsonl, so they skip a .jsonseq shard's records silently: 0 read, 0 damaged;
  • they read every line shard whole;
  • its citation search found none of the packed records, because a packed line starts with the separator rather than {"id":, and found every record in a line shard.
• The default: packed writing is off (PACKED_BY_DEFAULT = false). COHERENCE_PACKED_RECORDS=1 turns it on for one writer. Whether to turn the default on is the owner's call at release time, in the release after the one that ships these readers. The steps are in docs/records-layout.md ("When to write packed shards") and in d-4e3a2564.

Readers changed

• storeLines/recordTexts now split on the newline and on U+001E, and isStoreFile accepts both extensions. Content ordering and dedupe of a record kept twice still hold across shards.
• loadJournal/loadJournalFiles, journalFileKeys (the peer feed; its seen-memo is per file and works per shard), loadWork and loadRuns.
• The run index: it reads files by extension, and offsets end at the separator in a packed shard.
• The citation search (branches.ts) matches each record within a grep hit.
• The vocabulary check (check.ts: record files and their lines) and the vocabulary state's record keys.
• The CI records summary (.github/scripts/records-summary.mjs).
• The status line (STATUS_SOURCES) now reads every one of the session's shards, not <session>.jsonl.
• The verbs' "recorded in …" lines name the shard.
• No change was needed in Scope live (it maps by folder), the mirror reading (it leaves .coherence out) or telemetry (it reads no store).
• Follow-up for the Journal pane's agent, not edited here as asked: add '.jsonseq' to RECORD_FILE_EXTENSIONS in mods/coherence/hooks/journal.ts. Its splitter already handles U+001E.

.coherence/.gitattributes keeps merge=union for the legacy files old releases still append to. Nothing already committed is rewritten.

Measurements (bench/records/shipped.ts, shipped.json; darwin arm64, node 22.21.1)

loadJournal 1× (1,146 records, 89 files) / 10× (11,460 records, 890 files)	5.4 ms / 59 ms
1.7.x append (appendWhole to one file)	0.03 ms
choosing the shard and keeping its state	0.20 ms (100 tracked files), 0.24 ms (20,000 files, 1.6 MB index)
appendRecord line / packed	0.24 / 0.24 ms; 0.28 / 0.29 ms at 20,000 files
appendRun line / packed	0.47 / 0.46 ms; 0.56 / 0.55 ms
append when the index was rewritten before every append (so it is searched every time)	0.29 ms; 0.43 ms line / 0.56 ms packed at 20,000 files

This PR on GitHub's terms: node bench/records/shipped.ts --merge origin/main HEAD reads attributes from the empty tree, so merge=union does not apply. Run before merging main in, this branch landed on origin/main (bf7f77a) by merge, squash and rebase with no conflicts. The control (two appends to one legacy file) conflicts with no attributes and merges cleanly with merge=union.

Specs (spec --check: 453 bullets, all invariants, 0 problems)

Each new bullet has a test named by its via line and a witnessed refutation (break staged, refute, restored with git checkout, green batched and alone with --each):

• Journal / a committed shard is never appended again (the rotation rule). Break: reuse the last shard whatever HEAD, the branch, the reflog or the index say. Result: red, a committed shard was appended to.
• Journal / lost shard state never reuses a committed shard. Break: trust the state's index identity alone. Result: red.
• Journal / readers read packed shards. Break: recordTexts splits on the newline only. Result: red.
• Journal / packed writing is off by default. Break: PACKED_BY_DEFAULT = true. Result: red.
• Lifecycle / record stores merge with no driver, with the bench's six landings (parallel and stacked, each by merge, squash and rebase, no attributes) as a test. Break: the branch-only rule. Result: red on stacked plus squash, conflicts in all three stores and 21 of 30 records loaded, as the bench predicted.
• Lifecycle / record stores merge by union keeps its name, is scoped to legacy files, and was witnessed again. Its test now appends to legacy files directly.
• Lifecycle / record readers tolerate what a union merge keeps was witnessed again for the same reason.

The refutation of "packed writing is off by default" ran with the default staged on, so it wrote a real packed shard: .coherence/runs/a79f7dfa2c80ee6b9.b5839fdc.jsonseq. It is kept, because records are never removed (d-dada22ac).

Bench and probe

docs/records-layout.md and bench/records/ come from #121. The doc gains an As built section, the steps for turning packed writing on, the measurements and the spec changes. Two scripts are new: shipped.ts and old-readers.ts. The probe folder (bench/records/probe/) is left out. Its GitHub counts are recorded in github-counts.json and stay readable from #121.

Vocabulary

shard is declared as a property of journal, not as a new concept, so the compact injection is unchanged (d-ea90469a, d-386a2748). lexicon --check reports 0 rejected names and 0 unknown nouns.

Journal (session a79f7dfa2c80ee6b9, agent sealed-shards)

• d-adcc9c7f: rotation and state design
• d-7f00519e: packed extension and byte append
• d-4e3a2564: packed off by default, flip next release at the owner's call
• d-451607dd: union bullet scoped to legacy files, plus the no-driver bullet
• d-dada22ac: keep the refutation's packed shard; probe left out
• d-ea90469a, d-386a2748: shard as a property of journal
• u-efd60e98: unable. c-e23e2e42 could not be resolved, and Records layout: a design that never conflicts on GitHub's merge, measured #121's records could not be cited: they are uncommitted in the Records layout: a design that never conflicts on GitHub's merge, measured #121 worktree. Resolve the conjecture as "no" once they land.
• Enactments: en-26668abd (witness a refutation), en-6c69f175 and en-6d85235f (the two lexicon practices).

Cites d-3a125a06, d-fe8a26f3 and d-d5b49709 (#121) in prose.

🤖 Generated with Claude Code