Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -242,6 +242,10 @@ after its public API and format compatibility policies are established.

### Changed

- Documentation refreshed after the version-two merge: the README, the
version-two overview status, the closure and recovery status lines, the
reconstruction contract's retention note, and the requirements ledger state
what is implemented and what remains planned in #19, #21, and #97.
- The version-two retention and recovery pages each split their largest
sections into `retention-publication.md` (closure admission and the
generation transition) and `migration-recovery.md` (the one-way migration
Expand Down
21 changes: 16 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,17 +58,27 @@ Keep is required to refuse all three, before mutating anything.
- **Version-2 retention and migration, forward path.** Explicit retention
roots, deterministic closure verification, a one-way 21-phase migration,
and a 17-phase retention publication — all with production filesystem
writers, all preserving every version-1 byte.
writers, all preserving every version-1 byte. Reopening a migrated store
jointly admits its marker, intent, and receipt, binds the root's device,
mount, and inode identity to the intent, and pins the directories it
admitted. Publication binds this store's own catalog head and the catalog
it selects, and refuses retained stages, superseded candidates, substituted
files, replaced protocol directories, and every namespace or capacity
violation before it writes anything. Each refusal is a typed value, not a
Comment on lines +65 to +67

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Qualify the pre-mutation refusal claim

When the candidate namespace changes after verify_current, this promise is false: execute_root writes and synchronizes root.next before calling admit_root_namespace (src/adapters/retention/publication_execution.rs:55-65). The regression at filesystem_retention_attempt_tests.rs:78-102 demonstrates exactly this ordering by creating the conflicting namespace after the stage write and then receiving NamespaceExpectationViolated; the retained stage subsequently requires recovery. Limit the claim to violations observed during preflight, or document that a later namespace refusal can leave durable stage evidence.

AGENTS.md reference: AGENTS.md:L84-L89

Useful? React with 👍 / 👎.

string.
Comment on lines +67 to +68

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Do not promise typed substitution refusals

For a byte-equal file substitution during publication, the specific refusal is still only a string: filesystem_retention_stage::retention_error maps ExactRecordRefusal to io::Error::new(..., &'static str) (src/adapters/retention/filesystem_retention_stage.rs:94-112), and execute_retention_publication merely wraps that in the generic RetentionPublicationError::Storage phase variant. Consequently callers cannot downcast or match the listed substituted-file condition as a typed refusal—the corresponding test at filesystem_retention_storage_tests.rs:84-102 can assert only InvalidData. Narrow this statement to the current-state refusals that actually carry RetentionCurrentStateRefusal, or preserve the exact typed source.

AGENTS.md reference: AGENTS.md:L71-L74

Useful? React with 👍 / 👎.


## What it does not do yet

Version 2 writes correctly from a clean start. It cannot yet pick up the
pieces if it dies partway through. Until it can, **version 1 is the only
store admitted for production.**
Version 2 writes correctly from a clean start and, if it finds the residue of
an interrupted publication, refuses rather than guesses. Nothing yet recovers
that residue, and readers have no fence, so **an interrupted version-2
publication waits for a human until #19 lands.** A version-1 store stays
admitted until its owner migrates it; migrate only if you accept that wait.
Comment on lines +74 to +76

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Warn that clean migrations may fail after remount

This migration advice mentions only the wait caused by an interrupted publication, but a completely successful migration can also become unavailable after an ordinary reboot, unmount/remount, or restore changes statx.stx_mnt_id. The intent persists that mount ID (docs/formats/segment-store-v2/recovery.md:121-123), and FilesystemVersionTwoAdmission::require_root_identity compares it exactly and returns RootIdentityChanged (src/adapters/filesystem_version_two_admission.rs:120-134). Until #97 supplies a restart-stable coordinate, operators need this consequence stated explicitly before being advised to migrate.

AGENTS.md reference: AGENTS.md:L152-L156

Useful? React with 👍 / 👎.


| Gap | Tracked |
| --- | --- |
| Restart recovery for retention publication and migration | [#19](https://github.com/flyingrobots/keep/issues/19) |
| Restart-stable root identity coordinate in the migration intent | [#97](https://github.com/flyingrobots/keep/issues/97) |
| Reader fence binding one consistent catalog + retention snapshot | [#19](https://github.com/flyingrobots/keep/issues/19) |
| Precise verification reports at explicit depths | [#20](https://github.com/flyingrobots/keep/issues/20) |
| Garbage collection and identity-preserving compaction | [#21](https://github.com/flyingrobots/keep/issues/21) |
Expand Down Expand Up @@ -208,7 +218,8 @@ has one job; this one is the front door.
| Understand what is proved and what is not | [`docs/invariants/`](docs/invariants/) |
| Read the byte-level formats | [`docs/formats/`](docs/formats/) |
| See the architecture and port boundaries | [`docs/architecture/`](docs/architecture/) |
| Follow the crash and recovery rules | [`segment-store-v1/recovery.md`](docs/formats/segment-store-v1/recovery.md) · [`segment-store-v2/recovery.md`](docs/formats/segment-store-v2/recovery.md) |
| Follow the crash and recovery rules | [`segment-store-v1/recovery.md`](docs/formats/segment-store-v1/recovery.md) · [`segment-store-v2/recovery.md`](docs/formats/segment-store-v2/recovery.md) · [`segment-store-v2/migration-recovery.md`](docs/formats/segment-store-v2/migration-recovery.md) |
| See how a retention generation is published | [`segment-store-v2/retention-publication.md`](docs/formats/segment-store-v2/retention-publication.md) |
| Check reproducible performance evidence | [`docs/benchmarks/`](docs/benchmarks/) |
| Run the language-neutral corpora | [`conformance/`](conformance/) |
| See what changed | [`CHANGELOG.md`](CHANGELOG.md) |
Expand Down
44 changes: 26 additions & 18 deletions docs/formats/segment-store-v2/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,21 +76,29 @@ re-encode them.

## Status

The format contract is frozen by ADR-0009 and this specification. Public core
types now admit exact namespace bytes, namespace digests, root and liveness
generations, registered realization profiles, bounded closure policies,
reconstruction anchors, and semantic roots. Canonical root, manifest, and head
codecs match their independent golden records. Storage-independent transition
planning, deterministic bounded closure verification against one pinned
catalog, their combined preflight proof, and the exact 17-phase publication
vocabulary with a blocking storage capability port are available.
Storage-independent preparation derives exact canonical manifest and head
successors from coherent preflight and current-manifest evidence. Ordered
storage-port orchestration revalidates authority and returns a complete receipt.
Fresh writer-locked filesystem migration execution now publishes all canonical
fixed records and the exact empty version-2 namespace without changing
version-1 immutable bytes. Partial-prefix restart recovery, production
filesystem retention publication, immutable reader snapshots, and garbage
collection do not exist yet. Requirements still in progress in issue #19 or
issue #21 are not complete evidence. A store must refuse version-2 state until the relevant
corruption, model-based, crash-injection, recovery, and fuzz evidence exists.
The format contract is frozen by ADR-0009 and this specification.

Implemented with executable evidence: public core types for namespaces,
generations, realization profiles, closure policies, anchors, and roots;
canonical root, manifest, and head codecs matching their golden records;
storage-independent transition planning, bounded closure verification against
one pinned catalog, preflight, preparation, and the 17-phase publication port;
fresh writer-locked filesystem migration through all 21 phases, refusing a
version-one store that still holds a retained stage;
`FilesystemVersionTwoAdmission::reopen`, which jointly admits the marker,
intent, and receipt, binds the root's device, mount, and inode identity to the
intent, and pins the retention directories it admitted; and
`FilesystemRetentionPublicationAuthority`, which publishes initial and
successor generations against the observed head, binds this store's catalog
head and the catalog it selects, and refuses superseded candidates, retained
stages, replaced protocol directories, and every namespace or capacity
violation before mutation, each as a typed `RetentionCurrentStateRefusal`.

Not implemented: retention publication recovery and `KEEP-CRASH-036..052`
process-death evidence, partial-prefix migration recovery and
`KEEP-CRASH-053..073`, the reader fence, model-based transition evidence, and
garbage collection. Issue #19 owns the first four and issue #21 the last;
issue #97 owns the restart-stable root identity coordinate. A version-1 store
remains admitted until its owner migrates it, and the
[requirements ledger](requirements.md) is the authority on which requirements
are proven.
4 changes: 3 additions & 1 deletion docs/formats/segment-store-v2/closure.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
# Closure Verification

- Status: Normative version-2 protocol; storage-independent verifier
implemented; publication integration planned in issue
implemented; publication binds this store's catalog head and the catalog it
selects to the verified closure; member re-verification under filesystem
authority is planned in issue
[#19](https://github.com/flyingrobots/keep/issues/19)
- Format coordinate: `keep.segment-store/v2`
- Requirement: [`KEEP-RETENTION-005`](requirements.md#retention-transitions)
Expand Down
5 changes: 3 additions & 2 deletions docs/formats/segment-store-v2/recovery.md
Original file line number Diff line number Diff line change
Expand Up @@ -174,8 +174,9 @@ recovery instead. Direct version-2 initialization is undefined.

The exact offsets and fixtures are requirement `KEEP-MIGRATION-002`. The fresh
writer emits only those canonical records; success is not restart evidence.
Version 2 remains unavailable as production until partial-prefix recovery and
`KEEP-MIGRATION-007` process-death evidence exist.
A migrated store is admitted for forward publication, but partial-prefix
recovery and `KEEP-MIGRATION-007` process-death evidence remain absent, so an
interrupted migration waits for recovery instead of continuing.

## Retention publication recovery

Expand Down
4 changes: 2 additions & 2 deletions docs/formats/segment-store-v2/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ case is not evidence.
| `KEEP-RETENTION-004` | Retain and release compare expected and observed generations and publish exact successors only | unforgeable readiness and preflight proofs in `tests/retention_transition.rs` and `tests/retention_preflight.rs`; exact successor preparation and complete receipt evidence in `tests/retention_publication_preparation.rs` and `tests/retention_publication_execution.rs`; writer-locked initial filesystem publication in `filesystem_retention_storage_tests`; observed-head successor publication, exact predecessor binding, and absent-head refusal in `filesystem_retention_successor_tests`; the store's catalog head must name the closure's catalog generation and digest before any forward write in `filesystem_retention_catalog_tests`; a head whose predecessor disagrees with its manifest refuses in `filesystem_retention_current_tests`; a successor reopens and decodes the manifest-selected predecessor root and refuses an absent or changed one in `filesystem_retention_expectation_tests` | Implemented |
| `KEEP-RETENTION-005` | Closure derivation is deterministic, bounded, cycle-safe, fail-closed, and verifies complete blob reconstruction | exact accounting, reconstruction, adversarial-catalog, and exhaustive model laws in `tests/retention_closure.rs`; corrupt members refuse through the inherited segment-record admission laws and seeded `segment_format` fuzz target routed by `closure-corruption.md` | Implemented |
| `KEEP-RETENTION-006` | Publication follows the exact ordered durability protocol, including new namespace-directory admission and retention of fixed-stage evidence until head commit, and returns only after cleanup synchronization | typed vocabulary and blocking port in `tests/retention_publication_phase.rs` and `tests/retention_publication_storage.rs`; ordered execution, conditional namespace sync, and all 17 exact storage-fault boundaries in `tests/retention_publication_execution.rs`; production 17-phase forward filesystem execution, exclusive staging, byte-equal inode-substitution refusal, and retained-stage recovery refusal in `filesystem_retention_storage_tests`; orphan namespace directories count against the 4,096 ceiling and refuse a new namespace before any stage is written in `filesystem_retention_capacity_tests`; crash injection remains | In progress in #19 |
| `KEEP-RETENTION-007` | Restart resolves every fixed-stage crash prefix to one documented lawful state or typed ambiguity | recovery-required refusals before any mutation in `filesystem_retention_expectation_tests`: an absent head over populated pools, a non-initial head prepared against an absent head, an orphan directory for a namespace expected absent, and an absent directory for a namespace expected current; debug and release crash matrix remains | In progress in #19 |
| `KEEP-RETENTION-007` | Restart resolves every fixed-stage crash prefix to one documented lawful state or typed ambiguity | recovery-required refusals before any mutation in `filesystem_retention_expectation_tests`: an absent head over populated pools, a non-initial head prepared against an absent head, an orphan directory for a namespace expected absent, and an absent directory for a namespace expected current; debug and release crash matrix remains; replaced protocol directories, an absent or changed head-selected catalog, an over-full census, zero-generation pool names, and a stage retained by a failed write refuse in `filesystem_retention_*_tests` | In progress in #19 |

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Stop claiming evidence for the changed-catalog refusal

The new evidence entry says that a changed head-selected catalog refuses in filesystem_retention_*_tests, but a repo-wide search for CatalogChanged finds only the production variant and branch (filesystem_retention_catalog.rs:70) plus prose; filesystem_retention_catalog_tests.rs:17-125 tests foreign, corrupt-head, and absent-catalog cases but never constructs a changed selected catalog or asserts CatalogChanged. The authoritative ledger therefore presents an untested case as evidence; add an exact regression for that variant or leave the case explicitly outstanding.

AGENTS.md reference: AGENTS.md:L123-L129

Useful? React with 👍 / 👎.

| `KEEP-RETENTION-008` | Readers double-collect catalog and retention heads and bind one complete catalog, manifest, and root-generation view under a `ReaderFence` | immutable snapshot and concurrency tests | Planned in #19 |
| `KEEP-RETENTION-009` | Exact already-committed retry is idempotent only while its successor remains current | byte-identical planning in `tests/retention_transition.rs`; authority-revalidated zero-mutation retry receipt in `tests/retention_publication_execution.rs`; exact already-committed filesystem retry with a byte-identical retention witness in `filesystem_retention_storage_tests`; superseded-candidate filesystem refusal with zero mutation in `filesystem_retention_successor_tests`; committed retry reopens the head-selected manifest entry and root pool bytes, refusing absent, changed, or corrupt evidence in `filesystem_retention_current_tests`; every refusal is a typed `RetentionCurrentStateRefusal` source, with superseded, committed-root-absent, committed-root-changed, and head-absent-with-artifacts pinned by downcast | Implemented |
| `KEEP-RETENTION-010` | Model operation sequences agree with a deterministic namespace-to-anchor-set map and never admit caller identity, paths, clocks, or application policy | model-based and source-architecture tests | Planned in #19 |
Expand All @@ -30,7 +30,7 @@ case is not evidence.
| --- | --- | --- | --- |
| `KEEP-MIGRATION-001` | Exact version-1 stores remain admitted until a durable migration artifact exists | compatibility fixtures | Planned in #19 |
| `KEEP-MIGRATION-002` | Format marker, intent, and receipt have complete fixed byte tables, named domains, bounds, checksums, deterministic store identity, and exact initial-state digests | exact admission in `tests/store_format_marker.rs`, `tests/store_migration_intent.rs`, and `tests/store_migration_receipt.rs`; canonical construction in `tests/store_migration_intent_encoding.rs` and `tests/store_migration_receipt_encoding.rs`; seeded `migration_format` fuzz target | Implemented |
| `KEEP-MIGRATION-003` | Migration revalidates version-1 head, catalog, pools, root identity, and writer authority before mutation | bounded canonical pool inventory in `tests/store_migration_inventory.rs`; writer-locked filesystem pool admission in `filesystem_inventory_*_tests`; exact authority observation and drift refusal in `filesystem_migration_authority_tests`; verification-first execution in `tests/store_migration_execution.rs`; fresh filesystem integration and post-publication drift refusal in `filesystem_migration_storage_tests` | Implemented |
| `KEEP-MIGRATION-003` | Migration revalidates version-1 head, catalog, pools, root identity, and writer authority before mutation | bounded canonical pool inventory in `tests/store_migration_inventory.rs`; writer-locked filesystem pool admission in `filesystem_inventory_*_tests`; exact authority observation and drift refusal in `filesystem_migration_authority_tests`; verification-first execution in `tests/store_migration_execution.rs`; fresh filesystem integration and post-publication drift refusal in `filesystem_migration_storage_tests`; a version-one store still holding a retained stage refuses before the intent is observed in `filesystem_migration_storage_tests` | Implemented |
| `KEEP-MIGRATION-004` | Every partial migration prefix continues idempotently under writer authority | state-machine and recovery tests | Planned in #19 |
| `KEEP-MIGRATION-005` | Unknown, out-of-order, substituted, corrupt, conflicting, or changed evidence is unrecoverable ambiguity | forward-execution stage preservation, byte-equal inode-substitution, out-of-order-prefix, and post-publication drift laws in `filesystem_migration_storage_tests`; unknown `retention` entries, non-digest namespace directories, and noncanonical pool names refuse before any retention stage is written in `filesystem_retention_namespace_tests`; restart corruption and mutation matrix remains | In progress in #19 |
| `KEEP-MIGRATION-006` | Migration never rewrites or deletes admitted version-1 immutable bytes | exact segment, catalog, and head before/after witness in `filesystem_migration_storage_tests`; restart-path evidence remains | In progress in #19 |
Expand Down
5 changes: 3 additions & 2 deletions docs/invariants/authenticated-reconstruction/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -220,8 +220,9 @@ A future operation claiming durable logical reconstruction must additionally:

The current durable segment, catalog, publication, and recovery surfaces do
not yet form this consolidated high-level `BlobId`-to-writer contract.
Retention remains planned; no current retention surface protects the evidence
closure required by this operation. These lower-level surfaces must not be
Retention publication now records verified closures as generation-checked
roots, but nothing collects or fences yet, so no current surface protects or
releases the evidence closure this operation requires. These lower-level surfaces must not be
described as an implemented durable logical reconstruction API.

## Current public evidence
Expand Down