From 46ead1f4450f1d77bbc67357d2f5836b8d9bf3b7 Mon Sep 17 00:00:00 2001 From: James Ross Date: Tue, 8 Sep 2026 01:16:12 -0700 Subject: [PATCH] Docs: state the post-merge version-two status across the documentation PR #78 landed one-way migration, version-two reopen, and forward retention publication with the review hardening on top, but several pages still spoke from before it: the version-two overview said production retention publication did not exist, the recovery page said version two was unavailable as production, the closure page called publication integration planned, the reconstruction contract said retention remained planned, and the README said version one was the only store admitted for production. Every page now states the same boundary. Implemented: migration, reopen with root-identity binding and pinned directories, publication binding this store's catalog head and selected catalog, and typed refusals for every namespace, capacity, stage, and substitution violation. Planned: retention publication recovery and KEEP-CRASH-036..052, partial-prefix migration recovery and KEEP-CRASH-053..073, the reader fence, model-based transition evidence (#19), garbage collection (#21), and the restart-stable root identity coordinate (#97). The README says plainly that an interrupted version-two publication waits for a human until #19 lands, routes to the split recovery and publication pages, and adds #97 to its gap table. The requirements ledger's RETENTION-007 and MIGRATION-003 evidence cells name the new laws. No claim was added that a test does not back. Refs #19 #21 #97 --- CHANGELOG.md | 4 ++ README.md | 21 ++++++--- docs/formats/segment-store-v2/README.md | 44 +++++++++++-------- docs/formats/segment-store-v2/closure.md | 4 +- docs/formats/segment-store-v2/recovery.md | 5 ++- docs/formats/segment-store-v2/requirements.md | 4 +- .../authenticated-reconstruction/README.md | 5 ++- 7 files changed, 57 insertions(+), 30 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index fd6d2f4..080ae98 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -242,6 +242,10 @@ after its public API and format compatibility policies are established. ### Changed +- Documentation refreshed after the version-two merge: the README, the + version-two overview status, the closure and recovery status lines, the + reconstruction contract's retention note, and the requirements ledger state + what is implemented and what remains planned in #19, #21, and #97. - The version-two retention and recovery pages each split their largest sections into `retention-publication.md` (closure admission and the generation transition) and `migration-recovery.md` (the one-way migration diff --git a/README.md b/README.md index 3a20ed0..51b601c 100644 --- a/README.md +++ b/README.md @@ -58,17 +58,27 @@ Keep is required to refuse all three, before mutating anything. - **Version-2 retention and migration, forward path.** Explicit retention roots, deterministic closure verification, a one-way 21-phase migration, and a 17-phase retention publication — all with production filesystem - writers, all preserving every version-1 byte. + writers, all preserving every version-1 byte. Reopening a migrated store + jointly admits its marker, intent, and receipt, binds the root's device, + mount, and inode identity to the intent, and pins the directories it + admitted. Publication binds this store's own catalog head and the catalog + it selects, and refuses retained stages, superseded candidates, substituted + files, replaced protocol directories, and every namespace or capacity + violation before it writes anything. Each refusal is a typed value, not a + string. ## What it does not do yet -Version 2 writes correctly from a clean start. It cannot yet pick up the -pieces if it dies partway through. Until it can, **version 1 is the only -store admitted for production.** +Version 2 writes correctly from a clean start and, if it finds the residue of +an interrupted publication, refuses rather than guesses. Nothing yet recovers +that residue, and readers have no fence, so **an interrupted version-2 +publication waits for a human until #19 lands.** A version-1 store stays +admitted until its owner migrates it; migrate only if you accept that wait. | Gap | Tracked | | --- | --- | | Restart recovery for retention publication and migration | [#19](https://github.com/flyingrobots/keep/issues/19) | +| Restart-stable root identity coordinate in the migration intent | [#97](https://github.com/flyingrobots/keep/issues/97) | | Reader fence binding one consistent catalog + retention snapshot | [#19](https://github.com/flyingrobots/keep/issues/19) | | Precise verification reports at explicit depths | [#20](https://github.com/flyingrobots/keep/issues/20) | | Garbage collection and identity-preserving compaction | [#21](https://github.com/flyingrobots/keep/issues/21) | @@ -208,7 +218,8 @@ has one job; this one is the front door. | Understand what is proved and what is not | [`docs/invariants/`](docs/invariants/) | | Read the byte-level formats | [`docs/formats/`](docs/formats/) | | See the architecture and port boundaries | [`docs/architecture/`](docs/architecture/) | -| Follow the crash and recovery rules | [`segment-store-v1/recovery.md`](docs/formats/segment-store-v1/recovery.md) · [`segment-store-v2/recovery.md`](docs/formats/segment-store-v2/recovery.md) | +| Follow the crash and recovery rules | [`segment-store-v1/recovery.md`](docs/formats/segment-store-v1/recovery.md) · [`segment-store-v2/recovery.md`](docs/formats/segment-store-v2/recovery.md) · [`segment-store-v2/migration-recovery.md`](docs/formats/segment-store-v2/migration-recovery.md) | +| See how a retention generation is published | [`segment-store-v2/retention-publication.md`](docs/formats/segment-store-v2/retention-publication.md) | | Check reproducible performance evidence | [`docs/benchmarks/`](docs/benchmarks/) | | Run the language-neutral corpora | [`conformance/`](conformance/) | | See what changed | [`CHANGELOG.md`](CHANGELOG.md) | diff --git a/docs/formats/segment-store-v2/README.md b/docs/formats/segment-store-v2/README.md index 7f65830..1f27b80 100644 --- a/docs/formats/segment-store-v2/README.md +++ b/docs/formats/segment-store-v2/README.md @@ -76,21 +76,29 @@ re-encode them. ## Status -The format contract is frozen by ADR-0009 and this specification. Public core -types now admit exact namespace bytes, namespace digests, root and liveness -generations, registered realization profiles, bounded closure policies, -reconstruction anchors, and semantic roots. Canonical root, manifest, and head -codecs match their independent golden records. Storage-independent transition -planning, deterministic bounded closure verification against one pinned -catalog, their combined preflight proof, and the exact 17-phase publication -vocabulary with a blocking storage capability port are available. -Storage-independent preparation derives exact canonical manifest and head -successors from coherent preflight and current-manifest evidence. Ordered -storage-port orchestration revalidates authority and returns a complete receipt. -Fresh writer-locked filesystem migration execution now publishes all canonical -fixed records and the exact empty version-2 namespace without changing -version-1 immutable bytes. Partial-prefix restart recovery, production -filesystem retention publication, immutable reader snapshots, and garbage -collection do not exist yet. Requirements still in progress in issue #19 or -issue #21 are not complete evidence. A store must refuse version-2 state until the relevant -corruption, model-based, crash-injection, recovery, and fuzz evidence exists. +The format contract is frozen by ADR-0009 and this specification. + +Implemented with executable evidence: public core types for namespaces, +generations, realization profiles, closure policies, anchors, and roots; +canonical root, manifest, and head codecs matching their golden records; +storage-independent transition planning, bounded closure verification against +one pinned catalog, preflight, preparation, and the 17-phase publication port; +fresh writer-locked filesystem migration through all 21 phases, refusing a +version-one store that still holds a retained stage; +`FilesystemVersionTwoAdmission::reopen`, which jointly admits the marker, +intent, and receipt, binds the root's device, mount, and inode identity to the +intent, and pins the retention directories it admitted; and +`FilesystemRetentionPublicationAuthority`, which publishes initial and +successor generations against the observed head, binds this store's catalog +head and the catalog it selects, and refuses superseded candidates, retained +stages, replaced protocol directories, and every namespace or capacity +violation before mutation, each as a typed `RetentionCurrentStateRefusal`. + +Not implemented: retention publication recovery and `KEEP-CRASH-036..052` +process-death evidence, partial-prefix migration recovery and +`KEEP-CRASH-053..073`, the reader fence, model-based transition evidence, and +garbage collection. Issue #19 owns the first four and issue #21 the last; +issue #97 owns the restart-stable root identity coordinate. A version-1 store +remains admitted until its owner migrates it, and the +[requirements ledger](requirements.md) is the authority on which requirements +are proven. diff --git a/docs/formats/segment-store-v2/closure.md b/docs/formats/segment-store-v2/closure.md index 6e1f0e1..3e0e79b 100644 --- a/docs/formats/segment-store-v2/closure.md +++ b/docs/formats/segment-store-v2/closure.md @@ -1,7 +1,9 @@ # Closure Verification - Status: Normative version-2 protocol; storage-independent verifier - implemented; publication integration planned in issue + implemented; publication binds this store's catalog head and the catalog it + selects to the verified closure; member re-verification under filesystem + authority is planned in issue [#19](https://github.com/flyingrobots/keep/issues/19) - Format coordinate: `keep.segment-store/v2` - Requirement: [`KEEP-RETENTION-005`](requirements.md#retention-transitions) diff --git a/docs/formats/segment-store-v2/recovery.md b/docs/formats/segment-store-v2/recovery.md index c389940..efa539b 100644 --- a/docs/formats/segment-store-v2/recovery.md +++ b/docs/formats/segment-store-v2/recovery.md @@ -174,8 +174,9 @@ recovery instead. Direct version-2 initialization is undefined. The exact offsets and fixtures are requirement `KEEP-MIGRATION-002`. The fresh writer emits only those canonical records; success is not restart evidence. -Version 2 remains unavailable as production until partial-prefix recovery and -`KEEP-MIGRATION-007` process-death evidence exist. +A migrated store is admitted for forward publication, but partial-prefix +recovery and `KEEP-MIGRATION-007` process-death evidence remain absent, so an +interrupted migration waits for recovery instead of continuing. ## Retention publication recovery diff --git a/docs/formats/segment-store-v2/requirements.md b/docs/formats/segment-store-v2/requirements.md index ea516fd..ae8100b 100644 --- a/docs/formats/segment-store-v2/requirements.md +++ b/docs/formats/segment-store-v2/requirements.md @@ -15,7 +15,7 @@ case is not evidence. | `KEEP-RETENTION-004` | Retain and release compare expected and observed generations and publish exact successors only | unforgeable readiness and preflight proofs in `tests/retention_transition.rs` and `tests/retention_preflight.rs`; exact successor preparation and complete receipt evidence in `tests/retention_publication_preparation.rs` and `tests/retention_publication_execution.rs`; writer-locked initial filesystem publication in `filesystem_retention_storage_tests`; observed-head successor publication, exact predecessor binding, and absent-head refusal in `filesystem_retention_successor_tests`; the store's catalog head must name the closure's catalog generation and digest before any forward write in `filesystem_retention_catalog_tests`; a head whose predecessor disagrees with its manifest refuses in `filesystem_retention_current_tests`; a successor reopens and decodes the manifest-selected predecessor root and refuses an absent or changed one in `filesystem_retention_expectation_tests` | Implemented | | `KEEP-RETENTION-005` | Closure derivation is deterministic, bounded, cycle-safe, fail-closed, and verifies complete blob reconstruction | exact accounting, reconstruction, adversarial-catalog, and exhaustive model laws in `tests/retention_closure.rs`; corrupt members refuse through the inherited segment-record admission laws and seeded `segment_format` fuzz target routed by `closure-corruption.md` | Implemented | | `KEEP-RETENTION-006` | Publication follows the exact ordered durability protocol, including new namespace-directory admission and retention of fixed-stage evidence until head commit, and returns only after cleanup synchronization | typed vocabulary and blocking port in `tests/retention_publication_phase.rs` and `tests/retention_publication_storage.rs`; ordered execution, conditional namespace sync, and all 17 exact storage-fault boundaries in `tests/retention_publication_execution.rs`; production 17-phase forward filesystem execution, exclusive staging, byte-equal inode-substitution refusal, and retained-stage recovery refusal in `filesystem_retention_storage_tests`; orphan namespace directories count against the 4,096 ceiling and refuse a new namespace before any stage is written in `filesystem_retention_capacity_tests`; crash injection remains | In progress in #19 | -| `KEEP-RETENTION-007` | Restart resolves every fixed-stage crash prefix to one documented lawful state or typed ambiguity | recovery-required refusals before any mutation in `filesystem_retention_expectation_tests`: an absent head over populated pools, a non-initial head prepared against an absent head, an orphan directory for a namespace expected absent, and an absent directory for a namespace expected current; debug and release crash matrix remains | In progress in #19 | +| `KEEP-RETENTION-007` | Restart resolves every fixed-stage crash prefix to one documented lawful state or typed ambiguity | recovery-required refusals before any mutation in `filesystem_retention_expectation_tests`: an absent head over populated pools, a non-initial head prepared against an absent head, an orphan directory for a namespace expected absent, and an absent directory for a namespace expected current; debug and release crash matrix remains; replaced protocol directories, an absent or changed head-selected catalog, an over-full census, zero-generation pool names, and a stage retained by a failed write refuse in `filesystem_retention_*_tests` | In progress in #19 | | `KEEP-RETENTION-008` | Readers double-collect catalog and retention heads and bind one complete catalog, manifest, and root-generation view under a `ReaderFence` | immutable snapshot and concurrency tests | Planned in #19 | | `KEEP-RETENTION-009` | Exact already-committed retry is idempotent only while its successor remains current | byte-identical planning in `tests/retention_transition.rs`; authority-revalidated zero-mutation retry receipt in `tests/retention_publication_execution.rs`; exact already-committed filesystem retry with a byte-identical retention witness in `filesystem_retention_storage_tests`; superseded-candidate filesystem refusal with zero mutation in `filesystem_retention_successor_tests`; committed retry reopens the head-selected manifest entry and root pool bytes, refusing absent, changed, or corrupt evidence in `filesystem_retention_current_tests`; every refusal is a typed `RetentionCurrentStateRefusal` source, with superseded, committed-root-absent, committed-root-changed, and head-absent-with-artifacts pinned by downcast | Implemented | | `KEEP-RETENTION-010` | Model operation sequences agree with a deterministic namespace-to-anchor-set map and never admit caller identity, paths, clocks, or application policy | model-based and source-architecture tests | Planned in #19 | @@ -30,7 +30,7 @@ case is not evidence. | --- | --- | --- | --- | | `KEEP-MIGRATION-001` | Exact version-1 stores remain admitted until a durable migration artifact exists | compatibility fixtures | Planned in #19 | | `KEEP-MIGRATION-002` | Format marker, intent, and receipt have complete fixed byte tables, named domains, bounds, checksums, deterministic store identity, and exact initial-state digests | exact admission in `tests/store_format_marker.rs`, `tests/store_migration_intent.rs`, and `tests/store_migration_receipt.rs`; canonical construction in `tests/store_migration_intent_encoding.rs` and `tests/store_migration_receipt_encoding.rs`; seeded `migration_format` fuzz target | Implemented | -| `KEEP-MIGRATION-003` | Migration revalidates version-1 head, catalog, pools, root identity, and writer authority before mutation | bounded canonical pool inventory in `tests/store_migration_inventory.rs`; writer-locked filesystem pool admission in `filesystem_inventory_*_tests`; exact authority observation and drift refusal in `filesystem_migration_authority_tests`; verification-first execution in `tests/store_migration_execution.rs`; fresh filesystem integration and post-publication drift refusal in `filesystem_migration_storage_tests` | Implemented | +| `KEEP-MIGRATION-003` | Migration revalidates version-1 head, catalog, pools, root identity, and writer authority before mutation | bounded canonical pool inventory in `tests/store_migration_inventory.rs`; writer-locked filesystem pool admission in `filesystem_inventory_*_tests`; exact authority observation and drift refusal in `filesystem_migration_authority_tests`; verification-first execution in `tests/store_migration_execution.rs`; fresh filesystem integration and post-publication drift refusal in `filesystem_migration_storage_tests`; a version-one store still holding a retained stage refuses before the intent is observed in `filesystem_migration_storage_tests` | Implemented | | `KEEP-MIGRATION-004` | Every partial migration prefix continues idempotently under writer authority | state-machine and recovery tests | Planned in #19 | | `KEEP-MIGRATION-005` | Unknown, out-of-order, substituted, corrupt, conflicting, or changed evidence is unrecoverable ambiguity | forward-execution stage preservation, byte-equal inode-substitution, out-of-order-prefix, and post-publication drift laws in `filesystem_migration_storage_tests`; unknown `retention` entries, non-digest namespace directories, and noncanonical pool names refuse before any retention stage is written in `filesystem_retention_namespace_tests`; restart corruption and mutation matrix remains | In progress in #19 | | `KEEP-MIGRATION-006` | Migration never rewrites or deletes admitted version-1 immutable bytes | exact segment, catalog, and head before/after witness in `filesystem_migration_storage_tests`; restart-path evidence remains | In progress in #19 | diff --git a/docs/invariants/authenticated-reconstruction/README.md b/docs/invariants/authenticated-reconstruction/README.md index 79910b0..98430be 100644 --- a/docs/invariants/authenticated-reconstruction/README.md +++ b/docs/invariants/authenticated-reconstruction/README.md @@ -220,8 +220,9 @@ A future operation claiming durable logical reconstruction must additionally: The current durable segment, catalog, publication, and recovery surfaces do not yet form this consolidated high-level `BlobId`-to-writer contract. -Retention remains planned; no current retention surface protects the evidence -closure required by this operation. These lower-level surfaces must not be +Retention publication now records verified closures as generation-checked +roots, but nothing collects or fences yet, so no current surface protects or +releases the evidence closure this operation requires. These lower-level surfaces must not be described as an implemented durable logical reconstruction API. ## Current public evidence