diff --git a/CHANGELOG.md b/CHANGELOG.md index fd6d2f4..080ae98 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -242,6 +242,10 @@ after its public API and format compatibility policies are established. ### Changed +- Documentation refreshed after the version-two merge: the README, the + version-two overview status, the closure and recovery status lines, the + reconstruction contract's retention note, and the requirements ledger state + what is implemented and what remains planned in #19, #21, and #97. - The version-two retention and recovery pages each split their largest sections into `retention-publication.md` (closure admission and the generation transition) and `migration-recovery.md` (the one-way migration diff --git a/README.md b/README.md index 3a20ed0..51b601c 100644 --- a/README.md +++ b/README.md @@ -58,17 +58,27 @@ Keep is required to refuse all three, before mutating anything. - **Version-2 retention and migration, forward path.** Explicit retention roots, deterministic closure verification, a one-way 21-phase migration, and a 17-phase retention publication — all with production filesystem - writers, all preserving every version-1 byte. + writers, all preserving every version-1 byte. Reopening a migrated store + jointly admits its marker, intent, and receipt, binds the root's device, + mount, and inode identity to the intent, and pins the directories it + admitted. Publication binds this store's own catalog head and the catalog + it selects, and refuses retained stages, superseded candidates, substituted + files, replaced protocol directories, and every namespace or capacity + violation before it writes anything. Each refusal is a typed value, not a + string. ## What it does not do yet -Version 2 writes correctly from a clean start. It cannot yet pick up the -pieces if it dies partway through. Until it can, **version 1 is the only -store admitted for production.** +Version 2 writes correctly from a clean start and, if it finds the residue of +an interrupted publication, refuses rather than guesses. Nothing yet recovers +that residue, and readers have no fence, so **an interrupted version-2 +publication waits for a human until #19 lands.** A version-1 store stays +admitted until its owner migrates it; migrate only if you accept that wait. | Gap | Tracked | | --- | --- | | Restart recovery for retention publication and migration | [#19](https://github.com/flyingrobots/keep/issues/19) | +| Restart-stable root identity coordinate in the migration intent | [#97](https://github.com/flyingrobots/keep/issues/97) | | Reader fence binding one consistent catalog + retention snapshot | [#19](https://github.com/flyingrobots/keep/issues/19) | | Precise verification reports at explicit depths | [#20](https://github.com/flyingrobots/keep/issues/20) | | Garbage collection and identity-preserving compaction | [#21](https://github.com/flyingrobots/keep/issues/21) | @@ -208,7 +218,8 @@ has one job; this one is the front door. | Understand what is proved and what is not | [`docs/invariants/`](docs/invariants/) | | Read the byte-level formats | [`docs/formats/`](docs/formats/) | | See the architecture and port boundaries | [`docs/architecture/`](docs/architecture/) | -| Follow the crash and recovery rules | [`segment-store-v1/recovery.md`](docs/formats/segment-store-v1/recovery.md) · [`segment-store-v2/recovery.md`](docs/formats/segment-store-v2/recovery.md) | +| Follow the crash and recovery rules | [`segment-store-v1/recovery.md`](docs/formats/segment-store-v1/recovery.md) · [`segment-store-v2/recovery.md`](docs/formats/segment-store-v2/recovery.md) · [`segment-store-v2/migration-recovery.md`](docs/formats/segment-store-v2/migration-recovery.md) | +| See how a retention generation is published | [`segment-store-v2/retention-publication.md`](docs/formats/segment-store-v2/retention-publication.md) | | Check reproducible performance evidence | [`docs/benchmarks/`](docs/benchmarks/) | | Run the language-neutral corpora | [`conformance/`](conformance/) | | See what changed | [`CHANGELOG.md`](CHANGELOG.md) | diff --git a/docs/formats/segment-store-v2/README.md b/docs/formats/segment-store-v2/README.md index 7f65830..1f27b80 100644 --- a/docs/formats/segment-store-v2/README.md +++ b/docs/formats/segment-store-v2/README.md @@ -76,21 +76,29 @@ re-encode them. ## Status -The format contract is frozen by ADR-0009 and this specification. Public core -types now admit exact namespace bytes, namespace digests, root and liveness -generations, registered realization profiles, bounded closure policies, -reconstruction anchors, and semantic roots. Canonical root, manifest, and head -codecs match their independent golden records. Storage-independent transition -planning, deterministic bounded closure verification against one pinned -catalog, their combined preflight proof, and the exact 17-phase publication -vocabulary with a blocking storage capability port are available. -Storage-independent preparation derives exact canonical manifest and head -successors from coherent preflight and current-manifest evidence. Ordered -storage-port orchestration revalidates authority and returns a complete receipt. -Fresh writer-locked filesystem migration execution now publishes all canonical -fixed records and the exact empty version-2 namespace without changing -version-1 immutable bytes. Partial-prefix restart recovery, production -filesystem retention publication, immutable reader snapshots, and garbage -collection do not exist yet. Requirements still in progress in issue #19 or -issue #21 are not complete evidence. A store must refuse version-2 state until the relevant -corruption, model-based, crash-injection, recovery, and fuzz evidence exists. +The format contract is frozen by ADR-0009 and this specification. + +Implemented with executable evidence: public core types for namespaces, +generations, realization profiles, closure policies, anchors, and roots; +canonical root, manifest, and head codecs matching their golden records; +storage-independent transition planning, bounded closure verification against +one pinned catalog, preflight, preparation, and the 17-phase publication port; +fresh writer-locked filesystem migration through all 21 phases, refusing a +version-one store that still holds a retained stage; +`FilesystemVersionTwoAdmission::reopen`, which jointly admits the marker, +intent, and receipt, binds the root's device, mount, and inode identity to the +intent, and pins the retention directories it admitted; and +`FilesystemRetentionPublicationAuthority`, which publishes initial and +successor generations against the observed head, binds this store's catalog +head and the catalog it selects, and refuses superseded candidates, retained +stages, replaced protocol directories, and every namespace or capacity +violation before mutation, each as a typed `RetentionCurrentStateRefusal`. + +Not implemented: retention publication recovery and `KEEP-CRASH-036..052` +process-death evidence, partial-prefix migration recovery and +`KEEP-CRASH-053..073`, the reader fence, model-based transition evidence, and +garbage collection. Issue #19 owns the first four and issue #21 the last; +issue #97 owns the restart-stable root identity coordinate. A version-1 store +remains admitted until its owner migrates it, and the +[requirements ledger](requirements.md) is the authority on which requirements +are proven. diff --git a/docs/formats/segment-store-v2/closure.md b/docs/formats/segment-store-v2/closure.md index 6e1f0e1..3e0e79b 100644 --- a/docs/formats/segment-store-v2/closure.md +++ b/docs/formats/segment-store-v2/closure.md @@ -1,7 +1,9 @@ # Closure Verification - Status: Normative version-2 protocol; storage-independent verifier - implemented; publication integration planned in issue + implemented; publication binds this store's catalog head and the catalog it + selects to the verified closure; member re-verification under filesystem + authority is planned in issue [#19](https://github.com/flyingrobots/keep/issues/19) - Format coordinate: `keep.segment-store/v2` - Requirement: [`KEEP-RETENTION-005`](requirements.md#retention-transitions) diff --git a/docs/formats/segment-store-v2/recovery.md b/docs/formats/segment-store-v2/recovery.md index c389940..efa539b 100644 --- a/docs/formats/segment-store-v2/recovery.md +++ b/docs/formats/segment-store-v2/recovery.md @@ -174,8 +174,9 @@ recovery instead. Direct version-2 initialization is undefined. The exact offsets and fixtures are requirement `KEEP-MIGRATION-002`. The fresh writer emits only those canonical records; success is not restart evidence. -Version 2 remains unavailable as production until partial-prefix recovery and -`KEEP-MIGRATION-007` process-death evidence exist. +A migrated store is admitted for forward publication, but partial-prefix +recovery and `KEEP-MIGRATION-007` process-death evidence remain absent, so an +interrupted migration waits for recovery instead of continuing. ## Retention publication recovery diff --git a/docs/formats/segment-store-v2/requirements.md b/docs/formats/segment-store-v2/requirements.md index ea516fd..ae8100b 100644 --- a/docs/formats/segment-store-v2/requirements.md +++ b/docs/formats/segment-store-v2/requirements.md @@ -15,7 +15,7 @@ case is not evidence. | `KEEP-RETENTION-004` | Retain and release compare expected and observed generations and publish exact successors only | unforgeable readiness and preflight proofs in `tests/retention_transition.rs` and `tests/retention_preflight.rs`; exact successor preparation and complete receipt evidence in `tests/retention_publication_preparation.rs` and `tests/retention_publication_execution.rs`; writer-locked initial filesystem publication in `filesystem_retention_storage_tests`; observed-head successor publication, exact predecessor binding, and absent-head refusal in `filesystem_retention_successor_tests`; the store's catalog head must name the closure's catalog generation and digest before any forward write in `filesystem_retention_catalog_tests`; a head whose predecessor disagrees with its manifest refuses in `filesystem_retention_current_tests`; a successor reopens and decodes the manifest-selected predecessor root and refuses an absent or changed one in `filesystem_retention_expectation_tests` | Implemented | | `KEEP-RETENTION-005` | Closure derivation is deterministic, bounded, cycle-safe, fail-closed, and verifies complete blob reconstruction | exact accounting, reconstruction, adversarial-catalog, and exhaustive model laws in `tests/retention_closure.rs`; corrupt members refuse through the inherited segment-record admission laws and seeded `segment_format` fuzz target routed by `closure-corruption.md` | Implemented | | `KEEP-RETENTION-006` | Publication follows the exact ordered durability protocol, including new namespace-directory admission and retention of fixed-stage evidence until head commit, and returns only after cleanup synchronization | typed vocabulary and blocking port in `tests/retention_publication_phase.rs` and `tests/retention_publication_storage.rs`; ordered execution, conditional namespace sync, and all 17 exact storage-fault boundaries in `tests/retention_publication_execution.rs`; production 17-phase forward filesystem execution, exclusive staging, byte-equal inode-substitution refusal, and retained-stage recovery refusal in `filesystem_retention_storage_tests`; orphan namespace directories count against the 4,096 ceiling and refuse a new namespace before any stage is written in `filesystem_retention_capacity_tests`; crash injection remains | In progress in #19 | -| `KEEP-RETENTION-007` | Restart resolves every fixed-stage crash prefix to one documented lawful state or typed ambiguity | recovery-required refusals before any mutation in `filesystem_retention_expectation_tests`: an absent head over populated pools, a non-initial head prepared against an absent head, an orphan directory for a namespace expected absent, and an absent directory for a namespace expected current; debug and release crash matrix remains | In progress in #19 | +| `KEEP-RETENTION-007` | Restart resolves every fixed-stage crash prefix to one documented lawful state or typed ambiguity | recovery-required refusals before any mutation in `filesystem_retention_expectation_tests`: an absent head over populated pools, a non-initial head prepared against an absent head, an orphan directory for a namespace expected absent, and an absent directory for a namespace expected current; debug and release crash matrix remains; replaced protocol directories, an absent or changed head-selected catalog, an over-full census, zero-generation pool names, and a stage retained by a failed write refuse in `filesystem_retention_*_tests` | In progress in #19 | | `KEEP-RETENTION-008` | Readers double-collect catalog and retention heads and bind one complete catalog, manifest, and root-generation view under a `ReaderFence` | immutable snapshot and concurrency tests | Planned in #19 | | `KEEP-RETENTION-009` | Exact already-committed retry is idempotent only while its successor remains current | byte-identical planning in `tests/retention_transition.rs`; authority-revalidated zero-mutation retry receipt in `tests/retention_publication_execution.rs`; exact already-committed filesystem retry with a byte-identical retention witness in `filesystem_retention_storage_tests`; superseded-candidate filesystem refusal with zero mutation in `filesystem_retention_successor_tests`; committed retry reopens the head-selected manifest entry and root pool bytes, refusing absent, changed, or corrupt evidence in `filesystem_retention_current_tests`; every refusal is a typed `RetentionCurrentStateRefusal` source, with superseded, committed-root-absent, committed-root-changed, and head-absent-with-artifacts pinned by downcast | Implemented | | `KEEP-RETENTION-010` | Model operation sequences agree with a deterministic namespace-to-anchor-set map and never admit caller identity, paths, clocks, or application policy | model-based and source-architecture tests | Planned in #19 | @@ -30,7 +30,7 @@ case is not evidence. | --- | --- | --- | --- | | `KEEP-MIGRATION-001` | Exact version-1 stores remain admitted until a durable migration artifact exists | compatibility fixtures | Planned in #19 | | `KEEP-MIGRATION-002` | Format marker, intent, and receipt have complete fixed byte tables, named domains, bounds, checksums, deterministic store identity, and exact initial-state digests | exact admission in `tests/store_format_marker.rs`, `tests/store_migration_intent.rs`, and `tests/store_migration_receipt.rs`; canonical construction in `tests/store_migration_intent_encoding.rs` and `tests/store_migration_receipt_encoding.rs`; seeded `migration_format` fuzz target | Implemented | -| `KEEP-MIGRATION-003` | Migration revalidates version-1 head, catalog, pools, root identity, and writer authority before mutation | bounded canonical pool inventory in `tests/store_migration_inventory.rs`; writer-locked filesystem pool admission in `filesystem_inventory_*_tests`; exact authority observation and drift refusal in `filesystem_migration_authority_tests`; verification-first execution in `tests/store_migration_execution.rs`; fresh filesystem integration and post-publication drift refusal in `filesystem_migration_storage_tests` | Implemented | +| `KEEP-MIGRATION-003` | Migration revalidates version-1 head, catalog, pools, root identity, and writer authority before mutation | bounded canonical pool inventory in `tests/store_migration_inventory.rs`; writer-locked filesystem pool admission in `filesystem_inventory_*_tests`; exact authority observation and drift refusal in `filesystem_migration_authority_tests`; verification-first execution in `tests/store_migration_execution.rs`; fresh filesystem integration and post-publication drift refusal in `filesystem_migration_storage_tests`; a version-one store still holding a retained stage refuses before the intent is observed in `filesystem_migration_storage_tests` | Implemented | | `KEEP-MIGRATION-004` | Every partial migration prefix continues idempotently under writer authority | state-machine and recovery tests | Planned in #19 | | `KEEP-MIGRATION-005` | Unknown, out-of-order, substituted, corrupt, conflicting, or changed evidence is unrecoverable ambiguity | forward-execution stage preservation, byte-equal inode-substitution, out-of-order-prefix, and post-publication drift laws in `filesystem_migration_storage_tests`; unknown `retention` entries, non-digest namespace directories, and noncanonical pool names refuse before any retention stage is written in `filesystem_retention_namespace_tests`; restart corruption and mutation matrix remains | In progress in #19 | | `KEEP-MIGRATION-006` | Migration never rewrites or deletes admitted version-1 immutable bytes | exact segment, catalog, and head before/after witness in `filesystem_migration_storage_tests`; restart-path evidence remains | In progress in #19 | diff --git a/docs/invariants/authenticated-reconstruction/README.md b/docs/invariants/authenticated-reconstruction/README.md index 79910b0..98430be 100644 --- a/docs/invariants/authenticated-reconstruction/README.md +++ b/docs/invariants/authenticated-reconstruction/README.md @@ -220,8 +220,9 @@ A future operation claiming durable logical reconstruction must additionally: The current durable segment, catalog, publication, and recovery surfaces do not yet form this consolidated high-level `BlobId`-to-writer contract. -Retention remains planned; no current retention surface protects the evidence -closure required by this operation. These lower-level surfaces must not be +Retention publication now records verified closures as generation-checked +roots, but nothing collects or fences yet, so no current surface protects or +releases the evidence closure this operation requires. These lower-level surfaces must not be described as an implemented durable logical reconstruction API. ## Current public evidence