feat!: establish runtime ownership and durable application history - #28
Merged
Conversation
GCdePaula
force-pushed
the
feature/review-ledger-and-tracks
branch
2 times, most recently
from
August 24, 2026 12:00
a8c23eb to
5233880
Compare
GCdePaula
force-pushed
the
feature/review-ledger-and-tracks
branch
2 times, most recently
from
September 9, 2026 20:42
35596a4 to
8c58d4a
Compare
…story foundation One process, one data directory, one way in: this lands the authority boundary the 2026-08 ADR designed, with admission derived from durable facts and history given real coordinates. Process ownership and containment: - Kernel-enforced one-process-per-data-dir (runtime/process_lock.rs); the controller retains the lock through settlement, nested blocking work retains clones until it actually stops. - RuntimeScope: the runtime-lifetime capability (sticky containment OnceLock, independent two-second terminal-abort watchdog, best-effort fault recorder), constructible only from a held lock. - The Authorized externalization token: acks, L1 sends, WS frames, and snapshot-stream starts require a token minted by consulting containment — forgetting the check is a compile error, not a convention. Commands and the exit-code contract: - commands/ owns the operator brackets (run + worker supervisor, setup + fill, flush); runtime/ is the capability substrate consumed crate-wide. - One CommandError taxonomy with the R4 exit projection (10 expect recovery / 20 transient / 30 terminal, do not restart, page / 40 setup needs recovery); terminality declared beside each worker error type. Fact-derived admission (reviews L2/L3): - Admission is three facts, each with one owner: the process lock, two-sided setup_complete, and the absorbing canonical_divergence marker. No lifecycle state machine, no operator acknowledgement; standard recovery is automatic and cockroach rebuild is the one manual path. - The only durable telemetry is the terminal_faults black box: append-only terminal-cause rows, written best-effort and verdict-neutrally. - Boot is prepare -> admit (the same pure reducer re-run over one consistent fact set) -> single-use AdmittedRuntime -> non-yielding launch. Startup recovery: - A pure reducer over one transactionally consistent inspection selects at most one phase; every completed phase returns to inspection; flush/sync witnesses are boot-local and never persist. - admission.tla proves terminal dominance, one-phase-per-inspection, witness requirements, crash/restart soundness, and capability soundness (TLC: 860 generated / 266 distinct / depth 13 / 0 violations). Durable history foundation: - (EraId, RecoveryGeneration, K) coordinates, each recorded at the moment it happens; the executed_inputs canonical projection; setup completion as a synchronous=FULL linearization point; snapshot lifecycle hardening (lease reset, GC, orphan sweep, promotion restamp). - sequencer-core: one shared typed execution boundary writes the consensus coordinate; raw hooks are structurally unreachable; app-boundary decode and genesis paths return typed refusals.
The documentation practice gains a lifecycle, now recorded in AGENTS.md
("Documentation Practice"): living docs are timeless — present tense,
reasoning inline, no dates, no amendment banners, no review codenames —
while history lives only in docs/review/ and commit messages, and a review
ledger is distilled when it closes. Conclusions with reasoning outlive the
path taken to them. Plus the comment rule: comment the non-obvious, never
restate what the code expresses.
- docs/review/register.md is the new hub: open findings (statuses verified
against the tree), owed tests with recipes and harness levers, open
maintainer decisions, settled decisions with each reasoning's current
home, refuted proposals (do-not-re-propose), and a historical codename
map so older commit messages stay decodable. CLAUDE.md/AGENTS.md point
there.
- All eight dated ledgers distilled to stubs or compact decision records
(docs/review/ 2,392 -> 502 lines); full originals remain in git history.
- The authority-boundary ADR rewritten present-tense (769 -> ~210):
context, the four mechanisms, rejected alternatives with their
arguments, revisit conditions; cutover chronology and raw benchmark
tables reduced to their conclusions. The superseded terminal-containment
plan is deleted; the coordination-tracks doc is rewritten to current
truth (Track 5's LUT rationale kept in full).
- Living docs scrubbed of tense leakage; the concepts formerly cited as
review codes now have real names homed in invariants.md, which also
gains the do-not-simplify list and I9's content-equal => effect-equal
argument. ~90 codename citations removed from code comments and the
schema.
- Adversarial review of the distillation (24 agents, three lenses):
23 confirmed findings fixed. The lost-content lens restored two real
casualties with updated homes: cockroach recovery's flush is best-effort
by construction (no watermark survives the wipe; the fail-safe
flush-floor option recorded) -> cockroach.md step 2 + register; and the
supervisor recipes (systemd RestartPreventExitStatus; on Kubernetes the
crash-loop bound is exit-code alerting, since Deployments cannot honor
exit codes and there is no boot gate) -> the operator runbook.
L2TxFeedConfig::default() set batch_submitter_address to None — "filter nothing", the I11-violating value — so the feed a test or future caller got by default fanned our own batch envelopes out to WS subscribers. Production was correct only because the runtime glue remembered to override the default. The address is now a required constructor argument and the Option is threaded out end to end: the filter compares plain addresses (matching the inclusion lane's spelling), and the unfiltered SQL arm in storage/egress.rs — reachable only through the deleted default — is gone. Fixtures that seed no own-batch rows pass a sentinel address that collides with no seeded sender; Address::ZERO is a real fixture sender and must not be used for that.
…he launch bracket
Simplification pass over the run-command startup glue, adversarially
reviewed: two verification rounds over the proposals and a fidelity
review over the landed diff. Behavior-preserving; operator-visible
deltas are limited to log fields (the "listening" line reports the
resolved bound address; drain warnings carry a structured `phase`).
- L1Config carries the pinned DeploymentIdentity verbatim, built once in
run(): exactly one route to identity below the gate. The poster's
start_block round-trip through the reader is gone, and the test
fixture builds one identity literal instead of hand-copied fields.
- PreparedRuntime is now exactly the launch-argument bundle: config
structs die in prepare (lane_config, api_config, snapshot_state,
bound_addr in; run_config, l1_config, db_path, dumps_dir out), and
launch is six spawns plus the "listening" log.
- The EIP-712 domain is derived in prepare from l1_config.identity —
one source, now that the identity travels whole.
- ApiConfig owns the deployment-varying ingress values (domain, payload
bound) via a mandatory constructor; the service limits stay module
constants, documented as deliberately not operator-tunable.
start_on_listener drops to six parameters.
- AdmittedRuntime is collapsed into launch(self, RuntimeAdmission): the
witness is the mechanism, the wrapper was presentation. Vocabulary
swept through AGENTS.md, the ADR, invariants, the recovery README and
admission.tla comments; the spec's admittedRuntime state names are
deliberately kept (TLC unchanged: 860/266/depth 13/0).
- finish's two drain loops merge into one two-phase loop with a single
precedence match (contained > primary > signal > first drain error).
The register's drain-merge refutation was scoped to the naive
re-awaiting sketch ("as sketched" in the 2026-08-18 source); the
entry is scope-narrowed with the source cited.
- select_first_exit destructures Self exhaustively, closing the one
per-worker site a seventh worker could silently skip.
- Startup snapshot hygiene moves to commands/run/startup_hygiene.rs;
its five order-critical steps were mis-documented as four.
- Fee-oracle bootstrap: RunFeeOracleBootstrapError deleted (exit codes
proven unchanged; double-prefixed misconfig messages fixed); the
transient connect arm reuses the provider it already built. The
previously untested Some-limb drain path gets a pin.
- The WORKER_LAUNCH_COUNT probe and its global test mutex are deleted:
the ProcessLock::acquire assertion is the stronger behavioral
detector, and the tests it serialized are now independent.
The submitter's signing key rode through three Debug-derived structs (KeyArgs inside RunConfig/FlushConfig, and L1Config) in plaintext hex — one future ?config log away from leaking. It now enters the process as SubmitterKey at the clap value_parser: Debug prints [redacted], no Display exists (a %key log line is a compile error), and the raw hex is reachable only through expose_secret(), whose call sites are the provider builders and address derivation. A test pins that a carrier struct's derived Debug never prints the secret. No public-address accessor: the key's public identity is the pinned identity.batch_submitter_address beside it, already in the startup log. Closes the register's Debug-derive open finding; the startup log's full RPC URL (token-bearing per the help-leak test's threat model) is a recorded, deferred tension.
Require and persist a fresh fee quote during setup, then retain the last price across transient runtime failures. Keep the process lock through cancellable feed preparation and make E2E mining follow the safe-block clock without synthesizing future L1 time.
Couple stopped-process outage time with L1 progress, update the backward-clock scenario to assert retryable admission denial, and keep the live stuck-Tip injection recoverable after its detector exit.
Add the dated ledger for the pre-merge stock-take of this branch: the verdict (proportionate; three residue pockets), the facts verified first-hand, the eight jury-confirmed and ten jury-refuted simplification proposals with their evidence, and the fifty-four proposals no jury examined, marked as such. The register gains findings 19–31, two owed-test entries, a 2026-09-03 refuted block with an evidence line per entry, and two hygiene fixes the audit found in the register itself: the fee-price-age refutation is re-homed under the commit that made it, and the boot-gate refutation's wording is narrowed to the argument it rests on.
… the log line The rollups-e2e job was red on `aging_open_tip_runtime_danger_zone_exit_test` for a deterministic reason: the assertion grepped the child's log for `status=TipInDanger(`, but tracing-subscriber styles field names and the `=` with ANSI escapes whenever `NO_COLOR` is unset (no TTY check), and the harness passes the parent environment through. It passed only in shells that export `NO_COLOR`. Both wallet-sequencer binaries now enable ANSI only when stdout is a terminal — the production fix: a daemon writing to a pipe, a file, or journald must not interleave escape codes into its operator log. The scenario asserts the exit code instead: 10 is the observed-danger class (`TipInDanger` / `ClosedBatchInDanger`), a clock fallback would exit 20, and the scenario has no closed batch — the same discrimination the log grep was reaching for, on a documented contract rather than a rendered line. Verified locally: the scenario passes and the log file carries no escape bytes.
Four independent reviews found the same defect class on this branch: comments that claim more than the code enforces. In a codebase where comments are the audit surface, an overstated guarantee is a missing check. - `authorize()`, the ADR, and the register's settled entry claimed four compile-forced externalization primitives; three functions take the token (ack, L1 send, WS emit). The snapshot-stream start, the `POST /tx` success body, and the lane's two mutation commits are hand-placed consults bounded by the exit contract, and now say so. - `Authorized`'s doc claimed a consult "at this effect boundary"; the token is `Copy` and lives for the scope borrow, so it proves a consult at some point in that borrow — the ADR's bounded lag, stated where a reader meets it. - The flush witness was called "non-clone" in the recovery README, the TLA header, and a test comment; `RecoveryProgress` derives `Copy`. The property is that it is boot-local. - The process-lock witness comment said a live witness means runtime work has not drained; it also covers the controller's own clone through settlement. - Three "journal" sites missed by the black-box rename, one dated rename history and a removed "acknowledge" command in the error module's header, a module doc arguing with a hypothetical simplifier, and a recovery README naming a `DangerDetectorExit` type that does not exist. - `preparation_outliving_clean_facts_cannot_launch` never calls `launch`; it asserts that final admission refuses over aged facts, and is now named so.
AGENTS.md's documentation practice keeps review-item codenames out of code comments and living docs; the distillation commit removed ~90 citations and missed fifteen. Each is replaced by the reason itself or by the invariant id it stood for (R1a → I14, R2 → I9/I15, "review R4 class 10" → the exit-code class), or by the plain word "regression" where the code was only a label. Three of the fifteen were added by this branch (L2, H6, D10); the rest predate it.
Five `Storage` methods were `pub` with only test callers, one of them an unguarded batch-tree mutation sitting beside its guarded replacement: `ensure_open_tip` (production opens the Tip through the reducer's guarded `EnsureOpenTip` phase), `close_frame_and_batch` (production closes through the pending-snapshot twin, I7), `latest_batch_index`, `ordered_l2_txs_for_batch`, and `promote_finalized` (production promotes inside the drain transaction, I6). All five are now `#[cfg(test)] pub(crate)`; their intra-doc links and the snapshot lifecycle doc name the production paths instead. Gated rather than deleted: the two index helpers have callers across three test modules. This is the benefit the in-crate integration-test move was meant to unlock.
The finality wait timing out and resubmitting is the mempool flush's ordinary rhythm on a slow chain, not a fault; it was logged at `error!`, so every healthy recovery boot paged.
The comment claimed a `debug_assert` that does not exist. The high limbs are dropped without a check; that is sound because every caller's inputs are table entries or partial products for an exponent already bounded by `MAX_EXPONENT` in `fee_to_linear_fixed`, and partial products never exceed the final value. The contract half of the fee-determinism finding (the LSB-first floor order) stays open.
…body
The one `POST /tx` 500 path that still echoed internals — an application
`Internal { reason }` or I/O error during execution — now answers with the
fixed "application internal error" the other internal paths already use.
The reason still travels on the lane's `ExecuteUserOp` error and the log,
which the test now pins alongside the fixed client message.
`ApplicationProgress::new` panicked on an incoherent pair and had only test callers; its own doc said to use `try_new` on the one path that constructs a pair from data. `try_new` is now the single constructor (a genesis instance starts from `Default`), tests use it with `expect`, and the should-panic test became a direct assertion on the fallible form.
The ledger gains a "Landed" section so a later session resumes from the tree. The register's findings keep stable numbers: closed entries (4, 5, 17, 21, 25, the comment half of 8, the gating half of 10) are reduced to one-line closure notes rather than deleted, so the ledger's citations stay valid.
…ites the black box once `RuntimeScope` loses the in-scope fault recorder: the `FaultRecorder` alias, the `OnceLock` field, `set_fault_recorder`, the recorder call inside containment, and its installer in `PreparedRuntime::prepare`. Containment is now three non-blocking steps — CAS-elect the cause, arm the two-second abort watchdog, request shutdown — and the "arm the watchdog before recording, either may block" ordering hazard goes with the second SQLite writer that caused it. The command bracket's settlement write (`record_terminal_fault_best_effort`, which already existed as the second of two writers) is the black box's one writer, so a contained run records one row, not two. The accepted loss, stated in the runbook where an operator reads it: any death that does not return through its bracket — an abort at the two-second deadline, a controller panic, SIGKILL — leaves only the process logs, and the single write has no second attempt. The row is telemetry; restart policy is the exit code, which is untouched. `run` gains a verdict-neutral startup read: `warn_on_previous_terminal_fault` logs the latest black-box row once, ahead of the admission preflight so a refused boot still says why the last one died; nothing branches on it, and a missing or unreadable table is a debug line. The black box thereby gets its first in-product reader. Tests: the two recorder-only tests are deleted; containment's first-reporter election is pinned without a callback, and the settlement write is pinned directly (a transient verdict leaves no row; a terminal one lands the error's display form). The invariants check policy, AGENTS.md, the ADR, the recovery README, the admission model's comments, and the runbook are restated to match. Closes register finding 24.
…n-optionally
`acquire_finalized_lease` returns `FinalizedLease { inclusion_block: u64,
dump: LeasedDump }`; `LeasedDump` no longer carries an `Option<u64>` that was
`Some` for one query and `None` for the other. `finalized_state` destructures
the lease, and its branch on an impossible `None` — which escalated a type
artifact to terminal containment, against the check policy's "no
Option-handling for can't-be-None" — is deleted. The column is `NOT NULL`
with a CHECK at the engine, and corrupt-row containment is unchanged (the
persistent-storage classifier and the decode panic, as the existing test
pins). Wire behaviour of `/finalized_state` and `/latest_snapshot` is
unchanged: headers, ETag, the 304 path, the 404, and the guard-after-commit
ordering. Closes register finding 26.
…its postcondition
`ensure_open_tip_for_recovery` guarded two conditions with one disjunction,
so an already-open Tip surfaced as `StaleDecision { expected: Safe, actual:
Safe }` — a line the test pinned as intended. The two checks are now
separate: a non-`Safe` danger is still `StaleDecision`; an already-open Tip
is the payload-free `TipAlreadyOpen`, paired with a retry reason so
`classify_mutation` carries it to the operator (retry, exit 20; a restart
inspects, finds the Tip, and admits).
The same phase is the only edge back into `Repaired` without a Tip, so it
is the reducer's one cycle, and no watchdog exists on the boot path. It now
re-reads `has_valid_open_batch` inside its own transaction after opening and
returns `TipMissingAfterOpen` — classified `refuse`, exit 30 — rather than
commit without one. A typed refuse, not a `debug_assert` (compiles out) and
not a retry (would relocate the spin into the supervisor). `drive_recovery`'s
doc records the termination argument: at most five phases per attempt, the
process lock making this the only writer between them. `admission.tla`'s
`hasOpenTip' = TRUE` is now an enforced postcondition, and says so.
A fault-injection test invalidates the new Tip the moment its first frame
lands (invalidating on the batch insert is caught one step earlier by the
schema), so the opener returns `Ok` with no valid open Tip, and asserts the
refuse plus full rollback. Closes register findings 20 and 23.
…on half `DangerDetector`, `InputReader`, and the fee-oracle worker used their `RuntimeScope` for exactly one thing — `wait_for_shutdown` — and each already holds a construction-required `ProcessLock` clone for its data-directory work. They now take `ShutdownSignal`, the scope's notification half; `PreparedRuntime::launch` passes `shutdown.signal()` for the three and the scope to the lane, HTTP server, and submitter, which externalize or contain. Nothing the scope carried is lost for them: the lock clone they own is a clone of the same descriptor the watchdog's weak witness observes, so a hung one still aborts at the deadline; the signal is a clone of the same notification state containment pokes, so a contained fault still stops them; and their terminal exits were always classified by the supervisor, never by worker-local containment. Their tests construct a bare signal instead of the default test scope, which leaked one temp-dir lock per use. The prose that claimed every worker held a scope clone is restated where a reader meets it: the `ShutdownOnDrop` doc, the launch-boundary test comment, the process-lock module doc, the scope's own rule, and the ADR's mechanism 1. A pre-existing broken intra-doc link in the shutdown module is fixed while the file is open. Closes register finding 22.
…failure one verdict
A missing, unreadable, or non-text batch-submitter key file now exits 30
like bad key content one call later, instead of restart-looping at 1;
environmental I/O still exits 1. The read returns a typed
`BootstrapError::KeyFile { path, source }` whose message names the path
and never the contents. Both halves are pinned, including the EIO kind
an operator actually hits.
`RecoveryFailure::Provider(String)` carried two verdicts, decided by its
construction site. It is split into `ProviderUnreachable` (retry) and
`SignerMisconfig` (refuse), classified at birth by
`classify_signer_provider` and pinned, payload and polarity, against the
`BootstrapError` projection it must agree with. `classify_input_reader`
now documents what it can actually receive (only the sync's
`create_provider` produces `Bootstrap` here; discovery-time facts are
`setup`-only) and why `Bootstrap` and `Join` flip polarity between the
startup phases and the live worker.
Register: findings 19 and 27 closed; finding 32 opened (the same L1
misconfiguration exits 1 under `setup` and 30 under `run`; recorded, not
fixed); the flatten-`RecoveryError` refutation annotated now that its
two-verdict premise is gone; drifted cites retargeted.
…alues The five per-class exit-code tests, the verdict test, the two startup- reader tests, the fee-oracle fatal-math test, and the two app-bootstrap tests fold into five class functions and one table test that asserts, for every row, the class and that the black-box verdict agrees with it (`is_terminal` exactly for class 30). Every shape, reason string, and rationale comment survives; two rows are new (`ProviderUnreachable` under retry, `SignerMisconfig` under refuse). A second test pins the five verdicts to the integers 10/20/30/40/1 through an exhaustive match, so a renumbered constant or an unnumbered sixth verdict fails here, not in a runbook. Two wire-value pins outside the table: `run` on a never-set-up data directory dispatches to 30 through the real command bracket, offline by construction (the admission preflight refuses before the key is read or a provider is built); and the harness gains an opt-in `stop_expecting_clean_exit` that asserts SIGTERM→0 on a healthy sequencer, used at the healthy stop of the stale-batches scenario. The runner's per-scenario shutdown stays lenient on purpose. Register: the owed exit-code test keeps only its per-variant e2e half; the `classify_input_reader` polarity pins landed with the previous commit.
Make AGENTS.md an architecture map and give the protocol contracts, invariant register, recovery guide, and ADR explicit ownership of their facts. Align documentation claims with the code and explain non-obvious behavior in module comments. Consolidate the review ledgers into the history register and record the settled decisions and validation from the documentation review.
Rely on SQLite's contiguous-offset trigger as the transactional enforcement point. Remove the duplicate Rust pre-query and validation loop while preserving fail-loud behavior for invalid history.
Wait for the current blocking append before returning a clean reader exit. This lets the worker observe committed divergence and append failures, and keeps process ownership held until the write has actually stopped.
Replace the phase driver and its mirrored progress state with an explicit startup sequence. Preserve guarded repairs, flush then post-flush sync before cascade, retry/refusal classification, and fresh admission after preparation. Update the admission model and tests around those ordering requirements.
Stop a diagnosed terminal runtime fault immediately. Ordinary operator, recovery, and operational shutdown still notify workers and drain their work. Remove the containment tokens and terminal-watchdog machinery whose purpose was to keep a partially failed runtime alive during that drain. BREAKING CHANGE: diagnosed terminal runtime faults abort with SIGABRT (usually shell status 134), without graceful drain or durable command settlement, rather than returning to an embedding caller. Supervisors must treat this as terminal. CommandError::StorageInvariantViolation is removed.
…eation Let the application own its persisted progress and return it by value. Shared execution preflights overflow and checks the exact successful transition. Keep Send, remove unused Clone and Sync bounds, and separate canonical inspection from native execution. Clarify durable, immutable checkpoints and independent restores for file or directory layouts. Collapse the inclusion lane's duplicate turn outcomes and drain-range state, and combine frame closure with optional promotion in the same transaction. Preserve ordering, rejection semantics, canonical bytes, and commit-before-ACK. BREAKING CHANGE: Application implementations provide progress() by value; apply hooks advance it themselves. Execution capabilities and the mutable progress accessor are removed. Validation returns Result<ValidationOutcome, AppError>, create_dump takes &mut self, and canonical_snapshot_bytes belongs to CanonicalState. The generic export_state method is removed.
Scope CORS to POST /tx and use Lua 5.4 consistently across the development, CI, and watchdog tooling. Preserve the existing Lua executable override. Record application integration validation and environment limits in the review notes, and correct the application-progress invariant link.
GCdePaula
force-pushed
the
feature/review-ledger-and-tracks
branch
from
September 10, 2026 15:31
8c58d4a to
ef5f640
Compare
GCdePaula
marked this pull request as ready for review
September 10, 2026 15:32
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The sequencer must stop issuing soft confirmations after a diagnosed terminal fault and recover from durable facts before serving again. This change makes that boundary explicit, simplifies startup recovery, and gives native applications a direct ownership contract without changing canonical transaction ordering.
Changes
Applicationown and report progress by value. Shared execution verifies successful count/clock transitions. Distinguish validation rejection from fatal engine failure; retainSendand remove hostClone + Syncrequirements.CanonicalStatetrait.Breaking changes
SIGABRT. Treat it, like exit 30, as requiring intervention rather than automatic restart.runtimeintocommands;L1Configmoves intol1, andCommandErrorreplacesRunError.Validation and review
Local workspace checks, formatting, strict Clippy, 697 Rust tests, and 62 watchdog tests passed. Restart/replay E2E passed. The local recovery E2E reached the resumed finalized snapshot but could not complete watchdog initialization because the installed Lua binding expected a newer Cartesi Machine archive.
Full CI passed, including Rust checks/tests, canonical guest tests, rollups/watchdog E2E, and watchdog Docker smoke tests. That run tested
35596a4; its Git tree is identical to the pushed8c58d4aafter commit rewording and squashing. The new CI run for the rewritten head is in progress.The reference C bridge and its fallible genesis-factory API are on a separate integration branch. Private DEX engine/scheduler conformance remains unverified; scheduler injection and the public history-version/feed API remain follow-up work.
Suggested review order: Application contract, invariants, runtime authority, recovery, then their implementation and tests.