feat(proof): generic dynamic RLM engine (DB-backed rules, per-topic VM boundary, artefacts, fail-closed runner registry) - #243
Merged
Conversation
echobt
marked this pull request as ready for review
September 8, 2026 13:11
Greptile SummarySummaryThis PR adds a generic, database-backed Proof RLM workflow with topic lifecycle management, runner registration, artifact persistence, executor planning, and automatic promotion. It also carries submitted artifact locators and runner-measured FLOP usage through custom evaluation, rejects missing or excessive measurements, serializes promotion through persistence, and requires a valid signed, baseline-bound topic before opening it. Confidence Score: 5/5Safe to merge; there are no outstanding blocking issues. No new correctness issues remain. The previously reported issues are fully addressed in the current implementation: custom submissions require and forward an artifact locator, runner-measured FLOP usage is required when budgeted and enforced against both the topic budget and declared cap, promotion protection remains in place through persistence, and sealed topics can open only after status, validation, signature, baseline-binding, and lifecycle checks pass. Reviews (2): Last reviewed commit: "fix(proof-rlm): hold the measured flops ..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Proof topics are the challenge; the code is not. The signed topic schema
gains only generic, shape-checked bindings whose values come from the
document:
- constraints.{firecracker_required, model_pin (vendor/model[:tag]),
task_slice (opaque), params (<=32 opaque slug -> value)}
- checklist: [{id, text}] anti-cheat rule vector (<=64, unique slug ids)
- eval_executor.{require_offer_commitment (64 hex), max_proof_deadline_s}
mirroring the inference judge pattern; the deadline is tighten-only
against a new pin knob max_proof_deadline_s (crate lock 21600 s)
- metric.custom_id is topic data: any [a-z0-9][a-z0-9_-]{1,63} drafts,
and a custom topic may open only when a runner is registered under its
id on the host (validate takes the registered set). The compiled-in
custom id list is gone.
proof-canon is a new tiny crate holding canonical JSON and the id shapes
shared by every Proof crate (moved out of proof-task, which sits near the
LOC cap). proof-score exposes relative_win / novelty_bar for reuse.
Signed topics need re-signing (the signing payload carries the new
default fields), as with the inference addition.
Drop the static supported_custom() list. Custom metric ids now come from the host's live scorer (LiveScorer::custom_ids, registered_custom): a topic mints its id, a runner registered under it makes the topic scorable. FamilyMux routes the whole custom family to one pluggable scorer and everything else to the digest-pinned harvest; with no custom scorer, every custom topic is RunnerUnwired (503), never a harvest fallback. LiveScorer gains ready_for_topic (per-topic refusal before any row or rent), custom_ids, and async auto_promote / on_persisted hooks so a family scorer can crown a pass and write artefacts once the row has its id. EvalError::RunnerUnwired names the missing runner as the root cause.
Core types for Proof as a dynamic agentic challenge system. Nothing here names a benchmark, metric, model, rule list, or repository (a self-check test scans the non-test source for such literals). - RuleSet / Checklist / SpendToken: rules are a versioned vector (v1 = the signed topic's checklist, later versions written by the RLM). A checklist binds to a rule version and digest and is green only when every rule is ticked with evidence and passes; the token has no public constructor, so no paid run can be reached from a red, incomplete, or stale checklist. - Lifecycle: draft -> owner_presend -> awaiting_owner_keys -> provisioning -> baselining -> open <-> evaluating -> promoting -> closed as an explicit table, with an askUser-style OwnerHook (refuses without a hook) and a presence-only owner key probe. - CustomRunner + RunnerRegistry: custom_id -> runner, empty by default; an unregistered id is RunnerError::Unregistered (the 503 root cause). - TopicVmOrchestrator: create / attach / run / teardown for per-topic VMs; VmJobs carry public data only (never a host path, key, or origin). UnwiredVmOrchestrator refuses and names the env vars; the generic VmBackedRunner turns inspect / evaluate into VM jobs and is only ever registered by an operator. - decide_promote: pass + green checklist + relative win over the bar, direction from the topic. Fixtures (fake orchestrator, placeholder topic) ship behind the test-fixtures feature for sibling crates' tests.
Migration 0020_proof_rlm.sql: proof_topic_version (signed documents per re-sign), proof_rule_version (RLM-written rule vectors, versioned with the topic, digest-bound), proof_checklist (every inspection, red or green), proof_lifecycle_event, proof_baseline_measurement, proof_artefact (zip metadata), proof_promotion_event (the learning continuum: baseline -> runs -> best -> what the next run must beat). Append-only for base_app; the current best is the newest promotion row. proof-rlm-store: the RlmStore contract, PgRlmStore over crates/db (plain runtime sqlx), and MemoryRlmStore for CI / local. One contract test runs against both; the Postgres half runs on an isolated migrated schema when DATABASE_URL is set and is skipped otherwise.
RlmScorer is the LiveScorer for the custom family: resolve custom_id in
the registry (unknown or unwired -> RunnerUnwired, 503, no row), load the
topic's current rule version from the store (seeding v1 from the signed
document), inspect through the runner -> checklist persisted, red ->
reject document with no paid inference, green -> spend token -> evaluate
-> custom_value. Evaluations are serialised per topic and every
lifecycle move is written to the store; a stale evaluating/promoting
mirror is recovered rather than refused forever.
ArtefactStore writes {root}/{topic_id}/{submission_id}.zip (manifest,
artifact/, report, checklist, baseline_ref, logs) deterministically with
path validation, best.json on promotion, and public events.jsonl;
metadata and promotion rows mirror into the RLM store.
TopicSetup drives draft -> owner_presend -> awaiting_owner_keys ->
provisioning -> baselining over the VM boundary: owner hook, key probe,
provision or attach, RLM ProposeRules -> rules vN in store, Baseline job
-> measurement in store; mark_sealed closes baselining -> open. A re-run
resumes from the persisted state.
Router-level tests cover the unregistered 503, score / red-reject /
below-best paths through the registry with a fake orchestrator, and the
setup driver end to end.
proof-http: submit refuses a topic whose runner is not registered or wired (503, no row) right after host readiness; publish validates custom ids against the host's registered set (draft with any id, open only with a runner); /v1/status gains scorable_topics and registered_custom, and can_score means at least one open topic is scorable here; a pass the family scorer crowns persists as champion and the async on_persisted hook fires with the row id. bins/proof-challenge: BASE_DATABASE_URL(_FILE) selects the Postgres RLM store (migrated at boot; configured-but-unreachable is fatal, unset falls back to memory with a warning); the live harvest is wrapped in FamilyMux with an RlmScorer over an empty RunnerRegistry, so every custom topic answers 503 until an operator / RLM registers a runner. No runner, benchmark, model, or repository is compiled in. PROOF_ARTEFACT_ROOT (default /artefacts) names where zips land.
PROOF.md: custom ids as topic data (draft freely, open with a registered runner, empty registry by default), rules as a topic-carried vector re-versioned by the RLM into the DB, the per-topic VM isolation boundary, the lifecycle, the registry, artefacts / best pointer / public events, scorable_topics + registered_custom on /v1/status, and the re-sign note. external-miner/proof.md: what a custom-family submission is, the topic-carried checklist a miner must pass, the new 503 / 201-rejected rows, champion state. COMPLETENESS.md: RLM engine row (generic / fail-closed). AGENTS.md: product rule (zero challenge content in git). deploy: PROOF_ARTEFACT_ROOT on the proof-artifacts volume plus the orchestrator / image / owner-key env var names (names only, no values).
cursor
Bot
force-pushed
the
cursor/proof-generic-rlm-engine-1fd2
branch
from
September 8, 2026 13:42
1aa2bc4 to
20b5f18
Compare
Greptile P1 on #243: the custom submission flow accepted `artifact_uri` but `RlmScorer` built `CustomRunRequest` with `None`, so the runner got the digest and no way to retrieve the bytes. - `LiveScorer::score`, `eval_after_freeze`, and `FamilyMux` take the miner's `artifact_uri`; `proof-http` forwards `body.artifact_uri`; the Lium harvest ignores it (it fetches by digest). - `RlmScorer::evaluate` threads it into `CustomRunRequest::from_topic`; the request normalises whitespace/empty to `None`. - e2e: the inspect and evaluate VM jobs carry the submitted locator with the digest; unit test for the normalisation.
Greptile P1 on #243: every custom-run verdict reported `flops_used: 0` while neither the run request nor the verified report carried a usage measurement, so the judge compared zero with the signed budget and an over-budget run could pass. - `CustomRunRequest` carries the topic's `flops_budget`; `CustomRunReport` carries the runner-measured `flops_used`. `flops_used_for` is the only way to read it: `None` against a budget is `ReportError::FlopsMissing` (report binding fails closed, 503, no row); zero is only a measurement. - `RlmScorer` puts that figure in the verdict. A measurement over the budget is `reject` + `flops_over_budget`, which the judge fails on; the pre-spend reject keeps `0` because nothing ran. - `TopicSetup::baseline` refuses a baseline that spent over the topic budget or measured nothing (`SetupError::BaselineOverBudget`). - Fixtures: `FakeOrchestrator::set_flops_used`; tests for the binding, the persisted over-budget reject, the 503, and the setup refusals.
Greptile P1 on #243: the per-topic guard ended when `score` returned, so two completed runs could both promote against the same old bar and the persist path appended both promotions and moved the best pointer unconditionally; a later, worse run could replace a better champion. - A run now holds its topic lease from `score` until `on_persisted` (the guard moves into the pending entry). The next run of the topic cannot score, decide, or persist until the previous row landed. - `auto_promote` decides under that lease against the harder of the caller's bar and the store's current best, and records the best it was decided against. - `persist` runs a compare-and-swap before writing: the best pointer must still be the one the decision saw and the run must be strictly better than it, else the crown is refused (`promotion_refused`, manifest `promoted: false`, no promotion row, pointer untouched). The promotion row is written to the RLM store whether or not a filesystem artefact store is configured. - A scored run whose row never lands releases its lease after `DEFAULT_LEASE_TTL` (5 min, `with_lease_ttl` for tests) so a failed insert cannot block a topic; the lifecycle is recovered on next score. - e2e: a second run blocks until the first persists and then loses against the new best with the stale bar; an externally moved best refuses a decided crown; an abandoned lease is reaped.
Greptile P1 on #243: `mark_sealed` advanced `baselining → open` and stored the supplied document as the new version without checking that it was open, sealed, structurally valid, or signed, so an invalid draft could become the version behind an open lifecycle. `TopicSetup::mark_sealed(topic, pin, registered_custom, sealed)` now refuses, in order and before anything is written or moved: a document that is not `status: open` (`SetupError::NotOpen`); one that does not validate as an open topic on this host — sealed baseline, registered `custom_id`, tighten-only floors (`SetupError::Topic`); one whose operator signature does not verify under the pin's topic key; a sealed `BaselineMeasurement` that does not bind to the document (commitment, holdout, image digest) or whose `custom_value` is not the primary the RLM measured in the topic VM (`SetupError::Seal`); and a lifecycle not at `baselining`. The version is stored, then the transition recorded. e2e: draft, unsigned, unregistered, mis-sealed, and mismatched documents all leave the lifecycle at `baselining` with version 1; the signed open document sealing the measured value opens as version 2; a second seal is illegal. Test-only `crypto` dev-dependency for the topic key.
Describe what the Greptile P1 fixes on #243 changed for operators and miners: the run request carries `artifact_uri` and `flops_budget`, the verdict carries the runner-measured `flops_used` (missing → 503, over budget → reject; `declared_flops` is never the enforced figure); promotion is decided and persisted under a per-topic lease with a compare-and-swap on the best pointer; `mark_sealed` opens only a signed, valid, open document sealing the RLM's measured value.
Follow-up to the Greptile P1 "artifact location discarded" on #243: the locator now reaches the runner, but a custom submission without one gave the runner nothing to retrieve at all — the request would still carry `None` for a real artefact. - `proof-http` intake: on a custom-family topic a missing / blank `artifact_uri` is a 400 (`artifact_uri is required for custom topics`) with no row and no rent; `nll` / `throughput` keep it optional (the digest-pinned image fetches by digest). - `RlmScorer::evaluate` refuses a request without a locator rather than hand the runner one it cannot act on; `CustomRunRequest.artifact_uri` is never `None` for a miner submission. - Tests: http 400 with no row and no scorer call (null and blank), the harvest path unaffected; e2e 400 before any VM job, every custom body carries a locator. Docs: required on custom topics.
Follow-up to the Greptile P1 "FLOP budget bypass" on #243: the verdict now carries the runner's measurement against the signed budget, but the miner's `declared_flops` (checked `<=` budget at intake) had no teeth — a run could spend far more than it declared and still pass. - `LiveScorer::score` / `eval_after_freeze` / `FamilyMux` carry `declared_flops`; `proof-http` forwards the body's value; the Lium harvest ignores it (its agent observes usage against the budget). - `CustomRunRequest.declared_flops`: the runner may enforce it as a hard cap inside the topic VM. The operator baseline declares the budget. - `RlmScorer::paid_verdict`: measured usage over the budget is `flops_over_budget`, over the declaration is the new `ProofCheatCode::FlopsUnderDeclared`; either is a `reject` the judge fails the row on. Missing usage stays a 503 (not evidence). - e2e: declared 1, measured 2 → persisted reject with `flops_under_declared` only; declared the budget, measured one over → `flops_over_budget`; measured exactly the declared budget → crowned. Docs: declare what you will use, up to the budget.
Contributor
Author
|
@greptileai please re-review after P1 fixes: artifact locator, measured flops fail-closed, promote lease+CAS, mark_sealed signed/valid only. |
Contributor
Author
|
@greptileai review |
This was referenced Sep 8, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Proof becomes a dynamic agentic challenge system in code: the binary is a generic orchestrator (schema, DB, isolation boundary, artefact store, runner registry). Zero challenge content is compiled in — no benchmark, metric, model, rule list, repository, or topic catalog. The first live topic is a signed document its RLM sets up, never a code branch.
Rebased onto
mainafter #242 (0d418243,EvalExecutorOffer); this PR consumes that contract and does not duplicate it (see "Executor" below).What this PR delivers
proof-task, newproof-canon):constraints.{firecracker_required, model_pin, task_slice (opaque), params}and achecklist: [{id, text}]rule vector, all shape-checked only and omitted from the signed payload when unset (same rule as feat(proof): EvalExecutorOffer contract + harvest 1x / deadline enforcement #242'seval_executor), so previously signed topics keep verifying.metric.custom_idis topic data: any well-formed id drafts; a custom topic may open only when a runner is registered under its id (the compiled-in custom id list is gone).proof-canonholds canonical JSON, id shapes, and the sharedConstraints/ChecklistRuleshapes (keepsproof-taskunder the LOC cap).crates/db/migrations/0020_proof_rlm.sql, newproof-rlm-store):proof_topic_version,proof_rule_version(RLM-written rules versioned with the topic, digest-bound),proof_checklist,proof_lifecycle_event,proof_baseline_measurement,proof_artefact,proof_promotion_event(best pointer + history). Append-only forbase_app.PgRlmStore+MemoryRlmStoreshare one contract test (Postgres half runs whenDATABASE_URLis set).proof-rlm): versionedRuleSet+Checklist+SpendToken(no paid inference behind a red/incomplete/stale checklist — the token has no public constructor), lifecycledraft → owner_presend → awaiting_owner_keys → provisioning → baselining → open ⇄ evaluating → promoting → closedwithaskUser-style owner hook and presence-only key probe,CustomRunner+RunnerRegistry(empty by default),TopicVmOrchestratorboundary (create/attach/run/teardown;VmJobs carry public data only) withUnwiredVmOrchestratorand the genericVmBackedRunner,decide_promote.proof-rlm-scorer):RlmScorer(LiveScorerfor the whole custom family, routed viaFamilyMux), topic-scopedArtefactStore(/artefacts/{topic_id}/{submission_id}.zip,best.json,events.jsonl),TopicSetupdriver over the VM boundary.proof-http(per-topic 503 before any row/rent,scorable_topics+registered_customon/v1/status, automaticchampionvia the family scorer, async persist hook);bins/proof-challenge(Postgres RLM store fromBASE_DATABASE_URL(_FILE), migrated at boot; empty registry;PROOF_ARTEFACT_ROOT).Executor (#242) — consumed, not duplicated
TopicEvalExecutor, the pin ceilings,EvalExecutorOffer,ExecutorPlan, and the harvest 1x / deadline enforcement are used as merged.FamilyMuxforwardsLiveScorer::plan; on the custom path each run request records the resolved plan's deadline (tighter of topic and plan) andconfig_commitmentas provenance (CustomRunRequest::with_executor_plan), and rows stampexecutor_commitmentlike every other scored row.Fail-closed by construction
custom_id→ 503, no row, no rent, never a harvest fallback.opencustom topic without a registered runner → 400 at publish; the same document drafts.artifact_uri→ 400 at intake, no row; the runner never gets a request without a locator.flops_usedagainst a topic budget → 503, no row; measured over the signed budget (flops_over_budget) or over the miner'sdeclared_flops(flops_under_declared) → persisted reject.mark_sealedopens only astatus: open, structurally valid, operator-signed document whose sealed measurement is the one the RLM produced.Greptile P1 fixes (round 1, head
20b5f18b) — all four threads resolvedLiveScorer::score/eval_after_freeze/FamilyMuxtakeartifact_uri;proof-httpforwards it;RlmScorerputs it inCustomRunRequest. Then: a custom submission without a locator is a 400 at intake (no row) and the scorer refuses a request without one — neverNonefor a miner submission. e2e asserts the inspect + evaluate VM jobs carry the submitted locator.CustomRunRequest.flops_budget+CustomRunReport.flops_used(runner-measured).flops_used_foris the only read:Noneagainst a budget isFlopsMissing(binding fails closed → 503). Verdict carries the measurement; over budget →reject+flops_over_budget. Then: the miner'sdeclared_flopstravels in the request (runner may enforce it as a hard cap) and a measurement over it isflops_under_declared→reject. Setup refuses an over-budget / unmeasured baseline.scorethroughon_persisted;auto_promotedecides against the harder of the caller's bar and the store's best;persistcompare-and-swaps on the best pointer and requires strictly better, elsepromotion_refused(no row, pointer untouched). Abandoned leases reaped afterDEFAULT_LEASE_TTL(5 min).mark_sealed(topic, pin, registered_custom, sealed)refuses non-open, invalid, unsigned, mis-sealed, or mismatched documents before anything moves or is stored.Removed (per direction)
All earlier challenge-specific work was stripped before this branch was pushed: the fixed topic id / metric id / model pin / task-slice constants and
if-branches, the hardcoded eight-rule checklist, the example draft JSON catalog underdocs/, the pre-existing compiled-in custom id list, and the family-specific runner.proof-rlmcarries a test asserting no such literals appear in its non-test source.Out of scope
A live VM orchestrator and runner registration mechanics (interfaces + fail-closed stubs only). No proof-eval digest was invented.
Greptile
Every PR is reviewed by Greptile before merge. Config:
.greptile/.@greptileai reviewTest plan
cargo test --workspaceon the current head (1258 passed, 0 failed) — includes router-level e2e for unregistered-503, score / red-reject / below-best through the registry, the setup driver, and the P1 coverage: locator passthrough + 400 without one, over-budget and under-declared rejects + unmeasured 503, lease blocking + stale-bar loss + CAS-refused crown + reaped lease, and everymark_sealedrefusalSQLX_OFFLINE=true DATABASE_URL=… cargo test -p proof-rlm-storeagainst a local Postgres 16: migration0020applies in the full chain and both stores honour the contract; the binary boots against Postgres, runs migration 20, creates the seven tables with INSERT+SELECT-only grantscargo fmt --all -- --checkcargo clippy --workspace --all-targets -- -D warnings,cargo deny check,xtask loc-cap / consensus-lint / spec-check / design-check / external-docs-checkproof-rlmanti-hardcode test (no_challenge_content_is_compiled_in) greenRisk
New crates and one additive migration;
Constraintsis no longerCopy(callers clone) and now lives inproof-canon(re-exported fromproof-task, no call-site change). New topic knobs are omitted when unset, so existing signatures hold. NoBASE_*rename, no crypto domain change, no emission semantics change. Behaviour change: anopencustom topic without a registered runner is now a publish 400 (it drafts), and custom topics answer 503 until a runner is registered — both fail-closed. On custom topicsartifact_uriis now required (400 without) and the runner's measured usage must stay within the miner'sdeclared_flops;nll/throughputintake and scoring are unchanged.LiveScorer::scoregainedartifact_urianddeclared_flopsparameters (the Lium harvest ignores both);CustomRunReportgainedflops_used(required whenever the topic has a budget);ProofCheatCodegainedflops_under_declared;TopicSetup::mark_sealedtakes the pin, the registered custom ids, and the sealed measurement.Naming
I did not rename
BASE_*environment variables, deployed host paths(
/opt/base,/run/base, …), GHCRbaseintelligence/basepackage names, orbase-*-v1cryptographic domain tags.