feat(proof): generic dynamic RLM engine (DB-backed rules, per-topic VM boundary, artefacts, fail-closed runner registry)

AGENTS.md

@@ -71,13 +71,13 @@ When verifying a challenge (local-e2e, staging, or focused tests), **simulate a
 2. Edge / failure probes: bad harness, sanitize reject, quota, wrong routes/auth.
 3. **Bounty — pair + report:** `ctx bounty pair --hotkey <ss58> --account-id <id> --accept-terms`, then `POST /v1/pair` (terms + signature) and `POST /v1/reports`. Operator bearer `POST /v1/admin/adjudicate` (`valid` / `already_fixed_not_prod` / `invalid_malicious` / `duplicate`). Scoring **reads** CortexLM/backend public JSON (`BOUNTY_BACKEND_PUBLIC_URL`); do not serve `/v1/public/*` from this repo.
 4. **Bounty — fail-closed scorer:** the CortexLM/backend public feed is the only scorer. With no readable `BOUNTY_BACKEND_PUBLIC_URL`, `POST /v1/reports` must answer **503** and the emitter must pay **nobody** — it still covers `E` with `NoScore(ChallengeInternal)`, because a paid challenge with no leaves 409s the seal for every challenge. `BOUNTY_FORCE_SIM` is retired — do not reintroduce an offline bounty scorer. See [`docs/BOUNTY.md`](docs/BOUNTY.md).
-5. **Proof — submit:** `POST /v1/submissions` with a `topic_id`. Missing/unknown/not-open → **400** (no row). Empty `eval_image_digest`, missing/closed/misconfigured RLM judge `InferenceOffer`, missing judge API key, spoofed topic origin, missing/closed/non-`1x` `EvalExecutorOffer` (Lium path), zero open topics, or an unsealed baseline → **503**. Miners submit claim + code + FLOPs + artifact; they do not bind the judge offer or the executor offer. Contamination / empty manifest persist **rejected** without rent. `GET /v1/proof/topics` must never leak holdout records.
+5. **Proof — submit:** `POST /v1/submissions` with a `topic_id`. Missing/unknown/not-open → **400** (no row); a custom topic without `artifact_uri` → **400** (no row). Empty `eval_image_digest`, missing/closed/misconfigured RLM judge `InferenceOffer`, missing judge API key, spoofed topic origin, missing/closed/non-`1x` `EvalExecutorOffer` (Lium path), zero open topics, or an unsealed baseline → **503**. Miners submit claim + code + FLOPs + artifact; they do not bind the judge offer or the executor offer. Contamination / empty manifest persist **rejected** without rent; on custom topics the runner's measured `flops_used` over the budget or over the miner's `declared_flops` persists **rejected** after the run, and a report without a measurement is **503** (no row). `GET /v1/proof/topics` must never leak holdout records.
 6. **Proof — executor:** `GET /v1/proof/executor` is always 200 (`ready` + `reason`); `POST /v1/admin/proof/executor` (operator bearer) rotates or closes the live `1x` offer and 400s anything the pin refuses. Harvest rents the offer's `lium_template_id` at exactly `1x` (any other `rent_gpu_count` aborts before the rent) under `max_proof_deadline_s`; a run cut at the deadline is **503 + `stdout_tail`**. `PROOF_HARVEST_*` env only hot-swaps under the pin ceilings. Never a live Lium rent in CI.
 7. Leaf emission → `POST /v1/weights/raw` → seal → `GET /v1/weights/latest` with **`sealed: true`** (burn fallback alone is not a real seal).
 
 **Never host Sim in staging/prod** for live scoring. `PROOF_FORCE_SIM=1` is CI/local opt-in only (`deploy/scripts/assert-compose-matrix.sh` fails if a droplet overlay sets one). Live Proof rent requires a digest pin in `config/proof-pin.toml` plus miner BYOK (`LIUM_API_KEY` / `X-Lium-Api-Key`). Never log or commit that key. Do not invent `eval_image_digest`.
 
-**Bounty product rules:** pay is precision x severity, an unpriced `valid` row is not creditable, and the triage-noise ratio stays off the visible score. **Proof product rules (do not weaken):** topics are operator-published signed documents, not a git catalog; a topic may tighten a floor never loosen it; a baseline must be sealed to open; each topic is `wta` (winner takes the topic mass) or `discovery` (pass floor + novelty); global miner score is the **sum** of per-topic masses, not a mean of binary lattices; empty open set / empty eval digest fails closed (`503`); `custom` unknown ids refuse at publish; `harness_success_rate` is listed and fail-closes until the real harness exists; the eval executor is exactly `1x` (pin `gpu_class`), a topic may only tighten `eval_executor.max_proof_deadline_s` / pin `require_offer_commitment`, and there is no per-topic `machine_id`.
+**Bounty product rules:** pay is precision x severity, an unpriced `valid` row is not creditable, and the triage-noise ratio stays off the visible score. **Proof product rules (do not weaken):** topics are operator-published signed documents, not a git catalog; a topic may tighten a floor never loosen it; a baseline must be sealed to open; each topic is `wta` (winner takes the topic mass) or `discovery` (pass floor + novelty); global miner score is the **sum** of per-topic masses, not a mean of binary lattices; empty open set / empty eval digest fails closed (`503`); `custom` ids are **topic data** (any well-formed id drafts; a custom topic may **open** only when a runner is registered under its id, and the registry is **empty by default**); anti-cheat rules are a **vector carried by the signed topic** (re-versioned by the topic's RLM into the DB) and are ticked **before any paid inference** (one red item = persisted reject, no spend); the RLM runs **inside a per-topic VM** behind the `TopicVmOrchestrator` boundary (unwired stub = `503`), never on the control-plane host; the eval executor is exactly `1x` (pin `gpu_class`), a topic may only tighten `eval_executor.max_proof_deadline_s` / pin `require_offer_commitment`, and there is no per-topic `machine_id`. **Zero challenge content in git:** no benchmark, metric, model, rule list, repository, or topic catalog is compiled in — the first live topic is a signed document its RLM sets up (`crates/proof-rlm*`, `docs/PROOF.md` § Dynamic agentic engine).
 
 Local smoke automates the weights seal step via `weights-smoke` inside `./deploy/scripts/local-e2e.sh --smoke` (see [`deploy/AGENTS.md`](deploy/AGENTS.md) and [`docs/runbooks/local-testnet-e2e.md`](docs/runbooks/local-testnet-e2e.md)).
 

bins/proof-challenge/Cargo.toml

@@ -16,11 +16,15 @@ path = "src/main.rs"
 axum = { version = "0.8", default-features = false, features = ["http1", "tokio"] }
 challenge-keys = { path = "../../crates/challenge-keys" }
 clap = { version = "4", features = ["derive", "env"] }
+db = { path = "../../crates/db" }
 harvest-pod = { path = "../../crates/harvest-pod" }
 prism-lium = { path = "../../crates/prism-lium" }
 proof-challenge = { path = "../../crates/proof-challenge" }
 proof-eval = { path = "../../crates/proof-eval" }
 proof-harvest = { path = "../../crates/proof-harvest" }
+proof-rlm = { path = "../../crates/proof-rlm" }
+proof-rlm-scorer = { path = "../../crates/proof-rlm-scorer" }
+proof-rlm-store = { path = "../../crates/proof-rlm-store" }
 proof-task = { path = "../../crates/proof-task" }
 serde_json = "1"
 telemetry = { path = "../../crates/telemetry" }

bins/proof-challenge/src/main.rs

@@ -23,8 +23,11 @@ use proof_challenge::{
     BaselineMeasurement, EvalBackend, EvalExecutorOffer, HarvestOverrides, InferenceOffer,
     LiveScorer, MemoryStore, ProofPin, TopicDocument, CHALLENGE_ID, SCORING_VERSION,
 };
-use proof_eval::supported_custom;
+use proof_eval::{custom_ids_ref, registered_custom, FamilyMux};
 use proof_harvest::{HarvestLimits, LiumProofHarvest};
+use proof_rlm::RunnerRegistry;
+use proof_rlm_scorer::{ArtefactStore, RlmScorer};
+use proof_rlm_store::{MemoryRlmStore, PgRlmStore, RlmStore};
 use tokio::net::TcpListener;
 
 /// Operator Proof challenge service CLI.
@@ -88,6 +91,16 @@ struct Cli {
     /// Holdout shard bytes (`<content_sha256>` files). Not the record catalog.
     #[arg(long, env = "PROOF_HOLDOUT_STORE")]
     holdout_store: Option<PathBuf>,
+    /// Root for per-submission artefact zips (`{root}/{topic_id}/{submission_id}.zip`).
+    #[arg(long, env = "PROOF_ARTEFACT_ROOT", default_value = "/artefacts")]
+    artefact_root: PathBuf,
+    /// Postgres URL for the RLM store (topic versions, rule versions,
+    /// checklists, lifecycle, artefact metadata, promotions). Unset → in-memory.
+    #[arg(long, env = "BASE_DATABASE_URL")]
+    database_url: Option<String>,
+    /// File holding the Postgres URL (preferred on a droplet).
+    #[arg(long, env = "BASE_DATABASE_URL_FILE")]
+    database_url_file: Option<PathBuf>,
 }
 
 fn main() -> ExitCode {
@@ -129,19 +142,6 @@ fn run(cli: &Cli) -> Result<(), String> {
         );
     }
 
-    let store = MemoryStore::new();
-    match load_topics(&store, &pin, cli.topics_file.as_deref()) {
-        Ok(n) => tracing::info!(topics = n, "signed topics loaded"),
-        Err(e) => tracing::warn!("topics unavailable ({e}); submissions will 400/503 until fixed"),
-    }
-    match load_holdouts(&store, cli.holdout_file.as_deref()) {
-        Ok(n) => tracing::info!(topics = n, "holdouts verified against topic commitments"),
-        Err(e) => tracing::warn!("holdouts unavailable ({e}); submissions will 503 until fixed"),
-    }
-    match load_baselines(&store, &pin, cli.baseline_file.as_deref()) {
-        Ok(n) => tracing::info!(topics = n, "sealed baselines recorded"),
-        Err(e) => tracing::warn!("baselines unavailable ({e}); submissions will 503 until fixed"),
-    }
     let offer = match load_offer(&pin, cli.inference_offer_file.as_deref()) {
         Ok(o) => {
             tracing::info!(offer_id = %o.offer_id, status = ?o.status, "inference offer loaded");
@@ -169,16 +169,24 @@ fn run(cli: &Cli) -> Result<(), String> {
         .build()
         .map_err(|e| e.to_string())?;
 
+    let rlm_store = rt.block_on(resolve_rlm_store(cli))?;
     let live_scorer = build_live_scorer(
         backend,
         cli.eval_timeout_secs,
         judge_api_key.clone(),
         cli.proxy_model_dir.clone(),
         cli.holdout_store.clone(),
-    );
+    )
+    .map(|harvest| with_custom_family(harvest, rlm_store, &cli.artefact_root));
     match backend {
         EvalBackend::Lium if live_scorer.is_some() => {
             tracing::info!("live harvest wired: digest-pinned proof-eval image on Lium");
+            tracing::info!(
+                registered_custom = ?registered_custom(live_scorer.as_deref()),
+                artefact_root = %cli.artefact_root.display(),
+                "custom-family topics route to the rlm scorer; an id with no registered runner \
+                 answers 503 (no runner is compiled in)"
+            );
         }
         EvalBackend::Lium => tracing::warn!(
             "live harvest not wired; every submission will 503. Set the Lium credentials \
@@ -187,6 +195,21 @@ fn run(cli: &Cli) -> Result<(), String> {
         EvalBackend::Sim => {}
     }
 
+    let store = MemoryStore::new();
+    let registered = registered_custom(live_scorer.as_deref());
+    match load_topics(&store, &pin, cli.topics_file.as_deref(), &registered) {
+        Ok(n) => tracing::info!(topics = n, "signed topics loaded"),
+        Err(e) => tracing::warn!("topics unavailable ({e}); submissions will 400/503 until fixed"),
+    }
+    match load_holdouts(&store, cli.holdout_file.as_deref()) {
+        Ok(n) => tracing::info!(topics = n, "holdouts verified against topic commitments"),
+        Err(e) => tracing::warn!("holdouts unavailable ({e}); submissions will 503 until fixed"),
+    }
+    match load_baselines(&store, &pin, cli.baseline_file.as_deref()) {
+        Ok(n) => tracing::info!(topics = n, "sealed baselines recorded"),
+        Err(e) => tracing::warn!("baselines unavailable ({e}); submissions will 503 until fixed"),
+    }
+
     let state = AppState {
         store,
         pin,
@@ -249,6 +272,62 @@ fn build_live_scorer(
     ))
 }
 
+/// Route the `custom` metric family to the RLM scorer over the default harvest.
+///
+/// The runner registry starts **empty**: no benchmark, model, or repository is
+/// compiled in, so every custom topic answers 503 (`RunnerUnwired`) until an
+/// operator or the topic's RLM registers a runner under its `custom_id`. It
+/// never falls back to the digest-pinned harvest and never spends.
+fn with_custom_family(
+    harvest: Arc<dyn LiveScorer>,
+    rlm_store: Arc<dyn RlmStore>,
+    artefact_root: &Path,
+) -> Arc<dyn LiveScorer> {
+    let scorer = RlmScorer::new(Arc::new(RunnerRegistry::new()), rlm_store)
+        .with_artefacts(Some(ArtefactStore::new(artefact_root)));
+    Arc::new(FamilyMux::new(harvest).with_custom_family(Arc::new(scorer)))
+}
+
+fn database_url(cli: &Cli) -> Result<Option<String>, String> {
+    if let Some(url) = cli.database_url.as_deref().map(str::trim) {
+        if !url.is_empty() {
+            return Ok(Some(url.to_owned()));
+        }
+    }
+    let Some(path) = cli.database_url_file.as_deref() else {
+        return Ok(None);
+    };
+    let raw = std::fs::read_to_string(path).map_err(|e| format!("read {}: {e}", path.display()))?;
+    let trimmed = raw.trim();
+    if trimmed.is_empty() {
+        return Err("BASE_DATABASE_URL_FILE is empty".into());
+    }
+    Ok(Some(trimmed.to_owned()))
+}
+
+/// Postgres RLM store when a database is configured, in-memory otherwise.
+///
+/// A configured but unreachable database is fatal: falling back to memory
+/// would silently drop every rule version, checklist, and promotion on
+/// restart.
+async fn resolve_rlm_store(cli: &Cli) -> Result<Arc<dyn RlmStore>, String> {
+    let Some(url) = database_url(cli)? else {
+        tracing::warn!(
+            "no database configured; rlm rules, checklists, lifecycle, and promotions are not \
+             persisted across restarts"
+        );
+        return Ok(Arc::new(MemoryRlmStore::new()));
+    };
+    let pool = db::connect(&url)
+        .await
+        .map_err(|e| format!("database connect failed: {e}"))?;
+    db::migrate(&pool)
+        .await
+        .map_err(|e| format!("database migrate failed: {e}"))?;
+    tracing::info!("rlm store persists to postgres");
+    Ok(Arc::new(PgRlmStore::new(pool)))
+}
+
 fn load_inference_api_key(path: Option<&Path>) -> Option<String> {
     let p = path?;
     std::fs::read_to_string(p)
@@ -278,13 +357,18 @@ fn load_pin(path: Option<&Path>) -> Result<ProofPin, String> {
     Ok(pin)
 }
 
-fn load_topics(store: &MemoryStore, pin: &ProofPin, path: Option<&Path>) -> Result<usize, String> {
+fn load_topics(
+    store: &MemoryStore,
+    pin: &ProofPin,
+    path: Option<&Path>,
+    registered_custom: &[String],
+) -> Result<usize, String> {
     let p = path.ok_or("PROOF_TOPICS_FILE not set")?;
     let body = std::fs::read_to_string(p).map_err(|e| format!("read {}: {e}", p.display()))?;
     let docs = TopicDocument::many_from_json(&body).map_err(|e| e.to_string())?;
     let n = docs.len();
     for doc in docs {
-        doc.validate(pin, &supported_custom())
+        doc.validate(pin, &custom_ids_ref(registered_custom))
             .map_err(|e| format!("topic {}: {e}", doc.id))?;
         doc.verify_signature(pin)
             .map_err(|e| format!("topic {}: {e}", doc.id))?;
@@ -502,6 +586,69 @@ mod tests {
         std::env::remove_var("LIUM_SSH_PUBLIC_KEY_FILE");
     }
 
+    /// No runner is compiled in: every custom id refuses through the mux
+    /// (`RunnerUnwired`, the 503 root cause), the harvest still owns the
+    /// nll / throughput route, and nothing is registered.
+    #[test]
+    fn live_scorer_registers_no_custom_runner_and_refuses_every_custom_id() {
+        let _guard = LIUM_ENV
+            .lock()
+            .unwrap_or_else(std::sync::PoisonError::into_inner);
+        let pubkey = stub_ssh_pubkey("proof-families");
+        std::env::set_var("LIUM_API_KEY", "test-key-not-a-real-secret");
+        std::env::set_var("LIUM_SSH_PUBLIC_KEY_FILE", &pubkey);
+        let harvest =
+            build_live_scorer(EvalBackend::Lium, 900, None, None, None).expect("harvest wired");
+        std::env::remove_var("LIUM_API_KEY");
+        std::env::remove_var("LIUM_SSH_PUBLIC_KEY_FILE");
+
+        let root = std::env::temp_dir().join("proof-families-artefacts");
+        let mux = with_custom_family(harvest, Arc::new(MemoryRlmStore::new()), &root);
+        assert!(registered_custom(Some(mux.as_ref())).is_empty());
+        for id in ["any_metric", "another_metric"] {
+            let mut custom = TopicDocument::default();
+            custom.metric.family = proof_task::MetricFamily::Custom;
+            custom.metric.custom_id = id.into();
+            let err = mux.ready_for_topic(&custom).expect_err("unregistered");
+            assert!(
+                matches!(err, proof_eval::EvalError::RunnerUnwired { .. }),
+                "{err}"
+            );
+            assert!(err.to_string().contains("no registered runner"), "{err}");
+        }
+        // The default route is the harvest itself, whose readiness is about
+        // proxy weights and holdout shards, not the runner registry.
+        let nll = TopicDocument::default();
+        let err = mux.ready_for_topic(&nll).expect_err("no proxy dir staged");
+        assert!(
+            matches!(err, proof_eval::EvalError::ProxyModelMissing),
+            "{err}"
+        );
+    }
+
+    #[test]
+    fn database_url_comes_from_the_value_or_the_file_or_nowhere() {
+        let mut cli = Cli::try_parse_from(["proof-challenge"]).expect("cli");
+        assert_eq!(database_url(&cli).expect("none"), None);
+        cli.database_url = Some("  ".into());
+        assert_eq!(database_url(&cli).expect("blank is none"), None);
+        let dir = std::env::temp_dir().join(format!("proof-db-url-{}", std::process::id()));
+        std::fs::create_dir_all(&dir).expect("dir");
+        let file = dir.join("url");
+        std::fs::write(&file, "postgres://placeholder/db\n").expect("write");
+        cli.database_url_file = Some(file.clone());
+        assert_eq!(
+            database_url(&cli).expect("file"),
+            Some("postgres://placeholder/db".into())
+        );
+        std::fs::write(&file, "\n").expect("write");
+        assert!(
+            database_url(&cli).is_err(),
+            "an empty file is a config error"
+        );
+        let _ = std::fs::remove_dir_all(&dir);
+    }
+
     #[test]
     fn inference_api_key_file_is_read_not_existence_only() {
         let dir = std::env::temp_dir().join(format!(

bins/proof-challenge/tests/submit_e2e.rs

@@ -88,6 +88,7 @@ fn dt_topic() -> TopicDocument {
             no_nvlink: true,
             no_nccl_fast_fabric: true,
             max_inter_node_gbps: Some(12.5),
+            ..Constraints::default()
         },
         metric: MetricSpec {
             family: MetricFamily::Throughput,