feat(proof): live FirecrackerOrchestrator + KVM-host agent for topic VMs (sister miner guests) - #244
Merged
Merged
Conversation
…ocol proof-rlm (additive): InspectOutcome / RunOutcome / VmJobOutput derive serde (adjacently tagged output/body) so an orchestrator answers over the wire with the same types; VmJob::topic_id / deadline_s / requires_firecracker helpers for the hard topic bind and per-job timeouts. proof-vm-proto: the HTTPS contract between the control plane and the proof-vm-orchestrator agent (create / attach / run / teardown, ErrorBody + codes, SisterAttestation) and the vsock framing + messages the agent speaks to the RLM guest (jobs, secret staging) and the sister miner guest (the run, no network). Types only; no challenge content. proof-vm-agent: bearer-from-file auth (constant-time, re-read per request, never logged), one running VM per topic_id, request topic and job topic must both equal the VM's, per-VM job lock, Hypervisor trait, and host stamping of paid outputs: sandboxed and flops_used come from the sister guest the host booted, never from the RLM's report. FakeHypervisor + in-process FakeAgent behind test-fixtures; nothing here spawns a process. proof-vm-fc: FirecrackerOrchestrator, the live TopicVmOrchestrator. Reads PROOF_VM_ORCHESTRATOR_URL / _TOKEN_FILE / PROOF_RLM_VM_IMAGE_DIGEST (4 vCPU / 8192 MiB default), https only (plain http on loopback for tests), from_env is None when unset (host keeps UnwiredVmOrchestrator), missing token or unpinned digest is NotWired naming the env var, agent down or bearer refused is Backend — all 503, no host fallback. Refuses a job for another topic before any request, an echo for another vm, a created vm on another image, and a firecracker_required run without the host's sister attestation. Co-authored-by: Mathis <echobt@users.noreply.github.com>
…uest The Hypervisor the proof-vm-orchestrator agent drives on a dedicated KVM host. Per topic VM: resolve <image_dir>/sha256-<hex>.ext4 for the digest the control plane pinned and re-verify its bytes (cached by len+mtime), build the jail (kernel copy, read-only rootfs copy, fresh ext4 scratch, vm-config.json with jail-relative paths, vsock at /v.sock), a TAP on its own /30 plus a per-VM nftables table that forwards only the operator's egress allowlist (empty list = no egress) and masquerades out the uplink, exec Firecracker through the jailer (no --daemonize / --new-pid-ns so the child handle is the VM), Hello over vsock, stage owner key material read from the host's own owner_key_dir (the control plane never sees it). Paid jobs (Baseline / Evaluate) listen on v.sock_5001 for the RLM's sister request: topic must match the VM's, the artefact tar must hash to the stated digest, then a second microVM boots from the pinned sister image with no network interface, receives the bytes and the run over vsock, is held to the deadline (+grace, host kill as backstop), and is destroyed. The host writes the SisterAttestation (sandboxed, network none, guest-measured flops_used, wall, exit) that the agent stamps onto the report. One sister per job; none for inspection or rule proposals. Teardown kills the VM, drops the table + TAP, then rm -rf (Destroy) or moves the jail under retain_dir (Retain). Every host command goes through a Shell trait; tests assert the exact argv with a recording shell and prove ready() refuses on a host without firecracker/jailer//dev/kvm — nothing in CI boots a VM. proto: SisterAnswer (result | refused) for the sister channel. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Thin main over proof-vm-agent + proof-fc-host: clap flags with PROOF_VM_AGENT_* env twins for the systemd EnvironmentFile, kernel and sister image pins required at boot (malformed = exit 1, never invented), egress allowlist entries CIDR[:port[/tcp|udp]], owner key dir staged over vsock only. TLS via axum-server (rustls) from operator cert + key; a non-loopback bind without them exits 1 so the bearer never crosses a network in clear; plain http only on loopback. A missing token file does not stop the process — every request is refused until it exists (re-read per request, rotation without restart). Graceful shutdown on ctrl-c. Co-authored-by: Mathis <echobt@users.noreply.github.com>
…gured The custom-family runner registry is no longer hard-wired empty. The host resolves its topic-vm orchestrator once at boot: PROOF_VM_ORCHESTRATOR_URL + PROOF_VM_ORCHESTRATOR_TOKEN_FILE + PROOF_RLM_VM_IMAGE_DIGEST select the live FirecrackerOrchestrator (4 vCPU / 8192 MiB RLM VM by default); URL unset, or set but refused (plain http off loopback, no token-file env), keeps UnwiredVmOrchestrator with the reason logged. Token and digest are checked at ready(), so a missing bearer file or an unpinned image is a 503 naming the env var, fixable without a restart, never a boot error and never a host fallback. The generic VmBackedRunner is registered under exactly the custom ids the operator lists in PROOF_VM_RUNNER_CUSTOM_IDS (comma-separated; malformed ids skipped with a warning). No ids → empty registry → every custom topic 503 (registration stays an operator action, no runner is compiled in). Tests cover the unset / half-configured / fully configured paths and the registry. Co-authored-by: Mathis <echobt@users.noreply.github.com>
deploy: systemd unit for proof-vm-orchestrator (dedicated KVM host only — ConditionPathExists=/dev/kvm, hardened, NET_ADMIN for TAP + nftables), env example with every PROOF_VM_AGENT_* knob (pins left empty: computed from staged files, never invented), proof-challenge env block for the client side (URL / token file / CA / image digest / locked 4 vCPU 8192 MiB / runner ids), secrets README rows for the token, CA, and presence-only owner key. docs: runbook (what runs where, locked rules table, host prerequisites and layout, install, CP wiring, mandatory end-to-end submission verification with the 503 probes, operate, security model, v1 limitations); PROOF.md isolation boundary now describes both orchestrators, the sister guest, and host stamping; COMPLETENESS row for the orchestrator (implemented / operator-gated, no digest pinned yet); ARCHITECTURE topology + binary row; AGENTS.md key-roles row for the bearer file and verification item 7 (topic VMs, zero live Firecracker in CI); deploy/AGENTS.md section; miner page: your code runs offline in a sister guest, the host stamps sandboxed and flops_used. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Firecracker connects to <jail root>/v.sock_5001 as the jail uid while the agent binds it as root; a 0755 socket inode would refuse every guest-initiated sister request. chmod 0666 after bind (the jail root directory, owned by the jail uid and not world-searchable, is what confines it); test asserts the mode. Co-authored-by: Mathis <echobt@users.noreply.github.com>
echobt
marked this pull request as ready for review
September 8, 2026 17:15
Contributor
Author
|
@greptileai please complete review — CI fmt/clippy/test already SUCCESS; Architecte GO conditional on Greptile Review SUCCESS. |
Greptile SummaryThis update completes the Firecracker host lifecycle hardening by keeping jail cleanup ownership active until the VM registry insertion succeeds. The cancellation regression covers a boot paused on the registry lock and verifies that the process, network resources, and jail directory are released. Confidence Score: 5/5Safe to merge; there are no outstanding blocking concerns. No accepted new findings or outstanding previous findings remain. The prior evidence-binding issue is fixed by binding the paid job, report, and sister-attestation identities before results are stamped or accepted. Failed boots, cancelled sister runs, and dead VMs have cleanup and recovery paths. The registry-lock cancellation case is fixed because cleanup ownership remains armed until process handoff and registry insertion complete. Reviews (3): Last reviewed commit: "fix(proof-fc-host): keep the jail guard ..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Greptile P1 "Bind sister evidence" (crates/proof-vm-proto/src/lib.rs): SisterAttestation now carries the topic_id, submission_digest and artifact_digest the host verified before it booted the sister, and bind_evidence() is the one fail-closed check both the agent (before stamping) and the control plane (before accepting) run: the report and the attestation must name exactly the paid job's identities, a sister on a job that runs no miner code is refused, and ErrorCode::EvidenceMismatch (502) names the refusal on the wire. Also adds VmState::Crashed for the agent's dead-VM reaping (P1 "Recover dead VMs"): a VM whose process exited outside a teardown is never advertised as running. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Greptile P1 "Bind sister evidence": run_job calls bind_evidence before stamp_output, so a hypervisor (or a compromised guest behind it) that presents artefact A's sister attestation for a paid job on artefact B, or a report naming another submission than the job, gets 502 evidence_mismatch and nothing is stamped. Greptile P1 "Recover dead VMs" (crates/proof-vm-agent/src/router.rs): Hypervisor::alive() probes the VM process; attach, create, run and health probe every Running record and reap a dead one per its retain policy (teardown), record it Crashed, and stop advertising it, so the topic gets a fresh VM instead of a 409 forever. A VM that dies under a job is reaped by that job on its way out (it holds the lock). Fake hypervisor gains set_sister_replay, set_rlm_report_submission, kill and set_dies_under_job; tests cover replayed evidence (502), the dead-VM recreate path, death under a job, and the health sweep. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Greptile P1 "Clean failed boots" (lib.rs ~259): from jail::prepare on, the jail is owned by a JailGuard until the VM is registered. A failure at TAP setup, rules load, spawn, guest hello or staging destroys the process, the TAP, the nftables table and the jail directory before the error returns (boot_verified); a dropped request releases them through Drop. prepare() itself removes a half-built jail and still refuses to touch one that already exists. Greptile P1 "Clean cancelled sisters" (lib.rs ~343): run_job no longer aborts the sister task. It fires a CancellationToken (also via a drop guard) and waits for the task, and sister::run races the guest against that token and always runs jail.destroy() — kill + rm — before it returns, so a timed-out or finished job never leaves a sister jail or scratch on the host. HvError::Cancelled names the cut run. Greptile P1 "Bind sister evidence": serve_sisters carries the paid job's EvidenceBinding; check_request refuses a SisterRequest naming any other topic / submission / artefact before a jail is built, and the attestation copies those verified identities. Greptile P1 "Recover dead VMs": alive() = child.try_wait() is None. Tests (no Firecracker, no KVM): injected ip-tuntap and nft-f failures and a never-answering stand-in process all end in rm -rf of the jail; a dropped guard releases on the runtime; a cancelled sister is destroyed before run() returns; mismatched sister requests never prepare a jail. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Greptile P1 "Bind sister evidence": the client runs bind_evidence on every RunJobResponse before it accepts the stamps, so an attestation or report naming another topic / submission / artefact than the job is a VmError::Backend (503, no row) even if an agent ever emitted it. The live-agent test drives a replayed attestation through the fake agent (502 evidence_mismatch surfaces as Backend) and checks the client-side refusal on the same body. Co-authored-by: Mathis <echobt@users.noreply.github.com>
PROOF.md isolation boundary, the KVM-host runbook (verification probes, operate table, security model, limitations) and COMPLETENESS.md describe the bound SisterAttestation, the jail guard, cooperative sister cancellation and crashed-VM reaping. Co-authored-by: Mathis <echobt@users.noreply.github.com>
|
@greptileai review Head d0835a1 addresses all four P1s (bind sister evidence, clean failed boots, clean cancelled sisters, recover dead VMs) — see the per-thread replies for the commit map. |
Greptile P1 "Keep Cleanup Ownership" (lib.rs ~279): boot_verified took the registry lock after jail.keep() had disarmed the guard, so a request cancelled while that lock was contended killed the child (kill_on_drop) but left the jail directory, the TAP and the nftables table behind. The registry lock is now taken while the guard still owns the jail and nothing awaits between the hand-over and the insert; keep() returns the still-armed guard when there is no process to hand over, so no path disarms without registering. New test boots against a fake RLM guest over the jail's vsock UDS with a sleeping stand-in process, cancels the boot while the registry lock is held, and asserts the table, TAP and jail are released with nothing registered; the same boot then completes, is alive, and tears down when the lock is free. Co-authored-by: Mathis <echobt@users.noreply.github.com>
|
@greptileai review Head 68984c0: the "Keep Cleanup Ownership" P1 is fixed (registry lock taken while the jail guard is still armed; |
This was referenced Sep 8, 2026
cursor Bot
pushed a commit
that referenced
this pull request
Sep 8, 2026
…kvm works Architecte nit on #247: `cortex-staging` (nested DigitalOcean virtualisation) did boot Firecracker and the § 4 fail-closed matrix came back green, so the runbook must not claim a Droplet can never be the KVM host. Corrected wording everywhere the PR (and the #244 lines next to it) said so: - staging may colocate the agent on the control-plane droplet when /dev/kvm works — validated on cortex-staging; - production prefers a dedicated DO bare-metal / KVM host; - nested KVM remains fragile — if the boot fails or /dev/kvm goes away, provision metal, do not patch around it (fragility signs listed); - never Lium, never an emulator, never without /dev/kvm. Runbook § DigitalOcean staging gains the colocated specifics: the CP container reaches the agent on the droplet's VPC address (loopback in the container is the container; FcConfig accepts plain http on loopback only), agent bound on that address with TLS + SAN, :8200 firewalled to the compose network, sizing on top of the compose stack, a local `install` of the bearer copy; sign-off records the placement. The fail-closed probes, the matrix, and the harness are unchanged. Same correction in AGENTS.md item 7, deploy/AGENTS.md, both staging overlays, the generic env examples, the systemd unit comment, docs/PROOF.md, docs/ARCHITECTURE.md, docs/COMPLETENESS.md (staging boot + § 4 green recorded; § 5 / § 6 still to be recorded), and the agent / binary module docs. Co-authored-by: Mathis <echobt@users.noreply.github.com>
cursor Bot
pushed a commit
that referenced
this pull request
Sep 8, 2026
…kvm works Architecte nit on #247: `cortex-staging` (nested DigitalOcean virtualisation) did boot Firecracker and the § 4 fail-closed matrix came back green, so the runbook must not claim a Droplet can never be the KVM host. Corrected wording everywhere the PR (and the #244 lines next to it) said so: - staging may colocate the agent on the control-plane droplet when /dev/kvm works — validated on cortex-staging; - production prefers a dedicated DO bare-metal / KVM host; - nested KVM remains fragile — if the boot fails or /dev/kvm goes away, provision metal, do not patch around it (fragility signs listed); - never Lium, never an emulator, never without /dev/kvm. Runbook § DigitalOcean staging gains the colocated specifics: the CP container reaches the agent on the droplet's VPC address (loopback in the container is the container; FcConfig accepts plain http on loopback only), agent bound on that address with TLS + SAN, :8200 firewalled to the compose network, sizing on top of the compose stack, a local `install` of the bearer copy; sign-off records the placement. The fail-closed probes, the matrix, and the harness are unchanged. Same correction in AGENTS.md item 7, deploy/AGENTS.md, both staging overlays, the generic env examples, the systemd unit comment, docs/PROOF.md, docs/ARCHITECTURE.md, docs/COMPLETENESS.md (staging boot + § 4 green recorded; § 5 / § 6 still to be recorded), and the agent / binary module docs. Co-authored-by: Mathis <echobt@users.noreply.github.com>
echobt
added a commit
that referenced
this pull request
Sep 8, 2026
* feat(proof): admin probe for the topic-vm orchestrator wire
GET /v1/admin/proof/vm-orchestrator (operator bearer, read-only, no VM,
no spend) reports what proof-challenge resolved for the topic-VM
orchestrator and whether the KVM-host agent answers, through the very
client the runner drives: `ready()` (bearer file present, RLM image
pinned — re-read now), the locked template, one agent health call, and
the host's own gates (live_harvest_wired, registered_custom). A broken
wire is data, not an error: bearer refused, agent unreachable, digest
unpinned, and URL unset each show up by name so a staging operator can
prove the wire without cargo on the droplet. Never the bearer value.
proof-challenge resolves the orchestrator once per process and shares
the Arc between the runner registry and the probe; the registry is
still built only over a wired harvest, so log lines are unchanged.
Tests: route is 401 / 503 auth_unconfigured / 200; unwired hosts name
the env vars; against the in-process fake agent the report shows ready
+ fake hypervisor, then a bearer rotated on one side, a stopped agent,
and an emptied bearer file, each as data.
Co-authored-by: Mathis <echobt@users.noreply.github.com>
* feat(deploy): proof-vm-wire-check.sh staging harness + fake-agent test
Operator harness for the Proof topic-VM wire on the staging master
droplet: bash + curl + python3, no cargo, bearer only ever in a 0600
curl config, production hosts refused.
env CP env file: https URL (loopback http = WARN), bearer
file non-empty via --path-map (compose bind mount), RLM
image pin is sha256:<64 hex> (empty = FAIL, never
invented), optional CA is PEM, custom ids well-formed,
locked 4/8192 shape, PROOF_FORCE_SIM off
agent GET /v1/health with the bearer (ready / reason /
hypervisor / vms); no bearer and wrong bearer → 401
cp /v1/status gates (lium, live_harvest_wired,
registered_custom ⊇ ids), no URL / token / path leak,
/v1/proof/topics holdout leak, executor readiness, and
the admin vm-orchestrator probe (the CP's own rustls
client: ready, agent health, agent_error)
boot-probe create → attach → 409 → 409 topic_mismatch → destroy →
404 for ONE RLM VM (no job, no spend); Ctrl-C tears
the VM down
submit-probe POST /v1/submissions on a custom topic asserting the
fail-closed code + reason; 2xx refused without
--allow-live-run; a scored row must carry the
sister-measured flops_used
matrix the fail-closed flips with the expected 503 reasons
Integration test (proof-vm-fc, fake agent on loopback, skipped without
bash/curl/python3): env / agent / boot-probe speak the router's JSON,
the bearer never appears in the output, one boot + one Destroy, a dead
agent fails the check; unpinned digest and an emptied bearer file fail
closed.
Co-authored-by: Mathis <echobt@users.noreply.github.com>
* fix(proof-vm-fc): keep the agent url out of the unreachable error
`orchestrator unreachable (...)` travels into the miner-facing 503 body
when the KVM-host agent is down; reqwest's Display would print the
agent's URL with it. Keep the method + route, strip the URL
(`Error::without_url`). Test asserts the route stays and the address
does not.
Co-authored-by: Mathis <echobt@users.noreply.github.com>
* docs(deploy): staging env overlays for the proof topic-vm wire
deploy/env/proof-challenge.staging-vm.example: the client-side keys to
append to proof-challenge.env on the staging master (https agent URL
on the VPC, bearer file, private CA, RLM image pin, locked 4/8192
shape, custom ids, admin tokens file).
deploy/env/proof-vm-orchestrator.staging.example: /etc/proof-vm/
orchestrator.env for the dedicated KVM host serving staging (private
bind + TLS, bearer file, kernel + sister pins, sizes, egress allowlist
for the judge origin / artefact hosts / resolver).
Placeholders only, and every REPLACE_WITH_* value fails closed as
written: the wire check flags each one, proof-challenge stays unwired
or answers 503, the agent refuses to boot. No digest is real.
Co-authored-by: Mathis <echobt@users.noreply.github.com>
* feat(deploy): wire-check submit-probe --no-artifact-uri for the 400 row
Co-authored-by: Mathis <echobt@users.noreply.github.com>
* docs(runbook): proof topic-vm orchestrator — digitalocean staging wire + probes
New § DigitalOcean staging: the CP stays on the existing staging master
droplet; the agent needs a KVM host with /dev/kvm — a DO Droplet is not
one (no nested virtualisation; the unit's ConditionPathExists refuses by
design; no nested-FC workaround), so use DO bare-metal / dedicated
hardware or any bare-metal KVM host on the staging VPC / private
network, private bind, TLS + bearer. Host pre-checks, per-topic budget,
CP preconditions (lium, live_harvest_wired, judge, executor, a signed
custom topic with a sealed baseline), KVM-host and CP steps with the
staging env overlays, the wire check (env / agent / cp / boot-probe),
the fail-closed matrix (flip → restart? → expected 503 reason → what
the admin probe shows), the happy path with the exact log lines and
row / artefact evidence (sister sandboxed + flops_used, evidence bind,
destroy teardown), a sign-off checklist that records unknown / not run
rather than a green box, and rollback.
§ Wire the control plane documents GET /v1/admin/proof/vm-orchestrator
(field → root cause table; loopback only — staging's public API is
cleartext). Operate table gains "Is the wire up?".
Co-authored-by: Mathis <echobt@users.noreply.github.com>
* docs: point operators at the proof topic-vm staging harness
AGENTS.md verification item 7, deploy/AGENTS.md (staging wire section),
docs/AGENTS.md runbook index, docs/PROOF.md HTTP surface (the admin
vm-orchestrator probe), deploy/secrets/README.md (admin_tokens now also
gates the probe), docs/COMPLETENESS.md (probe + harness + overlays;
still not on any host, sign-off unfilled).
Co-authored-by: Mathis <echobt@users.noreply.github.com>
* test(proof-rlm-scorer): rlm_e2e app state carries vm_probe
Co-authored-by: Mathis <echobt@users.noreply.github.com>
* docs(runbook): staging may colocate the fc agent on the cp when /dev/kvm works
Architecte nit on #247: `cortex-staging` (nested DigitalOcean
virtualisation) did boot Firecracker and the § 4 fail-closed matrix came
back green, so the runbook must not claim a Droplet can never be the KVM
host. Corrected wording everywhere the PR (and the #244 lines next to
it) said so:
- staging may colocate the agent on the control-plane droplet when
/dev/kvm works — validated on cortex-staging;
- production prefers a dedicated DO bare-metal / KVM host;
- nested KVM remains fragile — if the boot fails or /dev/kvm goes away,
provision metal, do not patch around it (fragility signs listed);
- never Lium, never an emulator, never without /dev/kvm.
Runbook § DigitalOcean staging gains the colocated specifics: the CP
container reaches the agent on the droplet's VPC address (loopback in
the container is the container; FcConfig accepts plain http on loopback
only), agent bound on that address with TLS + SAN, :8200 firewalled to
the compose network, sizing on top of the compose stack, a local
`install` of the bearer copy; sign-off records the placement. The
fail-closed probes, the matrix, and the harness are unchanged.
Same correction in AGENTS.md item 7, deploy/AGENTS.md, both staging
overlays, the generic env examples, the systemd unit comment,
docs/PROOF.md, docs/ARCHITECTURE.md, docs/COMPLETENESS.md (staging
boot + § 4 green recorded; § 5 / § 6 still to be recorded), and the
agent / binary module docs.
Co-authored-by: Mathis <echobt@users.noreply.github.com>
* docs(proof-vm): staging colo = allowed proven exception; prod = dedicated do metal
Architecte GO conditionnel on #247: one wording across every placement
statement (runbook, AGENTS.md item 7, deploy/AGENTS.md, both staging
overlays, the generic env examples, the systemd unit comment,
docs/PROOF.md, docs/ARCHITECTURE.md, docs/COMPLETENESS.md, agent and
binary module docs):
- staging: colocating the agent on the CP droplet with nested /dev/kvm
is an allowed exception, proven on cortex-staging (§ 4 green); nested
stays fragile — if the boot fails, provision metal;
- production: dedicated DO metal preferred — never colocated on the CP.
No "a Droplet is not one / no nested" claim remains. Fail-closed probes,
matrix, and harness unchanged.
Co-authored-by: Mathis <echobt@users.noreply.github.com>
* fix(deploy): wire-check — case-insensitive prod guard, strand-proof boot-probe, budget flops
Greptile on #247:
P1 production guard bypass: refuse_prod parsed nothing and matched the
URL case-sensitively, so https://NETWORK.CORTEX.FOUNDATION reached
authenticated create / submit requests. The guard now parses the host
(scheme, userinfo, port, path, query, trailing dot stripped; IPv6 kept
bracketed), lower-cases it, and refuses a protected host or any
subdomain of one, plus a case-insensitive whole-URL match as belt and
braces. It runs at env load, before any other check, on every probe.
P2 lost response strands VM: PROBE_TOPIC_LIVE is set before the create
goes out; a create whose answer is 000 / 5xx / unparseable is reconciled
through GET /v1/vms/by-topic and any VM the agent reports is destroyed
in line (probe_reconcile_destroy), and the EXIT trap does the same for
anything still in flight (Ctrl-C, unconfirmed teardown). Topic state is
cleared only after a confirmed destroy followed by a 404, so a retry on
the probe topic is never blocked. PROOF_VM_WIRE_CHECK_FAULT=
lose-create-answer is a test-only hook that drops the create's answer.
P1 live probe under-declares FLOPs: submit-probe declared 1, so a live
run measuring more was a flops_under_declared reject. Fail-closed probes
still send 1 (nothing runs); --expect 2xx declares the topic's
flops_budget read from GET /v1/proof/topics/<id> (--declared-flops N
overrides; unreadable budget = FAIL naming the flag).
Tests (fake agent): uppercase / userinfo / trailing-dot / subdomain
production origins exit 2 before any request, for submit-probe and for
the agent URL at env load; the lost-answer create boots one VM that is
found by topic and destroyed, the topic is free, a retry passes.
Runbook § 3 / § 5 updated.
Co-authored-by: Mathis <echobt@users.noreply.github.com>
* style(proof-vm-fc): rustfmt the wire-check script test
Co-authored-by: Mathis <echobt@users.noreply.github.com>
* test(proof-vm-fc): split the lost-create reconcile probe into its own test
Co-authored-by: Mathis <echobt@users.noreply.github.com>
* fix(deploy): wire-check gates custom on custom_family_wired / custom_ready, not lium
After #246 the custom family is wired from the topic-VM env alone and
`live_harvest_wired` is the Lium harvest only (nll / throughput), so a
custom-only host reads false by design. The harness `cp` step no longer
FAILs on it: the flag is logged as informational, and the custom family
is gated on `custom_family_wired` (FAIL when ids are set but the family
is not routed), `registered_custom` ⊇ ids, and `custom_ready` ⊇ ids
(registered but not ready = bearer file / image pin on this host). The
admin probe log line shows the same three fields.
`VmOrchestratorReport` carries `custom_family_wired` next to
`registered_custom`, and its `live_harvest_wired` is the host's Lium-only
answer. Runbook (§ Wire the control plane, admin-probe field table, § 0
preconditions, § 3 cp row), the CP staging overlay, docs/PROOF.md, and
docs/COMPLETENESS.md say the same. Verified against a custom-only
loopback proof-challenge wired to a loopback agent: custom gates PASS
with live_harvest_wired=false logged, not failed.
Co-authored-by: Mathis <echobt@users.noreply.github.com>
* fix(deploy): wire-check refuses a zero declared_flops override
Greptile P2 on #247: `--declared-flops 0` passed the override check and
reached the control plane for a live probe, where any measured usage is
a flops_under_declared reject instead of the documented awaiting_admin
path. Explicit overrides must now be positive integers; zero, negative,
and non-numeric values exit 1 with an actionable message before any
request. Test covers 0 / abc / -1 on a live-run invocation.
Co-authored-by: Mathis <echobt@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Mathis <echobt@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the live
TopicVmOrchestratorfrom #243 — theUnwiredVmOrchestrator503 path is now the fail-closed default, not the only path. Each Proof custom topic gets an isolated Firecracker RLM microVM on a dedicated KVM host; every miner run happens in a sister Firecracker guest with no network beside it. TheTopicVmOrchestratorAPI from #243 is implemented, not redesigned.Crates / bins
crates/proof-rlm(additive)InspectOutcome/RunOutcome/VmJobOutput;VmJob::topic_id/deadline_s/requires_firecrackercrates/proof-vm-protocreate/attach/run/teardown,ErrorBody,SisterAttestation,bind_evidence) + vsock framing and messages agent ↔ RLM guest / sister guestcrates/proof-vm-fcFirecrackerOrchestrator—TopicVmOrchestratorimpl;from_env()isNonewhen the URL is unset; https only (plain http on loopback for tests); bearer from file, re-read per request, never loggedcrates/proof-vm-agentHypervisortrait (withaliveprobe), host stamping of paid outputs afterbind_evidence, dead-VM reaping;FakeHypervisor+ in-processFakeAgentbehindtest-fixturescrates/proof-fc-hostFirecrackerHypervisor: digest-verified images (images/sha256-<hex>.ext4, re-hashed, cached by len+mtime), jail layout +vm-config.jsonunder aJailGuard, jailer exec (no--daemonize/--new-pid-nsso the child handle is the VM), TAP + per-VM nftables table, vsock job channel, owner-key staging, sister guest flow (bound to the job, cooperatively cancelled), destroy-or-retain teardown. Every host command goes through aShelltraitbins/proof-vm-orchestratorPROOF_VM_AGENT_*env, TLS viaaxum-server(rustls), non-loopback bind without cert/key exits 1, pins required at bootbins/proof-challengePROOF_VM_ORCHESTRATOR_URL+_TOKEN_FILEare set (+PROOF_RLM_VM_IMAGE_DIGESTforready());PROOF_VM_RUNNER_CUSTOM_IDSregisters the genericVmBackedRunner(unset = empty registry)Locked decisions honoured
ConditionPathExists=/dev/kvm; nothing inproof-challengecan exec).VmJobpayloads cross the wire (tests assert no path / key / origin in any body); owner key material is staged by the agent fromPROOF_VM_AGENT_OWNER_KEY_DIR, CP only presence-probes.PROOF_RLM_VM_IMAGE_DIGESTleft empty / env-required, fail-closed (ready()→NotWirednaming the var). No digest invented anywhere (kernel / RLM / sister pins are computed by the operator from staged files).PROOF_VM_ORCHESTRATOR_URL+PROOF_VM_ORCHESTRATOR_TOKEN_FILE, bearer file first (mTLS follow-up), never logged.PROOF_VM_AGENT_SISTER_*) + topic deadline, never by the RLM.retainmoves the jail underretain_dir.topic_id ↔ VMbind on both sides.firecracker_requiredrun without the host's sister attestation → 503, no row, no host fallback.Host-stamped facts (the anti-cheat core)
The agent overwrites two report fields from what it booted:
sandboxed = trueonly when a sister guest ran;flops_used= the sister guest's measurement. An RLM claiming a sandbox without a sister is corrected tofalse(CP then refuses viaReportError::NotSandboxed); a sister that measured nothing yieldsflops_used: null→FlopsMissing503, never a substituted number. A guestFailed(nothing ran) isSome(0)so the RLM can write a persisted reject rather than a 503.Review hardening (Greptile P1s, head
68984c02)SisterAttestationcarriestopic_id/submission_digest/artifact_digestcopied from theSisterRequestthe host verified against the paid job before building the sister jail (a request naming anything else is refused with no jail).proof_vm_proto::bind_evidenceis the single fail-closed check: the agent runs it before stamping (502 evidence_mismatch, nothing stamped) and the CP runs it before accepting (503, no row). Evidence for artefact A never scores artefact B.jail::prepareon, aJailGuardowns the jail until the VM is registered; a failure at TAP / rules / spawn / hello / staging releases process, TAP, nftables table and directory before the error returns, and a dropped request releases them throughDrop— including one cancelled while parked on the VM registry lock, which is taken while the guard is still armed (keep()cannot disarm without a process to hand over).prepareremoves its own half-built jail.run_jobnever aborts the sister task: it fires aCancellationToken(also via a drop guard) and waits;sister::runraces the guest against it and always kills + destroys before returning.Hypervisor::aliveprobes the process on attach / create / run / health; a dead VM is released per its retain policy, recordedcrashed, and its topic may create a fresh VM (no 409 forever). A VM that dies under a job is reaped by that job.CI: zero live Firecracker
Every test uses
FakeHypervisor/RecordingShell(plus a test-onlyFailingShellto inject one host failure).FirecrackerHypervisor::ready()refuses on a host withoutfirecracker,jailer,/dev/kvm, and a test asserts nothing was spawned. The lifecycle tests that need a live child use a sleeping/bin/shstand-in for the jailer and a fake RLM guest on the jail's vsock UDS — no Firecracker, no KVM, no VM.proof-vm-fcis tested end to end against an in-process agent over the fake.Deploy / docs
deploy/systemd/proof-vm-orchestrator.service,deploy/env/proof-vm-orchestrator.env.example,deploy/env/proof-challenge.env.example(client block),deploy/secrets/README.md,docs/runbooks/proof-vm-orchestrator.md(host layout, install, CP wiring, mandatory end-to-end submission verification with 503 + cleanup probes, security model, v1 limitations),docs/PROOF.md,docs/COMPLETENESS.md,docs/ARCHITECTURE.md,AGENTS.md(+deploy/,docs/),docs/external-miner/proof.md(your code runs offline in a sister guest).Out of scope (as specified)
Challenge content,
EvalExecutorOfferchanges, nested FC, multi-GPU, inventing image digests, guest images (RLM / sister agents implementproof_vm_proto::guestoutside this repo), mTLS.Greptile
Every PR is reviewed by Greptile before merge. Config:
.greptile/.d0835a1f, follow-up P1 at68984c02; every thread answered with its fix commit and resolved)@greptileai reviewTest plan
cargo test --workspacecargo fmt --all -- --checkcargo clippy --workspace --all-targets -- -D warningscargo deny check(new dep:axum-server 0.8MIT,arc-swap,fs-err;tokio-util 0.7was already in the tree)xtask loc-cap(proof-fc-host1401,proof-rlm1286, others well under 1500),consensus-lint,spec-check,design-check,external-docs-checkRisk
No
BASE_*rename, no signature-domain or emission change, no compose change (the agent is a systemd unit on the KVM host, not a droplet service). Live custom submits still 503 until an operator stages images, pins their digests, and lists runner ids — which is the intended fail-closed state.Naming
I did not rename
BASE_*environment variables, deployed host paths(
/opt/base,/run/base, …), GHCRbaseintelligence/basepackage names, orbase-*-v1cryptographic domain tags. New names arePROOF_VM_*/PROOF_VM_AGENT_*/PROOF_RLM_VM_*only.