Repository navigation
Conversation
Adds the storage and provider layer for VM-backed Experiments, with no routes or UI yet: - Migration 167 adds vm_provider_configs, experiment_vms, experiment_previews and vm_operations, each with a per-table SQL module and QueryRecords schema. - experiment_vms.state and vm_operations.state have one owner each (src/shared/vm-state.ts) and one writer (src/server/vm/machine.ts), documented in docs/state-machines.md with a test that keeps the tables in step. - runVmOperation writes a receipt before every provider call, replays a repeated id, refuses a reused id with a different request, and keeps any unobserved outcome uncertain. reconcileVmCreate settles an uncertain create by the VM's deterministic name and never creates. - The worker tick marks receipts still pending after the longest provider call uncertain. - VmProvider plus a Sailboxes adapter over the HTTPS API (no @sailresearch/sdk, which ships native binaries). - The experiment image is a golden Sailbox checkpoint built from an install script, because Sail builds custom images only through its SDK. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pgrade test Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ery kind Review pass 1 fixes: - A reconcile adopts the remote id, settles the receipt and applies the observed VM state in one transaction, rolled back when another reconcile settled the receipt first. - A late provider answer no longer contradicts a receipt a reconcile settled; the caller gets the stored outcome. - A listing without a data array or has_more flag is uncertain, never proof that a VM is absent. - Adoption can find a VM already asleep or paused (provisioning -> sleeping/paused). - Exec output honours backpressure: the event stream pauses while Loop's consumer is behind. - Every uncertain operation kind can be reconciled read-only: sleep, resume and destroy from the VM status, expose and unexpose from the published port. vm_operations keeps request_json for that. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t out in-flight calls Review pass 2 fixes: - A reconcile observation that an effect did not happen (no VM, no port, VM not asleep) settles nothing until the longest provider call has passed since the receipt, so a still-running original call cannot be overtaken by a later operation. - runVmOperation takes effects that commit with the settled receipt; createdVmEffects adopts the provider id and observed state in that same transaction. - A create or fork whose VM boots into failed fails its receipt with Sail's reason and keeps the VM id; a reconcile that finds only a failed VM does the same. - A destroy that a reconcile finds unfinished moves the VM from destroying to failed, which allows another destroy. - Exec backpressure waits for every full stream, not only the last. Also from pass 2's open concerns: - The golden image is built with no volume mounted, so third-party installers never run beside harness credentials; experiments mount the auth volume after start (VmProvider.mountVolume). - experiment_vms inserts refuse another owner's folder or source experiment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…conciles Review pass 3 fixes: - A negative observation counts only when its lookup began after the in-flight window, not when its answer arrived. - A create or fork whose VM is still starting (or in an unrecognised state) stays uncertain instead of succeeding. - vmOperationEffects holds each kind's local writes: provider id and VM state for create/fork/sleep/resume/destroy, preview rows for expose and unexpose, and disabling every preview on destroy. runVmOperation and reconcileVmOperation apply the same effects in the settling transaction. - experiment_previews upserts require the owner's experiment. Also from pass 3's open concerns: a refused destroy moves a destroying VM to failed, and a 408 answer is uncertain rather than a refusal. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review pass 4 fixes: - A reconcile settles only from a lookup that began after the in-flight window, whatever it saw; only adopting an existing VM for a create or fork settles at once, since a late original create reaches the same VM. This replaces the per-case "did not happen" flag that left positive observations (an absent port after unexpose) open to a late call. - A create or fork answered while its VM is still starting keeps its receipt uncertain, with the provider id adopted, for a reconcile. - failed -> sleeping/paused: a retry or reconcile can adopt a VM found asleep after an earlier attempt failed. - A listing entry without a name is uncertain, not skipped. From pass 4's open concerns: - vm_operations ids are unique per owner (PRIMARY KEY (owner_id, id)), so one owner's ids neither collide with nor reveal another's; preview ids are derived from the experiment and port. - Each owner gets its own auth volume name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eadable exec Review pass 5 fixes: - A create answered while its VM is still starting keeps the provider id in the receipt only, so a reconcile that finds it absent leaves the experiment free for a retry. Once a VM id is recorded, another create or fork is refused (vm_exists). - sleeping <-> paused: every live state reaches every other, so a reconcile never drops an observed status. - An exec event Loop cannot read makes the exit code null rather than reporting truncated output as a clean exit; unknown event types are still ignored. From pass 5's open concerns: provider requests refuse redirects, so the API key, file bodies and exec environments are never resent elsewhere. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…comes Review pass 6 fixes: - loop-vm-init moves any regular credential file onto the auth volume before relinking, so a harness that refreshed by replacing its link keeps the new credential instead of being pointed back at the old one. - A resume answer other than running/already_running/terminal_unavailable is uncertain, not success. From pass 6's open concerns: a destroy sent while the VM was still running ends destroyed through destroying, and an exec event longer than 16 MB stops the stream and reports no exit code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
j-asefa
force-pushed
the
feat/vm-experiments-1-storage
branch
from
November 20, 2026 19:26
d5f81b3 to
937685a
Compare
…init Review pass 7 fixes: - loop-vm-init stops before touching credentials unless the auth volume is mounted, so a skipped mount cannot leave credentials on the VM disk. - The workspace's initial commit is checked by HEAD, not by .git, so a run stopped after git init finishes next time. - The 16 MB exec limit applies to the unfinished event only, after a chunk's complete events are handled. From pass 7's open concerns: runVmOperation applies the kind's own effects unless a caller passes others, so a create cannot settle without recording its VM, and operations other than create/fork are refused (no_vm) before the experiment has a VM. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cloud VM experiments 1/5: storage, state machines and Sailboxes provider
First of five PRs for VM-backed Experiments ("Pocket Lab"). This slice adds the storage and the provider layer only. No route, tool, UI or caller in the API uses it yet, so nothing is reachable from a browser.
What changes
migrations/167-experiment-vms.sqladds four tables. Each has a per-table module insrc/server/sql/and aQueryRecordsentry, anddocs/sql-index.mdis regenerated.vm_provider_configs: the owner's provider account. The key column holds a sealed value; the module never sees the key.experiment_vms: one VM per project folder (folder_id UNIQUE).remote_nameis unique and chosen before any provider call.experiment_previewsvm_operationsexperiment_vms.stateandvm_operations.stateeach have one owner (src/shared/vm-state.ts) and one writer (src/server/vm/machine.ts). The writer re-reads the row inside one immediate transaction (tx). Both are documented indocs/state-machines.md.test/vm-state-documentation.test.ts, added tonpm run check, keeps the doc tables equal to the transition tables.runVmOperation(src/server/vm/operations.ts) commits apendingreceipt before calling the provider. Its id is the caller's idempotency key.VmProviderErrorwith outcomerefused) marks a receiptfailed. Timeouts, 5xx responses, dropped connections, unreadable bodies and adapter exceptions leave ituncertain.reconcileVmOperationsettles an uncertain operation by reading from the provider. It never creates, sends or retries anything.remote_name. One match is adopted, and no match fails the receipt so the owner can start a new operation. A failed lookup, more than one match, or a listing Loop cannot read leaves it uncertain.vm_operations.request_jsonkeeps the request so a reconcile can check it.failedfails its receipt with Sail's reason and keeps the VM id so it can be destroyed. An unfinished destroy moves the VM fromdestroyingtofailed.vmOperationEffectsholds each kind's local writes: provider id and VM state, preview rows for expose and unexpose, and every preview disabled on destroy. An answer and a reconcile both apply them in the transaction that settles the receipt.vm_exists): a failed VM is destroyed, not created again.src/server/worker.ts) callsmarkStaleVmOperationsUncertain. It marks receipts stillpendingafter the longest provider call (15 min, against a 10 min boot timeout)uncertain. Nothing is resent, and a late answer settles the receipt only if nothing else has.VmProvider(src/server/vm/provider.ts) andcreateSailboxProvider(src/server/vm/sailbox-provider.ts) talk to the Sailboxes HTTPS API with an injectedfetch. Paths and fields come from Sail's published OpenAPI document.@sailresearch/sdkis not added, because it ships per-platform native binaries.PUT …/stdinat absolute offsets.sh -cwrapper. Sail appliescwdonly to string commands, and Loop never shell-parses a command.src/server/vm/sailbox-image.tsdefines the install script and the per-VMloop-vm-initscript.BASE_IMAGE_DEVBOX(Debian with systemd and Docker) with no volume mounted, run the install script, and take a filesystem-only checkpoint that experiments start from. The third-party installers never run beside the credentials.loop-authafter it starts (mountVolume). Checkpoints do not version volume contents, so credentials never land in an image or a fork.SAILBOX_IMAGE_VERSIONis a hash of the script.Owner constraints applied (2026-11-20)
--public-url(PR 2), and tool-bridge tickets carry only Loop's built-in VM tools (PR 2). Neither is in this slice.Evidence for review
experiment_vmsinserts andexperiment_previewsupserts refuse another owner's records.experiment_vmsinserts refuse another owner's folder or source experiment. Every query takes the owner and filtersowner_id, except the stale-receipt sweep. Operation ids are unique per owner (PRIMARY KEY (owner_id, id)), and each owner has its own auth volume name.vm-operations.test.ts"keeps another owner's experiment and operation ids out of reach".vm-operations-worker.test.tsexercises the production caller.Authorizationheader, and provider requests refuse redirects. The fixture checks it is absent from the request body. The init test checks that only credential files link to the auth volume and session directories stay local.statement();sql-inventoryandrow-schema-columnspass.docs/state-machines.mdkeeps its existing table layout.Verification
All results below are fixture coverage. No Sailbox was contacted: there is no
SAIL_API_KEYon this machine, and live Sailbox validation is still to be done. The install script has only been syntax-checked (bash -n), not run on a Sailbox. The init script was executed locally against temporary directories.npm run check: exit 0 on headafabb2a65cd27a00825bf3af3bba83e117342aed(rebased onto87525aed0; contracts 447 tests) (assets, format, typecheck, openapi, sql-index, contracts 39 files / 447 tests).test/vm-operations.test.ts(44),test/vm-sailbox-provider.test.ts(19),test/vm-sailbox-image.test.ts(5),test/vm-operations-worker.test.ts(1),test/vm-state-documentation.test.ts(2).npm test -- --shard=N/3: shard 1 exit 0. Shard 2 failed onlyfleet-credentials.test.tsand shard 3 failed onlyfleet-handoff-http.test.ts, both with "Test timed out in 5000ms" at load average ~10. Rerunning both files alone: 33/33 passed, exit 0.npm run test:native: exit 0, 512/512 passed, afternpm run build. A first run withoutdist/failed the compiled tests with ERR_MODULE_NOT_FOUND.Known limits, for later slices
exitwith no code; reconnecting with the exec idempotency key and resume cursors is PR 2's transport work.loop-vm-initmoves it back onto the volume, so another VM can sign in with the older copy in between. Each harness's refresh behaviour needs live validation.Review
f399c8d1): 3 high and 3 medium findings, all confirmed and fixed in16ba7e5c, each with a regression test named after it.1babd635): 2 high and 3 medium findings, all fixed ine7c1d439. Two of its open concerns are also addressed: the image build no longer mounts credentials, and inserts are owner-checked.0f9f3f84): 1 high and 3 medium findings, all fixed in24e19980. Two open concerns are also addressed: a refused destroy fails adestroyingVM, and a 408 is uncertain. Still open: binding the reconcile provider and app id to the owner's account belongs to the PR 3/4 callers.8db42160): 2 high and 2 medium findings, fixed ine27d382a. The per-case timing flags are replaced by the single timing rule above. Two open concerns are also addressed: operation ids are scoped to the owner, and auth volumes are named per owner.7779394a): 1 P1 and 2 P2 findings, fixed ina7ce63b1. One open concern is also addressed: redirects are refused.38b25fd2): 1 P1 and 3 P2 findings. The P2s are fixed inafabb2a6: init requires the auth mount, finishes an interrupted initial commit, and bounds only the unfinished exec event. The P1 (a Sail call landing after Loop's 15-minute window) is open and waiting on an owner decision; it is the same trade-off as passes 3–6.da68be23): 1 high and 1 medium finding, fixed ind5f81b3c. Two open concerns are also addressed: a destroy sent while the VM is running ends destroyed, and exec events are capped at 16 MB.🤖 Generated with Claude Code