Skip to content

Cloud VM experiments 1/5: storage, state machines and Sailboxes provider - #409

Open
j-asefa wants to merge 9 commits into
mainfrom
feat/vm-experiments-1-storage
Open

j-asefa wants to merge 9 commits into
mainfrom
feat/vm-experiments-1-storage

Conversation

@j-asefa

@j-asefa j-asefa commented Nov 20, 2026 •

Copy link
Copy Markdown
Owner

Cloud VM experiments 1/5: storage, state machines and Sailboxes provider

First of five PRs for VM-backed Experiments ("Pocket Lab"). This slice adds the storage and the provider layer only. No route, tool, UI or caller in the API uses it yet, so nothing is reachable from a browser.

What changes

  • Storage. migrations/167-experiment-vms.sql adds four tables. Each has a per-table module in src/server/sql/ and a QueryRecords entry, and docs/sql-index.md is regenerated.
    • vm_provider_configs: the owner's provider account. The key column holds a sealed value; the module never sees the key.
    • experiment_vms: one VM per project folder (folder_id UNIQUE). remote_name is unique and chosen before any provider call.
    • experiment_previews
    • vm_operations
  • State machines. experiment_vms.state and vm_operations.state each have one owner (src/shared/vm-state.ts) and one writer (src/server/vm/machine.ts). The writer re-reads the row inside one immediate transaction (tx). Both are documented in docs/state-machines.md. test/vm-state-documentation.test.ts, added to npm run check, keeps the doc tables equal to the transition tables.
  • Receipts before effects. runVmOperation (src/server/vm/operations.ts) commits a pending receipt before calling the provider. Its id is the caller's idempotency key.
    • A repeated id with the same request returns the stored outcome without a call. The same id with a different request is refused.
    • So is an id another owner holds, and so is a new operation while an earlier one on the same experiment is unsettled.
    • Only a provider answer (VmProviderError with outcome refused) marks a receipt failed. Timeouts, 5xx responses, dropped connections, unreadable bodies and adapter exceptions leave it uncertain.
  • Reconcile before retry. reconcileVmOperation settles an uncertain operation by reading from the provider. It never creates, sends or retries anything.
    • A create or fork is settled by listing the VM's exact remote_name. One match is adopted, and no match fails the receipt so the owner can start a new operation. A failed lookup, more than one match, or a listing Loop cannot read leaves it uncertain.
    • Sleep, resume and destroy are settled from the VM's status, and expose and unexpose from the published port. vm_operations.request_json keeps the request so a reconcile can check it.
    • The receipt, the adopted id and the observed VM state are written in one transaction. That transaction rolls back if another reconcile settled the receipt first, and a late provider answer then gets the stored outcome.
    • An observation that the effect did not happen settles nothing until 15 minutes after the receipt, because the original call could still be in flight.
    • A create whose VM boots into failed fails its receipt with Sail's reason and keeps the VM id so it can be destroyed. An unfinished destroy moves the VM from destroying to failed.
    • vmOperationEffects holds each kind's local writes: provider id and VM state, preview rows for expose and unexpose, and every preview disabled on destroy. An answer and a reconcile both apply them in the transaction that settles the receipt.
    • Timing rule: a reconcile settles only from a lookup that began more than 15 minutes after the receipt, because until then the original call may still land and change what the lookup saw. The only exception is adopting an existing VM for a create or fork, which settles at once: a late original create reaches the same VM through its experiment-scoped provider key. The 15 minutes is Loop's own call timeout; Sail does not document a deadline for applying an accepted request.
    • A create answered, or found, while its VM is still starting stays uncertain, and its provider id is kept only in the receipt until a reconcile sees the VM. Once a VM id is recorded, another create or fork is refused (vm_exists): a failed VM is destroyed, not created again.
    • Every live state (running, sleeping, paused) can reach every other, so an observed status is never dropped.
  • Stale receipt sweep. The worker tick (src/server/worker.ts) calls markStaleVmOperationsUncertain. It marks receipts still pending after the longest provider call (15 min, against a 10 min boot timeout) uncertain. Nothing is resent, and a late answer settles the receipt only if nothing else has.
  • Exec backpressure. The NDJSON event stream pauses while Loop's reader is behind, so a verbose command cannot fill server memory.
  • Provider. VmProvider (src/server/vm/provider.ts) and createSailboxProvider (src/server/vm/sailbox-provider.ts) talk to the Sailboxes HTTPS API with an injected fetch. Paths and fields come from Sail's published OpenAPI document. @sailresearch/sdk is not added, because it ships per-platform native binaries.
    • Exec uses the NDJSON event stream plus PUT …/stdin at absolute offsets.
    • Exec's working directory is set by a fixed sh -c wrapper. Sail applies cwd only to string commands, and Loop never shell-parses a command.
  • Experiment image. src/server/vm/sailbox-image.ts defines the install script and the per-VM loop-vm-init script.
    • Sail builds custom images only through its SDK. Loop will therefore boot BASE_IMAGE_DEVBOX (Debian with systemd and Docker) with no volume mounted, run the install script, and take a filesystem-only checkpoint that experiments start from. The third-party installers never run beside the credentials.
    • Each experiment mounts loop-auth after it starts (mountVolume). Checkpoints do not version volume contents, so credentials never land in an image or a fork.
    • SAILBOX_IMAGE_VERSION is a hash of the script.
    • The build step itself lands with the config routes in PR 3.

Owner constraints applied (2026-11-20)

  • Direct HTTP API, no SDK. Reconcile filters the list endpoint by exact name.
  • The public tool bridge needs an explicit stable --public-url (PR 2), and tool-bridge tickets carry only Loop's built-in VM tools (PR 2). Neither is in this slice.

Evidence for review

Rule Evidence
Owner/profile isolation experiment_vms inserts and experiment_previews upserts refuse another owner's records. experiment_vms inserts refuse another owner's folder or source experiment. Every query takes the owner and filters owner_id, except the stale-receipt sweep. Operation ids are unique per owner (PRIMARY KEY (owner_id, id)), and each owner has its own auth volume name. vm-operations.test.ts "keeps another owner's experiment and operation ids out of reach".
Dispatch identity The receipt is committed before the call ("writes the pending receipt before the provider is called").
Uncertain effects "does not call the provider again for an uncertain operation or start another beside it"; "treats an error that is not a provider answer as uncertain"; the worker-tick regression in vm-operations-worker.test.ts exercises the production caller.
Native context and credentials The API key is sent only in the Authorization header, and provider requests refuse redirects. The fixture checks it is absent from the request body. The init test checks that only credential files link to the auth volume and session directories stay local.
SQL query modules Four new modules using lazy statement(); sql-inventory and row-schema-columns pass.
Historical data Additive migration. "adds the experiment tables to a database a released version wrote, leaving its rows alone" migrates through 166 with a folder row, then runs the chain.
Formatting Pinned prettier on src/test/scripts; docs/state-machines.md keeps its existing table layout.

Verification

All results below are fixture coverage. No Sailbox was contacted: there is no SAIL_API_KEY on this machine, and live Sailbox validation is still to be done. The install script has only been syntax-checked (bash -n), not run on a Sailbox. The init script was executed locally against temporary directories.

  • npm run check: exit 0 on head afabb2a65cd27a00825bf3af3bba83e117342aed (rebased onto 87525aed0; contracts 447 tests) (assets, format, typecheck, openapi, sql-index, contracts 39 files / 447 tests).
  • New tests: test/vm-operations.test.ts (44), test/vm-sailbox-provider.test.ts (19), test/vm-sailbox-image.test.ts (5), test/vm-operations-worker.test.ts (1), test/vm-state-documentation.test.ts (2).
  • Full suite: npm test -- --shard=N/3: shard 1 exit 0. Shard 2 failed only fleet-credentials.test.ts and shard 3 failed only fleet-handoff-http.test.ts, both with "Test timed out in 5000ms" at load average ~10. Rerunning both files alone: 33/33 passed, exit 0.
  • npm run test:native: exit 0, 512/512 passed, after npm run build. A first run without dist/ failed the compiled tests with ERR_MODULE_NOT_FOUND.

Known limits, for later slices

  • A dropped exec event stream resolves exit with no code; reconnecting with the exec idempotency key and resume cursors is PR 2's transport work.
  • The install script runs the vendors' latest installers and unpinned packages (with no credentials mounted). Pinning harness versions and checksums is a follow-up before the image build ships in PR 3.
  • A harness that refreshes a credential by replacing its link writes a local file until the next loop-vm-init moves it back onto the volume, so another VM can sign in with the older copy in between. Each harness's refresh behaviour needs live validation.
  • Any unsettled operation blocks new operations on that experiment until the owner reconciles it. Every kind can be reconciled.
  • Replacing a saved Sail key must not let reconcile treat a VM in the old account as absent. PR 3's config route will refuse a key change while experiments exist, or reset the saved resources.

Review

  • Pass 1 (f399c8d1): 3 high and 3 medium findings, all confirmed and fixed in 16ba7e5c, each with a regression test named after it.
  • Pass 2 (1babd635): 2 high and 3 medium findings, all fixed in e7c1d439. Two of its open concerns are also addressed: the image build no longer mounts credentials, and inserts are owner-checked.
  • Pass 3 (0f9f3f84): 1 high and 3 medium findings, all fixed in 24e19980. Two open concerns are also addressed: a refused destroy fails a destroying VM, and a 408 is uncertain. Still open: binding the reconcile provider and app id to the owner's account belongs to the PR 3/4 callers.
  • Pass 4 (8db42160): 2 high and 2 medium findings, fixed in e27d382a. The per-case timing flags are replaced by the single timing rule above. Two open concerns are also addressed: operation ids are scoped to the owner, and auth volumes are named per owner.
  • Pass 5 (7779394a): 1 P1 and 2 P2 findings, fixed in a7ce63b1. One open concern is also addressed: redirects are refused.
  • Pass 7 (38b25fd2): 1 P1 and 3 P2 findings. The P2s are fixed in afabb2a6: init requires the auth mount, finishes an interrupted initial commit, and bounds only the unfinished exec event. The P1 (a Sail call landing after Loop's 15-minute window) is open and waiting on an owner decision; it is the same trade-off as passes 3–6.
  • Pass 6 (da68be23): 1 high and 1 medium finding, fixed in d5f81b3c. Two open concerns are also addressed: a destroy sent while the VM is running ends destroyed, and exec events are capped at 16 MB.

🤖 Generated with Claude Code

j-asefa and others added 8 commits November 20, 2026 19:25
Adds the storage and provider layer for VM-backed Experiments, with no
routes or UI yet:

- Migration 167 adds vm_provider_configs, experiment_vms,
  experiment_previews and vm_operations, each with a per-table SQL module
  and QueryRecords schema.
- experiment_vms.state and vm_operations.state have one owner each
  (src/shared/vm-state.ts) and one writer (src/server/vm/machine.ts),
  documented in docs/state-machines.md with a test that keeps the tables in
  step.
- runVmOperation writes a receipt before every provider call, replays a
  repeated id, refuses a reused id with a different request, and keeps any
  unobserved outcome uncertain. reconcileVmCreate settles an uncertain
  create by the VM's deterministic name and never creates.
- The worker tick marks receipts still pending after the longest provider
  call uncertain.
- VmProvider plus a Sailboxes adapter over the HTTPS API (no
  @sailresearch/sdk, which ships native binaries).
- The experiment image is a golden Sailbox checkpoint built from an install
  script, because Sail builds custom images only through its SDK.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pgrade test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ery kind

Review pass 1 fixes:
- A reconcile adopts the remote id, settles the receipt and applies the
  observed VM state in one transaction, rolled back when another reconcile
  settled the receipt first.
- A late provider answer no longer contradicts a receipt a reconcile
  settled; the caller gets the stored outcome.
- A listing without a data array or has_more flag is uncertain, never proof
  that a VM is absent.
- Adoption can find a VM already asleep or paused (provisioning ->
  sleeping/paused).
- Exec output honours backpressure: the event stream pauses while Loop's
  consumer is behind.
- Every uncertain operation kind can be reconciled read-only: sleep, resume
  and destroy from the VM status, expose and unexpose from the published
  port. vm_operations keeps request_json for that.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t out in-flight calls

Review pass 2 fixes:
- A reconcile observation that an effect did not happen (no VM, no port,
  VM not asleep) settles nothing until the longest provider call has passed
  since the receipt, so a still-running original call cannot be overtaken
  by a later operation.
- runVmOperation takes effects that commit with the settled receipt;
  createdVmEffects adopts the provider id and observed state in that same
  transaction.
- A create or fork whose VM boots into failed fails its receipt with Sail's
  reason and keeps the VM id; a reconcile that finds only a failed VM does
  the same.
- A destroy that a reconcile finds unfinished moves the VM from destroying
  to failed, which allows another destroy.
- Exec backpressure waits for every full stream, not only the last.

Also from pass 2's open concerns:
- The golden image is built with no volume mounted, so third-party
  installers never run beside harness credentials; experiments mount the
  auth volume after start (VmProvider.mountVolume).
- experiment_vms inserts refuse another owner's folder or source experiment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…conciles

Review pass 3 fixes:
- A negative observation counts only when its lookup began after the
  in-flight window, not when its answer arrived.
- A create or fork whose VM is still starting (or in an unrecognised
  state) stays uncertain instead of succeeding.
- vmOperationEffects holds each kind's local writes: provider id and VM
  state for create/fork/sleep/resume/destroy, preview rows for expose and
  unexpose, and disabling every preview on destroy. runVmOperation and
  reconcileVmOperation apply the same effects in the settling transaction.
- experiment_previews upserts require the owner's experiment.

Also from pass 3's open concerns: a refused destroy moves a destroying VM
to failed, and a 408 answer is uncertain rather than a refusal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review pass 4 fixes:
- A reconcile settles only from a lookup that began after the in-flight
  window, whatever it saw; only adopting an existing VM for a create or
  fork settles at once, since a late original create reaches the same VM.
  This replaces the per-case "did not happen" flag that left positive
  observations (an absent port after unexpose) open to a late call.
- A create or fork answered while its VM is still starting keeps its
  receipt uncertain, with the provider id adopted, for a reconcile.
- failed -> sleeping/paused: a retry or reconcile can adopt a VM found
  asleep after an earlier attempt failed.
- A listing entry without a name is uncertain, not skipped.

From pass 4's open concerns:
- vm_operations ids are unique per owner (PRIMARY KEY (owner_id, id)), so
  one owner's ids neither collide with nor reveal another's; preview ids
  are derived from the experiment and port.
- Each owner gets its own auth volume name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eadable exec

Review pass 5 fixes:
- A create answered while its VM is still starting keeps the provider id
  in the receipt only, so a reconcile that finds it absent leaves the
  experiment free for a retry. Once a VM id is recorded, another create or
  fork is refused (vm_exists).
- sleeping <-> paused: every live state reaches every other, so a
  reconcile never drops an observed status.
- An exec event Loop cannot read makes the exit code null rather than
  reporting truncated output as a clean exit; unknown event types are
  still ignored.

From pass 5's open concerns: provider requests refuse redirects, so the
API key, file bodies and exec environments are never resent elsewhere.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…comes

Review pass 6 fixes:
- loop-vm-init moves any regular credential file onto the auth volume
  before relinking, so a harness that refreshed by replacing its link
  keeps the new credential instead of being pointed back at the old one.
- A resume answer other than running/already_running/terminal_unavailable
  is uncertain, not success.

From pass 6's open concerns: a destroy sent while the VM was still running
ends destroyed through destroying, and an exec event longer than 16 MB
stops the stream and reports no exit code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@j-asefa
j-asefa force-pushed the feat/vm-experiments-1-storage branch from d5f81b3 to 937685a Compare November 20, 2026 19:26
…init

Review pass 7 fixes:
- loop-vm-init stops before touching credentials unless the auth volume
  is mounted, so a skipped mount cannot leave credentials on the VM disk.
- The workspace's initial commit is checked by HEAD, not by .git, so a run
  stopped after git init finishes next time.
- The 16 MB exec limit applies to the unfinished event only, after a
  chunk's complete events are handled.

From pass 7's open concerns: runVmOperation applies the kind's own effects
unless a caller passes others, so a create cannot settle without recording
its VM, and operations other than create/fork are refused (no_vm) before
the experiment has a VM.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant