Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions docs/sql-index.md
Original file line number Diff line number Diff line change
Expand Up @@ -511,6 +511,24 @@ Generated by `npm run sql-index` from [src/server/sql](../src/server/sql/); do n
- `start(db, owner, command, method, digest, generation, created, updated)`
- `unknown(db, time, owner, command, generation)`

## [experiment-previews.ts](../src/server/sql/experiment-previews.ts)

- `byExperiment(db, owner, experiment)`
- `byPort(db, owner, experiment, port)`
- `deactivate(db, owner, experiment, port, now)`
- `deactivateAll(db, owner, experiment, now)` — A destroyed VM serves nothing, so every link of the experiment is disabled.
- `upsert(db, preview)` — One preview per experiment port; exposing the port again reactivates it.

## [experiment-vms.ts](../src/server/sql/experiment-vms.ts)

- `adoptRemote(db, owner, id, remoteId)` — Records the provider's id once; a different id for the same VM is refused.
- `byFolder(db, owner, folder)`
- `byId(db, owner, id)`
- `insert(db, vm)` — Inserts the owner's experiment.
- `owned(db, owner)`
- `setState(db, owner, id, from, to, lastError, now)` — Moves `from` to `to`; returns false when the row is no longer in `from`.
- `touch(db, owner, id, now)`

## [feed-card-dirty.ts](../src/server/sql/feed-card-dirty.ts)

- `any(db)`
Expand Down Expand Up @@ -1639,6 +1657,22 @@ Generated by `npm run sql-index` from [src/server/sql](../src/server/sql/); do n
- `profile(db, owner)`
- `save(db, owner, name, handle, bio, updated)`

## [vm-operations.ts](../src/server/sql/vm-operations.ts)

- `byId(db, owner, id)`
- `insertPending(db, operation)`
- `pendingBefore(db, cutoff)` — Every owner's receipts still pending since before \`cutoff\`, for the recovery sweep only.
- `setState(db, owner, id, from, to, outcome, now)` — Moves `from` to `to`; returns false when the row is no longer in `from`.
- `unsettledByExperiment(db, owner, experiment)`

## [vm-provider-configs.ts](../src/server/sql/vm-provider-configs.ts)

- `byOwner(db, owner)`
- `remove(db, owner)`
- `saveKey(db, owner, provider, sealedKey, now)` — Stores the sealed key, keeping the resources already found for the owner.
- `setDefaultHarness(db, owner, harness, now)`
- `setResources(db, owner, resources, now)`

## [webhook-deliveries.ts](../src/server/sql/webhook-deliveries.ts)

- `activation(db, subscription)`
Expand Down
38 changes: 38 additions & 0 deletions docs/state-machines.md
Original file line number Diff line number Diff line change
Expand Up @@ -365,6 +365,44 @@ rolls back local work before recording either a refusal or another attempt. This
an uncertain harness delivery. Retry records cascade with their source run or
mail; ineligible results retain their metadata until source deletion.

## Cloud VM experiments

### `experiment_vms.state`

Owner: `VM_TRANSITIONS` in [vm-state.ts](../src/shared/vm-state.ts), applied
only by `transitionVm()` in [vm/machine.ts](../src/server/vm/machine.ts), which
re-reads the owner's row inside one immediate transaction. A destroyed VM
leaves its project, threads, results and published files readable.

| From | To | Trigger and guard |
| ---- | -- | ----------------- |
| `provisioning` | `running` / `sleeping` / `paused` / `failed` / `destroying` | The provider reported the VM up (a reconcile can find it already asleep or paused), refused the create, or the owner destroyed it; **tx** |
| `running` | `sleeping` / `paused` / `destroying` / `failed` | Sleep observed or requested, pause, owner destroy, or a provider failure observed; **tx** |
| `sleeping` | `running` / `paused` / `destroying` / `failed` | Wake for a turn or a preview, the provider reporting it paused, owner destroy, or a failed wake; **tx** |
| `paused` | `running` / `sleeping` / `destroying` / `failed` | Resume, the provider reporting it asleep, owner destroy, or a failed resume; **tx** |
| `failed` | `provisioning` / `running` / `sleeping` / `paused` / `destroying` | The owner's reconcile or retry (which can adopt a VM found asleep or paused), or owner destroy; **tx** |
| `destroying` | `destroyed` / `failed` | The provider confirmed the VM is gone (a missing VM counts), refused, or a reconcile found the destroy did not finish; **tx** |
| **`destroyed`** | — | Terminal |

### `vm_operations.state`

Owner: `VM_OPERATION_TRANSITIONS` in the same file, applied only by
`transitionVmOperation()`. The row is a receipt: it is inserted `pending`
before the provider is called, and its id is the caller's idempotency key.

| From | To | Trigger and guard |
| ---- | -- | ----------------- |
| `pending` | `succeeded` / `failed` / `uncertain` | The provider answered, refused, or gave no observed answer (timeout, disconnect, server error, or the worker's sweep found the row still pending after the longest provider call); **tx** |
| `uncertain` | `succeeded` / `failed` | A late answer arrived, or the owner's reconcile observed the outcome: a create or fork by the VM's `remote_name`, any other kind by the VM's status or published port; **tx** |
| **`succeeded`** | — | Terminal |
| **`failed`** | — | Terminal |

An uncertain operation is never sent again automatically. Whichever settles the receipt first wins; a later answer that disagrees returns the stored outcome. `vmOperationEffects` in [vm/operations.ts](../src/server/vm/operations.ts) holds each kind's local writes (provider id, VM state, preview rows), and both an answer and a reconcile apply them in the transaction that settles the receipt. A reconcile settles only from a lookup that began after the longest provider call had passed since the receipt was written, because the original call may still land and change what the lookup saw. The exception is a create or fork whose VM exists, which settles at once: the original create reaches the same VM through its experiment-scoped provider key. A create answered while its VM is still starting stays uncertain and records nothing on the VM; once the VM's provider id is recorded, a create or fork for that experiment is refused, so a failed VM is destroyed rather than created again. The window is Loop's own call timeout; Sail does not document a deadline for applying a request it accepted. A create whose VM boots into a failed state fails its receipt and keeps the VM's id, so the owner can destroy it. A retry is a new
operation with a new id, and is refused while an earlier one for the same
experiment is unsettled. Creates and forks send the provider an idempotency
key scoped to the experiment, so a retry after an "absent" reconcile reaches
the VM a still-running first attempt made instead of starting a second one.

## Fleets (experimental)

| Record | States and edges | Guards |
Expand Down
79 changes: 79 additions & 0 deletions migrations/167-experiment-vms.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
-- Cloud VM Experiments. One Experiment is one provider VM linked 1:1 to a
-- project folder. Conversations, results and published files stay in this
-- database and its object store, so no row here is needed to read history.
-- Additive: releases before this migration wrote none of these tables.

-- The owner's provider account. The key is sealed with the owner-bound
-- secret store and never leaves this host.
CREATE TABLE vm_provider_configs (
owner_id TEXT PRIMARY KEY,
provider TEXT NOT NULL,
encrypted_api_key TEXT NOT NULL,
app_id TEXT,
auth_volume_id TEXT,
base_image_ref TEXT,
base_image_version TEXT,
default_harness TEXT,
updated_at INTEGER NOT NULL
);

-- remote_name is chosen before the provider is called, so an uncertain create
-- can be reconciled by looking the name up instead of creating a second VM.
CREATE TABLE experiment_vms (
id TEXT PRIMARY KEY,
owner_id TEXT NOT NULL,
folder_id TEXT NOT NULL UNIQUE REFERENCES folders(id),
provider TEXT NOT NULL,
remote_id TEXT,
remote_name TEXT NOT NULL UNIQUE,
state TEXT NOT NULL,
auto_approve_local INTEGER NOT NULL DEFAULT 1,
vcpu_limit INTEGER NOT NULL DEFAULT 4,
memory_limit_gib INTEGER NOT NULL DEFAULT 16,
disk_limit_gib INTEGER NOT NULL DEFAULT 64,
auto_sleep_seconds INTEGER NOT NULL DEFAULT 900,
source_experiment_id TEXT REFERENCES experiment_vms(id),
source_checkpoint_id TEXT,
clone_url TEXT,
last_error TEXT,
last_active_at INTEGER NOT NULL,
created_at INTEGER NOT NULL,
destroyed_at INTEGER
);
CREATE INDEX experiment_vms_owner ON experiment_vms(owner_id, created_at);

CREATE TABLE experiment_previews (
id TEXT PRIMARY KEY,
owner_id TEXT NOT NULL,
experiment_id TEXT NOT NULL REFERENCES experiment_vms(id),
port INTEGER NOT NULL,
label TEXT,
path TEXT NOT NULL DEFAULT '/',
url TEXT NOT NULL,
active INTEGER NOT NULL DEFAULT 1,
created_at INTEGER NOT NULL,
updated_at INTEGER NOT NULL,
UNIQUE(experiment_id, port)
);

-- A receipt written before every provider call. id is the caller's
-- idempotency key, unique within the owner; request_digest refuses the same key for a different
-- request, and request_json keeps the request so a reconcile can check what
-- was asked. An operation left pending by a crash becomes uncertain and is
-- never sent again without the owner asking.
CREATE TABLE vm_operations (
id TEXT NOT NULL,
owner_id TEXT NOT NULL,
experiment_id TEXT NOT NULL REFERENCES experiment_vms(id),
kind TEXT NOT NULL,
state TEXT NOT NULL,
request_digest TEXT NOT NULL,
request_json TEXT NOT NULL,
result_json TEXT,
error_message TEXT,
created_at INTEGER NOT NULL,
updated_at INTEGER NOT NULL,
PRIMARY KEY (owner_id, id)
);
CREATE INDEX vm_operations_experiment ON vm_operations(experiment_id, created_at);
CREATE INDEX vm_operations_unsettled ON vm_operations(state) WHERE state IN ('pending', 'uncertain');
1 change: 1 addition & 0 deletions scripts/check.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ const tests = [
"test/json-column-boundaries.test.ts",
"test/discriminating-fields.test.ts",
"test/run-state-documentation.test.ts",
"test/vm-state-documentation.test.ts",
"test/guidance-links.test.ts",
"test/migration-policy.test.ts",
"test/row-schema-columns.test.ts",
Expand Down
96 changes: 96 additions & 0 deletions src/server/sql/experiment-previews.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
import type Database from "better-sqlite3";
import { QueryRecords } from "../query-records.js";
import { parseRecord } from "../row-schemas.js";
import { statement } from "./statement.js";

// Queries for the experiment_previews table (docs/architecture.md#recipe-add-or-extend-a-sql-table-module).

const parse = (row: unknown) =>
parseRecord(QueryRecords.experiment_previews, row, "experiment_previews");

// INSERT … SELECT needs a WHERE before ON CONFLICT so SQLite does not read
// the upsert clause as a join constraint.
const upsertStatement = statement(
"INSERT INTO experiment_previews(id,owner_id,experiment_id,port,label,path,url,active,created_at,updated_at) SELECT ?,?,?,?,?,?,?,1,?,? WHERE EXISTS(SELECT 1 FROM experiment_vms WHERE id=? AND owner_id=?) ON CONFLICT(experiment_id,port) DO UPDATE SET label=excluded.label,path=excluded.path,url=excluded.url,active=1,updated_at=excluded.updated_at WHERE experiment_previews.owner_id=excluded.owner_id",
);
/** One preview per experiment port; exposing the port again reactivates it.
* Returns false, writing nothing, when the experiment is not the owner's. */
export function upsert(
db: Database.Database,
preview: {
id: string;
owner: string;
experimentId: string;
port: number;
label: string | null;
path: string;
url: string;
now: number;
},
) {
return (
upsertStatement(db).run(
preview.id,
preview.owner,
preview.experimentId,
preview.port,
preview.label,
preview.path,
preview.url,
preview.now,
preview.now,
preview.experimentId,
preview.owner,
).changes === 1
);
}

const byExperimentStatement = statement(
"SELECT * FROM experiment_previews WHERE owner_id=? AND experiment_id=? ORDER BY port",
);
export function byExperiment(
db: Database.Database,
owner: string,
experiment: string,
) {
return byExperimentStatement(db).all(owner, experiment).map(parse);
}

const byPortStatement = statement(
"SELECT * FROM experiment_previews WHERE owner_id=? AND experiment_id=? AND port=?",
);
export function byPort(
db: Database.Database,
owner: string,
experiment: string,
port: number,
) {
const row = byPortStatement(db).get(owner, experiment, port);
return row === undefined ? undefined : parse(row);
}

const deactivateStatement = statement(
"UPDATE experiment_previews SET active=0,updated_at=? WHERE owner_id=? AND experiment_id=? AND port=?",
);
export function deactivate(
db: Database.Database,
owner: string,
experiment: string,
port: number,
now: number,
) {
return deactivateStatement(db).run(now, owner, experiment, port).changes;
}

const deactivateAllStatement = statement(
"UPDATE experiment_previews SET active=0,updated_at=? WHERE owner_id=? AND experiment_id=? AND active=1",
);
/** A destroyed VM serves nothing, so every link of the experiment is disabled. */
export function deactivateAll(
db: Database.Database,
owner: string,
experiment: string,
now: number,
) {
return deactivateAllStatement(db).run(now, owner, experiment).changes;
}
Loading