diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md new file mode 100644 index 0000000..32a1f7e --- /dev/null +++ b/ARCHITECTURE.md @@ -0,0 +1,408 @@ +# Mist architecture + +Mist is UTMIST's research compute platform. It works like the batch systems on +national HPC clusters. Each researcher gets a folder inside each of their team +folders and moves files with `scp`/`rsync`. Code runs as **jobs**: batch scripts +or short interactive shells. A scheduler places jobs on the GPU, CPU and +Tenstorrent hardware. Each job runs as the researcher's own Linux user, in their +own folder. Researchers never write Dockerfiles or declare mounts. + +This document is the system overview and the coordination point between +component owners. Each component has a document with the contracts it must +honour. The reasons behind major choices are in the +[decision log](docs/agents/architecture/decisions.md). + +> **Status:** target design, revised 27 October 2026 after architecture review. This design replaces the +> platform described in `docs/content/` (container-image jobs, dataset and +> result pools, team namespaces). It starts fresh: no users or data are migrated. + +## Goals + +1. **Development feels like a workstation.** A researcher sets up a Python + environment once, in their own folder, and every job uses it. Files a job + writes are simply there afterwards. +2. **Access follows teams.** Teammates can read each other's work. Other teams + cannot see it at all. Some projects are under IP agreements, so this boundary + is a hard requirement. +3. **Code runs only inside jobs.** Login sessions move files and manage jobs. + Compute and accelerators are reached only through a scheduled job. +4. **Adding a node is cheap.** Adding a third or fourth machine means configuring + that node, not redesigning the system. +5. **The A4000 GPUs come first.** The NVIDIA partition ships first; Tenstorrent + follows on the same design. + +## System at a glance + +```mermaid +flowchart LR + subgraph Users + SSH[ssh / scp / rsync] + Browser[Browser] + end + + subgraph TT["utmist-tt (QuietBox)"] + Gate[Login gateway
sshd + mist-gate] + Relay[mist-relay] + Prov[Provisioner agent] + NFS[("/mist
(durable storage)")] + CPUP[partition: cpu] + TTP[partition: tt
4× n300] + end + + subgraph NV["utmist-z1opa08 (NVIDIA)"] + Portal[Web portal] + API[mistd API] + Sched[mistd scheduler] + DB[(Postgres)] + K3s[k3s server] + GPUP[partition: gpu
2× RTX A4000] + end + + Misty[(misty
people and teams)] + + SSH -->|Tailscale| Gate --> Relay --> API + Browser -->|Tailscale| Portal --> API + API --> DB + API --> Misty + Sched --> DB + Sched -->|Executor| K3s + K3s --> GPUP & CPUP & TTP + Prov -->|desired state| API + Prov --> NFS + GPUP -. "NFS over 10 GbE direct link" .-> NFS +``` + +## Components + +| Component | Responsibility | Document | Proposed owner | +|---|---|---|---| +| Identity and accounts | People and teams from misty; Mist accounts, UIDs, GIDs, SSH keys; sign-up; provisioning Linux accounts and folders | [identity-and-accounts.md](docs/agents/architecture/identity-and-accounts.md) | Lucas | +| Storage | `/mist` layout and permissions, NFS, volumes, usage reporting, quotas, scratch, backups | [storage.md](docs/agents/architecture/storage.md) | Ansh | +| Login gateway | Restricted SSH (file transfer and `mist` commands only), identity relay, the `mist` CLI | [login-gateway.md](docs/agents/architecture/login-gateway.md) | TBD | +| API | The one HTTP API used by the portal, CLI and agents; authentication and authorization | [api.md](docs/agents/architecture/api.md) | Shared; each owner defines their endpoints | +| Scheduler | Partitions, QoS, limits, queue, priority, fairshare, placement, accounting | [scheduler.md](docs/agents/architecture/scheduler.md) | Brandon | +| Executors | Run an admitted job on a node; stream I/O; report status. K3s executor first, native node executor later | [executors.md](docs/agents/architecture/executors.md) | Brandon | +| Runtime environment | What a job sees: images, software, environment variables, caches, outputs, signals | [runtime-environment.md](docs/agents/architecture/runtime-environment.md) | Scott (to confirm) | +| Web portal | Lightweight UI over the API: account, SSH keys, teams, jobs, queue, usage, admin | [portal.md](docs/agents/architecture/portal.md) | Andrei | +| Deployment and network | Hosts, network paths, services, secrets, admin access, adding nodes | [deployment-and-network.md](docs/agents/architecture/deployment-and-network.md) | Ethan, with Ansh | + +Owners should confirm or reassign rows here. Before implementing a component, +read [implementation-contracts.md](docs/agents/architecture/implementation-contracts.md). +It defines cross-component invariants, required acceptance evidence, and decisions +that must be resolved before a milestone can ship. Component documents are the +normative contracts; the implementation plan orders the work. D18–D23 supersede +specific older decisions as listed in the decision log. + +## How the parts fit together + +Each box is one owner's area; every arrow between boxes is a contract listed in +the table below. Deployment and network (hosts, the 10 GbE link, Tailscale, +secrets) sits under all of it; see its own diagram in +[deployment-and-network.md](docs/agents/architecture/deployment-and-network.md). + +```mermaid +flowchart TB + R([Researcher]) + Misty[(misty)] + + subgraph GW["Login gateway · owner TBD"] + sshd[sshd] + gate[mist-gate] + cli[mist CLI] + relay[mist-relay] + end + + subgraph PO["Portal · Andrei"] + ui[Web UI] + end + + subgraph API["mistd api · shared"] + mw[Auth middleware + routing] + end + + subgraph ID["Identity · Lucas"] + dir[misty client / stub] + acct[(accounts, teams, SSH keys)] + prov[Provisioner agent] + end + + subgraph SC["Scheduler · Brandon"] + sched[Scheduling cycle] + ledger[(jobs, allocations, usage)] + end + + subgraph EX["Executors + k3s · Brandon"] + k3sx[K3s executor] + entry[mist-job-entry] + end + + subgraph RT["Runtime · Scott"] + img[Runtime images] + sw["/mist/software"] + end + + subgraph ST["Storage · Ansh"] + vol[("/mist volumes")] + scratch[(Node scratch)] + end + + R -->|"ssh, scp, rsync"| sshd --> gate --> cli + gate -->|"file transfer as the user"| vol + cli -->|unix socket| relay + sshd -.->|AuthorizedKeysCommand| relay + relay -->|"HTTP /v1 + asserted user"| mw + R -->|browser| ui -->|"HTTP /v1 + session"| mw + mw -->|identity.Authorize| acct + mw -->|job endpoints| sched + dir --> Misty + dir --> acct + prov -->|"GET provision/state"| mw + prov -->|"users, groups, folders"| vol + sched --> ledger + sched -->|Executor interface| k3sx + acct -.->|per-job mist-nss projection| k3sx + k3sx -->|"pod: uid, team gid, devices"| entry + k3sx -->|"mounts team folder + software"| vol + k3sx --> scratch + img -.->|ships| entry + img -.-> sw +``` + +| Boundary | Provider → consumer | Contract | Defined in | +|---|---|---|---| +| Who is registered, which teams | misty → Identity | `Directory` interface (stub until the real API) | [identity](docs/agents/architecture/identity-and-accounts.md#misty) | +| Who may do what | Identity → every API handler | `identity.Authorize(principal, action, resource)` | [identity](docs/agents/architecture/identity-and-accounts.md#roles) | +| Linux accounts and folders | Identity → Provisioner → Storage | `/internal/v1/provision/state`, layout rules | [identity](docs/agents/architecture/identity-and-accounts.md#provisioner), [storage](docs/agents/architecture/storage.md#layout) | +| CLI calls as the researcher | Gateway → API | Relay socket, `X-Mist-Asserted-User` + relay token | [login gateway](docs/agents/architecture/login-gateway.md#identity-relay) | +| Everything clients see | API → Portal, CLI | `/v1` endpoints and wire types | [api](docs/agents/architecture/api.md) | +| Running a job | Scheduler → Executors | `executor.Executor`, `LaunchSpec`, events | [executors](docs/agents/architecture/executors.md#interface) | +| What a job sees | Executors + Runtime → researcher code | Job environment contract, `mist-job-entry` | [runtime](docs/agents/architecture/runtime-environment.md) | +| Where files live | Storage → Executors, Gateway | Volumes, mount paths, permissions | [storage](docs/agents/architecture/storage.md#volumes-and-growth) | + +## Core concepts + +- **Person.** A human registered in misty. Misty is the source of truth for + people, teams and membership. +- **Account.** A person's Mist identity: one immutable `username`, one stable + Linux UID, and SSH keys. It is created when a registered person signs up. +- **Team.** A misty team. On Mist, each team has a Linux group (`team-`) and + a folder `/mist/teams//`. +- **Partition.** A named pool of nodes that share one kind of hardware: `gpu` + (NVIDIA), `cpu` (CPU-only, used for data prep and downloads) and `tt` + (Tenstorrent). A job targets exactly one partition. +- **QoS.** A class of service, `normal` or `interactive`, carrying limits and a + priority weight. +- **Job.** A request to run a script (batch) or a shell (interactive). It names + one partition, one QoS, one **charged team**, its resources and a time limit. + It runs as the submitter's UID. Accelerators are allocated whole and one at a + time: two 1-GPU jobs run side by side on the A4000 box. +- **Charged team.** The team a job is billed to and scoped to. The job sees only + that team's folder and the shared software. It cannot see the submitter's other + teams. +- **Volume.** One durable storage location, an NFS export on one server. Each + team folder lives entirely on one volume. There is one volume today. + +## The researcher's journey + +One researcher, Alice on team `vision`, goes from sign-up to results. The boxes +mark which owner's component handles each step. + +```mermaid +sequenceDiagram + autonumber + actor R as Alice + box Portal · Andrei + participant W as Portal + end + box Login gateway · TBD + participant G as Gateway + relay + end + box API · shared, Identity · Lucas + participant A as mistd api + participant P as Provisioner + end + box Scheduler + executors · Brandon + participant S as Scheduler + participant E as K3s executor + end + box Runtime · Scott, Storage · Ansh + participant J as Job + participant F as /mist + end + + Note over R,F: Join + R->>W: Sign in + W->>A: Is Alice registered in misty? + A-->>W: Yes, teams [vision]. Account alice, uid 20007 + R->>W: Add SSH public key + P->>A: Poll desired state + P->>F: Create user, groups, /mist/teams/vision/alice + + Note over R,F: Bring code and data + R->>G: ssh, then rsync ./project + G->>F: Write to /mist/teams/vision/alice/project + R->>G: mist run -p cpu -- hf download some-model + G->>A: Submit command job (cpu) + A->>S: Job queued + S->>E: Launch on utmist-tt + E->>J: Start as alice, team vision + J->>F: Model saved to team .cache + + Note over R,F: Set up an environment + R->>G: mist shell -p gpu --gpus 1 + G->>A: Submit interactive job + S->>E: Launch on utmist-z1opa08 (1 GPU) + E->>J: Start shell holder + G->>A: Attach terminal (WebSocket) + A->>E: Attach + R->>J: uv sync (venv lands in her folder) + R->>J: exit, job completes + + Note over R,F: Train + R->>G: mist submit train.sh + G->>A: Submit batch job 412 + A-->>R: 412 PENDING (RESOURCES) + S->>E: GPU frees up, launch 412 + E->>J: Start train.sh + J->>F: Stream mist-412.out, write checkpoints + R->>G: mist logs -f 412 + J-->>E: Exit 0 + E-->>S: Finished (CLEANING; allocation held) + E->>E: Persist result, delete execution, confirm node cleanup + E-->>S: Exited (cleanup receipt) + S->>A: 412 COMPLETED + + Note over R,F: Take results home + R->>G: rsync checkpoints back to laptop + G->>F: Read as alice +``` + +## Key flows + +### Sign-up and first login + +1. A person signs in to the portal. Identity confirms with misty that they are + registered and loads their teams. +2. Mist creates their account: it reserves a username and a UID and records team + memberships. +3. The person adds an SSH public key in the portal. +4. The provisioner agent on the storage/login host reconciles the change. It + creates the Linux user and groups, and `/mist/teams///` for each + team. +5. The person runs `ssh @`. They land in a folder with one + entry per team. + +### Moving files + +`scp`, `rsync` and `sftp` go through the login gateway, which sits on the storage +host and writes straight to disk. Unix permissions decide what each person can +read and write. + +Downloads from the internet (Hugging Face, S3, Kaggle and so on) run as jobs on +the `cpu` partition, using `mist run -p cpu -- ` or `mist shell -p cpu`. + +### Batch job + +```mermaid +sequenceDiagram + participant U as Researcher (login) + participant G as mist-gate + relay + participant A as mistd API + participant S as Scheduler + participant E as Executor (K3s) + participant N as Node + U->>G: mist submit train.sh + G->>A: POST /v1/jobs (relay-asserted user) + A->>A: validate spec, membership, limits + A-->>U: job 412 PENDING + S->>S: order by priority, check limits and free capacity + S->>E: Launch(job 412, node, resources) + E->>N: start as uid, team gid, team folder, devices + N-->>E: running … exited(0) + E-->>S: Finished, then Exited after confirmed cleanup + S->>A: job 412 COMPLETED; allocation released + U->>G: tail -f mist-412.out +``` + +The job starts in the folder it was submitted from. Its output streams to +`mist-.out` there. Checkpoints and results go wherever the code writes them, +in `/mist` by default. + +### Interactive job + +`mist shell --gpus 1` submits a job with QoS `interactive`. When the job starts, +the API attaches the researcher's terminal to a shell inside it. The job ends +when the shell exits. If the connection drops, the job waits for a grace period +so the researcher can run `mist attach`, then it is cancelled. A browser terminal +will later reuse the same attach endpoint. + +## Data ownership + +Postgres on the control node is the single source of truth for Mist state. +Kubernetes objects, Linux accounts and folders are *derived* state: agents +reconcile them from the database. + +| Data | Authority | Owner | +|---|---|---| +| People, teams, membership | misty (cached in Mist) | Identity | +| Accounts, UIDs, GIDs, SSH keys | Mist DB | Identity | +| Linux users, groups, folders | Derived by the provisioner | Identity, Storage | +| Volumes, team placement, usage | Mist DB | Storage | +| Partitions, QoS, limits | Mist DB (admin-configured) | Scheduler | +| Jobs, job events, usage records | Mist DB | Scheduler | +| Pods and other execution objects | Derived by the executor; cleanup acknowledged before reuse | Executors | +| Files | `/mist` | Storage | + +Each owner owns their tables and migrations. Other components go through that +owner's Go package or API, not their tables. Packages participating in admission +and revocation accept the same database transaction so authorization and job +changes serialize; an HTTP authorization check followed by an unrelated commit +is not sufficient. Durable job intents and executor observations survive restarts. +User files are not derived state and must never be recreated or deleted by job +reconciliation. + +## Code layout + +```text +cmd/mistd/ api, scheduler, provision, cleanup-report subcommands +cmd/mist/ CLI (on the login gateway; laptops later) +cmd/mist-gate/ SSH ForceCommand dispatcher +cmd/mist-relay/ login-node identity relay and SSH key lookup +cmd/mist-job-entry/ in-job wrapper: output file, signals, exit reasons +internal/api/ HTTP server, middleware, shared wire types +internal/identity/ accounts, misty client and stub, provisioning state +internal/storage/ layout rules, volumes, usage +internal/scheduler/ queue, policy, placement, accounting +internal/executor/ Executor interface; k3s/ and node/ implementations +internal/db/ connection and migrations (one directory per owner) +images/ job runtime images +deploy/ k3s manifests, systemd units, host setup +web-interface/ portal +``` + +The existing `src/` and `cli/` modules belong to the platform being replaced. +They are removed once the new path serves jobs. + +## Milestones + +| Milestone | Outcome | +|---|---| +| **M1: GPU batch** | Sign-up (misty stubbed), SSH login, file transfer, `mist submit/run/jobs/show/logs/cancel` on `gpu` and `cpu`, verified permissions/revocation/network isolation, crash recovery and cleanup accounting, 10 GbE direct link | +| **M2: Interactive** | `mist shell` and `mist attach` with reattach grace; `interactive` QoS | +| **M3: Fair scheduling** | Priority, fairshare by team, backfill, accounting, admin limits, queue views; backups in place | +| **M4: Tenstorrent** | `tt` partition, TT runtime image, board reset between jobs | +| **M5: Native executor and web terminal** | Node executor without containers, behind the same interface; browser terminal; port forwarding into jobs | + +Each component document lists its work per milestone. + +## Open questions + +| Question | Owner | +|---|---| +| Real misty API: shape, and how membership changes reach Mist (polling or webhooks) | Identity | +| How people prove who they are at sign-in (Google or UofT SSO, or email magic link) | Identity | +| Enforced storage quotas: needs `/mist` on a dedicated filesystem with project quotas | Storage | +| Retention after access revocation: archive or transfer departed members’ folders (access revocation itself is specified) | Identity, Storage | +| Backup tool, schedule and retention | Storage | +| Default limits per partition and per user once real usage is known | Scheduler | +| Monitoring and alerting beyond the portal | Deployment | diff --git a/README.md b/README.md index aec78c8..c658ef7 100644 --- a/README.md +++ b/README.md @@ -7,6 +7,15 @@ collect persistent results through the same website or CLI. **Private website:** [http://100.73.139.66:8088](http://100.73.139.66:8088), accessible through authorized Tailscale connectivity. A Mist account and team membership are also required. +## Architecture redesign — October 26 + +Mist is moving to an HPC-style design: per-team folders, restricted SSH login, +and jobs that run as the researcher's own Linux user under Mist's own scheduler. +Jobs no longer need images or mounts. Start with [ARCHITECTURE.md](ARCHITECTURE.md); +component contracts and the decision log are in +[docs/agents/architecture/](docs/agents/architecture/decisions.md). The +sections below and `docs/content/` describe the platform being replaced. + ## Storage update — October 21 New submissions use ordinary team dataset/result folders on QuietBox, accessed diff --git a/docs/agents/architecture/api.md b/docs/agents/architecture/api.md new file mode 100644 index 0000000..74698ad --- /dev/null +++ b/docs/agents/architecture/api.md @@ -0,0 +1,239 @@ +# API + +`mistd api` is the only way clients and agents reach Mist state. This document +fixes the conventions every endpoint follows, how callers authenticate, and the +job wire types shared by the CLI, portal and scheduler. Each component document +lists the endpoints it owns. Overview: [ARCHITECTURE.md](../../../ARCHITECTURE.md). + +## How it fits + +Every request passes the same middleware, then reaches the handlers of the +component that owns it: + +```mermaid +flowchart LR + portal["Portal
session cookie"] --> mw + relay["mist-relay
relay token + asserted user"] --> mw + agents["Provisioner, key lookup
host token"] --> mw + mw["Auth middleware
resolves principal"] --> authz{identity.Authorize} + authz --> idh["/v1/me, /v1/teams
Identity"] + authz --> sth["/v1/teams/slug/usage
Storage"] + authz --> jh["/v1/jobs, /v1/partitions, /v1/queue
Scheduler"] + authz --> adm["/v1/admin/...
Identity + Scheduler"] + mw --> inth["/internal/v1/...
Identity"] + idh & sth & jh & adm & inth --> db[(Postgres)] + jh -->|"logs, attach"| ex[Executor] +``` + +## Conventions + +- **Format.** JSON over HTTP. Paths are versioned: `/v1/...` for users and + admins, `/internal/v1/...` for host agents. +- **Placement.** One `mistd api` process on the control node (NVIDIA box). The + portal's Nginx forwards `/api/*` to it and strips the prefix. Agents reach it + on the direct link (`192.168.240.1`). +- **Logic lives in the API.** Clients don't compute permissions, defaults or + states; the API returns ready-to-display fields (for example `pending_reason` + and `can_cancel`). Clients may poll: every 5–10 s is fine. +- **Errors** use one shape and a stable `code`: + + ```json + { "error": { "code": "limit_exceeded", "message": "Team vision already has 2 GPUs running", "details": {} } } + ``` + + Codes: `unauthenticated`, `forbidden`, `not_found`, `invalid`, + `limit_exceeded`, `conflict`, `unavailable`, `internal`. +- **Lists** take `?limit=` and `?cursor=` and return `{ "items": [...], "next_cursor": "..." }`. +- **Times** are RFC 3339 UTC strings. **Durations** are integer seconds. + **Sizes** are integer bytes. +- **Streams** (log follow, interactive attach) use WebSockets on the same paths + with `?follow=1` or an `Upgrade` header. + +## Authentication + +| Caller | Credential | Principal | +|---|---|---| +| Portal | Session cookie set at sign-in | The signed-in account | +| CLI on the login gateway | Relay token + `X-Mist-Asserted-User` (see [login-gateway.md](login-gateway.md)) | The asserted account | +| Provisioner, relay key lookup | Per-host agent token | The host; `/internal/*` only | +| Laptop CLI (later) | Personal API token | The token's account | + +One middleware resolves the principal, and authorization goes through +`identity.Authorize` (see [identity-and-accounts.md](identity-and-accounts.md)). +Handlers never read identity headers directly. Check active account and current +membership on every request and stream establishment; revoke existing streams +when the authorization generation changes. The relay cannot assert disabled +accounts. Agent tokens are scoped by host and operation, not interchangeable +administrator credentials. + +Production uses HTTPS, `Secure`, `HttpOnly`, `SameSite=Lax` session cookies, +CSRF protection on mutations, and an Origin allowlist for browser WebSockets. +Nginx must strip user-supplied identity headers. Keep dev principal headers out +of production binaries. See Identity for bounded directory freshness/leases. + +Authorization matrix: owners may view/cancel their currently authorized jobs and +attach to their own interactive jobs; current team members may view team job +metadata/scripts/logs; team leads may cancel team jobs; admins may view/cancel +all jobs but cannot attach as the user. Former membership or ownership alone +never grants continued team access. Non-team callers receive `not_found` on job +and log endpoints; `/queue` provides only the explicitly public summary. + +`POST /v1/jobs` accepts a required `Idempotency-Key` from CLI clients. Store +(account, key, canonical client request hash, job ID, resolved submission spec) +atomically with submission. Look up the key before applying current defaults: +same key/body returns the original job even if defaults changed; differing body +returns `conflict`. Reauthorize access before returning a cached result. Retain keys +for at least the job metadata lifetime. Cancellation is intrinsically idempotent +and returns the current job after recording stop intent; it does not wait for +termination or release resources. + +## Endpoint index + +| Area | Endpoints | Owner doc | +|---|---|---| +| Account | `GET/PATCH /v1/me`, `/v1/me/ssh-keys` | identity | +| Teams | `GET /v1/teams`, `GET /v1/teams/{slug}` | identity | +| Storage | `GET /v1/teams/{slug}/usage`, `GET /v1/volumes` (admin) | storage | +| Jobs | `POST /v1/jobs`, `GET /v1/jobs`, `GET /v1/jobs/{id}`, `POST /v1/jobs/{id}/cancel`, `GET /v1/jobs/{id}/log`, `GET /v1/jobs/{id}/attach` (WebSocket) | scheduler / executors | +| Cluster | `GET /v1/partitions`, `GET /v1/queue` | scheduler | +| Admin | `/v1/admin/accounts`, `/v1/admin/partitions`, `/v1/admin/qos`, `/v1/admin/limits`, `/v1/admin/nodes` | identity, scheduler | +| Agents | `/internal/v1/provision/*`, `/internal/v1/ssh-keys/{username}` | identity | +| Cleanup | `POST /internal/v1/executions/{execution_id}/cleanup` | executor; node-scoped authenticated receipt | + +## Job wire types + +### Submitting: `POST /v1/jobs` + +```json +{ + "team": "vision", + "partition": "gpu", + "qos": "normal", + "name": "resnet-sweep-3", + "kind": "batch", + "script": "#!/bin/bash\n#MIST --gpus=1\npython train.py --lr 3e-4\n", + "args": [], + "workdir": "/mist/teams/vision/alice/resnet", + "output": "mist-%j.out", + "env": { "WANDB_PROJECT": "resnet" }, + "resources": { + "cpus": 8, + "memory_bytes": 34359738368, + "gpus": 1, + "tt_boards": 0, + "scratch_bytes": 107374182400 + }, + "time_limit_seconds": 14400, + "signal_grace_seconds": 60 +} +``` + +- `kind`: + - `batch` needs `script`. + - `command` (from `mist run`) takes `argv` instead of `script`. + - `interactive` (from `mist shell`) takes neither. +- **The API fills in missing fields** from partition and QoS defaults and + returns the complete, normalised spec. Defaults live only in the database, + never in clients. +- **Validation:** + - The caller must be active and a current member of `team`; directory state + must be fresh. Admission rechecks this atomically, not just at submission. + - `workdir` must be inside the charged team. Omitted workdir uses the personal + home for an explicit team; otherwise derive team from the submitted canonical + path. Never accept a login landing folder as a job workdir. + - Resolve relative output under workdir and expand only `%j`. Require output + inside the charged team's durable tree. Executor revalidates inside the + restricted mount namespace as the user, handling symlinks/path races without + privileged traversal. Missing/unwritable paths produce a clear launch error. + - CPU, memory, scratch and time are positive bounded integers; accelerators + are nonnegative whole counts. API `cpus` is a whole-core count, converted + once to executor millicores (1 CPU = 1000 mCPU). Reject overflow and incompatible partition + resources. Enforce partition/QoS maximums, including signal grace 1–120 s. + - `gpu` needs GPUs >= 1 and TT boards = 0; `cpu` requires both = 0; `tt` needs + boards >= 1 and GPUs = 0. A request must fit at least one configured node. + - `interactive` kind requires interactive QoS; batch/command use normal QoS. + Users cannot obtain interactive priority for unattached batch jobs. + - Optional `runtime` selects only an approved retained partition profile; + resolve it to a digest/profile version at admission and freeze it. + - Reject unknown/mutually exclusive command fields. Bound script size (1 MiB), + env total (64 KiB), argument total (128 KiB) and name length (128 characters). + Preserve batch `args` separately; command uses `argv`, without a shell. + - User env cannot override identity, allocation, device visibility, internal + attach socket or timing fields. Runtime documents the protected names. +- **Reply:** `201` with the job object, in state `PENDING`. + +### Job object: `GET /v1/jobs/{id}` + +```json +{ + "id": 412, + "name": "resnet-sweep-3", + "user": "alice", + "team": "vision", + "partition": "gpu", + "qos": "normal", + "kind": "batch", + "state": "PENDING", + "pending_reason": "RESOURCES", + "pending_detail": "Waiting for 1 GPU; estimated start after job 409 ends", + "exit_reason": null, + "exit_code": null, + "resources": { "cpus": 8, "memory_bytes": 34359738368, "gpus": 1, "tt_boards": 0, "scratch_bytes": 107374182400 }, + "time_limit_seconds": 14400, + "node": null, + "workdir": "/mist/teams/vision/alice/resnet", + "output_path": "/mist/teams/vision/alice/resnet/mist-412.out", + "submitted_at": "2026-11-02T15:04:05Z", + "started_at": null, + "ended_at": null, + "can_cancel": true, + "can_attach": false +} +``` + +**States and reasons** are defined in [scheduler.md](scheduler.md#job-states). +The object also exposes `cleanup_pending`, `cleanup_detail`, `output_incomplete`, +`authorization_generation`, and (interactive only) `attached`, `detached_at`, +`reattach_deadline`. The API computes `can_cancel` and `can_attach` using state +and current authorization. `CLEANING` is nonterminal; a `NODE_FAIL` job may be +terminal with cleanup still pending and capacity quarantined. + +Other teams' jobs appear only in `GET /v1/queue` as an allowlisted summary: +job ID, partition, QoS, state, resource counts, age and a public pending-reason +code. Set `user`, `team`, `name`, `workdir` and `output_path` to null. Exclude +scripts, env, arbitrary error text, event actors and detailed pending messages; +redacting a few top-level names does not sanitize embedded paths or secrets. + +### Logs: `GET /v1/jobs/{id}/log` + +- **Running job:** returns the live output stream from the executor. Add + `?follow=1` to stream it over a WebSocket. +- **Finished job:** returns the stored tail (last 64 KiB) from the database. +- **The full log normally resides in `/mist`.** Include `output_path` and + `output_incomplete`. Node failure, OOM/SIGKILL, or storage errors can prevent a + final footer or file creation; the DB retains the observed outcome and last + available tail. API log viewing never reads arbitrary user-supplied paths. + +### Attach: `GET /v1/jobs/{id}/attach` (WebSocket) + +Binary frames carry terminal bytes. Text frames carry control messages: + +```jsonl +{ "type": "resize", "cols": 120, "rows": 40 } +{ "type": "signal", "name": "SIGINT" } +{ "type": "exit", "code": 0 } +``` + +Only the currently authorized job owner can attach, while the interactive job +is `RUNNING`. The holder allows one attachment; a second receives `conflict`. +Network closure detaches, explicit `POST .../cancel` cancels, and shell exit ends +the job. Resize and allowed terminal signals target the existing PTY/foreground +process group; callers cannot specify a PID, executable or alternate command. +The holder owns grace timing, so API restart cannot reset it. The CLI's `mist +shell` and later browser terminal use this same endpoint. + +## Versioning + +Additive changes (new fields, endpoints or enum values) don't change the +version. Clients must ignore unknown fields and show unknown enum values as +plain text. Breaking changes go to `/v2`. diff --git a/docs/agents/architecture/decisions.md b/docs/agents/architecture/decisions.md new file mode 100644 index 0000000..1f31dff --- /dev/null +++ b/docs/agents/architecture/decisions.md @@ -0,0 +1,279 @@ +# Architecture decisions + +Each entry records the context, the decision, and what follows from it. To +change a decision, add a new entry that supersedes the old one rather than +editing it. The system overview is [ARCHITECTURE.md](../../../ARCHITECTURE.md). + +## D1. Mist has its own scheduler; Kubernetes only runs jobs + +**Context.** The previous platform made every job a container image with +declared mounts, and that friction is the reason for the redesign. An existing +HPC scheduler (Slurm) was considered. The team chose to build the scheduler as +part of Mist, and the cluster is expected to grow beyond two nodes. + +**Decision.** Mist's scheduler owns the queue, policy and placement. A pluggable +**executor** turns an admitted job into running processes. The first executor +uses k3s; the Kubernetes scheduler never decides when or where a job runs. + +**Consequences.** Mist implements queueing, limits, priority and accounting. +Kubernetes provides process supervision, device allocation (NVIDIA device plugin, +Tenstorrent DRA), cgroup limits and node membership. If both systems made +placement decisions, jobs could sit pending with no visible reason; pinning pods +to the scheduler's chosen node prevents that. + +## D2. Executors sit behind one interface, and each node has exactly one executor + +**Decision.** Executors implement the interface in [executors.md](executors.md). +The K3s executor ships first. A native node executor (systemd units and cgroups, +no containers) follows behind the same interface. Each partition maps to one +executor, and each node belongs to one executor, because two systems cannot both +own a node's devices. + +**Consequences.** The scheduler and API never import Kubernetes types. A shared +conformance test suite defines executor behaviour. Moving a node between +executors means draining it first. + +## D3. Postgres is the source of truth + +**Context.** The previous platform kept queue and policy state in ConfigMaps and +suspended Jobs. + +**Decision.** All Mist state lives in Postgres. Pods, Linux accounts and folders +are derived and reconciled from it. + +**Consequences.** Restarting `mistd` or wiping execution objects loses nothing; +the executor rebuilds its view by comparing the database with what is actually +running. Usage history for fairshare and accounting has a natural home. + +## D4. One Linux UID per person + +**Context.** Creating a Linux user per job was considered, so that only jobs +would get privileges. + +**Decision.** Each person has one stable UID, the same on every host. Jobs run +as that UID. Privileges (devices, write access to the right folders) come from +the job's cgroup, mounts and supplementary groups, not from a different user. + +**Consequences.** Files keep one owner across login, jobs and teammates. There +is no ownership mess between jobs. UID and GID allocation is centralised in the +Mist DB, in fixed ranges that don't collide with local system accounts. + +## D5. Each job is scoped to one charged team + +**Decision.** Every job names a charged team; by default it is the team whose +folder the job was submitted from. The job is given only that team's group and +that team's folder, plus shared read-only software. + +**Consequences.** Someone in two teams cannot accidentally mix IP-bound data +between them inside a job. They can still copy files by hand, since they can +read both teams; no technical control prevents a person from moving data they +can read. A person's home is per-team for the same reason (D8). + +## D6. Login moves files and manages jobs; code runs only in jobs + +**Decision.** Researcher SSH sessions are restricted by `ForceCommand` to: +SFTP/scp, `rsync`, a small set of built-in file viewing commands, and the `mist` +CLI. Anything that executes user code is a job. + +**Consequences.** The login host (also a storage and compute node) is protected +from stray compute. Internet downloads become jobs (D11). This is stricter than +typical HPC login nodes, and that is deliberate on a cluster with no dedicated +login hardware. + +## D7. Login and storage on the TT box; control plane on the NVIDIA box + +**Decision.** + +- **TT box:** the durable storage (`/mist`), the login gateway and the + provisioner. +- **NVIDIA box:** the k3s server, `mistd`, Postgres and the portal. + +**Consequences.** Uploads land on local disk with no network hop. The TT box is +the single point of failure for logins and file access. The control plane and +the database sit on separate hardware from the bulk storage. + +## D8. A researcher's team folder is their home + +**Decision.** `/mist/teams///` is both `$HOME` and the default +working directory for jobs charged to that team. It is team-readable (`2750`). +`.cache`, `.config`, `.local` and `.ssh` inside it are private (`0700`): the +provisioner creates them that way, and the job entry wrapper resets them on +every job. A few large shareable caches are mapped to the team's `.cache/` by +environment variables. + +**Consequences.** Every tool's cache and config lands somewhere sensible without +per-tool mapping. Credentials stored in the usual hidden folders stay private. +**Accepted risk:** a tool that writes a secret as a top-level file without +restricting its permissions exposes it to teammates (never to other teams). +Someone in two teams keeps two homes. Researchers need this explained, so it +appears under "What researchers see" in [storage.md](storage.md). + +## D9. Durable storage plus local scratch; growth by adding volumes + +**Decision.** `/mist` is the only durable storage. Node-local scratch is +temporary and wiped after each job. Durable capacity grows by adding **volumes** +(separate NFS exports), and each team folder lives entirely on one volume. No +distributed filesystem is used. + +**Consequences.** Researchers never have to work out which machine holds a file. +Adding storage is a database record plus an export. Spare disk on compute nodes +goes to backups first, then scratch, and only then to new volumes. + +## D10. The direct link runs at 10 GbE on existing network cards + +**Context.** The direct cable uses Intel I210 ports, which max out at 1 Gb/s +(about 110 MB/s). Loading a 14 GB model took around two minutes. + +**Decision.** Move the link to the 10GBASE-T ports already installed: the +Aquantia AQC107 (`ens6`) on the NVIDIA box and the Intel X710 +(`enp10s0f0np0`) on the TT box. Use Cat6a cable. + +**Consequences.** NFS reaches roughly 700–1000 MB/s, so the separate node-local +model cache is dropped from the design. The second X710 port can take one more +direct-connected node. A fourth node needs a 10 GbE switch. + +## D11. Downloads are ordinary jobs on a `cpu` partition + +**Decision.** The TT box's CPUs form a `cpu` partition, which sits next to the +storage. `mist run -p cpu -- ` and `mist shell -p cpu` cover every download +tool. There is no dedicated download service. + +**Consequences.** Any source works (Hugging Face, S3, Kaggle, `git lfs`, `wget`), +using the researcher's own tokens. GPUs are never held idle during downloads. + +## D12. Job outputs go to `/mist`; scratch is opt-in + +**Decision.** + +- **Output.** Stdout and stderr stream to `mist-.out` in the submit folder. + Mist appends its own messages there too (node, cancellation, time limit, + out-of-memory). +- **Job record.** The database keeps the job's metadata, a copy of the script, + and a tail of the log. +- **Scratch.** `$TMPDIR` points at node-local scratch, which is deleted when the + job ends. +- **Time limits.** At the limit the job receives SIGTERM, then SIGKILL after a + grace period (default 60 s). + +**Consequences.** Nothing a researcher writes by default is ever lost. The +portal can show logs without reading team files. + +## D13. Accelerator device files stay world-accessible; isolation comes from cgroups + +**Context.** `/dev/nvidia*` and `/dev/tenstorrent/*` are mode `0666` on both +hosts. + +**Decision.** Keep the host modes. Inside a job, the executor's device cgroup +(device plugin, or `DeviceAllow` in the node executor) exposes only the +allocated devices. Host-level access is prevented by D6 and by removing +unmanaged GPU consumers (for example an `ollama` service). + +**Consequences.** This matches Slurm's model. If the device files were tightened +to `0660`, containers could no longer open the devices they were given, because +non-root jobs inherit the host permissions. + +## D14. Misty owns people and teams; Mist owns accounts + +**Decision.** Misty is authoritative for who exists, which teams exist, and who +belongs to them. Mist caches that data and adds what is specific to Mist: +usernames, UIDs and GIDs, SSH keys, limits and roles. Until misty's API is +available, a stub backed by a file stands in for it. + +**Consequences.** Removing someone from a team in misty eventually removes their +Mist access to that team. Mist never edits people or teams. + +## D15. Other teams' jobs are anonymised in queue views + +**Decision.** Queue and partition views show every job's state, resources and +age, so researchers can see why they are waiting. The user, team, job name and +paths are shown only for the viewer's own teams. + +**Consequences.** Job names and paths can't leak IP-bound project details +between teams. + +## D16. One Go binary, plain Kubernetes objects + +**Decision.** `mistd` is one Go binary with subcommands (`api`, `scheduler`, +`provision`). It talks to Kubernetes through `client-go` and creates plain Pods. +It uses no CRDs, operators, `kubectl` shell-outs or Kubernetes Jobs. + +**Consequences.** Mist's own state machine stays in Mist's code and database, +which keeps it easy to understand and to swap executors. + +## D17. Accelerators are whole devices; CPU and memory are capped per GPU + +**Decision.** + +- **Whole devices.** Jobs request whole GPUs or TT boards; two 1-GPU jobs share + the A4000 box side by side. +- **Per-GPU caps.** Each GPU partition caps CPUs and memory per allocated GPU, + so one job cannot leave another GPU unusable. +- **No GPU-less jobs on GPU nodes.** `gpu` partition jobs must request at least + one GPU. +- **No GPU sharing.** Fractional GPUs (time-slicing) are not offered, because + the A4000 has no hardware partitioning and time-slicing gives no memory + isolation. + +**Consequences.** Placement only counts devices, CPUs and memory. Time-slicing +could later be added as an opt-in for small debug work. + +## D18. Mist admits jobs; Kubernetes may allocate DRA claims + +**Supersedes:** D1's unconditional scheduler bypass. Mist alone owns queue order, +policy, admission and the target node. GPU/CPU Pods use `nodeName`. TT Pods omit +it and use an exact unique node selector so Kubernetes can allocate/reserve DRA +claims on the chosen node. A claim stall is a bounded launch failure, never an +invitation to move the job. Pin and test the installed DRA driver and board reset +scope before M4. Source: [Kubernetes DRA pre-scheduled Pods](https://kubernetes.io/docs/concepts/resource-management/dynamic-resource-allocation/how-dra-works/). + +## D19. Personal work defaults to read-only for teammates + +**Clarifies D8 and supersedes conflicting implementation defaults.** Jobs and +uploads use `0027`; ordinary personal files are `0640`, directories inherit +setgid and exclude group write. Only shared/cache areas have group-write default +ACLs. Private dot-directories remain `0700`. Owners can explicitly change modes; +provisioning never recursively rewrites researcher content. Verify read/write +behavior as another UID, not merely the mode of the top-level directory. + +## D20. Access revocation includes live sessions and queued work + +**Refines D6 and D14.** `mist-users` is permanent; disabled accounts never lose +the marker enforcing restricted SSH. Directory snapshots and gateway leases have +a maximum 5 min validity, with no stale-cache extension. Known removal invalidates +API sessions, stops affected pending/launching/running jobs and kills gateway +sessions before updating groups. Existing process credentials do not change when +membership files change. A root local deny store and watchdog enforce expiry even +if the API is down. Revocation completion is acknowledged, not assumed while a +node is unreachable. Sources: [Linux credentials](https://man7.org/linux/man-pages/man7/credentials.7.html), +[usermod password locking](https://man7.org/linux/man-pages/man8/usermod.8.html). + +## D21. Execution intent, process outcome and cleanup are separate durable facts + +**Refines D3 and D12.** Commit immutable admission/launch intent before invoking an +executor; recover the same execution ID after ambiguous responses. Never rerun +started work automatically. A finished process enters `CLEANING`; resources stay +charged until node cleanup is proven. Lost nodes retain quarantined allocations. +Store result/log tail before object deletion and retain receipts/tombstones. +Postgres preserves control state, not running processes or user files: wiping +Pods can lose execution and scratch. OOM/SIGKILL can prevent an output footer; +the API reports authoritative status and incomplete output explicitly. + +## D22. Interactive jobs own a persistent terminal + +**Decision.** One holder owns one shell/PTY for the execution lifetime. Attach is +transport to that PTY, never a new shell. One client at a time, bounded replay, +10 min local detach grace, and unchanged walltime apply under both executors. +API/relay restart may detach but cannot reset grace or shell state. Holder death +fails the job. User code stays inside the one allocation/cgroup. + +## D23. Isolation must hold on the host as well as in the job spec + +**Refines D4, D5 and D13.** NetworkPolicy is paired with host filtering, including +job traffic to its own node, explicit DNS exceptions and tested IPv6 behavior. +Native execution explicitly replaces supplementary groups and drops credentials +before user code; systemd `SupplementaryGroups=` alone only extends NSS groups. +The same real-node conformance suite gates K3s and native execution. Adding nodes +uses a switch unless a separate bridge/routing design has been verified; D10's +second-port suggestion is not a complete network design. Sources: +[Kubernetes NetworkPolicy](https://kubernetes.io/docs/concepts/services-networking/network-policies/), +[systemd.exec](https://www.freedesktop.org/software/systemd/man/258/systemd.exec.html). diff --git a/docs/agents/architecture/deployment-and-network.md b/docs/agents/architecture/deployment-and-network.md new file mode 100644 index 0000000..56a6951 --- /dev/null +++ b/docs/agents/architecture/deployment-and-network.md @@ -0,0 +1,179 @@ +# Deployment and network + +Covers which host runs what, how hosts and users connect, where configuration +and secrets live, admin access, and how nodes are added. Overview: +[ARCHITECTURE.md](../../../ARCHITECTURE.md). Related decisions: D7, D10, D13. + +## Topology + +```mermaid +flowchart LR + Users([Researchers + admins]) -->|Tailscale| TSN + Users -->|Tailscale| TST + subgraph NV["utmist-z1opa08 · 56 CPU · 157 GiB · 2× A4000"] + TSN["tailscale0
100.73.139.66"] -->|"portal HTTPS :443"| NVsvc["k3s server · mistd api + scheduler
Postgres · Nginx portal"] + NVgpu[gpu partition] + ens6["ens6 · AQC107 10GBASE-T
192.168.240.1"] + end + subgraph TT["utmist-tt · 32 CPU · 503 GiB · 4× n300"] + TST["tailscale0
100.95.175.37"] -->|"ssh :22"| TTsvc["k3s agent · NFS /mist · sshd + mist-gate
mist-relay · provisioner"] + TTpart[cpu + tt partitions] + x710["enp10s0f0np0 · X710 10GBASE-T
192.168.240.2"] + end + ens6 <-->|"Cat6a direct link, 10 GbE
k3s, NFS, agents to API"| x710 + LAN[("Home LAN / internet
10.0.0.0/24")] + NV --- LAN + TT --- LAN +``` + +## Hosts + +| | `utmist-z1opa08` (NVIDIA) | `utmist-tt` (QuietBox) | +|---|---|---| +| Hardware | 56 CPUs, 157 GiB RAM, 2× RTX A4000 16 GB, 2× 447 GB SATA SSD | 32 CPUs, 503 GiB RAM, 4× n300 (8 chips), 3.6 TB NVMe | +| OS | Ubuntu 22.04, kernel 6.8, cgroup v2 | same | +| Roles | k3s server; `mistd api` + `mistd scheduler`; Postgres; portal (Nginx); `gpu` partition; backup target (M3) | k3s agent; `/mist` volume 1 + NFS server; login gateway (sshd, `mist-gate`, `mist-relay`); `mistd provision`; `cpu` and `tt` partitions | + +## Networks + +| Network | Addresses | Carries | +|---|---|---| +| **Direct link (10 GbE)** | NVIDIA `ens6` `192.168.240.1/24`, TT `enp10s0f0np0` `192.168.240.2/24` | k3s node traffic and Flannel, NFS, relay and provisioner → API, backups | +| LAN | NVIDIA `10.0.0.175` (currently over Wi-Fi), TT `10.0.0.112` | Internet access only | +| Tailscale | NVIDIA `100.73.139.66`, TT `100.95.175.37` | Users: portal (NVIDIA HTTPS :443), SSH login (TT :22); admin SSH | + +**Moving the direct link to 10 GbE (M1):** + +- **Cable:** Cat6a (or Cat6 under about 50 m) between the NVIDIA box's `ens6` + (Aquantia AQC107) and the TT box's `enp10s0f0np0` (Intel X710). Both are + 10GBASE-T (RJ45) ports. +- **Addresses:** move `192.168.240.1/.2` from the 1 GbE I210 ports to those two + ports, using a `/24` so more nodes can join. +- **MTU:** jumbo frames (9000) on both ends are optional, but help NFS. +- **Check:** `ethtool ` shows `Speed: 10000Mb/s`. An `iperf3` run should + reach above 9 Gb/s. + +**Wired LAN.** The NVIDIA box should also reach the LAN over wired Ethernet +rather than Wi-Fi; one of its I210 ports frees up once the direct link moves. + +**k3s** is installed fresh, with `--node-ip` and `--flannel-iface` on the direct +link, so cluster traffic never uses the LAN or Wi-Fi. + +## Services + +| Service | Host | Runs as | +|---|---|---| +| k3s server and agent | both | systemd | +| NVIDIA GPU Operator (device plugin + container toolkit) | NVIDIA | k3s | +| Tenstorrent DRA driver (M4) | TT | k3s | +| Postgres | NVIDIA | k3s StatefulSet, data on local SSD (never NFS) | +| `mistd api`, `mistd scheduler` | NVIDIA | k3s Deployments (1 replica each), reachable on `192.168.240.1` for agents | +| Portal (Nginx + static build) | NVIDIA | systemd, HTTPS on Tailscale (production), proxies `/api`; `:8088` development only | +| NFS server | TT | systemd | +| sshd + `mist-gate` | TT | systemd | +| `mist-relay` | TT | systemd (root) | +| `mistd provision` | TT (later every node) | systemd (root), includes gateway revocation/watchdog integration | +| `mistd cleanup-report` | both compute nodes | root-owned node service; narrowly scoped cleanup evidence | +| `mist-noded` (M5) | nodes moved to the node executor | systemd (root) | + +## Configuration and secrets + +- **Configuration in Git:** `deploy/` (manifests, systemd units, sshd and + exports snippets, host setup scripts). +- **Settings in the database:** partitions, QoS, limits, volumes. Admins change + them through the API. +- **Secrets never go in Git:** + - Postgres credentials, the relay token and per-host agent tokens. + - Kubernetes Secrets for in-cluster use; `/etc/mist/*.token` (0600 root) on + hosts. + - Rotate any secret that has been pasted into a chat or document. + +## Admin access + +- **Separate admin accounts** for each admin, with SSH keys only, `sudo`, and + not in `mist-users`. Researcher accounts never have `sudo` or `docker`. +- **Password SSH disabled** on both hosts. The shared `utmist` and `utmist-tt` + passwords are retired once admin accounts exist. +- **Clean up existing host state** after checking with whoever created it: + - leftover local accounts (`dev`, `terrymist`, `labadmin`, `labmember`) + - the `ollama` service and group + - Docker +- **Device files stay `0666`** (D13). This is safe only because researcher + logins can't run code and unmanaged GPU consumers are removed. + +## Starting fresh + +Before M1 goes live: + +1. Stop and remove the previous platform's API, namespaces (`mist-team-*`), + ConfigMaps and Jobs. +2. Keep the old storage folders untouched until a human confirms they can be + deleted. +3. Reinstall k3s on the direct link. + +## Adding a node + +1. **Connect it to the direct-link network.** For three or more nodes, use a + 10 GbE switch with all nodes on `192.168.240.0/24`. A second direct cable is + not automatically the same L2 network: an alternative needs an explicitly + designed bridge or routed subnet and routing/firewall tests before use. +2. **Install Ubuntu** with the drivers for its accelerators. +3. **K3s executor:** join it as a k3s agent and install its device plugin. Node + executor: install `mist-noded` and `mistd provision`. +4. **Register it in Mist:** add it to a partition (new or existing) and set its + reserved host share. +5. **Run the executor conformance suite** on it, then mark it `up`. + +## Work by milestone + +- **M1:** + - 10 GbE link and wired LAN + - fresh k3s + - Postgres + - deploy `mistd` and the portal + - gateway host setup + - admin accounts and secret rotation + - remove the old platform + - job/host firewall enforcement, authorization lease watchdog and cleanup reporter + - basic monitoring: disk headroom, stale directory/provisioning generations, + failed cleanup, quarantined nodes, scheduler leadership and launch timeouts + - pin k3s/kernel/runtime/plugins and record tested feature support +- **M3:** backup target, retention and restore exercises; richer monitoring UI. + +## Open questions + +- Monitoring and alerting stack. +- Whether the NVIDIA box's `sdb` is free for scratch and backups. +- A stable login hostname. + +## Job network boundary (M1) + +Deployment owns host firewall rules; Executors owns job namespace/NetworkPolicy +configuration. Both must pass the same conformance checks before a node is up. + +- Tag job traffic by its network interface/source before NAT. Filter both host + INPUT and routed/FORWARD paths; a pod's own host is not implicitly trusted. + Deny new job connections to every host address (direct-link, LAN, Tailscale, + pod gateway), Kubernetes service/pod CIDRs, RFC1918, CGNAT and link-local ranges. + Include locally bound public addresses, if added later. Allow only approved + DNS resolvers on TCP/UDP 53 and public internet egress; allow established + response traffic. Deny unsolicited inbound traffic to jobs. +- Host-origin NFS mounts, control-plane traffic, cleanup reporting and admin SSH + are separate trusted flows. Never open NFS from the job network to make a + mount work; mounts are performed by the host. Block job access to host runtime + sockets and Unix sockets through the filesystem too. +- M1 job networking is IPv4-only; disable IPv6 in job namespaces/routing. Enable + it only after equivalent IPv6/link-local rules pass tests. Startup must fail + closed if policy installation fails; rule reload and k3s restart must not + create an allow-all interval. Pin/test the actual CNI and host firewall ordering. +- Test from CPU jobs on the storage host and GPU jobs on the control host: own + host and peer SSH/NFS/API/Postgres/kubelet ports, service IP aliases, pod peers, + LAN/Tailscale peers, DNS, internet HTTPS, and IPv6 bypass attempts. Also verify + legitimate host NFS mounts and admin access remain functional. + +Use authenticated TLS for relay/provisioner/cleanup traffic on the direct link; +per-host bearer tokens authorize operations but do not encrypt them. Expose the +production portal through a stable Tailscale HTTPS name with a trusted certificate. +Record the selected hostname/certificate provisioning in deployment configuration. +Restrict internal API listeners to trusted hosts; user-facing proxy routes never +expose `/internal`. A clean restart must restore policies before admitting jobs. diff --git a/docs/agents/architecture/executors.md b/docs/agents/architecture/executors.md new file mode 100644 index 0000000..1e13f03 --- /dev/null +++ b/docs/agents/architecture/executors.md @@ -0,0 +1,469 @@ +# Executors + +An executor turns an admitted job into processes on a node, streams their I/O, +and reports what happened. The scheduler decides; the executor carries the +decision out. Overview: [ARCHITECTURE.md](../../../ARCHITECTURE.md). Related +decisions: D1, D2, D13, D16, D17. + +There are two executors: + +- the **K3s executor**, used from M1 +- the **node executor** (M5), which runs jobs as systemd units without + containers + +Both implement the same interface and pass the same conformance tests. Each node +is managed by exactly one executor. + +## How it fits + +Both executors take the same `LaunchSpec` and end at the same `mist-job-entry`. +Only the middle differs: + +```mermaid +flowchart LR + S[Scheduler] -->|LaunchSpec| I{{Executor interface}} + I --> K["K3s executor (M1)"] + I -.-> NE["Node executor (M5)"] + + K -->|client-go| KAPI[k3s API server] + KAPI --> Pod["Pod job-ID
uid, team gid, chosen node,
requests = limits, no token"] + KAPI --> CM["ConfigMap job-ID
script + job.json"] + Pod --> Vol["Volumes: team folder, software,
scratch, /dev/shm, mist-nss"] + Pod --> Dev["Device plugin / DRA
allocated GPUs or boards"] + Pod --> Entry[mist-job-entry] + + NE -.->|gRPC + mTLS| Agent[mist-noded] + Agent -.->|D-Bus| Unit["systemd unit mist-job-ID
User, DeviceAllow, MemoryMax, BindPaths"] + Unit -.-> Entry + + Entry --> Work[Script, command or shell] + K -->|"Started, Exited, Lost"| S +``` + +## Interface + +```go +package executor + +type Executor interface { + Name() string + + // Nodes reports schedulable capacity and health for nodes this executor manages. + Nodes(ctx context.Context) ([]Node, error) + + // Launch ensures the one execution exists on spec.Node. Same ID/spec is + // repeat-safe; never recreate an execution with a recorded native object + // UID or evidence that it has started/ended after that object disappears. + Launch(ctx context.Context, spec LaunchSpec) error + + // Terminate sends SIGTERM, then SIGKILL after grace. Idempotent. + Terminate(ctx context.Context, jobID int64, grace time.Duration, reason string) error + + // Attach connects a terminal to a running interactive job; returns when the session ends. + Attach(ctx context.Context, jobID int64, term Terminal) error + + // Logs streams the job's live output (stdout+stderr as written to the .out file). + Logs(ctx context.Context, jobID int64, follow bool) (io.ReadCloser, error) + + // List returns every execution this executor knows about, for reconciliation. + List(ctx context.Context) ([]Execution, error) + + // Events is an at-least-once notification stream. Persist and dedupe by + // EventID; List/observation receipts recover missed notifications. + Events() <-chan Event +} + +type LaunchSpec struct { + JobID int64 + ExecutionID string // immutable UUID; one execution per job + SpecHash string // immutable normalized spec digest + AuthGeneration int64 // authorization used at admission + LaunchDeadline time.Time // admitted_at + launch timeout + Kind Kind // Batch | Command | Interactive + Node string + User User // Name, UID, PersonalGID + TeamSlug string + TeamGID int + ExtraGIDs []int // mist-users + Resources Resources // CPUs, MemoryBytes, GPUs, TTBoards, ScratchBytes + DeviceIDs []string // optional; set when the scheduler assigns specific devices + Mounts []Mount // resolved team folder + software (volume, source, target, readOnly) + Workdir string + OutputPath string + Script string // Batch + Args []string // Batch arguments (never shell-concatenated) + Argv []string // Command + Env map[string]string // frozen base; work-start/deadline fields derived at startup + Runtime string // partition runtime image / host profile, pinned + TimeLimit time.Duration // executor sets a backstop slightly beyond this + SignalGrace time.Duration +} + +type Event struct { + EventID string // stable across redelivery + JobID int64 + ExecutionID string + Kind EventKind // Started | Finished | Exited | LaunchFailed | Lost + Node string + ExitCode *int + Reason ExitReason // Normal | OutOfMemory | ScratchExceeded | Deadline | Signaled | ... + Detail string // human-readable, shown to users + At time.Time +} + +type Node struct { + Name string + Partitions []string + Capacity Resources + Devices []Device // optional per-device inventory + Healthy bool + Detail string +} +``` + +The scheduler and API depend only on this package. Kubernetes and systemd types +never leak outside their executor packages. Define the supporting wire types in +this package before component implementations: + +- `Execution`: JobID, ExecutionID, SpecHash, Node, observed phase, native object + UID, started/finished timestamps, exit result, cleanup status and evidence. +- `Resources`: CPU millicores, memory/scratch bytes, whole GPU/board counts, + hugepage bytes (M4). `Mount`: volume ID, source, absolute target, read-only. +- `Terminal`: byte input/output, resize and allowed signal controls, plus a + cancellation context. Stream cancellation detaches; it does not cancel a job. +- `List` must distinguish unavailable/incomplete observations from authoritative + absence. Failed calls are errors, never an empty successful list. Merge live + objects with durable execution receipts; a watch stream is not a history store. + +The executor persists create intent, object UID, start/finish observations and +cleanup receipts in executor-owned database tables through its package. Persist +finish metadata and bounded log tail before deleting execution objects. Scheduler +state changes and event consumption are transactional; events carry stable IDs. +Only the scheduler issues Launch/Terminate; the API uses read/log/attach methods. +A mismatched SpecHash for an existing ExecutionID is a hard conflict. M1 never +reruns an execution that has started, even if the process ended before a watch +observed it. A recorded native object UID also forbids recreating a missing Pod: +it may have run before deletion without a start notification. Without evidence +that an ambiguous creation never ran, report loss/quarantine rather than replay. +Durable tombstones prevent ID reuse after cleanup. + +## Guarantees every executor must provide + +These guarantees are tested by a shared **conformance suite** +(`internal/executor/conformance`) that runs against each implementation on real +nodes. + +1. **Identity.** Processes run as `User.UID`. The primary group is `TeamGID`. + Supplementary groups may contain only `ExtraGIDs`, `PersonalGID` and a + redundant `TeamGID` (some runtimes include the primary group there). No other + team/system groups are allowed. There is no privilege escalation or capability. +2. **Filesystem view.** + - **Visible:** only the given mounts, at `/mist/teams/` and + `/mist/software` (read-only), plus `/scratch`. Other teams' folders are + absent, not merely denied. + - **Writable:** only the team folder (as permissions allow), `/scratch`, + `/tmp` and memory-backed `/dev/shm`. `/tmp` lives inside scratch. +3. **Devices.** Only allocated GPUs/TT boards can be opened. CUDA presents the + allocated GPU set with logical indices from 0; host device filenames need not + be renumbered. Do not hard-code CUDA_VISIBLE_DEVICES=0 for every physical GPU; + verify the plugin/runtime's UUID or index mapping for both A4000s. Native + execution uses allocated physical IDs/UUIDs; CUDA remaps that visible set. +4. **Limits.** CPU and memory limits apply, swap is off, and scratch is capped. + Hitting the memory limit gives `OutOfMemory`; detected scratch excess gives + `ScratchExceeded`. K3s ephemeral storage limits are eviction thresholds, not + synchronous write quotas. Reserve disk headroom and reject admissions under + disk pressure. Include `/dev/shm` in the memory budget. +5. **Time limits.** `Terminate` delivers SIGTERM to the job's processes, then + SIGKILL after the grace period. A backstop kills the job even if the + scheduler is down. +6. **Network.** + - **Allowed:** outbound internet and DNS. + - **Blocked:** the LAN, the direct link, the Tailscale range, cluster + services and the Kubernetes API. + - **Inbound:** none. +7. **Output.** Output goes to `OutputPath` through `mist-job-entry` (below), and + live output is also available from `Logs`. +8. **Cleanup.** When the job ends, all of its processes are gone, scratch is + deleted and devices are free. `Finished` reports process exit; `Exited` + acknowledges completed cleanup. Allocation release requires `Exited` or an + equivalent durable cleanup receipt. Failed cleanup keeps capacity reserved; + node loss quarantines it until reconciliation proves cleanup. +9. **Repeat-safe and recoverable.** Repeating `Launch` or `Terminate` is safe. + After a restart, `List` reports what is actually running. + +## `mist-job-entry` + +A small static Go binary that is the first process of every job, under any +executor. It is the single place that implements the job contract in +[runtime-environment.md](runtime-environment.md): + +1. **Set up:** + - set `umask 0027` + - create, or reset to `0700`, `$HOME/{.cache,.config,.local,.ssh}` + - create the output file and write a header (job ID, node, resources, start + time) +2. **Run the work:** + - **Batch:** `bash