From 7365f37e4579f5164df9ee8c90156638a2ec7401 Mon Sep 17 00:00:00 2001 From: Brandon Li Date: Mon, 26 Oct 2026 14:09:09 -0400 Subject: [PATCH 1/3] Add HPC-style architecture design docs Describe the redesign agreed on October 26: per-team folders, restricted SSH login, and jobs that run as the researcher's own UID under Mist's own scheduler, with a pluggable executor (K3s first, native node executor later). Adds the system overview, one contract doc per component, a decision log, and the jobs/scheduling implementation plan. README points to the new design and notes that docs/content describes the old platform. Co-Authored-By: Claude Opus 5.5 --- ARCHITECTURE.md | 237 ++++++++++++++ README.md | 9 + docs/agents/architecture/api.md | 157 +++++++++ docs/agents/architecture/decisions.md | 218 +++++++++++++ .../architecture/deployment-and-network.md | 120 +++++++ docs/agents/architecture/executors.md | 307 ++++++++++++++++++ .../architecture/identity-and-accounts.md | 186 +++++++++++ docs/agents/architecture/login-gateway.md | 162 +++++++++ docs/agents/architecture/portal.md | 45 +++ .../architecture/runtime-environment.md | 152 +++++++++ docs/agents/architecture/scheduler.md | 250 ++++++++++++++ docs/agents/architecture/storage.md | 185 +++++++++++ docs/agents/plans/jobs-and-scheduling.md | 185 +++++++++++ 13 files changed, 2213 insertions(+) create mode 100644 ARCHITECTURE.md create mode 100644 docs/agents/architecture/api.md create mode 100644 docs/agents/architecture/decisions.md create mode 100644 docs/agents/architecture/deployment-and-network.md create mode 100644 docs/agents/architecture/executors.md create mode 100644 docs/agents/architecture/identity-and-accounts.md create mode 100644 docs/agents/architecture/login-gateway.md create mode 100644 docs/agents/architecture/portal.md create mode 100644 docs/agents/architecture/runtime-environment.md create mode 100644 docs/agents/architecture/scheduler.md create mode 100644 docs/agents/architecture/storage.md create mode 100644 docs/agents/plans/jobs-and-scheduling.md diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md new file mode 100644 index 0000000..04af45a --- /dev/null +++ b/ARCHITECTURE.md @@ -0,0 +1,237 @@ +# Mist architecture + +Mist is UTMIST's research compute platform. It works like the batch systems on +national HPC clusters. Each researcher gets a folder inside each of their team +folders and moves files with `scp`/`rsync`. Code runs as **jobs**: batch scripts +or short interactive shells. A scheduler places jobs on the GPU, CPU and +Tenstorrent hardware. Each job runs as the researcher's own Linux user, in their +own folder. Researchers never write Dockerfiles or declare mounts. + +This document is the system overview and the coordination point between +component owners. Each component has a document with the contracts it must +honour. The reasons behind major choices are in the +[decision log](docs/agents/architecture/decisions.md). + +> **Status:** target design, agreed October 2026. This design replaces the +> platform described in `docs/content/` (container-image jobs, dataset and +> result pools, team namespaces). It starts fresh: no users or data are migrated. + +## Goals + +1. **Development feels like a workstation.** A researcher sets up a Python + environment once, in their own folder, and every job uses it. Files a job + writes are simply there afterwards. +2. **Access follows teams.** Teammates can read each other's work. Other teams + cannot see it at all. Some projects are under IP agreements, so this boundary + is a hard requirement. +3. **Code runs only inside jobs.** Login sessions move files and manage jobs. + Compute and accelerators are reached only through a scheduled job. +4. **Adding a node is cheap.** Adding a third or fourth machine means configuring + that node, not redesigning the system. +5. **The A4000 GPUs come first.** The NVIDIA partition ships first; Tenstorrent + follows on the same design. + +## System at a glance + +```mermaid +flowchart LR + subgraph Users + SSH[ssh / scp / rsync] + Browser[Browser] + end + + subgraph TT["utmist-tt (QuietBox)"] + Gate[Login gateway
sshd + mist-gate] + Relay[mist-relay] + Prov[Provisioner agent] + NFS[("/mist
(durable storage)")] + CPUP[partition: cpu] + TTP[partition: tt
4× n300] + end + + subgraph NV["utmist-z1opa08 (NVIDIA)"] + Portal[Web portal] + API[mistd API] + Sched[mistd scheduler] + DB[(Postgres)] + K3s[k3s server] + GPUP[partition: gpu
2× RTX A4000] + end + + Misty[(misty
people and teams)] + + SSH -->|Tailscale| Gate --> Relay --> API + Browser -->|Tailscale| Portal --> API + API --> DB + API --> Misty + Sched --> DB + Sched -->|Executor| K3s + K3s --> GPUP & CPUP & TTP + Prov -->|desired state| API + Prov --> NFS + GPUP -. "NFS over 10 GbE direct link" .-> NFS +``` + +## Components + +| Component | Responsibility | Document | Proposed owner | +|---|---|---|---| +| Identity and accounts | People and teams from misty; Mist accounts, UIDs, GIDs, SSH keys; sign-up; provisioning Linux accounts and folders | [identity-and-accounts.md](docs/agents/architecture/identity-and-accounts.md) | Lucas | +| Storage | `/mist` layout and permissions, NFS, volumes, usage reporting, quotas, scratch, backups | [storage.md](docs/agents/architecture/storage.md) | Ansh | +| Login gateway | Restricted SSH (file transfer and `mist` commands only), identity relay, the `mist` CLI | [login-gateway.md](docs/agents/architecture/login-gateway.md) | TBD | +| API | The one HTTP API used by the portal, CLI and agents; authentication and authorization | [api.md](docs/agents/architecture/api.md) | Shared; each owner defines their endpoints | +| Scheduler | Partitions, QoS, limits, queue, priority, fairshare, placement, accounting | [scheduler.md](docs/agents/architecture/scheduler.md) | Brandon | +| Executors | Run an admitted job on a node; stream I/O; report status. K3s executor first, native node executor later | [executors.md](docs/agents/architecture/executors.md) | Brandon | +| Runtime environment | What a job sees: images, software, environment variables, caches, outputs, signals | [runtime-environment.md](docs/agents/architecture/runtime-environment.md) | Scott (to confirm) | +| Web portal | Lightweight UI over the API: account, SSH keys, teams, jobs, queue, usage, admin | [portal.md](docs/agents/architecture/portal.md) | Andrei | +| Deployment and network | Hosts, network paths, services, secrets, admin access, adding nodes | [deployment-and-network.md](docs/agents/architecture/deployment-and-network.md) | Ethan, with Ansh | + +Owners should confirm or reassign rows here. + +## Core concepts + +- **Person.** A human registered in misty. Misty is the source of truth for + people, teams and membership. +- **Account.** A person's Mist identity: one immutable `username`, one stable + Linux UID, and SSH keys. It is created when a registered person signs up. +- **Team.** A misty team. On Mist, each team has a Linux group (`team-`) and + a folder `/mist/teams//`. +- **Partition.** A named pool of nodes that share one kind of hardware: `gpu` + (NVIDIA), `cpu` (CPU-only, used for data prep and downloads) and `tt` + (Tenstorrent). A job targets exactly one partition. +- **QoS.** A class of service, `normal` or `interactive`, carrying limits and a + priority weight. +- **Job.** A request to run a script (batch) or a shell (interactive). It names + one partition, one QoS, one **charged team**, its resources and a time limit. + It runs as the submitter's UID. Accelerators are allocated whole and one at a + time: two 1-GPU jobs run side by side on the A4000 box. +- **Charged team.** The team a job is billed to and scoped to. The job sees only + that team's folder and the shared software. It cannot see the submitter's other + teams. +- **Volume.** One durable storage location, an NFS export on one server. Each + team folder lives entirely on one volume. There is one volume today. + +## Key flows + +### Sign-up and first login + +1. A person signs in to the portal. Identity confirms with misty that they are + registered and loads their teams. +2. Mist creates their account: it reserves a username and a UID and records team + memberships. +3. The person adds an SSH public key in the portal. +4. The provisioner agent on the storage/login host reconciles the change. It + creates the Linux user and groups, and `/mist/teams///` for each + team. +5. The person runs `ssh @`. They land in a folder with one + entry per team. + +### Moving files + +`scp`, `rsync` and `sftp` go through the login gateway, which sits on the storage +host and writes straight to disk. Unix permissions decide what each person can +read and write. + +Downloads from the internet (Hugging Face, S3, Kaggle and so on) run as jobs on +the `cpu` partition, using `mist run -p cpu -- ` or `mist shell -p cpu`. + +### Batch job + +```mermaid +sequenceDiagram + participant U as Researcher (login) + participant G as mist-gate + relay + participant A as mistd API + participant S as Scheduler + participant E as Executor (K3s) + participant N as Node + U->>G: mist submit train.sh + G->>A: POST /v1/jobs (relay-asserted user) + A->>A: validate spec, membership, limits + A-->>U: job 412 PENDING + S->>S: order by priority, check limits and free capacity + S->>E: Launch(job 412, node, resources) + E->>N: start as uid, team gid, team folder, devices + N-->>E: running … exited(0) + E-->>S: status events + S->>A: job 412 COMPLETED + U->>G: tail -f mist-412.out +``` + +The job starts in the folder it was submitted from. Its output streams to +`mist-.out` there. Checkpoints and results go wherever the code writes them, +in `/mist` by default. + +### Interactive job + +`mist shell --gpus 1` submits a job with QoS `interactive`. When the job starts, +the API attaches the researcher's terminal to a shell inside it. The job ends +when the shell exits. If the connection drops, the job waits for a grace period +so the researcher can run `mist attach`, then it is cancelled. A browser terminal +will later reuse the same attach endpoint. + +## Data ownership + +Postgres on the control node is the single source of truth for Mist state. +Kubernetes objects, Linux accounts and folders are *derived* state: agents +reconcile them from the database. + +| Data | Authority | Owner | +|---|---|---| +| People, teams, membership | misty (cached in Mist) | Identity | +| Accounts, UIDs, GIDs, SSH keys | Mist DB | Identity | +| Linux users, groups, folders | Derived by the provisioner | Identity, Storage | +| Volumes, team placement, usage | Mist DB | Storage | +| Partitions, QoS, limits | Mist DB (admin-configured) | Scheduler | +| Jobs, job events, usage records | Mist DB | Scheduler | +| Pods and other execution objects | Derived by the executor | Executors | +| Files | `/mist` | Storage | + +Each owner owns their tables and migrations. Other components go through that +owner's Go package or API, not their tables. + +## Code layout + +```text +cmd/mistd/ one binary: `mistd api`, `mistd scheduler`, `mistd provision` +cmd/mist/ CLI (on the login gateway; laptops later) +cmd/mist-gate/ SSH ForceCommand dispatcher +cmd/mist-relay/ login-node identity relay and SSH key lookup +cmd/mist-job-entry/ in-job wrapper: output file, signals, exit reasons +internal/api/ HTTP server, middleware, shared wire types +internal/identity/ accounts, misty client and stub, provisioning state +internal/storage/ layout rules, volumes, usage +internal/scheduler/ queue, policy, placement, accounting +internal/executor/ Executor interface; k3s/ and node/ implementations +internal/db/ connection and migrations (one directory per owner) +images/ job runtime images +deploy/ k3s manifests, systemd units, host setup +web-interface/ portal +``` + +The existing `src/` and `cli/` modules belong to the platform being replaced. +They are removed once the new path serves jobs. + +## Milestones + +| Milestone | Outcome | +|---|---| +| **M1: GPU batch** | Sign-up (misty stubbed), SSH login, file transfer, `mist submit/run/jobs/show/logs/cancel` on `gpu` and `cpu`, team isolation, 10 GbE direct link | +| **M2: Interactive** | `mist shell` and `mist attach` with reattach grace; `interactive` QoS | +| **M3: Fair scheduling** | Priority, fairshare by team, backfill, accounting, admin limits, queue views; backups in place | +| **M4: Tenstorrent** | `tt` partition, TT runtime image, board reset between jobs | +| **M5: Native executor and web terminal** | Node executor without containers, behind the same interface; browser terminal; port forwarding into jobs | + +Each component document lists its work per milestone. + +## Open questions + +| Question | Owner | +|---|---| +| Real misty API: shape, and how membership changes reach Mist (polling or webhooks) | Identity | +| How people prove who they are at sign-in (Google or UofT SSO, or email magic link) | Identity | +| Enforced storage quotas: needs `/mist` on a dedicated filesystem with project quotas | Storage | +| Offboarding: what happens to a folder when someone leaves a team or UTMIST | Identity, Storage | +| Backup tool, schedule and retention | Storage | +| Default limits per partition and per user once real usage is known | Scheduler | +| Monitoring and alerting beyond the portal | Deployment | diff --git a/README.md b/README.md index aec78c8..c658ef7 100644 --- a/README.md +++ b/README.md @@ -7,6 +7,15 @@ collect persistent results through the same website or CLI. **Private website:** [http://100.73.139.66:8088](http://100.73.139.66:8088), accessible through authorized Tailscale connectivity. A Mist account and team membership are also required. +## Architecture redesign — October 26 + +Mist is moving to an HPC-style design: per-team folders, restricted SSH login, +and jobs that run as the researcher's own Linux user under Mist's own scheduler. +Jobs no longer need images or mounts. Start with [ARCHITECTURE.md](ARCHITECTURE.md); +component contracts and the decision log are in +[docs/agents/architecture/](docs/agents/architecture/decisions.md). The +sections below and `docs/content/` describe the platform being replaced. + ## Storage update — October 21 New submissions use ordinary team dataset/result folders on QuietBox, accessed diff --git a/docs/agents/architecture/api.md b/docs/agents/architecture/api.md new file mode 100644 index 0000000..8cf6f6c --- /dev/null +++ b/docs/agents/architecture/api.md @@ -0,0 +1,157 @@ +# API + +`mistd api` is the only way clients and agents reach Mist state. This document +fixes the conventions every endpoint follows, how callers authenticate, and the +job wire types shared by the CLI, portal and scheduler. Each component document +lists the endpoints it owns. Overview: [ARCHITECTURE.md](../../../ARCHITECTURE.md). + +## Conventions + +- **Format.** JSON over HTTP. Paths are versioned: `/v1/...` for users and + admins, `/internal/v1/...` for host agents. +- **Placement.** One `mistd api` process on the control node (NVIDIA box). The + portal's Nginx forwards `/api/*` to it and strips the prefix. Agents reach it + on the direct link (`192.168.240.1`). +- **Logic lives in the API.** Clients don't compute permissions, defaults or + states; the API returns ready-to-display fields (for example `pending_reason` + and `can_cancel`). Clients may poll: every 5–10 s is fine. +- **Errors** use one shape and a stable `code`: + + ```json + { "error": { "code": "limit_exceeded", "message": "Team vision already has 2 GPUs running", "details": {} } } + ``` + + Codes: `unauthenticated`, `forbidden`, `not_found`, `invalid`, + `limit_exceeded`, `conflict`, `unavailable`, `internal`. +- **Lists** take `?limit=` and `?cursor=` and return `{ "items": [...], "next_cursor": "..." }`. +- **Times** are RFC 3339 UTC strings. **Durations** are integer seconds. + **Sizes** are integer bytes. +- **Streams** (log follow, interactive attach) use WebSockets on the same paths + with `?follow=1` or an `Upgrade` header. + +## Authentication + +| Caller | Credential | Principal | +|---|---|---| +| Portal | Session cookie set at sign-in | The signed-in account | +| CLI on the login gateway | Relay token + `X-Mist-Asserted-User` (see [login-gateway.md](login-gateway.md)) | The asserted account | +| Provisioner, relay key lookup | Per-host agent token | The host; `/internal/*` only | +| Laptop CLI (later) | Personal API token | The token's account | + +One middleware resolves the principal, and authorization goes through +`identity.Authorize` (see [identity-and-accounts.md](identity-and-accounts.md)). +Handlers never read identity headers directly. + +## Endpoint index + +| Area | Endpoints | Owner doc | +|---|---|---| +| Account | `GET/PATCH /v1/me`, `/v1/me/ssh-keys` | identity | +| Teams | `GET /v1/teams`, `GET /v1/teams/{slug}` | identity | +| Storage | `GET /v1/teams/{slug}/usage`, `GET /v1/volumes` (admin) | storage | +| Jobs | `POST /v1/jobs`, `GET /v1/jobs`, `GET /v1/jobs/{id}`, `POST /v1/jobs/{id}/cancel`, `GET /v1/jobs/{id}/log`, `GET /v1/jobs/{id}/attach` (WebSocket) | scheduler / executors | +| Cluster | `GET /v1/partitions`, `GET /v1/queue` | scheduler | +| Admin | `/v1/admin/accounts`, `/v1/admin/partitions`, `/v1/admin/qos`, `/v1/admin/limits`, `/v1/admin/nodes` | identity, scheduler | +| Agents | `/internal/v1/provision/*`, `/internal/v1/ssh-keys/{username}` | identity | + +## Job wire types + +### Submitting: `POST /v1/jobs` + +```json +{ + "team": "vision", + "partition": "gpu", + "qos": "normal", + "name": "resnet-sweep-3", + "kind": "batch", + "script": "#!/bin/bash\n#MIST --gpus=1\npython train.py --lr 3e-4\n", + "args": [], + "workdir": "/mist/teams/vision/alice/resnet", + "output": "mist-%j.out", + "env": { "WANDB_PROJECT": "resnet" }, + "resources": { + "cpus": 8, + "memory_bytes": 34359738368, + "gpus": 1, + "tt_boards": 0, + "scratch_bytes": 107374182400 + }, + "time_limit_seconds": 14400, + "signal_grace_seconds": 60 +} +``` + +- `kind`: + - `batch` needs `script`. + - `command` (from `mist run`) takes `argv` instead of `script`. + - `interactive` (from `mist shell`) takes neither. +- **The API fills in missing fields** from partition and QoS defaults and + returns the complete, normalised spec. Defaults live only in the database, + never in clients. +- **Validation:** + - The caller must be a member of `team`. + - `workdir` must be inside `/mist/teams//`. + - Resources and time must be within partition and QoS maximums. + - Every `gpu` job needs `gpus >= 1`. +- **Reply:** `201` with the job object, in state `PENDING`. + +### Job object: `GET /v1/jobs/{id}` + +```json +{ + "id": 412, + "name": "resnet-sweep-3", + "user": "alice", + "team": "vision", + "partition": "gpu", + "qos": "normal", + "kind": "batch", + "state": "PENDING", + "pending_reason": "RESOURCES", + "pending_detail": "Waiting for 1 GPU; estimated start after job 409 ends", + "exit_reason": null, + "exit_code": null, + "resources": { "cpus": 8, "memory_bytes": 34359738368, "gpus": 1, "tt_boards": 0, "scratch_bytes": 107374182400 }, + "time_limit_seconds": 14400, + "node": null, + "workdir": "/mist/teams/vision/alice/resnet", + "output_path": "/mist/teams/vision/alice/resnet/mist-412.out", + "submitted_at": "2026-11-02T15:04:05Z", + "started_at": null, + "ended_at": null, + "can_cancel": true, + "can_attach": false +} +``` + +**States and reasons** are defined in [scheduler.md](scheduler.md#job-states). +Other teams' jobs appear in `GET /v1/queue` with `user`, `team`, `name`, +`workdir` and `output_path` set to `null` (D15). + +### Logs: `GET /v1/jobs/{id}/log` + +- **Running job:** returns the live output stream from the executor. Add + `?follow=1` to stream it over a WebSocket. +- **Finished job:** returns the stored tail (last 64 KiB) from the database. +- **The full log is always the `.out` file in `/mist`.** The response includes + `output_path` so clients can say where it is. + +### Attach: `GET /v1/jobs/{id}/attach` (WebSocket) + +Binary frames carry terminal bytes. Text frames carry control messages: + +```json +{ "type": "resize", "cols": 120, "rows": 40 } +{ "type": "signal", "name": "SIGINT" } +{ "type": "exit", "code": 0 } +``` + +Only the job's owner can attach. The CLI's `mist shell` and the later browser +terminal both use this endpoint. + +## Versioning + +Additive changes (new fields, endpoints or enum values) don't change the +version. Clients must ignore unknown fields and show unknown enum values as +plain text. Breaking changes go to `/v2`. diff --git a/docs/agents/architecture/decisions.md b/docs/agents/architecture/decisions.md new file mode 100644 index 0000000..d896490 --- /dev/null +++ b/docs/agents/architecture/decisions.md @@ -0,0 +1,218 @@ +# Architecture decisions + +Each entry records the context, the decision, and what follows from it. To +change a decision, add a new entry that supersedes the old one rather than +editing it. The system overview is [ARCHITECTURE.md](../../../ARCHITECTURE.md). + +## D1. Mist has its own scheduler; Kubernetes only runs jobs + +**Context.** The previous platform made every job a container image with +declared mounts, and that friction is the reason for the redesign. An existing +HPC scheduler (Slurm) was considered. The team chose to build the scheduler as +part of Mist, and the cluster is expected to grow beyond two nodes. + +**Decision.** Mist's scheduler owns the queue, policy and placement. A pluggable +**executor** turns an admitted job into running processes. The first executor +uses k3s; the Kubernetes scheduler never decides when or where a job runs. + +**Consequences.** Mist implements queueing, limits, priority and accounting. +Kubernetes provides process supervision, device allocation (NVIDIA device plugin, +Tenstorrent DRA), cgroup limits and node membership. If both systems made +placement decisions, jobs could sit pending with no visible reason; pinning pods +to the scheduler's chosen node prevents that. + +## D2. Executors sit behind one interface, and each node has exactly one executor + +**Decision.** Executors implement the interface in [executors.md](executors.md). +The K3s executor ships first. A native node executor (systemd units and cgroups, +no containers) follows behind the same interface. Each partition maps to one +executor, and each node belongs to one executor, because two systems cannot both +own a node's devices. + +**Consequences.** The scheduler and API never import Kubernetes types. A shared +conformance test suite defines executor behaviour. Moving a node between +executors means draining it first. + +## D3. Postgres is the source of truth + +**Context.** The previous platform kept queue and policy state in ConfigMaps and +suspended Jobs. + +**Decision.** All Mist state lives in Postgres. Pods, Linux accounts and folders +are derived and reconciled from it. + +**Consequences.** Restarting `mistd` or wiping execution objects loses nothing; +the executor rebuilds its view by comparing the database with what is actually +running. Usage history for fairshare and accounting has a natural home. + +## D4. One Linux UID per person + +**Context.** Creating a Linux user per job was considered, so that only jobs +would get privileges. + +**Decision.** Each person has one stable UID, the same on every host. Jobs run +as that UID. Privileges (devices, write access to the right folders) come from +the job's cgroup, mounts and supplementary groups, not from a different user. + +**Consequences.** Files keep one owner across login, jobs and teammates. There +is no ownership mess between jobs. UID and GID allocation is centralised in the +Mist DB, in fixed ranges that don't collide with local system accounts. + +## D5. Each job is scoped to one charged team + +**Decision.** Every job names a charged team; by default it is the team whose +folder the job was submitted from. The job is given only that team's group and +that team's folder, plus shared read-only software. + +**Consequences.** Someone in two teams cannot accidentally mix IP-bound data +between them inside a job. They can still copy files by hand, since they can +read both teams; no technical control prevents a person from moving data they +can read. A person's home is per-team for the same reason (D8). + +## D6. Login moves files and manages jobs; code runs only in jobs + +**Decision.** Researcher SSH sessions are restricted by `ForceCommand` to: +SFTP/scp, `rsync`, a small set of built-in file viewing commands, and the `mist` +CLI. Anything that executes user code is a job. + +**Consequences.** The login host (also a storage and compute node) is protected +from stray compute. Internet downloads become jobs (D11). This is stricter than +typical HPC login nodes, and that is deliberate on a cluster with no dedicated +login hardware. + +## D7. Login and storage on the TT box; control plane on the NVIDIA box + +**Decision.** + +- **TT box:** the durable storage (`/mist`), the login gateway and the + provisioner. +- **NVIDIA box:** the k3s server, `mistd`, Postgres and the portal. + +**Consequences.** Uploads land on local disk with no network hop. The TT box is +the single point of failure for logins and file access. The control plane and +the database sit on separate hardware from the bulk storage. + +## D8. A researcher's team folder is their home + +**Decision.** `/mist/teams///` is both `$HOME` and the default +working directory for jobs charged to that team. It is team-readable (`2750`). +`.cache`, `.config`, `.local` and `.ssh` inside it are private (`0700`): the +provisioner creates them that way, and the job entry wrapper resets them on +every job. A few large shareable caches are mapped to the team's `.cache/` by +environment variables. + +**Consequences.** Every tool's cache and config lands somewhere sensible without +per-tool mapping. Credentials stored in the usual hidden folders stay private. +**Accepted risk:** a tool that writes a secret as a top-level file without +restricting its permissions exposes it to teammates (never to other teams). +Someone in two teams keeps two homes. Researchers need this explained, so it +appears under "What researchers see" in [storage.md](storage.md). + +## D9. Durable storage plus local scratch; growth by adding volumes + +**Decision.** `/mist` is the only durable storage. Node-local scratch is +temporary and wiped after each job. Durable capacity grows by adding **volumes** +(separate NFS exports), and each team folder lives entirely on one volume. No +distributed filesystem is used. + +**Consequences.** Researchers never have to work out which machine holds a file. +Adding storage is a database record plus an export. Spare disk on compute nodes +goes to backups first, then scratch, and only then to new volumes. + +## D10. The direct link runs at 10 GbE on existing network cards + +**Context.** The direct cable uses Intel I210 ports, which max out at 1 Gb/s +(about 110 MB/s). Loading a 14 GB model took around two minutes. + +**Decision.** Move the link to the 10GBASE-T ports already installed: the +Aquantia AQC107 (`ens6`) on the NVIDIA box and the Intel X710 +(`enp10s0f0np0`) on the TT box. Use Cat6a cable. + +**Consequences.** NFS reaches roughly 700–1000 MB/s, so the separate node-local +model cache is dropped from the design. The second X710 port can take one more +direct-connected node. A fourth node needs a 10 GbE switch. + +## D11. Downloads are ordinary jobs on a `cpu` partition + +**Decision.** The TT box's CPUs form a `cpu` partition, which sits next to the +storage. `mist run -p cpu -- ` and `mist shell -p cpu` cover every download +tool. There is no dedicated download service. + +**Consequences.** Any source works (Hugging Face, S3, Kaggle, `git lfs`, `wget`), +using the researcher's own tokens. GPUs are never held idle during downloads. + +## D12. Job outputs go to `/mist`; scratch is opt-in + +**Decision.** + +- **Output.** Stdout and stderr stream to `mist-.out` in the submit folder. + Mist appends its own messages there too (node, cancellation, time limit, + out-of-memory). +- **Job record.** The database keeps the job's metadata, a copy of the script, + and a tail of the log. +- **Scratch.** `$TMPDIR` points at node-local scratch, which is deleted when the + job ends. +- **Time limits.** At the limit the job receives SIGTERM, then SIGKILL after a + grace period (default 60 s). + +**Consequences.** Nothing a researcher writes by default is ever lost. The +portal can show logs without reading team files. + +## D13. Accelerator device files stay world-accessible; isolation comes from cgroups + +**Context.** `/dev/nvidia*` and `/dev/tenstorrent/*` are mode `0666` on both +hosts. + +**Decision.** Keep the host modes. Inside a job, the executor's device cgroup +(device plugin, or `DeviceAllow` in the node executor) exposes only the +allocated devices. Host-level access is prevented by D6 and by removing +unmanaged GPU consumers (for example an `ollama` service). + +**Consequences.** This matches Slurm's model. If the device files were tightened +to `0660`, containers could no longer open the devices they were given, because +non-root jobs inherit the host permissions. + +## D14. Misty owns people and teams; Mist owns accounts + +**Decision.** Misty is authoritative for who exists, which teams exist, and who +belongs to them. Mist caches that data and adds what is specific to Mist: +usernames, UIDs and GIDs, SSH keys, limits and roles. Until misty's API is +available, a stub backed by a file stands in for it. + +**Consequences.** Removing someone from a team in misty eventually removes their +Mist access to that team. Mist never edits people or teams. + +## D15. Other teams' jobs are anonymised in queue views + +**Decision.** Queue and partition views show every job's state, resources and +age, so researchers can see why they are waiting. The user, team, job name and +paths are shown only for the viewer's own teams. + +**Consequences.** Job names and paths can't leak IP-bound project details +between teams. + +## D16. One Go binary, plain Kubernetes objects + +**Decision.** `mistd` is one Go binary with subcommands (`api`, `scheduler`, +`provision`). It talks to Kubernetes through `client-go` and creates plain Pods. +It uses no CRDs, operators, `kubectl` shell-outs or Kubernetes Jobs. + +**Consequences.** Mist's own state machine stays in Mist's code and database, +which keeps it easy to understand and to swap executors. + +## D17. Accelerators are whole devices; CPU and memory are capped per GPU + +**Decision.** + +- **Whole devices.** Jobs request whole GPUs or TT boards; two 1-GPU jobs share + the A4000 box side by side. +- **Per-GPU caps.** Each GPU partition caps CPUs and memory per allocated GPU, + so one job cannot leave another GPU unusable. +- **No GPU-less jobs on GPU nodes.** `gpu` partition jobs must request at least + one GPU. +- **No GPU sharing.** Fractional GPUs (time-slicing) are not offered, because + the A4000 has no hardware partitioning and time-slicing gives no memory + isolation. + +**Consequences.** Placement only counts devices, CPUs and memory. Time-slicing +could later be added as an opt-in for small debug work. diff --git a/docs/agents/architecture/deployment-and-network.md b/docs/agents/architecture/deployment-and-network.md new file mode 100644 index 0000000..eb125f8 --- /dev/null +++ b/docs/agents/architecture/deployment-and-network.md @@ -0,0 +1,120 @@ +# Deployment and network + +Covers which host runs what, how hosts and users connect, where configuration +and secrets live, admin access, and how nodes are added. Overview: +[ARCHITECTURE.md](../../../ARCHITECTURE.md). Related decisions: D7, D10, D13. + +## Hosts + +| | `utmist-z1opa08` (NVIDIA) | `utmist-tt` (QuietBox) | +|---|---|---| +| Hardware | 56 CPUs, 157 GiB RAM, 2× RTX A4000 16 GB, 2× 447 GB SATA SSD | 32 CPUs, 503 GiB RAM, 4× n300 (8 chips), 3.6 TB NVMe | +| OS | Ubuntu 22.04, kernel 6.8, cgroup v2 | same | +| Roles | k3s server; `mistd api` + `mistd scheduler`; Postgres; portal (Nginx); `gpu` partition; backup target (M3) | k3s agent; `/mist` volume 1 + NFS server; login gateway (sshd, `mist-gate`, `mist-relay`); `mistd provision`; `cpu` and `tt` partitions | + +## Networks + +| Network | Addresses | Carries | +|---|---|---| +| **Direct link (10 GbE)** | NVIDIA `ens6` `192.168.240.1/24`, TT `enp10s0f0np0` `192.168.240.2/24` | k3s node traffic and Flannel, NFS, relay and provisioner → API, backups | +| LAN | NVIDIA `10.0.0.175` (currently over Wi-Fi), TT `10.0.0.112` | Internet access only | +| Tailscale | NVIDIA `100.73.139.66`, TT `100.95.175.37` | Users: portal (NVIDIA :8088), SSH login (TT :22); admin SSH | + +**Moving the direct link to 10 GbE (M1):** + +- **Cable:** Cat6a (or Cat6 under about 50 m) between the NVIDIA box's `ens6` + (Aquantia AQC107) and the TT box's `enp10s0f0np0` (Intel X710). Both are + 10GBASE-T (RJ45) ports. +- **Addresses:** move `192.168.240.1/.2` from the 1 GbE I210 ports to those two + ports, using a `/24` so more nodes can join. +- **MTU:** jumbo frames (9000) on both ends are optional, but help NFS. +- **Check:** `ethtool ` shows `Speed: 10000Mb/s`. An `iperf3` run should + reach above 9 Gb/s. + +**Wired LAN.** The NVIDIA box should also reach the LAN over wired Ethernet +rather than Wi-Fi; one of its I210 ports frees up once the direct link moves. + +**k3s** is installed fresh, with `--node-ip` and `--flannel-iface` on the direct +link, so cluster traffic never uses the LAN or Wi-Fi. + +## Services + +| Service | Host | Runs as | +|---|---|---| +| k3s server and agent | both | systemd | +| NVIDIA GPU Operator (device plugin + container toolkit) | NVIDIA | k3s | +| Tenstorrent DRA driver (M4) | TT | k3s | +| Postgres | NVIDIA | k3s StatefulSet, data on local SSD (never NFS) | +| `mistd api`, `mistd scheduler` | NVIDIA | k3s Deployments (1 replica each), reachable on `192.168.240.1` for agents | +| Portal (Nginx + static build) | NVIDIA | systemd, `:8088` on Tailscale, proxies `/api` | +| NFS server | TT | systemd | +| sshd + `mist-gate` | TT | systemd | +| `mist-relay` | TT | systemd (root) | +| `mistd provision` | TT (later every node) | systemd (root) | +| `mist-noded` (M5) | nodes moved to the node executor | systemd (root) | + +## Configuration and secrets + +- **Configuration in Git:** `deploy/` (manifests, systemd units, sshd and + exports snippets, host setup scripts). +- **Settings in the database:** partitions, QoS, limits, volumes. Admins change + them through the API. +- **Secrets never go in Git:** + - Postgres credentials, the relay token and per-host agent tokens. + - Kubernetes Secrets for in-cluster use; `/etc/mist/*.token` (0600 root) on + hosts. + - Rotate any secret that has been pasted into a chat or document. + +## Admin access + +- **Separate admin accounts** for each admin, with SSH keys only, `sudo`, and + not in `mist-users`. Researcher accounts never have `sudo` or `docker`. +- **Password SSH disabled** on both hosts. The shared `utmist` and `utmist-tt` + passwords are retired once admin accounts exist. +- **Clean up existing host state** after checking with whoever created it: + - leftover local accounts (`dev`, `terrymist`, `labadmin`, `labmember`) + - the `ollama` service and group + - Docker +- **Device files stay `0666`** (D13). This is safe only because researcher + logins can't run code and unmanaged GPU consumers are removed. + +## Starting fresh + +Before M1 goes live: + +1. Stop and remove the previous platform's API, namespaces (`mist-team-*`), + ConfigMaps and Jobs. +2. Keep the old storage folders untouched until a human confirms they can be + deleted. +3. Reinstall k3s on the direct link. + +## Adding a node + +1. **Connect it to the direct-link network.** With three nodes, use the TT box's + second X710 port, or better a 10 GbE switch with all nodes on + `192.168.240.0/24`. Four or more nodes need the switch. +2. **Install Ubuntu** with the drivers for its accelerators. +3. **K3s executor:** join it as a k3s agent and install its device plugin. Node + executor: install `mist-noded` and `mistd provision`. +4. **Register it in Mist:** add it to a partition (new or existing) and set its + reserved host share. +5. **Run the executor conformance suite** on it, then mark it `up`. + +## Work by milestone + +- **M1:** + - 10 GbE link and wired LAN + - fresh k3s + - Postgres + - deploy `mistd` and the portal + - gateway host setup + - admin accounts and secret rotation + - remove the old platform +- **M3:** backup target, and basic monitoring (disk space, node health, failed + provisioner runs). + +## Open questions + +- Monitoring and alerting stack. +- Whether the NVIDIA box's `sdb` is free for scratch and backups. +- A stable login hostname. diff --git a/docs/agents/architecture/executors.md b/docs/agents/architecture/executors.md new file mode 100644 index 0000000..eeb9892 --- /dev/null +++ b/docs/agents/architecture/executors.md @@ -0,0 +1,307 @@ +# Executors + +An executor turns an admitted job into processes on a node, streams their I/O, +and reports what happened. The scheduler decides; the executor carries the +decision out. Overview: [ARCHITECTURE.md](../../../ARCHITECTURE.md). Related +decisions: D1, D2, D13, D16, D17. + +There are two executors: + +- the **K3s executor**, used from M1 +- the **node executor** (M5), which runs jobs as systemd units without + containers + +Both implement the same interface and pass the same conformance tests. Each node +is managed by exactly one executor. + +## Interface + +```go +package executor + +type Executor interface { + Name() string + + // Nodes reports schedulable capacity and health for nodes this executor manages. + Nodes(ctx context.Context) ([]Node, error) + + // Launch starts the job on spec.Node. Idempotent per spec.JobID. + Launch(ctx context.Context, spec LaunchSpec) error + + // Terminate sends SIGTERM, then SIGKILL after grace. Idempotent. + Terminate(ctx context.Context, jobID int64, grace time.Duration, reason string) error + + // Attach connects a terminal to a running interactive job; returns when the session ends. + Attach(ctx context.Context, jobID int64, term Terminal) error + + // Logs streams the job's live output (stdout+stderr as written to the .out file). + Logs(ctx context.Context, jobID int64, follow bool) (io.ReadCloser, error) + + // List returns every execution this executor knows about, for reconciliation. + List(ctx context.Context) ([]Execution, error) + + // Events delivers state changes. Delivery is at-least-once; consumers dedupe by (JobID, Kind). + Events() <-chan Event +} + +type LaunchSpec struct { + JobID int64 + Kind Kind // Batch | Command | Interactive + Node string + User User // Name, UID, PersonalGID + TeamSlug string + TeamGID int + ExtraGIDs []int // mist-users + Resources Resources // CPUs, MemoryBytes, GPUs, TTBoards, ScratchBytes + DeviceIDs []string // optional; set when the scheduler assigns specific devices + Mounts []Mount // resolved team folder + software (volume, source, target, readOnly) + Workdir string + OutputPath string + Script string // Batch + Argv []string // Command + Env map[string]string // full job environment, see runtime-environment.md + Runtime string // partition runtime image / host profile, pinned + TimeLimit time.Duration // executor sets a backstop slightly beyond this + SignalGrace time.Duration +} + +type Event struct { + JobID int64 + Kind EventKind // Started | Exited | LaunchFailed | Lost + Node string + ExitCode *int + Reason ExitReason // Normal | OutOfMemory | ScratchExceeded | Deadline | Signaled | ... + Detail string // human-readable, shown to users + Transient bool // for LaunchFailed: safe to retry once + At time.Time +} + +type Node struct { + Name string + Partitions []string + Capacity Resources + Devices []Device // optional per-device inventory + Healthy bool + Detail string +} +``` + +The scheduler and API depend only on this package. Kubernetes and systemd types +never leak outside their executor packages. + +## Guarantees every executor must provide + +These guarantees are tested by a shared **conformance suite** +(`internal/executor/conformance`) that runs against each implementation on real +nodes. + +1. **Identity.** Processes run as `User.UID`. The primary group is `TeamGID`. + The supplementary groups are exactly `ExtraGIDs` plus `PersonalGID`. There is + no privilege escalation and there are no capabilities. +2. **Filesystem view.** + - **Visible:** only the given mounts, at `/mist/teams/` and + `/mist/software` (read-only), plus `/scratch`. Other teams' folders are + absent, not merely denied. + - **Writable:** only the team folder (as permissions allow), `/scratch` and + `/tmp`. `/tmp` lives inside scratch. +3. **Devices.** Only the allocated GPUs or TT boards can be opened, and they + appear numbered from 0. +4. **Limits.** CPU and memory limits apply, swap is off, and scratch is capped. + Hitting the memory limit gives `OutOfMemory`; going over scratch gives + `ScratchExceeded`. +5. **Time limits.** `Terminate` delivers SIGTERM to the job's processes, then + SIGKILL after the grace period. A backstop kills the job even if the + scheduler is down. +6. **Network.** + - **Allowed:** outbound internet and DNS. + - **Blocked:** the LAN, the direct link, the Tailscale range, cluster + services and the Kubernetes API. + - **Inbound:** none. +7. **Output.** Output goes to `OutputPath` through `mist-job-entry` (below), and + live output is also available from `Logs`. +8. **Cleanup.** When the job ends, all of its processes are gone, scratch is + deleted and devices are free. Executors report `Exited` only once this is + true. +9. **Repeat-safe and recoverable.** Repeating `Launch` or `Terminate` is safe. + After a restart, `List` reports what is actually running. + +## `mist-job-entry` + +A small static Go binary that is the first process of every job, under any +executor. It is the single place that implements the job contract in +[runtime-environment.md](runtime-environment.md): + +1. **Set up:** + - set `umask 0007` + - create, or reset to `0700`, `$HOME/{.cache,.config,.local,.ssh}` + - create the output file and write a header (job ID, node, resources, start + time) +2. **Run the work:** + - **Batch:** `bash