diff --git a/.dockerignore b/.dockerignore
deleted file mode 100644
index c81d217..0000000
--- a/.dockerignore
+++ /dev/null
@@ -1,5 +0,0 @@
-.git
-web-interface/node_modules
-web-interface/dist
-logs
-**/__pycache__
diff --git a/.gitignore b/.gitignore
index 6bd8076..1f994cc 100644
--- a/.gitignore
+++ b/.gitignore
@@ -13,7 +13,6 @@ node_modules/
data/
/logs/
.direnv/
-.fake-backend-config.json
*.local
# Python caches
diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md
new file mode 100644
index 0000000..1904798
--- /dev/null
+++ b/ARCHITECTURE.md
@@ -0,0 +1,411 @@
+# Mist architecture
+
+Mist is UTMIST's research compute platform. It works like the batch systems on
+national HPC clusters. Each researcher gets a folder inside each of their team
+folders and moves files with `scp`/`rsync`. Code runs as **jobs**: batch scripts
+or short interactive shells. A scheduler places jobs on the GPU, CPU and
+Tenstorrent hardware. Each job runs as the researcher's own Linux user, in their
+own folder. Researchers never write Dockerfiles or declare mounts.
+
+This document is the system overview and the coordination point between
+component owners. Each component has a document with the contracts it must
+honour. The reasons behind major choices are in the
+[decision log](docs/agents/architecture/decisions.md).
+
+> **Status:** target design, revised 27 October 2026 after architecture review. This design replaces the
+> previous container-image platform (dataset and result pools, team
+> namespaces), which was never deployed and has been removed from the tree; it
+> remains in Git history before this change. It starts fresh: no users or data
+> are migrated.
+
+## Goals
+
+1. **Development feels like a workstation.** A researcher sets up a Python
+ environment once, in their own folder, and every job uses it. Files a job
+ writes are simply there afterwards.
+2. **Access follows teams.** Teammates can read each other's work. Other teams
+ cannot see it at all. Some projects are under IP agreements, so this boundary
+ is a hard requirement.
+3. **Code runs only inside jobs.** Login sessions move files and manage jobs.
+ Compute and accelerators are reached only through a scheduled job.
+4. **Adding a node is cheap.** Adding a third or fourth machine means configuring
+ that node, not redesigning the system.
+5. **The A4000 GPUs come first.** The NVIDIA partition ships first; Tenstorrent
+ follows on the same design.
+
+## System at a glance
+
+```mermaid
+flowchart LR
+ subgraph Users
+ SSH[ssh / scp / rsync]
+ Browser[Browser]
+ end
+
+ subgraph TT["utmist-tt (QuietBox)"]
+ Gate[Login gateway
sshd + mist-gate]
+ Relay[mist-relay]
+ Prov[Provisioner agent]
+ NFS[("/mist
(durable storage)")]
+ CPUP[partition: cpu]
+ TTP[partition: tt
4× n300]
+ end
+
+ subgraph NV["utmist-z1opa08 (NVIDIA)"]
+ Portal[Web portal]
+ API[mistd API]
+ Sched[mistd scheduler]
+ DB[(Postgres)]
+ K3s[k3s server]
+ GPUP[partition: gpu
2× RTX A4000]
+ end
+
+ Misty[(misty
people and teams)]
+
+ SSH -->|Tailscale| Gate --> Relay --> API
+ Browser -->|Tailscale| Portal --> API
+ API --> DB
+ API --> Misty
+ Sched --> DB
+ Sched -->|Executor| K3s
+ K3s --> GPUP & CPUP & TTP
+ Prov -->|desired state| API
+ Prov --> NFS
+ GPUP -. "NFS over 10 GbE direct link" .-> NFS
+```
+
+## Components
+
+| Component | Responsibility | Document | Proposed owner |
+|---|---|---|---|
+| Identity and accounts | People and teams from misty; Mist accounts, UIDs, GIDs, SSH keys; sign-up; provisioning Linux accounts and folders | [identity-and-accounts.md](docs/agents/architecture/identity-and-accounts.md) | Lucas |
+| Storage | `/mist` layout and permissions, NFS, volumes, usage reporting, quotas, scratch, backups | [storage.md](docs/agents/architecture/storage.md) | Ansh |
+| Login gateway | Restricted SSH (file transfer and `mist` commands only), identity relay, the `mist` CLI | [login-gateway.md](docs/agents/architecture/login-gateway.md) | TBD |
+| API | The one HTTP API used by the portal, CLI and agents; authentication and authorization | [api.md](docs/agents/architecture/api.md) | Shared; each owner defines their endpoints |
+| Scheduler | Partitions, QoS, limits, queue, priority, fairshare, placement, accounting | [scheduler.md](docs/agents/architecture/scheduler.md) | Brandon |
+| Executors | Run an admitted job on a node; stream I/O; report status. K3s executor first, native node executor later | [executors.md](docs/agents/architecture/executors.md) | Brandon |
+| Runtime environment | What a job sees: images, software, environment variables, caches, outputs, signals | [runtime-environment.md](docs/agents/architecture/runtime-environment.md) | Scott (to confirm) |
+| Web portal | Lightweight UI over the API: account, SSH keys, teams, jobs, queue, usage, admin | [portal.md](docs/agents/architecture/portal.md) | Andrei |
+| Deployment and network | Hosts, network paths, services, secrets, admin access, adding nodes | [deployment-and-network.md](docs/agents/architecture/deployment-and-network.md) | Ethan, with Ansh |
+
+Owners should confirm or reassign rows here. Before implementing a component,
+read [implementation-contracts.md](docs/agents/architecture/implementation-contracts.md).
+It defines cross-component invariants, required acceptance evidence, and decisions
+that must be resolved before a milestone can ship. Component documents are the
+normative contracts; the implementation plan orders the work. D18–D23 supersede
+specific older decisions as listed in the decision log.
+
+## How the parts fit together
+
+Each box is one owner's area; every arrow between boxes is a contract listed in
+the table below. Deployment and network (hosts, the 10 GbE link, Tailscale,
+secrets) sits under all of it; see its own diagram in
+[deployment-and-network.md](docs/agents/architecture/deployment-and-network.md).
+
+```mermaid
+flowchart TB
+ R([Researcher])
+ Misty[(misty)]
+
+ subgraph GW["Login gateway · owner TBD"]
+ sshd[sshd]
+ gate[mist-gate]
+ cli[mist CLI]
+ relay[mist-relay]
+ end
+
+ subgraph PO["Portal · Andrei"]
+ ui[Web UI]
+ end
+
+ subgraph API["mistd api · shared"]
+ mw[Auth middleware + routing]
+ end
+
+ subgraph ID["Identity · Lucas"]
+ dir[misty client / stub]
+ acct[(accounts, teams, SSH keys)]
+ prov[Provisioner agent]
+ end
+
+ subgraph SC["Scheduler · Brandon"]
+ sched[Scheduling cycle]
+ ledger[(jobs, allocations, usage)]
+ end
+
+ subgraph EX["Executors + k3s · Brandon"]
+ k3sx[K3s executor]
+ entry[mist-job-entry]
+ end
+
+ subgraph RT["Runtime · Scott"]
+ img[Runtime images]
+ sw["/mist/software"]
+ end
+
+ subgraph ST["Storage · Ansh"]
+ vol[("/mist volumes")]
+ scratch[(Node scratch)]
+ end
+
+ R -->|"ssh, scp, rsync"| sshd --> gate --> cli
+ gate -->|"file transfer as the user"| vol
+ cli -->|unix socket| relay
+ sshd -.->|AuthorizedKeysCommand| relay
+ relay -->|"HTTP /v1 + asserted user"| mw
+ R -->|browser| ui -->|"HTTP /v1 + session"| mw
+ mw -->|identity.Authorize| acct
+ mw -->|job endpoints| sched
+ dir --> Misty
+ dir --> acct
+ prov -->|"GET provision/state"| mw
+ prov -->|"users, groups, folders"| vol
+ sched --> ledger
+ sched -->|Executor interface| k3sx
+ acct -.->|per-job mist-nss projection| k3sx
+ k3sx -->|"pod: uid, team gid, devices"| entry
+ k3sx -->|"mounts team folder + software"| vol
+ k3sx --> scratch
+ img -.->|ships| entry
+ img -.-> sw
+```
+
+| Boundary | Provider → consumer | Contract | Defined in |
+|---|---|---|---|
+| Who is registered, which teams | misty → Identity | `Directory` interface (stub until the real API) | [identity](docs/agents/architecture/identity-and-accounts.md#misty) |
+| Who may do what | Identity → every API handler | `identity.Authorize(principal, action, resource)` | [identity](docs/agents/architecture/identity-and-accounts.md#roles) |
+| Linux accounts and folders | Identity → Provisioner → Storage | `/internal/v1/provision/state`, layout rules | [identity](docs/agents/architecture/identity-and-accounts.md#provisioner), [storage](docs/agents/architecture/storage.md#layout) |
+| CLI calls as the researcher | Gateway → API | Relay socket, `X-Mist-Asserted-User` + relay token | [login gateway](docs/agents/architecture/login-gateway.md#identity-relay) |
+| Everything clients see | API → Portal, CLI | `/v1` endpoints and wire types | [api](docs/agents/architecture/api.md) |
+| Running a job | Scheduler → Executors | `executor.Executor`, `LaunchSpec`, events | [executors](docs/agents/architecture/executors.md#interface) |
+| What a job sees | Executors + Runtime → researcher code | Job environment contract, `mist-job-entry` | [runtime](docs/agents/architecture/runtime-environment.md) |
+| Where files live | Storage → Executors, Gateway | Volumes, mount paths, permissions | [storage](docs/agents/architecture/storage.md#volumes-and-growth) |
+
+## Core concepts
+
+- **Person.** A human registered in misty. Misty is the source of truth for
+ people, teams and membership.
+- **Account.** A person's Mist identity: one immutable `username`, one stable
+ Linux UID, and SSH keys. It is created when a registered person signs up.
+- **Team.** A misty team. On Mist, each team has a Linux group (`team-`) and
+ a folder `/mist/teams//`.
+- **Partition.** A named pool of nodes that share one kind of hardware: `gpu`
+ (NVIDIA), `cpu` (CPU-only, used for data prep and downloads) and `tt`
+ (Tenstorrent). A job targets exactly one partition.
+- **QoS.** A class of service, `normal` or `interactive`, carrying limits and a
+ priority weight.
+- **Job.** A request to run a script (batch) or a shell (interactive). It names
+ one partition, one QoS, one **charged team**, its resources and a time limit.
+ It runs as the submitter's UID. Accelerators are allocated whole and one at a
+ time: two 1-GPU jobs run side by side on the A4000 box.
+- **Charged team.** The team a job is billed to and scoped to. The job sees only
+ that team's folder and the shared software. It cannot see the submitter's other
+ teams.
+- **Volume.** One durable storage location, an NFS export on one server. Each
+ team folder lives entirely on one volume. There is one volume today.
+
+## The researcher's journey
+
+One researcher, Alice on team `vision`, goes from sign-up to results. The boxes
+mark which owner's component handles each step.
+
+```mermaid
+sequenceDiagram
+ autonumber
+ actor R as Alice
+ box Portal · Andrei
+ participant W as Portal
+ end
+ box Login gateway · TBD
+ participant G as Gateway + relay
+ end
+ box API · shared, Identity · Lucas
+ participant A as mistd api
+ participant P as Provisioner
+ end
+ box Scheduler + executors · Brandon
+ participant S as Scheduler
+ participant E as K3s executor
+ end
+ box Runtime · Scott, Storage · Ansh
+ participant J as Job
+ participant F as /mist
+ end
+
+ Note over R,F: Join
+ R->>W: Sign in
+ W->>A: Is Alice registered in misty?
+ A-->>W: Yes, teams [vision]. Account alice, uid 20007
+ R->>W: Add SSH public key
+ P->>A: Poll desired state
+ P->>F: Create user, groups, /mist/teams/vision/alice
+
+ Note over R,F: Bring code and data
+ R->>G: ssh, then rsync ./project
+ G->>F: Write to /mist/teams/vision/alice/project
+ R->>G: mist run -p cpu -- hf download some-model
+ G->>A: Submit command job (cpu)
+ A->>S: Job queued
+ S->>E: Launch on utmist-tt
+ E->>J: Start as alice, team vision
+ J->>F: Model saved to team .cache
+
+ Note over R,F: Set up an environment
+ R->>G: mist shell -p gpu --gpus 1
+ G->>A: Submit interactive job
+ S->>E: Launch on utmist-z1opa08 (1 GPU)
+ E->>J: Start shell holder
+ G->>A: Attach terminal (WebSocket)
+ A->>E: Attach
+ R->>J: uv sync (venv lands in her folder)
+ R->>J: exit, job completes
+
+ Note over R,F: Train
+ R->>G: mist submit train.sh
+ G->>A: Submit batch job 412
+ A-->>R: 412 PENDING (RESOURCES)
+ S->>E: GPU frees up, launch 412
+ E->>J: Start train.sh
+ J->>F: Stream mist-412.out, write checkpoints
+ R->>G: mist logs -f 412
+ J-->>E: Exit 0
+ E-->>S: Finished (CLEANING; allocation held)
+ E->>E: Persist result, delete execution, confirm node cleanup
+ E-->>S: Exited (cleanup receipt)
+ S->>A: 412 COMPLETED
+
+ Note over R,F: Take results home
+ R->>G: rsync checkpoints back to laptop
+ G->>F: Read as alice
+```
+
+## Key flows
+
+### Sign-up and first login
+
+1. A person signs in to the portal. Identity confirms with misty that they are
+ registered and loads their teams.
+2. Mist creates their account: it reserves a username and a UID and records team
+ memberships.
+3. The person adds an SSH public key in the portal.
+4. The provisioner agent on the storage/login host reconciles the change. It
+ creates the Linux user and groups, and `/mist/teams///` for each
+ team.
+5. The person runs `ssh @`. They land in a folder with one
+ entry per team.
+
+### Moving files
+
+`scp`, `rsync` and `sftp` go through the login gateway, which sits on the storage
+host and writes straight to disk. Unix permissions decide what each person can
+read and write.
+
+Downloads from the internet (Hugging Face, S3, Kaggle and so on) run as jobs on
+the `cpu` partition, using `mist run -p cpu -- ` or `mist shell -p cpu`.
+
+### Batch job
+
+```mermaid
+sequenceDiagram
+ participant U as Researcher (login)
+ participant G as mist-gate + relay
+ participant A as mistd API
+ participant S as Scheduler
+ participant E as Executor (K3s)
+ participant N as Node
+ U->>G: mist submit train.sh
+ G->>A: POST /v1/jobs (relay-asserted user)
+ A->>A: validate spec, membership, limits
+ A-->>U: job 412 PENDING
+ S->>S: order by priority, check limits and free capacity
+ S->>E: Launch(job 412, node, resources)
+ E->>N: start as uid, team gid, team folder, devices
+ N-->>E: running … exited(0)
+ E-->>S: Finished, then Exited after confirmed cleanup
+ S->>A: job 412 COMPLETED; allocation released
+ U->>G: tail -f mist-412.out
+```
+
+The job starts in the folder it was submitted from. Its output streams to
+`mist-.out` there. Checkpoints and results go wherever the code writes them,
+in `/mist` by default.
+
+### Interactive job
+
+`mist shell --gpus 1` submits a job with QoS `interactive`. When the job starts,
+the API attaches the researcher's terminal to a shell inside it. The job ends
+when the shell exits. If the connection drops, the job waits for a grace period
+so the researcher can run `mist attach`, then it is cancelled. A browser terminal
+will later reuse the same attach endpoint.
+
+## Data ownership
+
+Postgres on the control node is the single source of truth for Mist state.
+Kubernetes objects, Linux accounts and folders are *derived* state: agents
+reconcile them from the database.
+
+| Data | Authority | Owner |
+|---|---|---|
+| People, teams, membership | misty (cached in Mist) | Identity |
+| Accounts, UIDs, GIDs, SSH keys | Mist DB | Identity |
+| Linux users, groups, folders | Derived by the provisioner | Identity, Storage |
+| Volumes, team placement, usage | Mist DB | Storage |
+| Partitions, QoS, limits | Mist DB (admin-configured) | Scheduler |
+| Jobs, job events, usage records | Mist DB | Scheduler |
+| Pods and other execution objects | Derived by the executor; cleanup acknowledged before reuse | Executors |
+| Files | `/mist` | Storage |
+
+Each owner owns their tables and migrations. Other components go through that
+owner's Go package or API, not their tables. Packages participating in admission
+and revocation accept the same database transaction so authorization and job
+changes serialize; an HTTP authorization check followed by an unrelated commit
+is not sufficient. Durable job intents and executor observations survive restarts.
+User files are not derived state and must never be recreated or deleted by job
+reconciliation.
+
+## Code layout
+
+```text
+cmd/mistd/ api, scheduler, provision, cleanup-report subcommands
+cmd/mist/ CLI (on the login gateway; laptops later)
+cmd/mist-gate/ SSH ForceCommand dispatcher
+cmd/mist-relay/ login-node identity relay and SSH key lookup
+cmd/mist-job-entry/ in-job wrapper: output file, signals, exit reasons
+internal/api/ HTTP server, middleware, shared wire types
+internal/identity/ accounts, misty client and stub, provisioning state
+internal/storage/ layout rules, volumes, usage
+internal/scheduler/ queue, policy, placement, accounting
+internal/executor/ Executor interface; k3s/ and node/ implementations
+internal/db/ connection and migrations (one directory per owner)
+images/ job runtime images
+deploy/ k3s manifests, systemd units, host setup
+web-interface/ portal
+```
+
+The previous platform's `src/`, `cli/`, `deploy/` and `web-interface/` were
+removed on 27 October 2026; the portal is rebuilt in `web-interface/` against
+the new API.
+
+## Milestones
+
+| Milestone | Outcome |
+|---|---|
+| **M1: GPU batch** | Sign-up (misty stubbed), SSH login, file transfer, `mist submit/run/jobs/show/logs/cancel` on `gpu` and `cpu`, verified permissions/revocation/network isolation, crash recovery and cleanup accounting, 10 GbE direct link |
+| **M2: Interactive** | `mist shell` and `mist attach` with reattach grace; `interactive` QoS |
+| **M3: Fair scheduling** | Priority, fairshare by team, backfill, accounting, admin limits, queue views; backups in place |
+| **M4: Tenstorrent** | `tt` partition, TT runtime image, board reset between jobs |
+| **M5: Native executor and web terminal** | Node executor without containers, behind the same interface; browser terminal; port forwarding into jobs |
+
+Each component document lists its work per milestone.
+
+## Open questions
+
+| Question | Owner |
+|---|---|
+| Real misty API: shape, and how membership changes reach Mist (polling or webhooks) | Identity |
+| How people prove who they are at sign-in (Google or UofT SSO, or email magic link) | Identity |
+| Enforced storage quotas: needs `/mist` on a dedicated filesystem with project quotas | Storage |
+| Retention after access revocation: archive or transfer departed members’ folders (access revocation itself is specified) | Identity, Storage |
+| Backup tool, schedule and retention | Storage |
+| Default limits per partition and per user once real usage is known | Scheduler |
+| Monitoring and alerting beyond the portal | Deployment |
diff --git a/README.md b/README.md
index aec78c8..63a9f22 100644
--- a/README.md
+++ b/README.md
@@ -1,96 +1,53 @@
# Mist
-UTMIST's research compute platform. Submit Python scripts or tagged container
-images, choose CPU, NVIDIA GPUs or Tenstorrent boards, upload datasets, and
-collect persistent results through the same website or CLI.
+UTMIST's research compute platform. Mist works like the batch systems on
+national HPC clusters: each researcher gets a folder inside their team folders,
+logs in over SSH, moves files with `scp`/`rsync`, and runs batch or interactive
+jobs that a scheduler places on NVIDIA GPU, CPU and Tenstorrent hardware.
-**Private website:** [http://100.73.139.66:8088](http://100.73.139.66:8088), accessible through authorized
-Tailscale connectivity. A Mist account and team membership are also required.
+## Status
-## Storage update — October 21
+Under redesign (October 2026). The previous container-image platform was never
+deployed and has been removed from the tree; see Git history before 27 October
+2026 if you need it. The new implementation starts from the design docs.
-New submissions use ordinary team dataset/result folders on QuietBox, accessed
-through the direct Ethernet link. Storage allowances are advisory; team access
-is enforced. See [current storage operations](deploy/k3s/storage/README.md) for
-layout, setup, API semantics, verification and rollback. The earlier operating
-snapshots describe the retired image-backed layout.
+## Design
-## Current installation
+- [ARCHITECTURE.md](ARCHITECTURE.md): system overview, components, owners,
+ milestones and code layout.
+- [docs/agents/architecture/](docs/agents/architecture/decisions.md): per-component
+ contracts and the decision log.
+- [docs/agents/plans/](docs/agents/plans/jobs-and-scheduling.md): implementation
+ plans per owner.
+## Cluster
| Machine | Role | LAN address | Tailscale address |
| ---------------- | --------------------------------------------------------- | ------------ | ----------------- |
-| `utmist-z1opa08` | k3s server, API/auth, website, CPU and two RTX A4000 GPUs | `10.0.0.175` | `100.73.139.66` |
-| `utmist-tt` | k3s worker, four n300 boards/eight chips, shared storage | `10.0.0.112` | `100.95.175.37` |
+| `utmist-z1opa08` | Two RTX A4000 GPUs, CPU | `10.0.0.175` | `100.73.139.66` |
+| `utmist-tt` | Four n300 Tenstorrent boards (eight chips), shared storage | `10.0.0.112` | `100.95.175.37` |
+## Development
-The cluster and NFS use the existing LAN. Tailscale provides private portal and
-administrator SSH access. Kubernetes/containerd run workload containers.
-
-## Documentation
-
-Start with the [documentation index](docs/README.md).
-
-- [Using the portal, teams and jobs](docs/content/department-rollout.md)
-- [k3s setup, accelerator allocation and diagnostics](docs/content/k3s-setup.md)
-- [Tailscale and SSH setup](docs/content/tailscale-ssh.md)
-- [Architecture and code organization](docs/content/architecture.md)
-- [Storage and upload limits](docs/content/storage.md)
-- [Deployment and operations](deploy/private/README.md)
-- [Developer setup](docs/content/development.md), [CLI](cli/docs/setup.md), [testing](docs/content/testing.md)
-
-Main contains the department release merged through
-[PR #110](https://github.com/UTMIST/Mist/pull/110). The earlier main and operating
-snapshots are preserved in the [historical archive](docs/content/archive/_index.md).
-
-## Development and checks
-
-Use Go **1.25.1** and Node.js **22.16.0 or newer**. The Go workspace contains the
-API (`src`) and CLI (`cli`). From the repository root:
+A Nix dev shell (`flake.nix`, `shell.nix`) provides Go, Node.js, Python, k3s and
+kubectl. Run the Go tests with:
```bash
-go test ./src/... ./cli/...
-npm --prefix web-interface ci
-npm --prefix web-interface test
-npm --prefix web-interface run lint
-npm --prefix web-interface run build
+go test ./...
```
-For a local UI against the authenticated deployed API, follow
-[developer setup](docs/content/development.md). For the existing installation, deploy with
-`bash deploy/private/build-and-deploy.sh`; read its [prerequisites](deploy/private/README.md)
-before running it. Fresh-host provisioning is covered separately by the k3s guide.
+Database tests start an embedded Postgres as the current user (downloaded to
+`~/.embedded-postgres-go` on first run). To use an existing server instead, set
+`MIST_TEST_DATABASE_URL` to a superuser URL; each test creates its own database.
-The current API uses Kubernetes only. The unfinished Redis/Docker supervisor,
-fake frontend account/chart and dummy CLI configuration have been retired.
-Docker remains build tooling for images; it does not schedule Mist jobs.
+Check documentation links with:
-Custom accelerator images need compatible training libraries. Credits, Jupyter,
-chaining, browser image archives/builds, private pull-credential management and
-distributed multi-machine training are outside this release. Automatic backups,
-HTTPS and monitoring/cleanup products were explicitly excluded. This is a private
-HTTP installation with one control plane, one auth instance and one storage host.
+```bash
+python3 scripts/check-docs.py
+```
### Documentation
Humans are responsible for editing the human facing docs. This is because the human facing docs will be read by humans, and it is rude to give them untested instructions to follow.
Documentation should include explanations of design decisions for later platform maintainers.
-
-## Frontend Testing
-
-how to use fake backend for testing frontend.
-
-(temporary, functionality, to be removed in future commit.)
-
-```bash
-# Terminal 1
-npm --prefix web-interface run dev -- --host 127.0.0.1 --port 3001 --strictPort
-
-# Terminal 2
-npm --prefix web-interface run fake-backend
-```
-
-
-
-
\ No newline at end of file
diff --git a/cli/cmd/api_client.go b/cli/cmd/api_client.go
deleted file mode 100644
index 555490f..0000000
--- a/cli/cmd/api_client.go
+++ /dev/null
@@ -1,83 +0,0 @@
-package cmd
-
-import (
- "bytes"
- "encoding/json"
- "fmt"
- "io"
- "net/http"
- "strings"
- "time"
-)
-
-func (ctx *AppContext) api(method, path string, body, out interface{}) error {
- if ctx == nil {
- return fmt.Errorf("missing application context")
- }
- base := ctx.baseURL()
- var data []byte
- var err error
- if body != nil {
- data, err = json.Marshal(body)
- if err != nil {
- return err
- }
- }
- req, err := http.NewRequest(method, strings.TrimRight(base, "/")+path, bytes.NewReader(data))
- if err != nil {
- return err
- }
- req.Header.Set("Content-Type", "application/json")
- if team := ctx.teamID(); team != "" {
- req.Header.Set("X-Mist-Team", team)
- }
- if ctx.Config != nil && ctx.Config.SessionCookie != "" {
- req.Header.Set("Cookie", ctx.Config.SessionCookie)
- }
- client := ctx.HTTPClient
- if client == nil {
- client = &http.Client{Timeout: 30 * time.Second}
- }
- response, err := client.Do(req)
- if err != nil {
- return err
- }
- defer response.Body.Close()
- if response.StatusCode < 200 || response.StatusCode >= 300 {
- data, _ := io.ReadAll(io.LimitReader(response.Body, 8192))
- var errorBody struct {
- Error string `json:"error"`
- }
- if json.Unmarshal(data, &errorBody) == nil && errorBody.Error != "" {
- return fmt.Errorf("API %d: %s", response.StatusCode, errorBody.Error)
- }
- return fmt.Errorf("API %d: %s", response.StatusCode, strings.TrimSpace(string(data)))
- }
- if out != nil {
- return json.NewDecoder(response.Body).Decode(out)
- }
- return nil
-}
-
-func (ctx *AppContext) baseURL() string {
- if ctx.APIBaseURL != "" {
- return strings.TrimRight(ctx.APIBaseURL, "/")
- }
- if ctx.Config != nil && ctx.Config.APIBaseURL != "" {
- return strings.TrimRight(ctx.Config.APIBaseURL, "/")
- }
- return "http://127.0.0.1:3000"
-}
-
-func (ctx *AppContext) teamID() string {
- if ctx.TeamID == "legacy" {
- return ""
- }
- if ctx.TeamID != "" {
- return ctx.TeamID
- }
- if ctx.Config != nil {
- return ctx.Config.TeamID
- }
- return ""
-}
diff --git a/cli/cmd/auth.go b/cli/cmd/auth.go
deleted file mode 100644
index 9da9f38..0000000
--- a/cli/cmd/auth.go
+++ /dev/null
@@ -1,13 +0,0 @@
-package cmd
-
-type AuthCmd struct {
- Login LoginCmd `cmd:"" help:"Log in to your account"`
- Logout LogoutCmd `cmd:"" help:"Sign out and revoke your session"`
- // Status AuthStatusCmd `cmd:"" help:"Check your authentication status" default:1`
-}
-
-func (a *AuthCmd) Run() error {
- // Possible fallback if no subcommand is provided
- // fmt.Println("(auth root) – try 'mist auth login|logout|status' or mist help")
- return nil
-}
diff --git a/cli/cmd/auth_login.go b/cli/cmd/auth_login.go
deleted file mode 100644
index 8fff8d0..0000000
--- a/cli/cmd/auth_login.go
+++ /dev/null
@@ -1,148 +0,0 @@
-package cmd
-
-import (
- "bytes"
- "encoding/json"
- "fmt"
- "golang.org/x/term"
- "io"
- "net/http"
- "net/url"
- "os"
- "path/filepath"
- "strings"
-)
-
-type LoginCmd struct {
- Email string `help:"Mist member email" required:""`
- PasswordStdin bool `help:"Read password from standard input (for automation)"`
-}
-
-func (l *LoginCmd) Run(ctx *AppContext) error {
- var password string
- if l.PasswordStdin {
- data, err := io.ReadAll(io.LimitReader(os.Stdin, 130))
- if err != nil {
- return err
- }
- password = strings.TrimSpace(string(data))
- } else {
- fmt.Print("Password: ")
- data, err := term.ReadPassword(int(os.Stdin.Fd()))
- fmt.Println()
- if err != nil {
- return fmt.Errorf("password input: %w (use --password-stdin for automation)", err)
- }
- password = string(data)
- }
- base := ctx.baseURL()
- u, err := url.Parse(base)
- if err != nil {
- return err
- }
- u.Path = "/auth/sign-in/email"
- u.RawQuery = ""
- body, _ := json.Marshal(map[string]string{"email": l.Email, "password": password})
- req, err := http.NewRequest(http.MethodPost, u.String(), bytes.NewReader(body))
- if err != nil {
- return err
- }
- req.Header.Set("Content-Type", "application/json")
- req.Header.Set("Origin", u.Scheme+"://"+u.Host)
- response, err := ctx.HTTPClient.Do(req)
- if err != nil {
- return err
- }
- defer response.Body.Close()
- if response.StatusCode != 200 {
- return fmt.Errorf("login failed (HTTP %d)", response.StatusCode)
- }
- cookies := []string{}
- for _, cookie := range response.Cookies() {
- if strings.Contains(cookie.Name, "session_token") && cookie.Value != "" {
- cookies = append(cookies, cookie.Name+"="+cookie.Value)
- }
- }
- if len(cookies) == 0 {
- return fmt.Errorf("login returned no session cookie")
- }
- if ctx.Config == nil {
- ctx.Config = &Config{}
- }
- ctx.Config.SessionCookie = strings.Join(cookies, "; ")
- ctx.Config.AccessToken = ""
- ctx.Config.APIBaseURL = base
- if err := ctx.saveConfig(); err != nil {
- return err
- }
- fmt.Println("Signed in. Session saved in protected CLI configuration.")
- return nil
-}
-func (ctx *AppContext) saveConfig() error {
- path := ctx.ConfigPath
- if path == "" {
- path = defaultConfigPath()
- }
- if err := os.MkdirAll(filepath.Dir(path), 0700); err != nil {
- return err
- }
- data, err := json.MarshalIndent(ctx.Config, "", " ")
- if err != nil {
- return err
- }
- file, err := os.CreateTemp(filepath.Dir(path), ".mist-config-")
- if err != nil {
- return err
- }
- defer os.Remove(file.Name())
- if err = file.Chmod(0600); err != nil {
- file.Close()
- return err
- }
- if _, err = file.Write(data); err != nil {
- file.Close()
- return err
- }
- if err = file.Close(); err != nil {
- return err
- }
- return os.Rename(file.Name(), path)
-}
-
-type LogoutCmd struct{}
-
-func (l *LogoutCmd) Run(ctx *AppContext) error {
- if ctx.Config == nil || ctx.Config.SessionCookie == "" {
- fmt.Println("Not signed in.")
- return nil
- }
- base, err := url.Parse(ctx.baseURL())
- if err != nil {
- return err
- }
- origin := base.Scheme + "://" + base.Host
- base.Path = "/auth/sign-out"
- base.RawQuery = ""
- req, err := http.NewRequest(http.MethodPost, base.String(), strings.NewReader("{}"))
- if err != nil {
- return err
- }
- req.Header.Set("Content-Type", "application/json")
- req.Header.Set("Origin", origin)
- req.Header.Set("Cookie", ctx.Config.SessionCookie)
- response, err := ctx.HTTPClient.Do(req)
- if err != nil {
- return err
- }
- response.Body.Close()
- if response.StatusCode != 200 {
- return fmt.Errorf("sign out failed (HTTP %d)", response.StatusCode)
- }
- ctx.Config.SessionCookie = ""
- ctx.Config.AccessToken = ""
- if err := ctx.saveConfig(); err != nil {
- return err
- }
- fmt.Println("Signed out.")
- return nil
-}
diff --git a/cli/cmd/config.go b/cli/cmd/config.go
deleted file mode 100644
index 3ece81d..0000000
--- a/cli/cmd/config.go
+++ /dev/null
@@ -1,15 +0,0 @@
-package cmd
-
-import (
- "os"
- "path/filepath"
-)
-
-func defaultConfigPath() string {
- dir, err := os.UserConfigDir()
- if err != nil {
- home, _ := os.UserHomeDir()
- return filepath.Join(home, ".config", "mist", "config.json")
- }
- return filepath.Join(dir, "mist", "config.json")
-}
diff --git a/cli/cmd/help.go b/cli/cmd/help.go
deleted file mode 100644
index cf571e2..0000000
--- a/cli/cmd/help.go
+++ /dev/null
@@ -1,16 +0,0 @@
-package cmd
-
-import "fmt"
-
-type HelpCmd struct{}
-
-func (h *HelpCmd) Run() error {
- fmt.Println("MIST CLI Help")
- fmt.Println("Usage: mist [command] [options]")
- fmt.Println("Commands:")
- fmt.Println(" auth Authentication commands")
- fmt.Println(" job Job management commands")
- fmt.Println(" config Configuration commands")
- fmt.Println(" help Show help information")
- return nil
-}
diff --git a/cli/cmd/job.go b/cli/cmd/job.go
deleted file mode 100644
index e5d7faf..0000000
--- a/cli/cmd/job.go
+++ /dev/null
@@ -1,9 +0,0 @@
-package cmd
-
-type JobCmd struct {
- Submit JobSubmitCmd `cmd:"" help:"Submit a new job"`
- Cancel JobCancelCmd `cmd:"" help:"Cancel an existing job"`
- Status JobStatusCmd `cmd:"" help:"Check the status of a job"`
- Logs JobLogsCmd `cmd:"" help:"Read workload logs"`
- List ListCmd `cmd:"" help:"List jobs"`
-}
diff --git a/cli/cmd/job_cancel.go b/cli/cmd/job_cancel.go
deleted file mode 100644
index 0cd7072..0000000
--- a/cli/cmd/job_cancel.go
+++ /dev/null
@@ -1,19 +0,0 @@
-package cmd
-
-import (
- "fmt"
- "net/url"
-)
-
-type JobCancelCmd struct {
- ID string `arg:"" help:"Job ID to cancel"`
-}
-
-func (j *JobCancelCmd) Run(ctx *AppContext) error {
- var job Job
- if err := ctx.api("POST", "/jobs/"+url.PathEscape(j.ID)+"/cancel", nil, &job); err != nil {
- return err
- }
- fmt.Printf("Job %s: %s\n", job.ID, job.Status)
- return nil
-}
diff --git a/cli/cmd/job_cancel_test.go b/cli/cmd/job_cancel_test.go
deleted file mode 100644
index 79da1e2..0000000
--- a/cli/cmd/job_cancel_test.go
+++ /dev/null
@@ -1,25 +0,0 @@
-package cmd
-
-import (
- "net/http"
- "net/http/httptest"
- "testing"
-)
-
-func TestJobCancelCallsAPIWithoutConfirmation(t *testing.T) {
- server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
- if r.Method != "POST" || r.URL.Path != "/jobs/cancel-id/cancel" {
- t.Errorf("wrong request: %s %s", r.Method, r.URL.Path)
- }
- _, _ = w.Write([]byte(`{"id":"cancel-id","job_state":"Cancelled"}`))
- }))
- defer server.Close()
- out := CaptureOutput(func() {
- if err := (&JobCancelCmd{ID: "cancel-id"}).Run(&AppContext{APIBaseURL: server.URL}); err != nil {
- t.Error(err)
- }
- })
- if !contains(out, "Cancelled") || contains(out, "Are you sure") {
- t.Fatal(out)
- }
-}
diff --git a/cli/cmd/job_list.go b/cli/cmd/job_list.go
deleted file mode 100644
index 0dfec2d..0000000
--- a/cli/cmd/job_list.go
+++ /dev/null
@@ -1,49 +0,0 @@
-package cmd
-
-import (
- "fmt"
- "os"
- "text/tabwriter"
- "time"
-)
-
-type ListCmd struct {
- All bool `help:"Include completed, failed, and cancelled jobs" short:"a"`
-}
-type Job struct {
- ID string `json:"id"`
- Name string `json:"name"`
- Status string `json:"job_state"`
- GPUType string `json:"accelerator"`
- CreatedAt time.Time `json:"created"`
- Node string `json:"node"`
- DeviceCount int `json:"device_count"`
- ExitCode *int `json:"exit_code"`
- Message string `json:"message"`
-}
-
-func printJobs(jobs []Job) {
- w := tabwriter.NewWriter(os.Stdout, 0, 0, 2, ' ', 0)
- fmt.Fprintln(w, "Job ID\tName\tStatus\tCompute\tDevices\tNode\tCreated")
- for _, job := range jobs {
- fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%d\t%s\t%s\n", job.ID, job.Name, job.Status, job.GPUType, job.DeviceCount, job.Node, job.CreatedAt.Format(time.RFC3339))
- }
- _ = w.Flush()
-}
-
-func (l *ListCmd) Run(ctx *AppContext) error {
- var response struct {
- Jobs []Job `json:"jobs"`
- }
- if err := ctx.api("GET", "/jobs", nil, &response); err != nil {
- return err
- }
- jobs := []Job{}
- for _, job := range response.Jobs {
- if l.All || job.Status == "Scheduled" || job.Status == "InProgress" {
- jobs = append(jobs, job)
- }
- }
- printJobs(jobs)
- return nil
-}
diff --git a/cli/cmd/job_list_test.go b/cli/cmd/job_list_test.go
deleted file mode 100644
index 070ce98..0000000
--- a/cli/cmd/job_list_test.go
+++ /dev/null
@@ -1,31 +0,0 @@
-package cmd
-
-import (
- "net/http"
- "net/http/httptest"
- "testing"
-)
-
-func TestJobListFiltersCompleted(t *testing.T) {
- server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
- _, _ = w.Write([]byte(`{"jobs":[{"id":"running-id","name":"work","job_state":"InProgress","accelerator":"nvidia"},{"id":"done-id","name":"old","job_state":"Success","accelerator":"cpu"}]}`))
- }))
- defer server.Close()
- ctx := &AppContext{APIBaseURL: server.URL}
- out := CaptureOutput(func() {
- if err := (&ListCmd{}).Run(ctx); err != nil {
- t.Error(err)
- }
- })
- if !contains(out, "running-id") || contains(out, "done-id") {
- t.Fatal(out)
- }
- out = CaptureOutput(func() {
- if err := (&ListCmd{All: true}).Run(ctx); err != nil {
- t.Error(err)
- }
- })
- if !contains(out, "done-id") {
- t.Fatal(out)
- }
-}
diff --git a/cli/cmd/job_logs.go b/cli/cmd/job_logs.go
deleted file mode 100644
index b183510..0000000
--- a/cli/cmd/job_logs.go
+++ /dev/null
@@ -1,21 +0,0 @@
-package cmd
-
-import (
- "fmt"
- "net/url"
-)
-
-type JobLogsCmd struct {
- ID string `arg:"" help:"Job ID"`
-}
-
-func (j *JobLogsCmd) Run(ctx *AppContext) error {
- var response struct {
- Logs string `json:"logs"`
- }
- if err := ctx.api("GET", "/jobs/"+url.PathEscape(j.ID)+"/logs", nil, &response); err != nil {
- return err
- }
- fmt.Print(response.Logs)
- return nil
-}
diff --git a/cli/cmd/job_status.go b/cli/cmd/job_status.go
deleted file mode 100644
index 9d15a01..0000000
--- a/cli/cmd/job_status.go
+++ /dev/null
@@ -1,25 +0,0 @@
-package cmd
-
-import (
- "fmt"
- "net/url"
-)
-
-type JobStatusCmd struct {
- ID string `arg:"" help:"Job ID"`
-}
-
-func (j *JobStatusCmd) Run(ctx *AppContext) error {
- var job Job
- if err := ctx.api("GET", "/jobs/"+url.PathEscape(j.ID), nil, &job); err != nil {
- return err
- }
- printJobs([]Job{job})
- if job.ExitCode != nil {
- fmt.Println("Exit code:", *job.ExitCode)
- }
- if job.Message != "" {
- fmt.Println(job.Message)
- }
- return nil
-}
diff --git a/cli/cmd/job_status_test.go b/cli/cmd/job_status_test.go
deleted file mode 100644
index 09e1f1d..0000000
--- a/cli/cmd/job_status_test.go
+++ /dev/null
@@ -1,32 +0,0 @@
-package cmd
-
-import (
- "net/http"
- "net/http/httptest"
- "testing"
-)
-
-func TestJobStatusPropagatesFailureAndExitCode(t *testing.T) {
- server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
- _, _ = w.Write([]byte(`{"id":"failed-id","name":"failed","job_state":"Failure","exit_code":17}`))
- }))
- defer server.Close()
- out := CaptureOutput(func() {
- if err := (&JobStatusCmd{ID: "failed-id"}).Run(&AppContext{APIBaseURL: server.URL}); err != nil {
- t.Error(err)
- }
- })
- if !contains(out, "Failure") || !contains(out, "Exit code: 17") {
- t.Fatal(out)
- }
-}
-func TestJobStatusDoesNotInventSuccessForMissingJob(t *testing.T) {
- server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
- w.WriteHeader(404)
- _, _ = w.Write([]byte(`{"error":"job not found"}`))
- }))
- defer server.Close()
- if err := (&JobStatusCmd{ID: "missing"}).Run(&AppContext{APIBaseURL: server.URL}); err == nil || !contains(err.Error(), "job not found") {
- t.Fatal(err)
- }
-}
diff --git a/cli/cmd/job_submit.go b/cli/cmd/job_submit.go
deleted file mode 100644
index a54945a..0000000
--- a/cli/cmd/job_submit.go
+++ /dev/null
@@ -1,63 +0,0 @@
-package cmd
-
-import (
- "fmt"
- "os"
- "path/filepath"
- "strings"
-)
-
-type JobSubmitCmd struct {
- Scope string `help:"Save outputs in common or your member folder (default: own folder)"`
- TTRuntime string `help:"Tenstorrent runtime: host or container"`
- Script string `arg:"" help:"Path to a Python or shell script"`
- Compute string `help:"CPU, NVIDIA, or TT" default:"CPU"`
- Devices int `help:"NVIDIA GPU count or Tenstorrent board count" default:"1"`
- Dataset string `help:"Owned ready dataset ID to mount at /inputs (optional)"`
- Image string `help:"Tagged image from an allowed registry (optional; defaults by compute type)"`
- CPU string `help:"Requested CPU quantity, e.g. 2 or 500m"`
- Memory string `help:"Requested memory, e.g. 4Gi"`
- Timeout int64 `help:"Maximum execution time in seconds" default:"600"`
- Name string `help:"Display name for the job"`
-}
-
-func (j *JobSubmitCmd) Run(ctx *AppContext) error {
- compute := strings.ToUpper(j.Compute)
- if compute == "" {
- compute = "CPU"
- }
- accelerator := "cpu"
- devices := j.Devices
- if devices == 0 {
- devices = 1
- }
- switch compute {
- case "CPU":
- devices = 0
- case "NVIDIA", "CUDA":
- accelerator = "nvidia"
- case "TT", "TENSTORRENT":
- accelerator = "tenstorrent"
- default:
- return fmt.Errorf("compute must be CPU, NVIDIA, or TT")
- }
- data, err := os.ReadFile(j.Script)
- if err != nil {
- return err
- }
- name := j.Name
- if name == "" {
- name = filepath.Base(j.Script)
- }
- request := map[string]interface{}{"type": "command", "name": name, "accelerator": accelerator,
- "storage_scope": j.Scope, "tt_runtime": j.TTRuntime, "device_count": devices, "script": string(data), "script_name": filepath.Base(j.Script),
- "dataset_id": j.Dataset, "image": j.Image, "cpu": j.CPU, "memory": j.Memory, "timeout_seconds": j.Timeout}
- var response struct {
- JobID string `json:"job_id"`
- }
- if err := ctx.api("POST", "/jobs", request, &response); err != nil {
- return err
- }
- fmt.Println("Job submitted:", response.JobID)
- return nil
-}
diff --git a/cli/cmd/job_submit_test.go b/cli/cmd/job_submit_test.go
deleted file mode 100644
index 0432adf..0000000
--- a/cli/cmd/job_submit_test.go
+++ /dev/null
@@ -1,43 +0,0 @@
-package cmd
-
-import (
- "encoding/json"
- "net/http"
- "net/http/httptest"
- "os"
- "path/filepath"
- "testing"
-)
-
-func TestJobSubmitUsesRealAPI(t *testing.T) {
- script := filepath.Join(t.TempDir(), "train.py")
- _ = os.WriteFile(script, []byte("print('train')"), 0600)
- server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
- if r.Method != "POST" || r.URL.Path != "/jobs" {
- t.Errorf("wrong request: %s %s", r.Method, r.URL.Path)
- }
- if r.Header.Get("X-Mist-Team") != "team-selected" {
- t.Errorf("missing selected workspace header")
- }
- var body map[string]interface{}
- _ = json.NewDecoder(r.Body).Decode(&body)
- if body["accelerator"] != "tenstorrent" || body["device_count"] != float64(2) || body["script"] != "print('train')" {
- t.Errorf("wrong submission: %+v", body)
- }
- w.WriteHeader(201)
- _, _ = w.Write([]byte(`{"job_id":"real-id"}`))
- }))
- defer server.Close()
- var err error
- output := CaptureOutput(func() {
- err = (&JobSubmitCmd{Script: script, Compute: "TT", Devices: 2}).Run(&AppContext{APIBaseURL: server.URL, Config: &Config{TeamID: "team-selected"}})
- })
- if err != nil || !contains(output, "real-id") || contains(output, "Are you sure") {
- t.Fatalf("%v: %s", err, output)
- }
-}
-func TestJobSubmitRejectsMissingFile(t *testing.T) {
- if err := (&JobSubmitCmd{Script: "/does-not-exist.py", Compute: "CPU"}).Run(&AppContext{}); err == nil {
- t.Fatal("missing file accepted")
- }
-}
diff --git a/cli/cmd/root.go b/cli/cmd/root.go
deleted file mode 100644
index 23a532d..0000000
--- a/cli/cmd/root.go
+++ /dev/null
@@ -1,84 +0,0 @@
-package cmd
-
-import (
- "encoding/json"
- "net/http"
- "os"
- "time"
-
- "github.com/alecthomas/kong"
-)
-
-type Config struct {
- AccessToken string `json:"access_token,omitempty"`
- SessionCookie string `json:"session_cookie,omitempty"`
- APIBaseURL string `json:"api_base_url,omitempty"`
- TeamID string `json:"team_id,omitempty"`
-}
-
-type AppContext struct {
- Config *Config
- HTTPClient *http.Client
- APIBaseURL string
- ConfigPath string
- TeamID string
-}
-
-type Globals struct {
- Workspace string `name:"team" env:"MIST_TEAM" help:"Team workspace ID, or legacy (overrides saved workspace)"`
- ConfigPath string `name:"config" help:"Path to config file" default:"${config_path}"`
- APIURL string `name:"api-url" env:"MIST_API_URL" help:"Mist API URL (default http://127.0.0.1:3000)"`
-}
-
-type CLI struct {
- Globals
-
- Team TeamCmd `cmd:"" help:"List and select team workspaces"`
- Auth AuthCmd `cmd:"" help:"Authentication commands"`
- Job JobCmd `cmd:"" help:"Job management commands"`
- Help HelpCmd `cmd:"" help:"Show help information"`
-}
-
-func loadConfig(path string) (*Config, error) {
- b, err := os.ReadFile(path)
- if err != nil {
- return nil, err
- }
-
- var cfg Config
- if err := json.Unmarshal(b, &cfg); err != nil {
- return nil, err
- }
-
- return &cfg, nil
-}
-
-func Main() {
- var cli CLI
- // Read command-line arguments
-
- appCtx := &AppContext{
- HTTPClient: &http.Client{Timeout: 10 * time.Second},
- }
-
- kctx := kong.Parse(&cli,
- kong.Name("mist"),
- kong.Description("MIST CLI - Manage your MIST jobs and configurations"),
- kong.UsageOnError(),
- kong.Vars{"config_path": defaultConfigPath()},
- kong.Bind(appCtx),
- )
-
- if cfg, err := loadConfig(cli.ConfigPath); err == nil {
- appCtx.Config = cfg
- } else if !os.IsNotExist(err) {
- // config file is present but broken
- kctx.FatalIfErrorf(err)
- }
-
- appCtx.APIBaseURL = cli.APIURL
- appCtx.TeamID = cli.Globals.Workspace
- appCtx.ConfigPath = cli.ConfigPath
- err := kctx.Run()
- kctx.FatalIfErrorf(err)
-}
diff --git a/cli/cmd/team.go b/cli/cmd/team.go
deleted file mode 100644
index 936d12f..0000000
--- a/cli/cmd/team.go
+++ /dev/null
@@ -1,84 +0,0 @@
-package cmd
-
-import "fmt"
-
-type TeamCmd struct {
- List TeamListCmd `cmd:"" help:"List your available team workspaces"`
- Use TeamUseCmd `cmd:"" help:"Save a team workspace selection; use legacy for historical account jobs"`
-}
-type TeamListCmd struct{}
-type TeamUseCmd struct {
- ID string `arg:"" help:"Team ID, or legacy"`
-}
-type teamSummary struct {
- ID string `json:"id"`
- Name string `json:"name"`
- Disabled bool `json:"disabled"`
- Members []struct {
- ID string `json:"id"`
- } `json:"members"`
-}
-
-func (j *TeamListCmd) Run(ctx *AppContext) error {
- var response struct {
- Teams []teamSummary `json:"teams"`
- }
- if err := ctx.api("GET", "/teams", nil, &response); err != nil {
- return err
- }
- fmt.Printf("%-24s %-28s %s\n", "TEAM ID", "NAME", "STATE")
- for _, t := range response.Teams {
- state := "Active"
- if t.Disabled {
- state = "Disabled"
- }
- if ctx.teamID() == t.ID {
- state += " · Selected"
- }
- fmt.Printf("%-24s %-28s %s\n", t.ID, t.Name, state)
- }
- return nil
-}
-func (j *TeamUseCmd) Run(ctx *AppContext) error {
- if j.ID != "legacy" {
- var response struct {
- Teams []teamSummary `json:"teams"`
- }
- if err := ctx.api("GET", "/teams", nil, &response); err != nil {
- return err
- }
- var session struct {
- User *struct {
- ID string `json:"id"`
- } `json:"user"`
- }
- if err := ctx.api("GET", "/session", nil, &session); err != nil {
- return err
- }
- found := false
- for _, t := range response.Teams {
- if t.ID == j.ID && !t.Disabled && session.User != nil {
- for _, m := range t.Members {
- if m.ID == session.User.ID {
- found = true
- }
- }
- }
- }
- if !found {
- return fmt.Errorf("select an active team you belong to; run team list")
- }
- }
- if ctx.Config == nil {
- ctx.Config = &Config{}
- }
- ctx.Config.TeamID = j.ID
- if j.ID == "legacy" {
- ctx.Config.TeamID = ""
- }
- if err := ctx.saveConfig(); err != nil {
- return err
- }
- fmt.Println("Workspace selected:", j.ID)
- return nil
-}
diff --git a/cli/cmd/testutil_test.go b/cli/cmd/testutil_test.go
deleted file mode 100644
index 6f60274..0000000
--- a/cli/cmd/testutil_test.go
+++ /dev/null
@@ -1,24 +0,0 @@
-package cmd
-
-import (
- "bytes"
- "io"
- "os"
-)
-
-func CaptureOutput(f func()) string {
- old := os.Stdout
- r, w, _ := os.Pipe()
- os.Stdout = w
- f()
- w.Close()
- os.Stdout = old
- var buf bytes.Buffer
- io.Copy(&buf, r)
- r.Close()
- return buf.String()
-}
-
-func contains(s, substr string) bool {
- return bytes.Contains([]byte(s), []byte(substr))
-}
diff --git a/cli/docs/setup.md b/cli/docs/setup.md
deleted file mode 100644
index 789a050..0000000
--- a/cli/docs/setup.md
+++ /dev/null
@@ -1,68 +0,0 @@
-# Mist CLI
-
-Requires Go 1.25.1. Build from the repository root:
-
-```bash
-go -C cli build -o ../bin/mist-cli .
-bin/mist-cli --help
-```
-
-The default API URL is `http://127.0.0.1:3000`. Override it using
-`MIST_API_URL`, `--api-url`, or the configuration's `api_base_url` field.
-The production endpoint is `http://100.73.139.66:8088/api`. Use it from a
-Tailnet device or either SSH host, then authenticate:
-
-```bash
-export MIST_API_URL=http://100.73.139.66:8088/api
-bin/mist-cli auth login --email your-member-email@example.org
-```
-
-The password prompt does not echo. Automation can use `--password-stdin`.
-The CLI saves a real session cookie in a mode-0600 config; `--config` selects
-an alternate file. Logout with `bin/mist-cli auth logout` revokes the session.
-
-Select your team before submitting new work:
-
-```bash
-bin/mist-cli team list
-read -r -p "Paste your team ID from the list: " TEAM_ID
-bin/mist-cli team use "$TEAM_ID"
-bin/mist-cli job submit /path/to/train.py --compute NVIDIA --devices 1 \
- --cpu 2 --memory 2Gi --timeout 1800 --name my-training
-bin/mist-cli job submit /path/to/tt_train.py --compute TT --devices 1
-bin/mist-cli job submit /path/to/task.sh --compute CPU
-bin/mist-cli job list # Waiting and running jobs
-bin/mist-cli job list --all # Includes completed, failed, and cancelled jobs
-read -r -p "Paste the job ID returned by submission: " JOB_ID
-bin/mist-cli job status "$JOB_ID"
-bin/mist-cli job logs "$JOB_ID"
-```
-
-Submission sends the actual file contents to the API. `.py` and `.sh` files
-are supported, up to 32 KiB. `--image` selects an image allowed by the API;
-otherwise the API chooses a runtime for the compute type. Upload datasets through the website, then pass `--dataset dataset-ID` to attach
-one read-only at `/inputs`. Dependencies belong in an image allowed by the team registry policy.
-Result files are downloadable from the website's job Files panel.
-
-NVIDIA counts are whole GPUs (one or two); TT counts are whole n300 boards
-(one to four, two chips each). Scripts must use the corresponding runtime.
-The default TT profile uses QuietBox's existing TT-Metal installation; a
-compatible custom image can select the container runtime. For a small
-built-in accelerator training check, use the Jobs page or the API's
-`training-smoke` submission type.
-
-Status, logs, cancellation, and errors come from the real API. Cancellation
-does not ask for confirmation. The deployed API authenticates each session and derives job/dataset ownership
-from the member account. Legacy fake-token authentication is removed.
-
-```bash
-go -C cli test ./...
-```
-
-The prompts above take actual IDs returned by `team list` and submission.
-To cancel that job while waiting/running, use `bin/mist-cli job cancel "$JOB_ID"`.
-`--team` or `MIST_TEAM` overrides the saved selection. Use `team use legacy` to
-read historical jobs; new work requires a team. `--scope common` requires common
-write permission. Custom tagged public images are allowed by team registry policy;
-TT custom images use `--tt-runtime container`. See the
-[department guide](../../docs/content/department-rollout.md) for limits and storage.
diff --git a/cli/go.mod b/cli/go.mod
deleted file mode 100644
index 3d70d07..0000000
--- a/cli/go.mod
+++ /dev/null
@@ -1,10 +0,0 @@
-module mist/cli
-
-go 1.24.0
-
-require (
- github.com/alecthomas/kong v1.12.1
- golang.org/x/term v0.35.0
-)
-
-require golang.org/x/sys v0.36.0 // indirect
diff --git a/cli/go.sum b/cli/go.sum
deleted file mode 100644
index 31447c8..0000000
--- a/cli/go.sum
+++ /dev/null
@@ -1,12 +0,0 @@
-github.com/alecthomas/assert/v2 v2.11.0 h1:2Q9r3ki8+JYXvGsDyBXwH3LcJ+WK5D0gc5E8vS6K3D0=
-github.com/alecthomas/assert/v2 v2.11.0/go.mod h1:Bze95FyfUr7x34QZrjL+XP+0qgp/zg8yS+TtBj1WA3k=
-github.com/alecthomas/kong v1.12.1 h1:iq6aMJDcFYP9uFrLdsiZQ2ZMmcshduyGv4Pek0MQPW0=
-github.com/alecthomas/kong v1.12.1/go.mod h1:p2vqieVMeTAnaC83txKtXe8FLke2X07aruPWXyMPQrU=
-github.com/alecthomas/repr v0.4.0 h1:GhI2A8MACjfegCPVq9f1FLvIBS+DrQ2KQBFZP1iFzXc=
-github.com/alecthomas/repr v0.4.0/go.mod h1:Fr0507jx4eOXV7AlPV6AVZLYrLIuIeSOWtW57eE/O/4=
-github.com/hexops/gotextdiff v1.0.3 h1:gitA9+qJrrTCsiCl7+kh75nPqQt1cx4ZkudSTLoUqJM=
-github.com/hexops/gotextdiff v1.0.3/go.mod h1:pSWU5MAI3yDq+fZBTazCSJysOMbxWL1BSow5/V2vxeg=
-golang.org/x/sys v0.36.0 h1:KVRy2GtZBrk1cBYA7MKu5bEZFxQk4NIDV6RLVcC8o0k=
-golang.org/x/sys v0.36.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
-golang.org/x/term v0.35.0 h1:bZBVKBudEyhRcajGcNc3jIfWPqV4y/Kt2XcoigOWtDQ=
-golang.org/x/term v0.35.0/go.mod h1:TPGtkTLesOwf2DE8CgVYiZinHAOuy5AYUYT1lENIZnA=
diff --git a/cli/main.go b/cli/main.go
deleted file mode 100644
index 038f096..0000000
--- a/cli/main.go
+++ /dev/null
@@ -1,5 +0,0 @@
-package main
-
-import "mist/cli/cmd"
-
-func main() { cmd.Main() }
diff --git a/cmd/mistd/main.go b/cmd/mistd/main.go
new file mode 100644
index 0000000..2b7d7d5
--- /dev/null
+++ b/cmd/mistd/main.go
@@ -0,0 +1,98 @@
+// Command mistd is the Mist control plane: one binary, several roles (D16).
+//
+// mistd api HTTP API
+// mistd scheduler queue, admission and executor calls (one active leader)
+//
+// Both connect to MIST_DATABASE_URL (or -database-url) and apply pending
+// migrations at start.
+package main
+
+import (
+ "context"
+ "errors"
+ "flag"
+ "fmt"
+ "io"
+ "log/slog"
+ "net/http"
+ "os"
+ "os/signal"
+ "syscall"
+ "time"
+
+ "github.com/UTMIST/Mist/internal/db"
+ "github.com/UTMIST/Mist/internal/scheduler"
+)
+
+func main() {
+ ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
+ defer stop()
+ if err := run(ctx, os.Args[1:], os.Stderr); err != nil {
+ fmt.Fprintln(os.Stderr, "mistd:", err)
+ os.Exit(1)
+ }
+}
+
+func run(ctx context.Context, args []string, stderr io.Writer) error {
+ if len(args) == 0 {
+ return errors.New("usage: mistd [flags]")
+ }
+ cmd, args := args[0], args[1:]
+ fs := flag.NewFlagSet("mistd "+cmd, flag.ContinueOnError)
+ fs.SetOutput(stderr)
+ dbURL := fs.String("database-url", os.Getenv("MIST_DATABASE_URL"), "Postgres URL")
+ listen := fs.String("listen", "127.0.0.1:8080", "API listen address (api)")
+ tick := fs.Duration("tick", 10*time.Second, "scheduling cycle period (scheduler)")
+ leaderCheck := fs.Duration("leader-check", 2*time.Second, "leadership check period (scheduler)")
+ if err := fs.Parse(args); err != nil {
+ return err
+ }
+ if *dbURL == "" {
+ return errors.New("set MIST_DATABASE_URL or -database-url")
+ }
+ log := slog.New(slog.NewTextHandler(stderr, nil)).With("role", cmd)
+
+ pool, err := db.Open(ctx, *dbURL)
+ if err != nil {
+ return err
+ }
+ defer pool.Close()
+ if err := db.Migrate(ctx, pool); err != nil {
+ return err
+ }
+ log.Info("database migrated")
+
+ switch cmd {
+ case "scheduler":
+ return scheduler.Run(ctx, scheduler.Config{
+ DatabaseURL: *dbURL,
+ Pool: pool,
+ Log: log,
+ Tick: *tick,
+ LeaderCheck: *leaderCheck,
+ })
+ case "api":
+ mux := http.NewServeMux()
+ mux.HandleFunc("GET /healthz", func(w http.ResponseWriter, r *http.Request) {
+ if err := pool.Ping(r.Context()); err != nil {
+ http.Error(w, "database unavailable", http.StatusServiceUnavailable)
+ return
+ }
+ io.WriteString(w, "ok\n")
+ })
+ srv := &http.Server{Addr: *listen, Handler: mux, ReadHeaderTimeout: 10 * time.Second}
+ go func() {
+ <-ctx.Done()
+ shut, cancel := context.WithTimeout(context.WithoutCancel(ctx), 5*time.Second)
+ defer cancel()
+ srv.Shutdown(shut)
+ }()
+ log.Info("listening", "addr", *listen)
+ if err := srv.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) {
+ return err
+ }
+ return nil
+ default:
+ return fmt.Errorf("unknown subcommand %q", cmd)
+ }
+}
diff --git a/cmd/mistd/main_test.go b/cmd/mistd/main_test.go
new file mode 100644
index 0000000..4d41be2
--- /dev/null
+++ b/cmd/mistd/main_test.go
@@ -0,0 +1,132 @@
+package main
+
+import (
+ "bytes"
+ "context"
+ "strings"
+ "sync"
+ "testing"
+ "time"
+
+ "github.com/jackc/pgx/v5"
+
+ "github.com/UTMIST/Mist/internal/db/dbtest"
+)
+
+func TestMain(m *testing.M) { dbtest.Main(m) }
+
+type syncBuffer struct {
+ mu sync.Mutex
+ b bytes.Buffer
+}
+
+func (s *syncBuffer) Write(p []byte) (int, error) {
+ s.mu.Lock()
+ defer s.mu.Unlock()
+ return s.b.Write(p)
+}
+
+func (s *syncBuffer) String() string {
+ s.mu.Lock()
+ defer s.mu.Unlock()
+ return s.b.String()
+}
+
+func waitFor(t *testing.T, what string, cond func() bool) {
+ t.Helper()
+ deadline := time.Now().Add(10 * time.Second)
+ for !cond() {
+ if time.Now().After(deadline) {
+ t.Fatalf("timed out waiting for %s", what)
+ }
+ time.Sleep(20 * time.Millisecond)
+ }
+}
+
+type instance struct {
+ log *syncBuffer
+ cancel context.CancelFunc
+ done chan error
+}
+
+func startScheduler(t *testing.T, url string) *instance {
+ ctx, cancel := context.WithCancel(context.Background())
+ in := &instance{log: &syncBuffer{}, cancel: cancel, done: make(chan error, 1)}
+ go func() {
+ in.done <- run(ctx, []string{"scheduler", "-database-url", url, "-leader-check", "100ms"}, in.log)
+ }()
+ t.Cleanup(func() { cancel(); <-in.done })
+ return in
+}
+
+// M1 step 1 check: the scheduler migrates an empty database and holds the
+// advisory lock; a second instance waits until the first stops.
+func TestSchedulerMigratesAndHoldsLock(t *testing.T) {
+ url := dbtest.URL(t)
+
+ first := startScheduler(t, url)
+ waitFor(t, "first instance to lead", func() bool {
+ return strings.Contains(first.log.String(), "holding scheduler lock")
+ })
+
+ ctx := context.Background()
+ conn, err := pgx.Connect(ctx, url)
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer conn.Close(ctx)
+ var n int
+ if err := conn.QueryRow(ctx, "SELECT count(*) FROM schema_migrations WHERE owner = 'scheduler'").Scan(&n); err != nil || n == 0 {
+ t.Fatalf("scheduler migrations not applied: n=%d err=%v", n, err)
+ }
+
+ second := startScheduler(t, url)
+ waitFor(t, "second instance to wait", func() bool {
+ return strings.Contains(second.log.String(), "waiting for scheduler lock")
+ })
+ time.Sleep(500 * time.Millisecond)
+ if strings.Contains(second.log.String(), "holding scheduler lock") {
+ t.Fatal("second instance acquired the lock while the first held it")
+ }
+
+ first.cancel()
+ if err := <-first.done; err != nil {
+ t.Fatalf("first instance: %v", err)
+ }
+ first.done <- nil // for Cleanup
+ waitFor(t, "second instance to take over", func() bool {
+ return strings.Contains(second.log.String(), "holding scheduler lock")
+ })
+}
+
+// Losing the database connection must stop the leader, not leave it
+// dispatching without the lock.
+func TestSchedulerExitsWhenConnectionLost(t *testing.T) {
+ url := dbtest.URL(t)
+ in := startScheduler(t, url)
+ waitFor(t, "instance to lead", func() bool {
+ return strings.Contains(in.log.String(), "holding scheduler lock")
+ })
+
+ ctx := context.Background()
+ conn, err := pgx.Connect(ctx, url)
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer conn.Close(ctx)
+ if _, err := conn.Exec(ctx, `
+ SELECT pg_terminate_backend(pid) FROM pg_locks
+ WHERE locktype = 'advisory' AND granted AND pid <> pg_backend_pid()`); err != nil {
+ t.Fatal(err)
+ }
+
+ select {
+ case err := <-in.done:
+ if err == nil || !strings.Contains(err.Error(), "leadership lost") {
+ t.Fatalf("want leadership lost error, got %v", err)
+ }
+ in.done <- nil
+ case <-time.After(10 * time.Second):
+ t.Fatal("scheduler kept running after losing its lock connection")
+ }
+}
diff --git a/deploy/k3s/Dockerfile.api b/deploy/k3s/Dockerfile.api
deleted file mode 100644
index 0cad2bf..0000000
--- a/deploy/k3s/Dockerfile.api
+++ /dev/null
@@ -1,7 +0,0 @@
-# Build the static binary from the repository root:
-# CGO_ENABLED=0 go -C src build -o ../bin/mist-api .
-# docker build -f deploy/k3s/Dockerfile.api -t mist-api:departments-20261009 .
-FROM alpine:3.16.3
-COPY bin/mist-api /usr/local/bin/mist-api
-USER 65532:65532
-ENTRYPOINT ["/usr/local/bin/mist-api"]
diff --git a/deploy/k3s/README.md b/deploy/k3s/README.md
deleted file mode 100644
index 55ceeb1..0000000
--- a/deploy/k3s/README.md
+++ /dev/null
@@ -1,34 +0,0 @@
-# Kubernetes deployment assets
-
-Current installation: two k3s nodes, authenticated team jobs, scoped NFS storage
-and private Tailscale portal. Start with the [k3s setup/handoff](../../docs/content/k3s-setup.md)
-and [deployment operations](../private/README.md).
-
-## Manifest and tooling map
-
-| Assets | Purpose |
-|---|---|
-| `mist-api.yaml` | Single API, built-in auth, and controller deployment |
-| `team-rbac.yaml`, `team-workload-policy.yaml` | Team provisioning authority and workload admission |
-| `nvidia-gpu-operator-values.yaml` | Pinned host-driver/CDI configuration |
-| `tenstorrent-operator-values.yaml`, `mist-tenstorrent-claims.yaml` | TT operator configuration and retained legacy claims |
-| `tenstorrent-workload-policy.yaml` | Admission protection for legacy `mist` diagnostics |
-| `storage/` | NFS manifests, bounded pool provisioning, migration/growth |
-| `training_smoke.py`, `examples/` | Accelerator checks and packaged-image examples |
-| `gpu-smoke-job.yaml`, `training-*-job.yaml`, `submit_tenstorrent_smoke.py` | Administrator diagnostics in legacy namespace |
-| `verify_department*`, `verify_split_storage.py`, `verify_large_dataset.py` | Current authenticated live verifiers |
-| `verify_tenstorrent_allocation.py` | Infrastructure-level TT allocation diagnostic |
-| `evidence/department-2026-10-09/` | Current release evidence |
-| Earlier `*-results.json` | Historical dated test evidence |
-
-Direct manifest diagnostics require cluster administrator access and bypass
-portal team admission. Researchers should use the authenticated portal or CLI.
-Archived pre-team API/browser verifiers have been retired from executable source.
-
-The application image build uses [Dockerfile.api](Dockerfile.api). API/auth images
-are imported into main's k3s containerd, not scheduled by a Docker supervisor.
-Use `bash deploy/private/build-and-deploy.sh` for the existing installation.
-
-[Old deployment history](../../docs/content/archive/k3s-deployment-history.md) and
-[original node setup](../../docs/content/archive/k3s-setup-history.md) preserve earlier
-observations; their unauthenticated/node-local instructions are superseded.
diff --git a/deploy/k3s/complete-results.json b/deploy/k3s/complete-results.json
deleted file mode 100644
index 641710a..0000000
--- a/deploy/k3s/complete-results.json
+++ /dev/null
@@ -1,104 +0,0 @@
-{
- "site": "http://100.73.139.66:8088",
- "started": "2026-10-09T03:36:21.090Z",
- "jobs": [
- {
- "id": "mist-22600b7fdeaa5e5f921b25e9",
- "backend": "cpu",
- "state": "Success",
- "node": "utmist-z1opa08",
- "files": [
- {
- "path": "dataset-check.json",
- "size": 37
- }
- ]
- },
- {
- "id": "mist-78b599aa6e10655067ab8d86",
- "backend": "nvidia",
- "state": "Success",
- "node": "utmist-z1opa08",
- "files": [
- {
- "path": "allocation.json",
- "size": 85
- },
- {
- "path": "nvidia-0.pt",
- "size": 5409
- },
- {
- "path": "nvidia-1.pt",
- "size": 5409
- },
- {
- "path": "results.json",
- "size": 1081
- }
- ]
- },
- {
- "id": "mist-5a2e0e2e99adcded65b1fc5d",
- "backend": "tenstorrent",
- "state": "Success",
- "node": "utmist-tt",
- "files": [
- {
- "path": "allocation.json",
- "size": 72
- },
- {
- "path": "results.json",
- "size": 977
- },
- {
- "path": "tenstorrent-0.pt",
- "size": 5843
- },
- {
- "path": "tenstorrent-1.pt",
- "size": 5843
- }
- ]
- }
- ],
- "checks": {
- "browser_login": true,
- "browser_upload_select_train_download": true,
- "member_password_change": true,
- "member_creation_isolation_deactivation": true,
- "anonymous_signup_csrf_denied": true
- },
- "dataset": {
- "id": "dataset-197f830f9dfd0a7b1d4cf7b3",
- "sha256": "98f61f87e9d30c36f0a8933028bbfcda06d56c393f12a4b5363cc1584decc01e",
- "size": 248398
- },
- "browser_errors": [],
- "finished": "2026-10-09T03:37:08.265Z",
- "quietbox_submission": {
- "submitted_from": "utmist-tt",
- "job_id": "mist-32bf668f4232ca4b3b3f9472",
- "node": "utmist-z1opa08",
- "state": "Success",
- "downloaded_metrics": {
- "initial": 0.08051128685474396,
- "final": 2.4448332247573035e-09,
- "cuda": true
- }
- },
- "operations": {
- "api_auth_restart_preserves_jobs_datasets_downloads": true,
- "legacy_outputs_downloadable": 10,
- "checked_zip_upload_and_unused_delete": true,
- "auth_backup_integrity": "ok",
- "checked_at": "2026-10-09T03:43:07Z"
- },
- "images": {
- "mist-api:complete-20261009": "sha256:de7b759498b0fdc157d846b36b5bebb560c6e7a1cc7f3c5d34904a6494273611",
- "mist-auth:members-20261009": "sha256:bec799095b8ca79faabe75457143e366b67d3ca0fe6cc3dc7de5c5c9c4517c64"
- },
- "scope": "Five-part foundation, privately hosted; HTTPS Serve prerequisite remains explicit",
- "evidence_directory": "/home/utmist/mist-complete-results-2026-10-09"
-}
diff --git a/deploy/k3s/evidence/department-2026-10-09/README.md b/deploy/k3s/evidence/department-2026-10-09/README.md
deleted file mode 100644
index a9596b2..0000000
--- a/deploy/k3s/evidence/department-2026-10-09/README.md
+++ /dev/null
@@ -1,94 +0,0 @@
-# Department verification — October 9, 2026
-
-Evidence from the actual two-node private pilot before repository cleanup.
-The original Go/Docker/logger counts below belong to that earlier release;
-[current cleanup verification](../../../../docs/content/repository-cleanup.md) records
-the maintained suites after their obsolete implementations were removed.
-Reports contain synthetic
-verification data and job/device IDs; no passwords, cookies or auth secrets.
-Scripts are in `deploy/k3s/verify_department*`. They require existing authorized
-local credentials and recorded machine addresses; inspect them before rerunning.
-They create bounded test fixtures and preserve files/history afterward.
-
-| Check | Evidence |
-|---|---|
-| Fourteen API/storage/queue/training/isolation/legacy checks | [results.json](results.json) |
-| Eleven browser checks, including pagination and mobile | [browser-results.json](browser-results.json) |
-| Streamed 2 GiB + 4 MiB dataset, SHA256 and mounted Job input | [large upload](large-dataset-results.json), [job log](large-dataset-job.log) |
-| Two concurrent TT jobs, distinct boards; failure/deadline | [runtime-results.json](runtime-results.json) |
-| SSH on TT → CLI → NVIDIA on main | [remote-cli-results.json](remote-cli-results.json), [status](remote-cli-status.txt), [log](remote-cli-job.log) |
-| Six server admission denials | [admission-results.json](admission-results.json) |
-| Installed host chains, synthetic untrusted source | [portal-firewall.log](portal-firewall.log) |
-| NVIDIA training metrics / actual allocation | [metrics](nvidia-results.json), [allocation](nvidia-allocation.json), [log](mist-8644eccafaf0b42c3d4add05.log) |
-| TT training metrics / actual allocation | [metrics](tenstorrent-results.json), [allocation](tenstorrent-allocation.json), [log](mist-e5c04ea2cc126a786d6fe783.log) |
-| Job writes fill only the bounded test filesystem | [capacity log](mist-1c4a852d366b9d2349e1cf7c.log) |
-| Actual container cannot reach private services/credentials | [isolation log](mist-b46534ddeb9f18852b75f645.log) |
-| Full release suites and final browser rerun | [release-validation.json](release-validation.json) |
-| Dependency audit, zero reported vulnerabilities | [web audit](web-audit.json), [auth audit](auth-audit.json) |
-
-Screenshots: [jobs desktop](final-jobs-desktop.png),
-[jobs mobile](final-jobs-mobile.png), [job composer](final-job-composer.png),
-[mobile composer](final-job-composer-mobile.png), [datasets](final-datasets-desktop.png),
-[overview](final-overview.png).
-
-Saved training weights were downloaded and checked during verification. They
-remain in the protected local evidence directory and team job outputs; binary
-weights and protected account/session state are not checked in.
-
-## Local checks
-
-```bash
-go -C src test ./... -run 'Test(Department|Kubernetes|Shared|Dataset|Auth|Member|Image|Hardware|Result)' -count=1
-go -C cli test ./...
-npm --prefix web-interface test
-npx --prefix web-interface tsc --noEmit -p web-interface/tsconfig.json
-```
-
-Full frontend ESLint, shell/Python syntax checks and the production frontend
-build passed. Use the pinned Go/Node versions in the deployment script. The final
-release run passed all 33 backend and 10 Docker module tests, with no skips, using
-a dedicated Redis container and the existing CPU Docker fixtures. CLI and logger
-suites, TypeScript and all 10 React tests passed. The final browser rerun passed
-all 11 checks with no browser errors. See [release validation](release-validation.json).
-
-The host firewall test used an isolated synthetic source on the actual installed
-chains, not an external researcher's laptop. Regression training demonstrates
-the tested runtimes/devices; arbitrary models, hostile-code VM isolation, HA,
-per-member hard storage quotas and excluded rollout features are not claimed.
-
-## Separate storage pools follow-up
-
-[Small-file storage report](split-storage-results.json): seven live checks passed,
-using a 260,729-byte dataset. Both NVIDIA GPUs and one TT board/two chips trained
-from the uploaded data and saved/downloaded outputs through the new model PVC.
-Existing accelerator outputs match their pre-migration evidence byte for byte.
-Both pool shrink requests and combined budget overcommit were rejected without
-changing persisted policy. The own small dataset was removed after both jobs.
-Logs: [NVIDIA](split-nvidia.log), [Tenstorrent](split-tenstorrent.log).
-
-Six isolated migration tests passed using only tiny temporary files: content,
-mode/ownership/link preservation, idempotence, retry after publication, conflicts,
-partial stages, safe manifests and API access through member-folder parents.
-Live NFS checks identified zero statfs filesystem IDs; readiness uses distinct
-mount device identities. Dataset/model exports have separate stable files.
-
-The eleven browser checks and nine React tests passed after migration. Large
-uploads and quota exhaustion were not repeated, at the user’s request. These
-checks do not demonstrate uploads at 200/500 GiB or every disk-full scenario.
-The initial unused 1.5 TiB bulk reservation was released using filesystem discard,
-preserving data. After reclaiming deleted test blocks, the sparse backing file
-used about 6 GiB physically and QuietBox had about 1.9 TiB free on its host disk.
-
-## Packaged image through the regular-member UI
-
-[Container mode report](container-mode-result.json), [filled form](container-mode-form.png),
-[completed job](container-mode-completed.png). The visible desktop browser selected
-NVIDIA → Container → one GPU and submitted the custom image with blank command
-and argument fields. Actual job manifest inspection confirmed the image startup
-was preserved and no Python script was injected. Real CUDA backpropagation,
-convergence, saved weights/metrics and checkpoint reload passed. The image was
-preloaded locally, so this does not claim a registry push/pull test.
-
-The walkthrough found and fixed an image-field editing bug: clearing an override
-now leaves the field blank while the member types a replacement. A regression
-test covers clearing, intermediate typing and preserving image startup defaults.
diff --git a/deploy/k3s/evidence/department-2026-10-09/admission-results.json b/deploy/k3s/evidence/department-2026-10-09/admission-results.json
deleted file mode 100644
index f39b83b..0000000
--- a/deploy/k3s/evidence/department-2026-10-09/admission-results.json
+++ /dev/null
@@ -1,26 +0,0 @@
-{
- "host_network": {
- "denied": true,
- "reason": "The pods \"admission-probe\" is invalid: : ValidatingAdmissionPolicy 'mist-team-boundary' with binding 'mist-team-boundary' denied request: Team workloads cannot join host namespaces."
- },
- "host_pid": {
- "denied": true,
- "reason": "The pods \"admission-probe\" is invalid: : ValidatingAdmissionPolicy 'mist-team-boundary' with binding 'mist-team-boundary' denied request: Team workloads cannot join host namespaces."
- },
- "service_account_token": {
- "denied": true,
- "reason": "The pods \"admission-probe\" is invalid: : ValidatingAdmissionPolicy 'mist-team-boundary' with binding 'mist-team-boundary' denied request: Team workloads must disable Kubernetes credentials."
- },
- "privileged_container": {
- "denied": true,
- "reason": "The pods \"admission-probe\" is invalid: : ValidatingAdmissionPolicy 'mist-team-boundary' with binding 'mist-team-boundary' denied request: Team containers must drop capabilities and forbid privilege escalation."
- },
- "root_host_path": {
- "denied": true,
- "reason": "The pods \"admission-probe\" is invalid: : ValidatingAdmissionPolicy 'mist-team-boundary' with binding 'mist-team-boundary' denied request: Team volumes allow PVCs, scripts, scratch space and read-only TT runtime mounts."
- },
- "capability_add": {
- "denied": true,
- "reason": "The pods \"admission-probe\" is invalid: : ValidatingAdmissionPolicy 'mist-team-boundary' with binding 'mist-team-boundary' denied request: Team containers must drop capabilities and forbid privilege escalation."
- }
-}
diff --git a/deploy/k3s/evidence/department-2026-10-09/auth-audit.json b/deploy/k3s/evidence/department-2026-10-09/auth-audit.json
deleted file mode 100644
index c9ea51a..0000000
--- a/deploy/k3s/evidence/department-2026-10-09/auth-audit.json
+++ /dev/null
@@ -1,22 +0,0 @@
-{
- "auditReportVersion": 2,
- "vulnerabilities": {},
- "metadata": {
- "vulnerabilities": {
- "info": 0,
- "low": 0,
- "moderate": 0,
- "high": 0,
- "critical": 0,
- "total": 0
- },
- "dependencies": {
- "prod": 26,
- "dev": 0,
- "optional": 0,
- "peer": 0,
- "peerOptional": 0,
- "total": 25
- }
- }
-}
diff --git a/deploy/k3s/evidence/department-2026-10-09/browser-results.json b/deploy/k3s/evidence/department-2026-10-09/browser-results.json
deleted file mode 100644
index b74736b..0000000
--- a/deploy/k3s/evidence/department-2026-10-09/browser-results.json
+++ /dev/null
@@ -1,19 +0,0 @@
-{
- "checks": {
- "admin_browser_login": true,
- "browser_account_creation": true,
- "browser_team_creation_membership_common_write_and_policy": true,
- "browser_upload_common_folder_browse_and_download": true,
- "job_defaults_follow_loaded_team_policy": true,
- "regular_browser_selfservice_image_submit_expand_logs_result_download": true,
- "regular_browser_admin_controls_denied": true,
- "browser_password_reset_revokes_sessions_new_login_works": true,
- "responsive_job_dialog_and_device_slider": true,
- "history_pagination_next_previous_real_legacy_jobs": true,
- "desktop_mobile_layout_no_overflow_or_browser_errors": true
- },
- "browser_errors": [],
- "started": "2026-10-09T07:54:23.922Z",
- "job_id": "mist-6b49a490b1bf833a1e194415",
- "finished": "2026-10-09T07:54:45.282Z"
-}
\ No newline at end of file
diff --git a/deploy/k3s/evidence/department-2026-10-09/cleanup-validation.json b/deploy/k3s/evidence/department-2026-10-09/cleanup-validation.json
deleted file mode 100644
index 59eeb18..0000000
--- a/deploy/k3s/evidence/department-2026-10-09/cleanup-validation.json
+++ /dev/null
@@ -1,56 +0,0 @@
-{
- "recorded_at": "2026-10-09T08:41:50.836023+00:00",
- "base_main_commit": "3431ffdb713fac72881fc7f61f2750a82fe3f916",
- "source_branch": "chore/main-cleanup-20261009",
- "go": {
- "race_detector": true,
- "api_tests_passed": 29,
- "cli_tests_passed": 6,
- "skipped_tests": 0,
- "vet": "passed",
- "api_cli_static_builds": "passed",
- "retired_executor_fail_fast": "passed",
- "deadcode_tool": "golang.org/x/tools/cmd/deadcode v0.38.0, with tests",
- "unreachable_functions_after_cleanup": 0
- },
- "frontend": {
- "react_tests_passed": 10,
- "typescript": "passed",
- "lint": "passed",
- "production_build": "passed",
- "npm_audit_vulnerabilities": 0
- },
- "migration": {
- "tests_passed": 6,
- "skipped_tests": 0,
- "environment": "disposable root container; read-only source, isolated tiny temporary files; no live storage"
- },
- "candidate_api_smoke": {
- "checks": {
- "compiled_api_startup_real_kubernetes_health": true,
- "real_auth_gateway_session": true,
- "anonymous_jobs_denied": true,
- "sigterm_graceful_shutdown": true
- },
- "temporary_listener": "localhost only, removed after test",
- "team_controller_enabled": false,
- "cluster_or_account_mutations": false,
- "production_redeployed": false
- },
- "infrastructure_audit": {
- "nodes_ready": 2,
- "api_auth_available": true,
- "operators_ready": true,
- "ssh_quietbox_connection": "passed",
- "tailscale_ssh_handler_enabled": false,
- "live_host_service_enablement_checked": true
- },
- "production_redeployed": false,
- "large_storage_tests_run": false,
- "fresh_training_or_browser_runs_claimed": false,
- "notes": [
- "Prior live training/browser/isolation evidence is for the pre-cleanup department release",
- "Main still requires independent pull-request review",
- "Legacy source removal preserves existing Kubernetes jobs/PVCs and host files"
- ]
-}
diff --git a/deploy/k3s/evidence/department-2026-10-09/container-mode-completed.png b/deploy/k3s/evidence/department-2026-10-09/container-mode-completed.png
deleted file mode 100644
index efe6177..0000000
Binary files a/deploy/k3s/evidence/department-2026-10-09/container-mode-completed.png and /dev/null differ
diff --git a/deploy/k3s/evidence/department-2026-10-09/container-mode-form.png b/deploy/k3s/evidence/department-2026-10-09/container-mode-form.png
deleted file mode 100644
index ab2243b..0000000
Binary files a/deploy/k3s/evidence/department-2026-10-09/container-mode-form.png and /dev/null differ
diff --git a/deploy/k3s/evidence/department-2026-10-09/container-mode-result.json b/deploy/k3s/evidence/department-2026-10-09/container-mode-result.json
deleted file mode 100644
index ac76a98..0000000
--- a/deploy/k3s/evidence/department-2026-10-09/container-mode-result.json
+++ /dev/null
@@ -1,22 +0,0 @@
-{
- "job_id": "mist-a939ac7bc458ac8fc6d6dbd6",
- "name": "My packaged CUDA container",
- "image": "docker.io/utmist/mist-cuda-student:demo-20261009",
- "mode": "Container",
- "submitted_through_ui": true,
- "regular_member": true,
- "status": "Success",
- "node": "utmist-z1opa08",
- "image_command_overridden": false,
- "script_injected": false,
- "gpu": "NVIDIA RTX A4000",
- "device": "cuda:0",
- "allocated_gpus": 1,
- "cuda_version": "12.4",
- "initial_loss": 14.088242530822754,
- "final_loss": 0.004935909993946552,
- "steps": 200,
- "checkpoint_reload_passed": true,
- "startup": "image CMD",
- "script_path": "/app/train.py"
-}
\ No newline at end of file
diff --git a/deploy/k3s/evidence/department-2026-10-09/final-datasets-desktop.png b/deploy/k3s/evidence/department-2026-10-09/final-datasets-desktop.png
deleted file mode 100644
index f455d80..0000000
Binary files a/deploy/k3s/evidence/department-2026-10-09/final-datasets-desktop.png and /dev/null differ
diff --git a/deploy/k3s/evidence/department-2026-10-09/final-job-composer-mobile.png b/deploy/k3s/evidence/department-2026-10-09/final-job-composer-mobile.png
deleted file mode 100644
index 529ccd6..0000000
Binary files a/deploy/k3s/evidence/department-2026-10-09/final-job-composer-mobile.png and /dev/null differ
diff --git a/deploy/k3s/evidence/department-2026-10-09/final-job-composer.png b/deploy/k3s/evidence/department-2026-10-09/final-job-composer.png
deleted file mode 100644
index 3765cac..0000000
Binary files a/deploy/k3s/evidence/department-2026-10-09/final-job-composer.png and /dev/null differ
diff --git a/deploy/k3s/evidence/department-2026-10-09/final-jobs-desktop.png b/deploy/k3s/evidence/department-2026-10-09/final-jobs-desktop.png
deleted file mode 100644
index 182e15e..0000000
Binary files a/deploy/k3s/evidence/department-2026-10-09/final-jobs-desktop.png and /dev/null differ
diff --git a/deploy/k3s/evidence/department-2026-10-09/final-jobs-mobile.png b/deploy/k3s/evidence/department-2026-10-09/final-jobs-mobile.png
deleted file mode 100644
index 2e771b2..0000000
Binary files a/deploy/k3s/evidence/department-2026-10-09/final-jobs-mobile.png and /dev/null differ
diff --git a/deploy/k3s/evidence/department-2026-10-09/final-overview.png b/deploy/k3s/evidence/department-2026-10-09/final-overview.png
deleted file mode 100644
index a9a5d9b..0000000
Binary files a/deploy/k3s/evidence/department-2026-10-09/final-overview.png and /dev/null differ
diff --git a/deploy/k3s/evidence/department-2026-10-09/large-dataset-job.log b/deploy/k3s/evidence/department-2026-10-09/large-dataset-job.log
deleted file mode 100644
index 9bdddba..0000000
--- a/deploy/k3s/evidence/department-2026-10-09/large-dataset-job.log
+++ /dev/null
@@ -1 +0,0 @@
-LARGE_DATASET_JOB_PASSED
diff --git a/deploy/k3s/evidence/department-2026-10-09/large-dataset-results.json b/deploy/k3s/evidence/department-2026-10-09/large-dataset-results.json
deleted file mode 100644
index 020df46..0000000
--- a/deploy/k3s/evidence/department-2026-10-09/large-dataset-results.json
+++ /dev/null
@@ -1,12 +0,0 @@
-{
- "checks": {
- "actual_upload_over_2_gib_checksum_verified": true,
- "large_file_mounted_readonly_real_job": true,
- "own_probe_removed_after_job_no_user_files_deleted": true
- },
- "team_id": "team-7acd03dd93cac1af",
- "dataset_id": "dataset-22a8a7f21644eebf07f8e9b0",
- "bytes": 2151677952,
- "upload_seconds": 65.0,
- "job_id": "mist-8444679f5946e19925cd4732"
-}
diff --git a/deploy/k3s/evidence/department-2026-10-09/mist-1c4a852d366b9d2349e1cf7c.log b/deploy/k3s/evidence/department-2026-10-09/mist-1c4a852d366b9d2349e1cf7c.log
deleted file mode 100644
index 2783429..0000000
--- a/deploy/k3s/evidence/department-2026-10-09/mist-1c4a852d366b9d2349e1cf7c.log
+++ /dev/null
@@ -1 +0,0 @@
-STORAGE_QUOTA_PASSED bytes=1027604480
diff --git a/deploy/k3s/evidence/department-2026-10-09/mist-28ba381f47c609714d91aafb.log b/deploy/k3s/evidence/department-2026-10-09/mist-28ba381f47c609714d91aafb.log
deleted file mode 100644
index 232354e..0000000
--- a/deploy/k3s/evidence/department-2026-10-09/mist-28ba381f47c609714d91aafb.log
+++ /dev/null
@@ -1,56 +0,0 @@
-2026-10-09 05:50:26.046 | DEBUG | ttnn::78 - Initial ttnn.CONFIG:
-Config{cache_path=/root/.cache/ttnn,model_cache_path=/root/.cache/ttnn/models,tmp_dir=/tmp/ttnn,enable_model_cache=false,enable_fast_runtime_mode=true,validate_program_args=false,throw_exception_on_fallback=false,enable_logging=false,enable_graph_report=false,enable_graph_python_stack_traces=false,enable_detailed_buffer_report=false,enable_detailed_tensor_report=false,enable_comparison_mode=false,comparison_mode_should_raise_exception=false,comparison_mode_pcc=0.9999,root_report_path=generated/ttnn/reports,report_name=std::nullopt,std::nullopt}
-2026-10-09 05:50:26.248 | info | Device | Opening user mode device driver (tt_cluster.cpp:228)
-2026-10-09 05:50:26.248 | info | UMD | Cluster constructor started. (cluster.cpp:389)
-2026-10-09 05:50:26.249 | info | UMD | Creating TopologyDiscovery for architecture: wormhole_b0 (topology_discovery.cpp:96)
-2026-10-09 05:50:26.249 | info | UMD | Starting topology discovery. (topology_discovery.cpp:115)
-2026-10-09 05:50:26.250 | info | UMD | Established firmware bundle version: 19.4.2 (topology_discovery.cpp:575)
-2026-10-09 05:50:26.258 | info | UMD | Completed topology discovery. (topology_discovery.cpp:119)
-2026-10-09 05:50:26.302 | warning | UMD | hwloc_set_area_membind(): failed for physical_device_id: 1 on NodeSet: [0] with errno: Operation not permitted (pid: 1 tid: 134777867474752) (cpuset_lib.cpp:440)
-2026-10-09 05:50:26.302 | warning | UMD | ---- ttSiliconDevice::init_hugepage: bind_area_to_memory_nodeset() failed (physical_device_id: 1 ch: 0). Hugepage allocation is not on NumaNode matching TT Device. Side-Effect is decreased Device->Host perf (Issue #893). (silicon_sysmem_manager.cpp:271)
-2026-10-09 05:50:26.302 | warning | UMD | hwloc_set_area_membind(): failed for physical_device_id: 1 on NodeSet: [0] with errno: Operation not permitted (pid: 1 tid: 134777867474752) (cpuset_lib.cpp:440)
-2026-10-09 05:50:26.302 | warning | UMD | ---- ttSiliconDevice::init_hugepage: bind_area_to_memory_nodeset() failed (physical_device_id: 1 ch: 1). Hugepage allocation is not on NumaNode matching TT Device. Side-Effect is decreased Device->Host perf (Issue #893). (silicon_sysmem_manager.cpp:271)
-2026-10-09 05:50:26.305 | info | UMD | Opening local chip ids/PCIe ids: {0}/[1] and remote chip ids {1} (cluster.cpp:173)
-2026-10-09 05:50:26.305 | info | UMD | IOMMU: disabled (cluster.cpp:147)
-2026-10-09 05:50:26.305 | info | UMD | KMD version: 2.5.0 (cluster.cpp:150)
-2026-10-09 05:50:26.305 | info | UMD | Cluster constructor completed. (cluster.cpp:648)
-2026-10-09 05:50:26.306 | info | UMD | Starting devices in cluster (cluster.cpp:1188)
-2026-10-09 05:50:26.310 | info | UMD | Starting devices in cluster completed. (cluster.cpp:1196)
-Tenstorrent devices=2; torch used for data/metrics=2.11.0+cpu
-Opening Tenstorrent device 0
-2026-10-09 05:50:26.386 | info | Distributed | Using auto discovery to generate mesh graph. (metal_env.cpp:437)
-2026-10-09 05:50:26.386 | info | Distributed | Constructing control plane using auto-discovery (no mesh graph descriptor). (metal_env.cpp:499)
-2026-10-09 05:50:26.386 | info | Fabric | Logical multi-mesh adjacency: intermesh degree histogram {0:1}; intra-mesh degree histograms mesh0 {1:2} (topology_mapper_utils.cpp:3329)
-2026-10-09 05:50:26.386 | info | Fabric | Physical multi-mesh adjacency: intermesh degree histogram {0:1}; intra-mesh degree histograms mesh0 {1:2} (topology_mapper_utils.cpp:3337)
-tt:0 step=0 mse=0.18213829
-tt:0 step=20 mse=0.00123700
-tt:0 step=40 mse=0.00011524
-tt:0 step=60 mse=0.00006454
-tt:0 step=80 mse=0.00006277
-{"backend": "tenstorrent", "device_id": 0, "initial_mse": 0.3193739652633667, "final_mse": 6.238709465833381e-05, "validation_mse": 0.00017507212760392576, "max_weight_change": 0.3046875, "checkpoint": "/checkpoints/mist-28ba381f47c609714d91aafb/tenstorrent-0.pt"}
-Opening Tenstorrent device 1
-2026-10-09 05:50:40.978 | info | Metal | Fabric config changed from FabricConfig::DISABLED to FabricConfig::FABRIC_1D, reinitializing control plane (metal_env.cpp:337)
-2026-10-09 05:50:40.978 | info | Distributed | Using auto discovery to generate mesh graph. (metal_env.cpp:437)
-2026-10-09 05:50:40.978 | info | Distributed | Constructing control plane using auto-discovery (no mesh graph descriptor). (metal_env.cpp:499)
-2026-10-09 05:50:40.978 | info | Fabric | Logical multi-mesh adjacency: intermesh degree histogram {0:1}; intra-mesh degree histograms mesh0 {1:2} (topology_mapper_utils.cpp:3329)
-2026-10-09 05:50:40.978 | info | Fabric | Physical multi-mesh adjacency: intermesh degree histogram {0:1}; intra-mesh degree histograms mesh0 {1:2} (topology_mapper_utils.cpp:3337)
-2026-10-09 05:50:40.979 | info | Metal | Enabling FabricConfig::FABRIC_1D only for dispatch. If your workload requires fabric, please set the fabric config accordingly. (device_manager.cpp:291)
-2026-10-09 05:50:40.979 | info | Metal | Dispatch on FabricConfig::FABRIC_1D with 1 Command Queues
- (device_manager.cpp:299)
-2026-10-09 05:50:40.980 | info | Metal | Initializing Fabric (fabric_firmware_initializer.cpp:296)
-2026-10-09 05:50:42.982 | info | Metal | Fabric initialized on 2 devices (fabric_firmware_initializer.cpp:444)
-2026-10-09 05:50:42.982 | info | Metal | Fabric Initialized with config FabricConfig::FABRIC_1D (fabric_firmware_initializer.cpp:313)
-tt:1 step=0 mse=0.18213829
-tt:1 step=20 mse=0.00123700
-tt:1 step=40 mse=0.00011524
-tt:1 step=60 mse=0.00006454
-tt:1 step=80 mse=0.00006277
-{"backend": "tenstorrent", "device_id": 1, "initial_mse": 0.3193739652633667, "final_mse": 6.238709465833381e-05, "validation_mse": 0.00017507212760392576, "max_weight_change": 0.3046875, "checkpoint": "/checkpoints/mist-28ba381f47c609714d91aafb/tenstorrent-1.pt"}
-TRAINING_PASSED backend=tenstorrent devices=2
-2026-10-09 05:50:56.847 | info | BuildKernels | JIT cache stats: 0/58 hits (0.0%) [0 cached, 10 build-once dedup, 0 merged artifacts, 0 merged genfiles] (build_cache_telemetry.cpp:207)
-2026-10-09 05:50:56.847 | info | BuildKernels | JIT telemetry: 2 registered TelemetryTokens (build_cache_telemetry.cpp:230)
-2026-10-09 05:50:56.852 | info | Device | Closing user mode device drivers (tt_cluster.cpp:513)
-2026-10-09 05:50:56.852 | info | UMD | Closing devices in cluster (cluster.cpp:1201)
-2026-10-09 05:50:56.860 | info | UMD | Closing devices in cluster completed. (cluster.cpp:1210)
-2026-10-09 05:50:56.860 | info | UMD | Cluster destructor started. (cluster.cpp:916)
-2026-10-09 05:50:56.860 | info | UMD | Cluster destructor completed. (cluster.cpp:919)
diff --git a/deploy/k3s/evidence/department-2026-10-09/mist-8644eccafaf0b42c3d4add05.log b/deploy/k3s/evidence/department-2026-10-09/mist-8644eccafaf0b42c3d4add05.log
deleted file mode 100644
index 3886ab9..0000000
--- a/deploy/k3s/evidence/department-2026-10-09/mist-8644eccafaf0b42c3d4add05.log
+++ /dev/null
@@ -1,18 +0,0 @@
-CUDA devices=2; torch=2.5.1+cu124
-Training on cuda:0: NVIDIA RTX A4000
-DATASET_LOADED file=/inputs/regression.json
-cuda:0 step=0 mse=0.08051129
-cuda:0 step=20 mse=0.00135718
-cuda:0 step=40 mse=0.00003812
-cuda:0 step=60 mse=0.00000139
-cuda:0 step=80 mse=0.00000006
-{"backend": "nvidia", "device_id": 0, "initial_mse": 0.08051128685474396, "final_mse": 2.4448332247573035e-09, "validation_mse": 5.969065242084071e-09, "max_weight_change": 0.09999077022075653, "checkpoint": "/outputs/nvidia-0.pt"}
-Training on cuda:1: NVIDIA RTX A4000
-DATASET_LOADED file=/inputs/regression.json
-cuda:1 step=0 mse=0.08051129
-cuda:1 step=20 mse=0.00135718
-cuda:1 step=40 mse=0.00003812
-cuda:1 step=60 mse=0.00000139
-cuda:1 step=80 mse=0.00000006
-{"backend": "nvidia", "device_id": 1, "initial_mse": 0.08051128685474396, "final_mse": 2.4448332247573035e-09, "validation_mse": 5.969065242084071e-09, "max_weight_change": 0.09999077022075653, "checkpoint": "/outputs/nvidia-1.pt"}
-TRAINING_PASSED backend=nvidia devices=2
diff --git a/deploy/k3s/evidence/department-2026-10-09/mist-aeba50ca436d70aa879381ce.log b/deploy/k3s/evidence/department-2026-10-09/mist-aeba50ca436d70aa879381ce.log
deleted file mode 100644
index 13c62c2..0000000
--- a/deploy/k3s/evidence/department-2026-10-09/mist-aeba50ca436d70aa879381ce.log
+++ /dev/null
@@ -1,56 +0,0 @@
-2026-10-09 05:50:26.124 | DEBUG | ttnn::78 - Initial ttnn.CONFIG:
-Config{cache_path=/root/.cache/ttnn,model_cache_path=/root/.cache/ttnn/models,tmp_dir=/tmp/ttnn,enable_model_cache=false,enable_fast_runtime_mode=true,validate_program_args=false,throw_exception_on_fallback=false,enable_logging=false,enable_graph_report=false,enable_graph_python_stack_traces=false,enable_detailed_buffer_report=false,enable_detailed_tensor_report=false,enable_comparison_mode=false,comparison_mode_should_raise_exception=false,comparison_mode_pcc=0.9999,root_report_path=generated/ttnn/reports,report_name=std::nullopt,std::nullopt}
-2026-10-09 05:50:26.323 | info | Device | Opening user mode device driver (tt_cluster.cpp:228)
-2026-10-09 05:50:26.323 | info | UMD | Cluster constructor started. (cluster.cpp:389)
-2026-10-09 05:50:26.324 | info | UMD | Creating TopologyDiscovery for architecture: wormhole_b0 (topology_discovery.cpp:96)
-2026-10-09 05:50:26.324 | info | UMD | Starting topology discovery. (topology_discovery.cpp:115)
-2026-10-09 05:50:26.325 | info | UMD | Established firmware bundle version: 19.4.2 (topology_discovery.cpp:575)
-2026-10-09 05:50:26.333 | info | UMD | Completed topology discovery. (topology_discovery.cpp:119)
-2026-10-09 05:50:26.366 | warning | UMD | hwloc_set_area_membind(): failed for physical_device_id: 2 on NodeSet: [0] with errno: Operation not permitted (pid: 1 tid: 125262848497472) (cpuset_lib.cpp:440)
-2026-10-09 05:50:26.366 | warning | UMD | ---- ttSiliconDevice::init_hugepage: bind_area_to_memory_nodeset() failed (physical_device_id: 2 ch: 0). Hugepage allocation is not on NumaNode matching TT Device. Side-Effect is decreased Device->Host perf (Issue #893). (silicon_sysmem_manager.cpp:271)
-2026-10-09 05:50:26.366 | warning | UMD | hwloc_set_area_membind(): failed for physical_device_id: 2 on NodeSet: [0] with errno: Operation not permitted (pid: 1 tid: 125262848497472) (cpuset_lib.cpp:440)
-2026-10-09 05:50:26.366 | warning | UMD | ---- ttSiliconDevice::init_hugepage: bind_area_to_memory_nodeset() failed (physical_device_id: 2 ch: 1). Hugepage allocation is not on NumaNode matching TT Device. Side-Effect is decreased Device->Host perf (Issue #893). (silicon_sysmem_manager.cpp:271)
-2026-10-09 05:50:26.369 | info | UMD | Opening local chip ids/PCIe ids: {0}/[2] and remote chip ids {1} (cluster.cpp:173)
-2026-10-09 05:50:26.369 | info | UMD | IOMMU: disabled (cluster.cpp:147)
-2026-10-09 05:50:26.369 | info | UMD | KMD version: 2.5.0 (cluster.cpp:150)
-2026-10-09 05:50:26.369 | info | UMD | Cluster constructor completed. (cluster.cpp:648)
-2026-10-09 05:50:26.371 | info | UMD | Starting devices in cluster (cluster.cpp:1188)
-2026-10-09 05:50:26.374 | info | UMD | Starting devices in cluster completed. (cluster.cpp:1196)
-Tenstorrent devices=2; torch used for data/metrics=2.11.0+cpu
-Opening Tenstorrent device 0
-2026-10-09 05:50:26.449 | info | Distributed | Using auto discovery to generate mesh graph. (metal_env.cpp:437)
-2026-10-09 05:50:26.449 | info | Distributed | Constructing control plane using auto-discovery (no mesh graph descriptor). (metal_env.cpp:499)
-2026-10-09 05:50:26.449 | info | Fabric | Logical multi-mesh adjacency: intermesh degree histogram {0:1}; intra-mesh degree histograms mesh0 {1:2} (topology_mapper_utils.cpp:3329)
-2026-10-09 05:50:26.449 | info | Fabric | Physical multi-mesh adjacency: intermesh degree histogram {0:1}; intra-mesh degree histograms mesh0 {1:2} (topology_mapper_utils.cpp:3337)
-tt:0 step=0 mse=0.18213829
-tt:0 step=20 mse=0.00123700
-tt:0 step=40 mse=0.00011524
-tt:0 step=60 mse=0.00006454
-tt:0 step=80 mse=0.00006277
-{"backend": "tenstorrent", "device_id": 0, "initial_mse": 0.3193739652633667, "final_mse": 6.238709465833381e-05, "validation_mse": 0.00017507212760392576, "max_weight_change": 0.3046875, "checkpoint": "/checkpoints/mist-aeba50ca436d70aa879381ce/tenstorrent-0.pt"}
-Opening Tenstorrent device 1
-2026-10-09 05:50:40.352 | info | Metal | Fabric config changed from FabricConfig::DISABLED to FabricConfig::FABRIC_1D, reinitializing control plane (metal_env.cpp:337)
-2026-10-09 05:50:40.352 | info | Distributed | Using auto discovery to generate mesh graph. (metal_env.cpp:437)
-2026-10-09 05:50:40.352 | info | Distributed | Constructing control plane using auto-discovery (no mesh graph descriptor). (metal_env.cpp:499)
-2026-10-09 05:50:40.352 | info | Fabric | Logical multi-mesh adjacency: intermesh degree histogram {0:1}; intra-mesh degree histograms mesh0 {1:2} (topology_mapper_utils.cpp:3329)
-2026-10-09 05:50:40.352 | info | Fabric | Physical multi-mesh adjacency: intermesh degree histogram {0:1}; intra-mesh degree histograms mesh0 {1:2} (topology_mapper_utils.cpp:3337)
-2026-10-09 05:50:40.353 | info | Metal | Enabling FabricConfig::FABRIC_1D only for dispatch. If your workload requires fabric, please set the fabric config accordingly. (device_manager.cpp:291)
-2026-10-09 05:50:40.353 | info | Metal | Dispatch on FabricConfig::FABRIC_1D with 1 Command Queues
- (device_manager.cpp:299)
-2026-10-09 05:50:40.354 | info | Metal | Initializing Fabric (fabric_firmware_initializer.cpp:296)
-2026-10-09 05:50:42.326 | info | Metal | Fabric initialized on 2 devices (fabric_firmware_initializer.cpp:444)
-2026-10-09 05:50:42.326 | info | Metal | Fabric Initialized with config FabricConfig::FABRIC_1D (fabric_firmware_initializer.cpp:313)
-tt:1 step=0 mse=0.18213829
-tt:1 step=20 mse=0.00123700
-tt:1 step=40 mse=0.00011524
-tt:1 step=60 mse=0.00006454
-tt:1 step=80 mse=0.00006277
-{"backend": "tenstorrent", "device_id": 1, "initial_mse": 0.3193739652633667, "final_mse": 6.238709465833381e-05, "validation_mse": 0.00017507212760392576, "max_weight_change": 0.3046875, "checkpoint": "/checkpoints/mist-aeba50ca436d70aa879381ce/tenstorrent-1.pt"}
-TRAINING_PASSED backend=tenstorrent devices=2
-2026-10-09 05:50:56.182 | info | BuildKernels | JIT cache stats: 0/58 hits (0.0%) [0 cached, 10 build-once dedup, 0 merged artifacts, 0 merged genfiles] (build_cache_telemetry.cpp:207)
-2026-10-09 05:50:56.182 | info | BuildKernels | JIT telemetry: 2 registered TelemetryTokens (build_cache_telemetry.cpp:230)
-2026-10-09 05:50:56.187 | info | Device | Closing user mode device drivers (tt_cluster.cpp:513)
-2026-10-09 05:50:56.187 | info | UMD | Closing devices in cluster (cluster.cpp:1201)
-2026-10-09 05:50:56.196 | info | UMD | Closing devices in cluster completed. (cluster.cpp:1210)
-2026-10-09 05:50:56.196 | info | UMD | Cluster destructor started. (cluster.cpp:916)
-2026-10-09 05:50:56.196 | info | UMD | Cluster destructor completed. (cluster.cpp:919)
diff --git a/deploy/k3s/evidence/department-2026-10-09/mist-b46534ddeb9f18852b75f645.log b/deploy/k3s/evidence/department-2026-10-09/mist-b46534ddeb9f18852b75f645.log
deleted file mode 100644
index 74e595e..0000000
--- a/deploy/k3s/evidence/department-2026-10-09/mist-b46534ddeb9f18852b75f645.log
+++ /dev/null
@@ -1 +0,0 @@
-ISOLATION_PASSED
diff --git a/deploy/k3s/evidence/department-2026-10-09/mist-e5c04ea2cc126a786d6fe783.log b/deploy/k3s/evidence/department-2026-10-09/mist-e5c04ea2cc126a786d6fe783.log
deleted file mode 100644
index aaeffb1..0000000
--- a/deploy/k3s/evidence/department-2026-10-09/mist-e5c04ea2cc126a786d6fe783.log
+++ /dev/null
@@ -1,58 +0,0 @@
-2026-10-09 05:24:57.648 | DEBUG | ttnn::78 - Initial ttnn.CONFIG:
-Config{cache_path=/root/.cache/ttnn,model_cache_path=/root/.cache/ttnn/models,tmp_dir=/tmp/ttnn,enable_model_cache=false,enable_fast_runtime_mode=true,validate_program_args=false,throw_exception_on_fallback=false,enable_logging=false,enable_graph_report=false,enable_graph_python_stack_traces=false,enable_detailed_buffer_report=false,enable_detailed_tensor_report=false,enable_comparison_mode=false,comparison_mode_should_raise_exception=false,comparison_mode_pcc=0.9999,root_report_path=generated/ttnn/reports,report_name=std::nullopt,std::nullopt}
-2026-10-09 05:24:57.853 | info | Device | Opening user mode device driver (tt_cluster.cpp:228)
-2026-10-09 05:24:57.853 | info | UMD | Cluster constructor started. (cluster.cpp:389)
-2026-10-09 05:24:57.854 | info | UMD | Creating TopologyDiscovery for architecture: wormhole_b0 (topology_discovery.cpp:96)
-2026-10-09 05:24:57.854 | info | UMD | Starting topology discovery. (topology_discovery.cpp:115)
-2026-10-09 05:24:57.856 | info | UMD | Established firmware bundle version: 19.4.2 (topology_discovery.cpp:575)
-2026-10-09 05:24:57.863 | info | UMD | Completed topology discovery. (topology_discovery.cpp:119)
-2026-10-09 05:24:57.900 | warning | UMD | hwloc_set_area_membind(): failed for physical_device_id: 1 on NodeSet: [0] with errno: Operation not permitted (pid: 1 tid: 136277795993408) (cpuset_lib.cpp:440)
-2026-10-09 05:24:57.900 | warning | UMD | ---- ttSiliconDevice::init_hugepage: bind_area_to_memory_nodeset() failed (physical_device_id: 1 ch: 0). Hugepage allocation is not on NumaNode matching TT Device. Side-Effect is decreased Device->Host perf (Issue #893). (silicon_sysmem_manager.cpp:271)
-2026-10-09 05:24:57.900 | warning | UMD | hwloc_set_area_membind(): failed for physical_device_id: 1 on NodeSet: [0] with errno: Operation not permitted (pid: 1 tid: 136277795993408) (cpuset_lib.cpp:440)
-2026-10-09 05:24:57.900 | warning | UMD | ---- ttSiliconDevice::init_hugepage: bind_area_to_memory_nodeset() failed (physical_device_id: 1 ch: 1). Hugepage allocation is not on NumaNode matching TT Device. Side-Effect is decreased Device->Host perf (Issue #893). (silicon_sysmem_manager.cpp:271)
-2026-10-09 05:24:57.903 | info | UMD | Opening local chip ids/PCIe ids: {0}/[1] and remote chip ids {1} (cluster.cpp:173)
-2026-10-09 05:24:57.903 | info | UMD | IOMMU: disabled (cluster.cpp:147)
-2026-10-09 05:24:57.903 | info | UMD | KMD version: 2.5.0 (cluster.cpp:150)
-2026-10-09 05:24:57.903 | info | UMD | Cluster constructor completed. (cluster.cpp:648)
-2026-10-09 05:24:57.904 | info | UMD | Starting devices in cluster (cluster.cpp:1188)
-2026-10-09 05:24:57.908 | info | UMD | Starting devices in cluster completed. (cluster.cpp:1196)
-Tenstorrent devices=2; torch used for data/metrics=2.11.0+cpu
-Opening Tenstorrent device 0
-2026-10-09 05:24:57.985 | info | Distributed | Using auto discovery to generate mesh graph. (metal_env.cpp:437)
-2026-10-09 05:24:57.985 | info | Distributed | Constructing control plane using auto-discovery (no mesh graph descriptor). (metal_env.cpp:499)
-2026-10-09 05:24:57.986 | info | Fabric | Logical multi-mesh adjacency: intermesh degree histogram {0:1}; intra-mesh degree histograms mesh0 {1:2} (topology_mapper_utils.cpp:3329)
-2026-10-09 05:24:57.986 | info | Fabric | Physical multi-mesh adjacency: intermesh degree histogram {0:1}; intra-mesh degree histograms mesh0 {1:2} (topology_mapper_utils.cpp:3337)
-DATASET_LOADED file=/inputs/regression.json
-tt:0 step=0 mse=0.06471165
-tt:0 step=20 mse=0.00108452
-tt:0 step=40 mse=0.00004048
-tt:0 step=60 mse=0.00003227
-tt:0 step=80 mse=0.00003213
-{"backend": "tenstorrent", "device_id": 0, "initial_mse": 0.08051129430532455, "final_mse": 3.2111081964103505e-05, "validation_mse": 5.255065480014309e-05, "max_weight_change": 0.0986328125, "checkpoint": "/outputs/tenstorrent-0.pt"}
-Opening Tenstorrent device 1
-2026-10-09 05:25:13.424 | info | Metal | Fabric config changed from FabricConfig::DISABLED to FabricConfig::FABRIC_1D, reinitializing control plane (metal_env.cpp:337)
-2026-10-09 05:25:13.424 | info | Distributed | Using auto discovery to generate mesh graph. (metal_env.cpp:437)
-2026-10-09 05:25:13.424 | info | Distributed | Constructing control plane using auto-discovery (no mesh graph descriptor). (metal_env.cpp:499)
-2026-10-09 05:25:13.424 | info | Fabric | Logical multi-mesh adjacency: intermesh degree histogram {0:1}; intra-mesh degree histograms mesh0 {1:2} (topology_mapper_utils.cpp:3329)
-2026-10-09 05:25:13.424 | info | Fabric | Physical multi-mesh adjacency: intermesh degree histogram {0:1}; intra-mesh degree histograms mesh0 {1:2} (topology_mapper_utils.cpp:3337)
-2026-10-09 05:25:13.425 | info | Metal | Enabling FabricConfig::FABRIC_1D only for dispatch. If your workload requires fabric, please set the fabric config accordingly. (device_manager.cpp:291)
-2026-10-09 05:25:13.425 | info | Metal | Dispatch on FabricConfig::FABRIC_1D with 1 Command Queues
- (device_manager.cpp:299)
-2026-10-09 05:25:13.427 | info | Metal | Initializing Fabric (fabric_firmware_initializer.cpp:296)
-2026-10-09 05:25:15.409 | info | Metal | Fabric initialized on 2 devices (fabric_firmware_initializer.cpp:444)
-2026-10-09 05:25:15.409 | info | Metal | Fabric Initialized with config FabricConfig::FABRIC_1D (fabric_firmware_initializer.cpp:313)
-DATASET_LOADED file=/inputs/regression.json
-tt:1 step=0 mse=0.06471165
-tt:1 step=20 mse=0.00108452
-tt:1 step=40 mse=0.00004048
-tt:1 step=60 mse=0.00003227
-tt:1 step=80 mse=0.00003213
-{"backend": "tenstorrent", "device_id": 1, "initial_mse": 0.08051129430532455, "final_mse": 3.2111081964103505e-05, "validation_mse": 5.255065480014309e-05, "max_weight_change": 0.0986328125, "checkpoint": "/outputs/tenstorrent-1.pt"}
-TRAINING_PASSED backend=tenstorrent devices=2
-2026-10-09 05:25:29.422 | info | BuildKernels | JIT cache stats: 0/58 hits (0.0%) [0 cached, 10 build-once dedup, 0 merged artifacts, 0 merged genfiles] (build_cache_telemetry.cpp:207)
-2026-10-09 05:25:29.422 | info | BuildKernels | JIT telemetry: 2 registered TelemetryTokens (build_cache_telemetry.cpp:230)
-2026-10-09 05:25:29.427 | info | Device | Closing user mode device drivers (tt_cluster.cpp:513)
-2026-10-09 05:25:29.427 | info | UMD | Closing devices in cluster (cluster.cpp:1201)
-2026-10-09 05:25:29.436 | info | UMD | Closing devices in cluster completed. (cluster.cpp:1210)
-2026-10-09 05:25:29.436 | info | UMD | Cluster destructor started. (cluster.cpp:916)
-2026-10-09 05:25:29.436 | info | UMD | Cluster destructor completed. (cluster.cpp:919)
diff --git a/deploy/k3s/evidence/department-2026-10-09/nvidia-allocation.json b/deploy/k3s/evidence/department-2026-10-09/nvidia-allocation.json
deleted file mode 100644
index d428085..0000000
--- a/deploy/k3s/evidence/department-2026-10-09/nvidia-allocation.json
+++ /dev/null
@@ -1,7 +0,0 @@
-{
- "device_nodes": [
- "/dev/nvidia0",
- "/dev/nvidia1"
- ],
- "gpu_count": 2
-}
diff --git a/deploy/k3s/evidence/department-2026-10-09/nvidia-results.json b/deploy/k3s/evidence/department-2026-10-09/nvidia-results.json
deleted file mode 100644
index c301509..0000000
--- a/deploy/k3s/evidence/department-2026-10-09/nvidia-results.json
+++ /dev/null
@@ -1,36 +0,0 @@
-[
- {
- "backend": "nvidia",
- "device_id": 0,
- "initial_mse": 0.08051128685474396,
- "final_mse": 2.4448332247573035e-09,
- "validation_mse": 5.969065242084071e-09,
- "max_weight_change": 0.09999077022075653,
- "checkpoint": "/outputs/nvidia-0.pt",
- "checkpoint_reload_mse": 5.969065242084071e-09,
- "device_name": "NVIDIA RTX A4000",
- "started_at": "2026-10-09T05:24:44.711442+00:00",
- "finished_at": "2026-10-09T05:24:45.079501+00:00",
- "device_nodes": [
- "/dev/nvidia0",
- "/dev/nvidia1"
- ]
- },
- {
- "backend": "nvidia",
- "device_id": 1,
- "initial_mse": 0.08051128685474396,
- "final_mse": 2.4448332247573035e-09,
- "validation_mse": 5.969065242084071e-09,
- "max_weight_change": 0.09999077022075653,
- "checkpoint": "/outputs/nvidia-1.pt",
- "checkpoint_reload_mse": 5.969065242084071e-09,
- "device_name": "NVIDIA RTX A4000",
- "started_at": "2026-10-09T05:24:45.310209+00:00",
- "finished_at": "2026-10-09T05:24:45.517661+00:00",
- "device_nodes": [
- "/dev/nvidia0",
- "/dev/nvidia1"
- ]
- }
-]
diff --git a/deploy/k3s/evidence/department-2026-10-09/portal-firewall.log b/deploy/k3s/evidence/department-2026-10-09/portal-firewall.log
deleted file mode 100644
index 1ad75d5..0000000
--- a/deploy/k3s/evidence/department-2026-10-09/portal-firewall.log
+++ /dev/null
@@ -1,5 +0,0 @@
-PASS untrusted source reaches portal
-PASS untrusted source denied 100.73.139.66:22
-PASS untrusted source denied 100.73.139.66:6443
-PASS untrusted source denied 100.73.139.66:30491
-PASS untrusted source denied 10.43.1.116:3000
diff --git a/deploy/k3s/evidence/department-2026-10-09/release-validation.json b/deploy/k3s/evidence/department-2026-10-09/release-validation.json
deleted file mode 100644
index bee973b..0000000
--- a/deploy/k3s/evidence/department-2026-10-09/release-validation.json
+++ /dev/null
@@ -1,101 +0,0 @@
-{
- "recorded_at": "2026-10-09T07:59:10.227717+00:00",
- "source_branch": "feat/local-job-foundation",
- "suites": {
- "backend": {
- "passed": true,
- "test_count": 33,
- "skipped": 0,
- "tests": [
- "TestContainerJobCPU",
- "TestRunContainerCPUIntegration",
- "TestHardwareCountsExternalScheduledPodsAndDRAReservations",
- "TestHardwareDoesNotAdvertiseOfflineOrIncompleteInventory",
- "TestGPURequestIncludesInitPeakAndSidecars",
- "TestHardwarePermissionFailureIsAnError",
- "TestCustomImagePreservesEntrypointAndWorkingDirectory",
- "TestTenstorrentCustomImageRestrictionIsInCatalogAndValidation",
- "TestIntegration",
- "TestDummySupervisors",
- "TestStatusRegistry_BasicOperations",
- "TestJobEnqueueAndSupervisor",
- "TestKubernetesRejectInvalidSubmission",
- "TestKubernetesReportsActualExitBeforeControllerCompletion",
- "TestKubernetesReportsCurrentWaitingReasonAfterPlacement",
- "TestKubernetesCancellationPersistsAcrossExecutorRestart",
- "TestKubernetesDoesNotExposeUnmanagedJobs",
- "TestKubernetesHTTPRejectsUnknownAndMultipleObjects",
- "TestSharedDatasetAtomicLimitsAndOwnership",
- "TestSharedDatasetConfiguredCeilingAndMissingStorage",
- "TestSharedZipRejectsTraversalSymlinksAndExpansion",
- "TestSharedJobMountAndEscapingOutput",
- "TestMemberAuthIsolationAndFailureClosed",
- "TestDepartmentTeamPermissionsAndScopedSharing",
- "TestDepartmentPolicyAndStorageBudget",
- "TestDepartmentDurableFairAdmission",
- "TestDepartmentAgedHeadDrainsCapacityWithoutStarvation",
- "TestDepartmentDemotionRevokesAdministratorWritePrivilege",
- "TestMemberProxyAddressCannotBeSpoofed",
- "TestDepartmentPendingPodClaimDoesNotDoubleReserveBoards",
- "TestDepartmentSplitDatasetAndModelPaths",
- "TestDepartmentMissingModelMountCannotBorrowDatasetCapacity",
- "TestDepartmentDefaultPoolsAndLegacyPolicy"
- ]
- },
- "docker_module": {
- "passed": true,
- "test_count": 10,
- "skipped": 0,
- "tests": [
- "TestCreateDeleteVolume",
- "TestCreateVolumeTwice",
- "TestRemoveNonexistentVolume",
- "TestRunContainerCPU",
- "TestRemoveVolumeInUse",
- "TestAttachNonexistentVolume",
- "TestTwoContainersSameVolume",
- "TestTwoContainersSameVolumeConcurrent",
- "TestVolumeLimit",
- "TestContainerLimit"
- ]
- }
- },
- "other_checks": {
- "cli_go_tests": "passed",
- "logger_go_tests": "passed",
- "react_tests": {
- "passed": true,
- "test_count": 10
- },
- "typescript": "passed",
- "full_frontend_eslint": "passed",
- "production_frontend_build": "passed",
- "migration_tests": {
- "passed": true,
- "test_count": 6
- },
- "live_browser": {
- "passed": true,
- "check_count": 11,
- "browser_errors": []
- },
- "live_cluster": {
- "nodes_ready": 2,
- "api_available_replicas": 1,
- "auth_available_replicas": 1
- }
- },
- "legacy_test_isolation": "Dedicated Redis container with no published ports; preview Redis and data preserved. Existing CPU Docker fixtures ran against host Docker; test containers and volumes removed.",
- "live_accelerator_evidence": [
- "split-storage-results.json",
- "runtime-results.json",
- "container-mode-result.json"
- ],
- "large_upload_or_exhaustion_repeated": false,
- "limitations": [
- "No 200/500 GiB transfer stress or exhaustive edge-case claim",
- "Custom CUDA demonstration image preloaded locally; no registry push tested in that demonstration",
- "Arbitrary research model compatibility, high availability and disk-failure recovery not verified",
- "Features explicitly excluded by the user are outside this release"
- ]
-}
diff --git a/deploy/k3s/evidence/department-2026-10-09/remote-cli-job.log b/deploy/k3s/evidence/department-2026-10-09/remote-cli-job.log
deleted file mode 100644
index 6a0ac8c..0000000
--- a/deploy/k3s/evidence/department-2026-10-09/remote-cli-job.log
+++ /dev/null
@@ -1 +0,0 @@
-REMOTE_CLI_NVIDIA_PASSED devices=1
diff --git a/deploy/k3s/evidence/department-2026-10-09/remote-cli-results.json b/deploy/k3s/evidence/department-2026-10-09/remote-cli-results.json
deleted file mode 100644
index bdcdcd4..0000000
--- a/deploy/k3s/evidence/department-2026-10-09/remote-cli-results.json
+++ /dev/null
@@ -1,9 +0,0 @@
-{
- "origin_node": "utmist-tt",
- "destination_node": "utmist-z1opa08",
- "team_id": "team-db3933031a6c2437",
- "login": true,
- "team_selection": true,
- "job_id": "mist-7d426bc89696c01e19474c00",
- "job_succeeded": true
-}
diff --git a/deploy/k3s/evidence/department-2026-10-09/remote-cli-status.txt b/deploy/k3s/evidence/department-2026-10-09/remote-cli-status.txt
deleted file mode 100644
index b87a496..0000000
--- a/deploy/k3s/evidence/department-2026-10-09/remote-cli-status.txt
+++ /dev/null
@@ -1,3 +0,0 @@
-Job ID Name Status Compute Devices Node Created
-mist-7d426bc89696c01e19474c00 remote-cli-nvidia Success nvidia 1 utmist-z1opa08 2026-10-09T05:39:46Z
-Exit code: 0
diff --git a/deploy/k3s/evidence/department-2026-10-09/results.json b/deploy/k3s/evidence/department-2026-10-09/results.json
deleted file mode 100644
index fa86627..0000000
--- a/deploy/k3s/evidence/department-2026-10-09/results.json
+++ /dev/null
@@ -1,87 +0,0 @@
-{
- "started": "2026-10-09T05:22:29.799Z",
- "site": "http://100.73.139.66:8088",
- "checks": {
- "team_filesystems_ready": true,
- "team_read_write_and_cross_team_denial": true,
- "scoped_readonly_sharing_and_running_job_revocation": true,
- "durable_fifo_team_concurrency_fair_turns_and_cancel": true,
- "policy_limits_and_legacy_bypass_denied": true,
- "live_container_private_network_and_credentials_denied": true,
- "training_writes_obey_filesystem_capacity": true,
- "nvidia_real_training_uploaded_data_and_downloaded_weights": true,
- "tenstorrent_real_training_uploaded_data_and_downloaded_weights": true,
- "account_deactivation_stops_running_jobs": true,
- "browser_rows_expand_and_saved_files_open": true,
- "browser_admin_workspace_and_shared_folders": true,
- "online_team_storage_growth": true,
- "legacy_history_preserved": true
- },
- "jobs": [
- {
- "id": "mist-75fac1f6c8220c1a3cf11109",
- "name": "Read-only shared input",
- "team": "team-1416c881530c4ea5"
- },
- {
- "id": "mist-53ad27bd38ebb05d81fd6035",
- "name": "Revoke shared input",
- "team": "team-1416c881530c4ea5"
- },
- {
- "id": "mist-29df16c58727bed96f5d4ed5",
- "name": "Queue holds capacity",
- "team": "team-db3933031a6c2437"
- },
- {
- "id": "mist-f27e6e3cc96da7e8541a6752",
- "name": "Queue survives restart",
- "team": "team-db3933031a6c2437"
- },
- {
- "id": "mist-835e04d572467f624e6724e6",
- "name": "Other team gets a turn",
- "team": "team-1416c881530c4ea5"
- },
- {
- "id": "mist-b46534ddeb9f18852b75f645",
- "name": "Workload network isolation",
- "team": "team-db3933031a6c2437"
- },
- {
- "id": "mist-1c4a852d366b9d2349e1cf7c",
- "name": "Filesystem capacity enforcement",
- "team": "team-1416c881530c4ea5"
- },
- {
- "id": "mist-8644eccafaf0b42c3d4add05",
- "name": "Department training nvidia",
- "team": "team-db3933031a6c2437",
- "node": "utmist-z1opa08"
- },
- {
- "id": "mist-e5c04ea2cc126a786d6fe783",
- "name": "Department training tenstorrent",
- "team": "team-db3933031a6c2437",
- "node": "utmist-tt"
- },
- {
- "id": "mist-5aa3c1f395918b993de4e076",
- "name": "Account deactivation releases job",
- "team": "team-1416c881530c4ea5"
- }
- ],
- "teams": [
- "team-db3933031a6c2437",
- "team-1416c881530c4ea5"
- ],
- "legacy_job_count_before": 47,
- "storage": {
- "available_bytes": 994050048,
- "capacity_bytes": 1021313024,
- "enabled": true,
- "upload_limit_bytes": 2147483648
- },
- "browser_errors": [],
- "finished": "2026-10-09T05:25:50.157Z"
-}
\ No newline at end of file
diff --git a/deploy/k3s/evidence/department-2026-10-09/runtime-results.json b/deploy/k3s/evidence/department-2026-10-09/runtime-results.json
deleted file mode 100644
index 4d5b421..0000000
--- a/deploy/k3s/evidence/department-2026-10-09/runtime-results.json
+++ /dev/null
@@ -1,41 +0,0 @@
-{
- "checks": {
- "concurrent_tt_jobs_distinct_boards_real_training": true,
- "nonzero_exit_visible_no_retry": true,
- "execution_deadline_failure_visible": true
- },
- "jobs": [
- {
- "id": "mist-28ba381f47c609714d91aafb",
- "name": "Concurrent TT board A"
- },
- {
- "id": "mist-aeba50ca436d70aa879381ce",
- "name": "Concurrent TT board B"
- },
- {
- "id": "mist-613d2049819ee13de76a57f1",
- "name": "Intentional failure"
- },
- {
- "id": "mist-34416e68372f1bb5048f40b9",
- "name": "Execution deadline"
- }
- ],
- "allocations": {
- "mist-28ba381f47c609714d91aafb": [
- {
- "driver": "tenstorrent.com",
- "pool": "utmist-tt",
- "device": "tt-0"
- }
- ],
- "mist-aeba50ca436d70aa879381ce": [
- {
- "driver": "tenstorrent.com",
- "pool": "utmist-tt",
- "device": "tt-1"
- }
- ]
- }
-}
diff --git a/deploy/k3s/evidence/department-2026-10-09/split-nvidia.log b/deploy/k3s/evidence/department-2026-10-09/split-nvidia.log
deleted file mode 100644
index 3886ab9..0000000
--- a/deploy/k3s/evidence/department-2026-10-09/split-nvidia.log
+++ /dev/null
@@ -1,18 +0,0 @@
-CUDA devices=2; torch=2.5.1+cu124
-Training on cuda:0: NVIDIA RTX A4000
-DATASET_LOADED file=/inputs/regression.json
-cuda:0 step=0 mse=0.08051129
-cuda:0 step=20 mse=0.00135718
-cuda:0 step=40 mse=0.00003812
-cuda:0 step=60 mse=0.00000139
-cuda:0 step=80 mse=0.00000006
-{"backend": "nvidia", "device_id": 0, "initial_mse": 0.08051128685474396, "final_mse": 2.4448332247573035e-09, "validation_mse": 5.969065242084071e-09, "max_weight_change": 0.09999077022075653, "checkpoint": "/outputs/nvidia-0.pt"}
-Training on cuda:1: NVIDIA RTX A4000
-DATASET_LOADED file=/inputs/regression.json
-cuda:1 step=0 mse=0.08051129
-cuda:1 step=20 mse=0.00135718
-cuda:1 step=40 mse=0.00003812
-cuda:1 step=60 mse=0.00000139
-cuda:1 step=80 mse=0.00000006
-{"backend": "nvidia", "device_id": 1, "initial_mse": 0.08051128685474396, "final_mse": 2.4448332247573035e-09, "validation_mse": 5.969065242084071e-09, "max_weight_change": 0.09999077022075653, "checkpoint": "/outputs/nvidia-1.pt"}
-TRAINING_PASSED backend=nvidia devices=2
diff --git a/deploy/k3s/evidence/department-2026-10-09/split-storage-results.json b/deploy/k3s/evidence/department-2026-10-09/split-storage-results.json
deleted file mode 100644
index 4958668..0000000
--- a/deploy/k3s/evidence/department-2026-10-09/split-storage-results.json
+++ /dev/null
@@ -1,37 +0,0 @@
-{
- "checks": {
- "both_pools_ready_with_independent_capacities": true,
- "both_shrinks_and_combined_overcommit_rejected_without_mutation": true,
- "old_accelerator_checkpoints_preserved_byte_for_byte": true,
- "small_dataset_uploaded_with_verified_checksum": true,
- "nvidia_uploaded_input_readonly_and_models_saved_in_separate_pvc": true,
- "tenstorrent_uploaded_input_readonly_and_models_saved_in_separate_pvc": true,
- "own_small_probe_removed_after_terminal_jobs": true
- },
- "jobs": [
- {
- "id": "mist-976551958cfdf6a5ad42427c",
- "backend": "nvidia",
- "node": "utmist-z1opa08"
- },
- {
- "id": "mist-68ba775f5254308450de267c",
- "backend": "tenstorrent",
- "node": "utmist-tt"
- }
- ],
- "bulk_testing": false,
- "storage": {
- "available_bytes": 2049966080,
- "capacity_bytes": 2077229056,
- "dataset_allocation_gib": 2,
- "enabled": true,
- "max_dataset_bytes": 1125899906842624,
- "model_allocation_gib": 2,
- "model_available_bytes": 2002780160,
- "model_capacity_bytes": 2040528896,
- "upload_limit_bytes": 1982857216
- },
- "preserved_checkpoint_files": 8,
- "dataset_bytes": 260729
-}
diff --git a/deploy/k3s/evidence/department-2026-10-09/split-tenstorrent.log b/deploy/k3s/evidence/department-2026-10-09/split-tenstorrent.log
deleted file mode 100644
index 6272129..0000000
--- a/deploy/k3s/evidence/department-2026-10-09/split-tenstorrent.log
+++ /dev/null
@@ -1,58 +0,0 @@
-2026-10-09 07:05:33.186 | DEBUG | ttnn::78 - Initial ttnn.CONFIG:
-Config{cache_path=/root/.cache/ttnn,model_cache_path=/root/.cache/ttnn/models,tmp_dir=/tmp/ttnn,enable_model_cache=false,enable_fast_runtime_mode=true,validate_program_args=false,throw_exception_on_fallback=false,enable_logging=false,enable_graph_report=false,enable_graph_python_stack_traces=false,enable_detailed_buffer_report=false,enable_detailed_tensor_report=false,enable_comparison_mode=false,comparison_mode_should_raise_exception=false,comparison_mode_pcc=0.9999,root_report_path=generated/ttnn/reports,report_name=std::nullopt,std::nullopt}
-2026-10-09 07:05:33.388 | info | Device | Opening user mode device driver (tt_cluster.cpp:228)
-2026-10-09 07:05:33.388 | info | UMD | Cluster constructor started. (cluster.cpp:389)
-2026-10-09 07:05:33.388 | info | UMD | Creating TopologyDiscovery for architecture: wormhole_b0 (topology_discovery.cpp:96)
-2026-10-09 07:05:33.388 | info | UMD | Starting topology discovery. (topology_discovery.cpp:115)
-2026-10-09 07:05:33.389 | info | UMD | Established firmware bundle version: 19.4.2 (topology_discovery.cpp:575)
-2026-10-09 07:05:33.397 | info | UMD | Completed topology discovery. (topology_discovery.cpp:119)
-2026-10-09 07:05:33.433 | warning | UMD | hwloc_set_area_membind(): failed for physical_device_id: 1 on NodeSet: [0] with errno: Operation not permitted (pid: 1 tid: 130452827985728) (cpuset_lib.cpp:440)
-2026-10-09 07:05:33.433 | warning | UMD | ---- ttSiliconDevice::init_hugepage: bind_area_to_memory_nodeset() failed (physical_device_id: 1 ch: 0). Hugepage allocation is not on NumaNode matching TT Device. Side-Effect is decreased Device->Host perf (Issue #893). (silicon_sysmem_manager.cpp:271)
-2026-10-09 07:05:33.433 | warning | UMD | hwloc_set_area_membind(): failed for physical_device_id: 1 on NodeSet: [0] with errno: Operation not permitted (pid: 1 tid: 130452827985728) (cpuset_lib.cpp:440)
-2026-10-09 07:05:33.433 | warning | UMD | ---- ttSiliconDevice::init_hugepage: bind_area_to_memory_nodeset() failed (physical_device_id: 1 ch: 1). Hugepage allocation is not on NumaNode matching TT Device. Side-Effect is decreased Device->Host perf (Issue #893). (silicon_sysmem_manager.cpp:271)
-2026-10-09 07:05:33.436 | info | UMD | Opening local chip ids/PCIe ids: {0}/[1] and remote chip ids {1} (cluster.cpp:173)
-2026-10-09 07:05:33.436 | info | UMD | IOMMU: disabled (cluster.cpp:147)
-2026-10-09 07:05:33.436 | info | UMD | KMD version: 2.5.0 (cluster.cpp:150)
-2026-10-09 07:05:33.436 | info | UMD | Cluster constructor completed. (cluster.cpp:648)
-2026-10-09 07:05:33.438 | info | UMD | Starting devices in cluster (cluster.cpp:1188)
-2026-10-09 07:05:33.441 | info | UMD | Starting devices in cluster completed. (cluster.cpp:1196)
-Tenstorrent devices=2; torch used for data/metrics=2.11.0+cpu
-Opening Tenstorrent device 0
-2026-10-09 07:05:33.515 | info | Distributed | Using auto discovery to generate mesh graph. (metal_env.cpp:437)
-2026-10-09 07:05:33.515 | info | Distributed | Constructing control plane using auto-discovery (no mesh graph descriptor). (metal_env.cpp:499)
-2026-10-09 07:05:33.516 | info | Fabric | Logical multi-mesh adjacency: intermesh degree histogram {0:1}; intra-mesh degree histograms mesh0 {1:2} (topology_mapper_utils.cpp:3329)
-2026-10-09 07:05:33.516 | info | Fabric | Physical multi-mesh adjacency: intermesh degree histogram {0:1}; intra-mesh degree histograms mesh0 {1:2} (topology_mapper_utils.cpp:3337)
-DATASET_LOADED file=/inputs/regression.json
-tt:0 step=0 mse=0.06471165
-tt:0 step=20 mse=0.00108452
-tt:0 step=40 mse=0.00004048
-tt:0 step=60 mse=0.00003227
-tt:0 step=80 mse=0.00003213
-{"backend": "tenstorrent", "device_id": 0, "initial_mse": 0.08051129430532455, "final_mse": 3.2111081964103505e-05, "validation_mse": 5.255065480014309e-05, "max_weight_change": 0.0986328125, "checkpoint": "/outputs/tenstorrent-0.pt"}
-Opening Tenstorrent device 1
-2026-10-09 07:05:48.403 | info | Metal | Fabric config changed from FabricConfig::DISABLED to FabricConfig::FABRIC_1D, reinitializing control plane (metal_env.cpp:337)
-2026-10-09 07:05:48.403 | info | Distributed | Using auto discovery to generate mesh graph. (metal_env.cpp:437)
-2026-10-09 07:05:48.403 | info | Distributed | Constructing control plane using auto-discovery (no mesh graph descriptor). (metal_env.cpp:499)
-2026-10-09 07:05:48.404 | info | Fabric | Logical multi-mesh adjacency: intermesh degree histogram {0:1}; intra-mesh degree histograms mesh0 {1:2} (topology_mapper_utils.cpp:3329)
-2026-10-09 07:05:48.404 | info | Fabric | Physical multi-mesh adjacency: intermesh degree histogram {0:1}; intra-mesh degree histograms mesh0 {1:2} (topology_mapper_utils.cpp:3337)
-2026-10-09 07:05:48.405 | info | Metal | Enabling FabricConfig::FABRIC_1D only for dispatch. If your workload requires fabric, please set the fabric config accordingly. (device_manager.cpp:291)
-2026-10-09 07:05:48.405 | info | Metal | Dispatch on FabricConfig::FABRIC_1D with 1 Command Queues
- (device_manager.cpp:299)
-2026-10-09 07:05:48.406 | info | Metal | Initializing Fabric (fabric_firmware_initializer.cpp:296)
-2026-10-09 07:05:50.399 | info | Metal | Fabric initialized on 2 devices (fabric_firmware_initializer.cpp:444)
-2026-10-09 07:05:50.399 | info | Metal | Fabric Initialized with config FabricConfig::FABRIC_1D (fabric_firmware_initializer.cpp:313)
-DATASET_LOADED file=/inputs/regression.json
-tt:1 step=0 mse=0.06471165
-tt:1 step=20 mse=0.00108452
-tt:1 step=40 mse=0.00004048
-tt:1 step=60 mse=0.00003227
-tt:1 step=80 mse=0.00003213
-{"backend": "tenstorrent", "device_id": 1, "initial_mse": 0.08051129430532455, "final_mse": 3.2111081964103505e-05, "validation_mse": 5.255065480014309e-05, "max_weight_change": 0.0986328125, "checkpoint": "/outputs/tenstorrent-1.pt"}
-TRAINING_PASSED backend=tenstorrent devices=2
-2026-10-09 07:06:04.775 | info | BuildKernels | JIT cache stats: 0/58 hits (0.0%) [0 cached, 10 build-once dedup, 0 merged artifacts, 0 merged genfiles] (build_cache_telemetry.cpp:207)
-2026-10-09 07:06:04.775 | info | BuildKernels | JIT telemetry: 2 registered TelemetryTokens (build_cache_telemetry.cpp:230)
-2026-10-09 07:06:04.779 | info | Device | Closing user mode device drivers (tt_cluster.cpp:513)
-2026-10-09 07:06:04.779 | info | UMD | Closing devices in cluster (cluster.cpp:1201)
-2026-10-09 07:06:04.788 | info | UMD | Closing devices in cluster completed. (cluster.cpp:1210)
-2026-10-09 07:06:04.788 | info | UMD | Cluster destructor started. (cluster.cpp:916)
-2026-10-09 07:06:04.788 | info | UMD | Cluster destructor completed. (cluster.cpp:919)
diff --git a/deploy/k3s/evidence/department-2026-10-09/tenstorrent-allocation.json b/deploy/k3s/evidence/department-2026-10-09/tenstorrent-allocation.json
deleted file mode 100644
index 591a9ba..0000000
--- a/deploy/k3s/evidence/department-2026-10-09/tenstorrent-allocation.json
+++ /dev/null
@@ -1,6 +0,0 @@
-{
- "device_nodes": [
- "/dev/tenstorrent/1"
- ],
- "chip_count": 2
-}
diff --git a/deploy/k3s/evidence/department-2026-10-09/tenstorrent-results.json b/deploy/k3s/evidence/department-2026-10-09/tenstorrent-results.json
deleted file mode 100644
index 6dfc9ed..0000000
--- a/deploy/k3s/evidence/department-2026-10-09/tenstorrent-results.json
+++ /dev/null
@@ -1,32 +0,0 @@
-[
- {
- "backend": "tenstorrent",
- "device_id": 0,
- "initial_mse": 0.08051129430532455,
- "final_mse": 3.2111081964103505e-05,
- "validation_mse": 5.255065480014309e-05,
- "max_weight_change": 0.0986328125,
- "checkpoint": "/outputs/tenstorrent-0.pt",
- "checkpoint_reload_mse": 5.255065480014309e-05,
- "started_at": "2026-10-09T05:25:02.341652+00:00",
- "finished_at": "2026-10-09T05:25:13.379742+00:00",
- "device_nodes": [
- "/dev/tenstorrent/1"
- ]
- },
- {
- "backend": "tenstorrent",
- "device_id": 1,
- "initial_mse": 0.08051129430532455,
- "final_mse": 3.2111081964103505e-05,
- "validation_mse": 5.255065480014309e-05,
- "max_weight_change": 0.0986328125,
- "checkpoint": "/outputs/tenstorrent-1.pt",
- "checkpoint_reload_mse": 5.255065480014309e-05,
- "started_at": "2026-10-09T05:25:25.167206+00:00",
- "finished_at": "2026-10-09T05:25:28.814332+00:00",
- "device_nodes": [
- "/dev/tenstorrent/1"
- ]
- }
-]
diff --git a/deploy/k3s/evidence/department-2026-10-09/web-audit.json b/deploy/k3s/evidence/department-2026-10-09/web-audit.json
deleted file mode 100644
index 65a3f25..0000000
--- a/deploy/k3s/evidence/department-2026-10-09/web-audit.json
+++ /dev/null
@@ -1,22 +0,0 @@
-{
- "auditReportVersion": 2,
- "vulnerabilities": {},
- "metadata": {
- "vulnerabilities": {
- "info": 0,
- "low": 0,
- "moderate": 0,
- "high": 0,
- "critical": 0,
- "total": 0
- },
- "dependencies": {
- "prod": 156,
- "dev": 195,
- "optional": 6,
- "peer": 57,
- "peerOptional": 0,
- "total": 355
- }
- }
-}
diff --git a/deploy/k3s/evidence/storage-2026-10-21/README.md b/deploy/k3s/evidence/storage-2026-10-21/README.md
deleted file mode 100644
index 4868077..0000000
--- a/deploy/k3s/evidence/storage-2026-10-21/README.md
+++ /dev/null
@@ -1,57 +0,0 @@
-# Shared-folder storage release verification
-
-Verified on the real two-machine installation October 21, 2026 (America/New_York).
-Source branch: `feat/simple-storage-ethernet-20261021`, based on main `ba2bb86`.
-
-## Evidence
-
-- [Validation summary](validation.json): source checks, small Ethernet sample,
- authentication prerequisites and operating boundaries.
-- [Live API/training results](results.json): eight checks including regular-member
- access, upload checksum/download, ZIP traversal rejection, team isolation,
- active-dataset deletion protection, both accelerators and sampled usage.
-- [Actual workload mounts](workload-volumes.json): direct Ethernet NFS exports,
- selected read-only inputs, scoped outputs and disabled service-account tokens.
-- [Deployed API artifact](deployed-api.json): image identity and binary checksum,
- matched to the built candidate.
-- [QuietBox audit](host-storage.json): ordinary ext4 directories, zero image files
- in the new root, service state, shared capacity and persistent Ethernet profile.
-- [Browser checks](browser-results.json), [dataset screen](datasets-browser.png),
- [job screen](jobs-browser.png): real regular-member login, advisory display,
- an actual browser upload and visible completed jobs; no JavaScript errors.
-- Training logs: [NVIDIA](nvidia.log), [Tenstorrent](tenstorrent.log).
-
-Training uploaded **255,854 bytes**, requested **one NVIDIA GPU** and **one n300
-board (two chips)**, read `/inputs/regression.json`, trained actual accelerator
-operations and saved/reloaded/downloaded checkpoints from `/outputs`. Loss and
-held-out validation checks passed. The browser added an 8-byte CSV afterward.
-
-All **43 API + 6 CLI Go tests** pass with race detection, alongside vet,
-**16 React tests**, TypeScript, lint/build and **6 Python folder tests**. The CLI
-entry package has no test files; no actual test was skipped. Vite/launch-editor
-were patched within the existing major version and npm reports zero vulnerabilities.
-
-The host is now configured with the new API/web release and local folder service.
-The separate old auth deployment remains scaled to zero for rollback. Existing
-auth credentials and deactivated-account state were preserved. No physical disk
-was formatted, old storage data deleted, large quota allocated, or bulk upload run.
-
-## Failures resolved during verification
-
-- Go-auth main previously accepted user-ID cookies, could not verify existing
- Better Auth password hashes, omitted an old schema's required `emailVerified`
- value, and did not honor old deactivated accounts. Targeted fixes and regression
- tests were required to deploy latest main safely.
-- The first dynamic team bindings retained the old export prefix. Verification
- caught the mount failure; the new test jobs were cancelled, only the new wrong
- bindings were replaced, and the correct Ethernet/folder exports were tested.
- The successful results linked above are from the final rerun.
-- NFS 4.2 reused an existing transport to the old LAN alias. New bindings use a
- separate NFS 4.1 connection, verified on the Ethernet path.
-
-Private credentials, sessions, database backups and rollback manifests are kept
-outside Git in `/home/utmist/mist-storage-validation-20261021`. The reusable
-[live verifier](../../verify_storage_folders.py) creates its own test account and
-workspaces; never publish its `member-private.json` file.
-
-Operations and rollback: [storage guide](../../storage/README.md).
diff --git a/deploy/k3s/evidence/storage-2026-10-21/browser-results.json b/deploy/k3s/evidence/storage-2026-10-21/browser-results.json
deleted file mode 100644
index 5a600f2..0000000
--- a/deploy/k3s/evidence/storage-2026-10-21/browser-results.json
+++ /dev/null
@@ -1,9 +0,0 @@
-{
- "checks": [
- "regular_member_login_and_advisory_storage_display",
- "real_browser_upload_with_verified_checksum",
- "both_real_training_jobs_visible_in_member_history",
- "no_browser_javascript_errors"
- ],
- "browser_errors": []
-}
\ No newline at end of file
diff --git a/deploy/k3s/evidence/storage-2026-10-21/datasets-browser.png b/deploy/k3s/evidence/storage-2026-10-21/datasets-browser.png
deleted file mode 100644
index 45ace93..0000000
Binary files a/deploy/k3s/evidence/storage-2026-10-21/datasets-browser.png and /dev/null differ
diff --git a/deploy/k3s/evidence/storage-2026-10-21/deployed-api.json b/deploy/k3s/evidence/storage-2026-10-21/deployed-api.json
deleted file mode 100644
index 66853f1..0000000
--- a/deploy/k3s/evidence/storage-2026-10-21/deployed-api.json
+++ /dev/null
@@ -1,6 +0,0 @@
-{
- "image": "mist-api:folders-20261021",
- "image_id": "sha256:350ced027c45a35e388f4ec5ef34f818074e856ff100c5e12c8f058fbbe8a9ef",
- "binary_sha256": "ef980d2e624a3f015a3f45d87f49563861475e6c56aa8f808fb85f76990e3204",
- "node": "utmist-z1opa08"
-}
diff --git a/deploy/k3s/evidence/storage-2026-10-21/host-storage.json b/deploy/k3s/evidence/storage-2026-10-21/host-storage.json
deleted file mode 100644
index 69b3477..0000000
--- a/deploy/k3s/evidence/storage-2026-10-21/host-storage.json
+++ /dev/null
@@ -1,11 +0,0 @@
-{
- "root": "/srv/mist-data",
- "filesystem": "/dev/nvme0n1p2 ext4",
- "image_files_in_new_root": 0,
- "logical_regular_file_bytes": 535003,
- "capacity_bytes": 3936290357248,
- "available_bytes": 2010709856256,
- "storage_service": "active",
- "ethernet_speed_mbps": "1000",
- "ethernet_profile_autoconnect": "yes"
-}
diff --git a/deploy/k3s/evidence/storage-2026-10-21/jobs-browser.png b/deploy/k3s/evidence/storage-2026-10-21/jobs-browser.png
deleted file mode 100644
index 32aa044..0000000
Binary files a/deploy/k3s/evidence/storage-2026-10-21/jobs-browser.png and /dev/null differ
diff --git a/deploy/k3s/evidence/storage-2026-10-21/nvidia.log b/deploy/k3s/evidence/storage-2026-10-21/nvidia.log
deleted file mode 100644
index e447011..0000000
--- a/deploy/k3s/evidence/storage-2026-10-21/nvidia.log
+++ /dev/null
@@ -1,10 +0,0 @@
-CUDA devices=1; torch=2.5.1+cu124
-Training on cuda:0: NVIDIA RTX A4000
-DATASET_LOADED file=/inputs/regression.json
-cuda:0 step=0 mse=0.32162201
-cuda:0 step=20 mse=0.00133983
-cuda:0 step=40 mse=0.00006271
-cuda:0 step=60 mse=0.00000431
-cuda:0 step=80 mse=0.00000035
-{"backend": "nvidia", "device_id": 0, "initial_mse": 0.32162201404571533, "final_mse": 3.100358014762605e-08, "validation_mse": 9.109567145060282e-08, "max_weight_change": 0.37507763504981995, "checkpoint": "/outputs/nvidia-0.pt"}
-TRAINING_PASSED backend=nvidia devices=1
diff --git a/deploy/k3s/evidence/storage-2026-10-21/results.json b/deploy/k3s/evidence/storage-2026-10-21/results.json
deleted file mode 100644
index 0a2e84e..0000000
--- a/deploy/k3s/evidence/storage-2026-10-21/results.json
+++ /dev/null
@@ -1,98 +0,0 @@
-{
- "checks": [
- "regular_member_login_and_shared_folder_readiness",
- "membership_and_common_write_permissions_enforced",
- "unsafe_archive_rejected",
- "small_upload_checksum_download_and_cross_team_isolation",
- "active_dataset_deletion_blocked",
- "nvidia_real_training_and_output_downloads",
- "tenstorrent_real_training_and_output_downloads",
- "host_sampled_dataset_and_model_usage"
- ],
- "jobs": [
- {
- "id": "mist-b5e74aec693e5111ae262449",
- "backend": "nvidia",
- "metrics": [
- {
- "backend": "nvidia",
- "device_id": 0,
- "initial_mse": 0.32162201404571533,
- "final_mse": 3.100358014762605e-08,
- "validation_mse": 9.109567145060282e-08,
- "max_weight_change": 0.37507763504981995,
- "checkpoint": "/outputs/nvidia-0.pt",
- "checkpoint_reload_mse": 9.109567145060282e-08,
- "device_name": "NVIDIA RTX A4000",
- "started_at": "2026-10-22T00:28:43.814031+00:00",
- "finished_at": "2026-10-22T00:28:44.148257+00:00",
- "device_nodes": [
- "/dev/nvidia1"
- ]
- }
- ],
- "node": "utmist-z1opa08",
- "status": "Success"
- },
- {
- "id": "mist-57208c18e561c8cef36e6697",
- "backend": "tenstorrent",
- "metrics": [
- {
- "backend": "tenstorrent",
- "device_id": 0,
- "initial_mse": 0.32162201404571533,
- "final_mse": 5.1175324188079685e-05,
- "validation_mse": 0.00011455668573034927,
- "max_weight_change": 0.369140625,
- "checkpoint": "/outputs/tenstorrent-0.pt",
- "checkpoint_reload_mse": 0.00011455668573034927,
- "started_at": "2026-10-22T00:28:49.539632+00:00",
- "finished_at": "2026-10-22T00:28:59.859064+00:00",
- "device_nodes": [
- "/dev/tenstorrent/1"
- ]
- },
- {
- "backend": "tenstorrent",
- "device_id": 1,
- "initial_mse": 0.32162201404571533,
- "final_mse": 5.1175324188079685e-05,
- "validation_mse": 0.00011455668573034927,
- "max_weight_change": 0.369140625,
- "checkpoint": "/outputs/tenstorrent-1.pt",
- "checkpoint_reload_mse": 0.00011455668573034927,
- "started_at": "2026-10-22T00:29:11.964071+00:00",
- "finished_at": "2026-10-22T00:29:15.617335+00:00",
- "device_nodes": [
- "/dev/tenstorrent/1"
- ]
- }
- ],
- "node": "utmist-tt",
- "status": "Success"
- }
- ],
- "bulk_testing": false,
- "dataset_bytes": 255854,
- "storage": {
- "available_bytes": 2010710540288,
- "capacity_bytes": 3936291389440,
- "dataset_allocation_gib": 1,
- "dataset_used_bytes": 256360,
- "enabled": true,
- "max_dataset_bytes": 1125899906842624,
- "model_allocation_gib": 1,
- "model_used_bytes": 6065,
- "storage_limits_enforced": false,
- "storage_low": false,
- "storage_mode": "shared-folders",
- "upload_limit_bytes": 2009636798464,
- "usage_observed_at": "2026-10-22T00:28:54Z",
- "usage_stale": false
- },
- "teams": [
- "team-a91af070ba2a6bdc",
- "team-71d2f7b638c597e1"
- ]
-}
diff --git a/deploy/k3s/evidence/storage-2026-10-21/tenstorrent.log b/deploy/k3s/evidence/storage-2026-10-21/tenstorrent.log
deleted file mode 100644
index 989051e..0000000
--- a/deploy/k3s/evidence/storage-2026-10-21/tenstorrent.log
+++ /dev/null
@@ -1,58 +0,0 @@
-2026-10-22 00:28:44.905 | DEBUG | ttnn::78 - Initial ttnn.CONFIG:
-Config{cache_path=/root/.cache/ttnn,model_cache_path=/root/.cache/ttnn/models,tmp_dir=/tmp/ttnn,enable_model_cache=false,enable_fast_runtime_mode=true,validate_program_args=false,throw_exception_on_fallback=false,enable_logging=false,enable_graph_report=false,enable_graph_python_stack_traces=false,enable_detailed_buffer_report=false,enable_detailed_tensor_report=false,enable_comparison_mode=false,comparison_mode_should_raise_exception=false,comparison_mode_pcc=0.9999,root_report_path=generated/ttnn/reports,report_name=std::nullopt,std::nullopt}
-2026-10-22 00:28:45.104 | info | Device | Opening user mode device driver (tt_cluster.cpp:228)
-2026-10-22 00:28:45.104 | info | UMD | Cluster constructor started. (cluster.cpp:389)
-2026-10-22 00:28:45.104 | info | UMD | Creating TopologyDiscovery for architecture: wormhole_b0 (topology_discovery.cpp:96)
-2026-10-22 00:28:45.104 | info | UMD | Starting topology discovery. (topology_discovery.cpp:115)
-2026-10-22 00:28:45.106 | info | UMD | Established firmware bundle version: 19.4.2 (topology_discovery.cpp:575)
-2026-10-22 00:28:45.114 | info | UMD | Completed topology discovery. (topology_discovery.cpp:119)
-2026-10-22 00:28:45.151 | warning | UMD | hwloc_set_area_membind(): failed for physical_device_id: 1 on NodeSet: [0] with errno: Operation not permitted (pid: 1 tid: 137514765571904) (cpuset_lib.cpp:440)
-2026-10-22 00:28:45.151 | warning | UMD | ---- ttSiliconDevice::init_hugepage: bind_area_to_memory_nodeset() failed (physical_device_id: 1 ch: 0). Hugepage allocation is not on NumaNode matching TT Device. Side-Effect is decreased Device->Host perf (Issue #893). (silicon_sysmem_manager.cpp:271)
-2026-10-22 00:28:45.151 | warning | UMD | hwloc_set_area_membind(): failed for physical_device_id: 1 on NodeSet: [0] with errno: Operation not permitted (pid: 1 tid: 137514765571904) (cpuset_lib.cpp:440)
-2026-10-22 00:28:45.151 | warning | UMD | ---- ttSiliconDevice::init_hugepage: bind_area_to_memory_nodeset() failed (physical_device_id: 1 ch: 1). Hugepage allocation is not on NumaNode matching TT Device. Side-Effect is decreased Device->Host perf (Issue #893). (silicon_sysmem_manager.cpp:271)
-2026-10-22 00:28:45.154 | info | UMD | Opening local chip ids/PCIe ids: {0}/[1] and remote chip ids {1} (cluster.cpp:173)
-2026-10-22 00:28:45.154 | info | UMD | IOMMU: disabled (cluster.cpp:147)
-2026-10-22 00:28:45.154 | info | UMD | KMD version: 2.5.0 (cluster.cpp:150)
-2026-10-22 00:28:45.154 | info | UMD | Cluster constructor completed. (cluster.cpp:648)
-2026-10-22 00:28:45.155 | info | UMD | Starting devices in cluster (cluster.cpp:1188)
-2026-10-22 00:28:45.158 | info | UMD | Starting devices in cluster completed. (cluster.cpp:1196)
-Tenstorrent devices=2; torch used for data/metrics=2.11.0+cpu
-Opening Tenstorrent device 0
-2026-10-22 00:28:45.209 | info | Distributed | Using auto discovery to generate mesh graph. (metal_env.cpp:437)
-2026-10-22 00:28:45.209 | info | Distributed | Constructing control plane using auto-discovery (no mesh graph descriptor). (metal_env.cpp:499)
-2026-10-22 00:28:45.210 | info | Fabric | Logical multi-mesh adjacency: intermesh degree histogram {0:1}; intra-mesh degree histograms mesh0 {1:2} (topology_mapper_utils.cpp:3329)
-2026-10-22 00:28:45.210 | info | Fabric | Physical multi-mesh adjacency: intermesh degree histogram {0:1}; intra-mesh degree histograms mesh0 {1:2} (topology_mapper_utils.cpp:3337)
-DATASET_LOADED file=/inputs/regression.json
-tt:0 step=0 mse=0.18711002
-tt:0 step=20 mse=0.00110440
-tt:0 step=40 mse=0.00008407
-tt:0 step=60 mse=0.00005286
-tt:0 step=80 mse=0.00005125
-{"backend": "tenstorrent", "device_id": 0, "initial_mse": 0.32162201404571533, "final_mse": 5.1175324188079685e-05, "validation_mse": 0.00011455668573034927, "max_weight_change": 0.369140625, "checkpoint": "/outputs/tenstorrent-0.pt"}
-Opening Tenstorrent device 1
-2026-10-22 00:28:59.905 | info | Metal | Fabric config changed from FabricConfig::DISABLED to FabricConfig::FABRIC_1D, reinitializing control plane (metal_env.cpp:337)
-2026-10-22 00:28:59.905 | info | Distributed | Using auto discovery to generate mesh graph. (metal_env.cpp:437)
-2026-10-22 00:28:59.905 | info | Distributed | Constructing control plane using auto-discovery (no mesh graph descriptor). (metal_env.cpp:499)
-2026-10-22 00:28:59.906 | info | Fabric | Logical multi-mesh adjacency: intermesh degree histogram {0:1}; intra-mesh degree histograms mesh0 {1:2} (topology_mapper_utils.cpp:3329)
-2026-10-22 00:28:59.906 | info | Fabric | Physical multi-mesh adjacency: intermesh degree histogram {0:1}; intra-mesh degree histograms mesh0 {1:2} (topology_mapper_utils.cpp:3337)
-2026-10-22 00:28:59.907 | info | Metal | Enabling FabricConfig::FABRIC_1D only for dispatch. If your workload requires fabric, please set the fabric config accordingly. (device_manager.cpp:291)
-2026-10-22 00:28:59.907 | info | Metal | Dispatch on FabricConfig::FABRIC_1D with 1 Command Queues
- (device_manager.cpp:299)
-2026-10-22 00:28:59.909 | info | Metal | Initializing Fabric (fabric_firmware_initializer.cpp:296)
-2026-10-22 00:29:01.929 | info | Metal | Fabric initialized on 2 devices (fabric_firmware_initializer.cpp:444)
-2026-10-22 00:29:01.929 | info | Metal | Fabric Initialized with config FabricConfig::FABRIC_1D (fabric_firmware_initializer.cpp:313)
-DATASET_LOADED file=/inputs/regression.json
-tt:1 step=0 mse=0.18711002
-tt:1 step=20 mse=0.00110440
-tt:1 step=40 mse=0.00008407
-tt:1 step=60 mse=0.00005286
-tt:1 step=80 mse=0.00005125
-{"backend": "tenstorrent", "device_id": 1, "initial_mse": 0.32162201404571533, "final_mse": 5.1175324188079685e-05, "validation_mse": 0.00011455668573034927, "max_weight_change": 0.369140625, "checkpoint": "/outputs/tenstorrent-1.pt"}
-TRAINING_PASSED backend=tenstorrent devices=2
-2026-10-22 00:29:16.207 | info | BuildKernels | JIT cache stats: 0/58 hits (0.0%) [0 cached, 10 build-once dedup, 0 merged artifacts, 0 merged genfiles] (build_cache_telemetry.cpp:207)
-2026-10-22 00:29:16.207 | info | BuildKernels | JIT telemetry: 2 registered TelemetryTokens (build_cache_telemetry.cpp:230)
-2026-10-22 00:29:16.212 | info | Device | Closing user mode device drivers (tt_cluster.cpp:513)
-2026-10-22 00:29:16.212 | info | UMD | Closing devices in cluster (cluster.cpp:1201)
-2026-10-22 00:29:16.220 | info | UMD | Closing devices in cluster completed. (cluster.cpp:1210)
-2026-10-22 00:29:16.220 | info | UMD | Cluster destructor started. (cluster.cpp:916)
-2026-10-22 00:29:16.220 | info | UMD | Cluster destructor completed. (cluster.cpp:919)
diff --git a/deploy/k3s/evidence/storage-2026-10-21/validation.json b/deploy/k3s/evidence/storage-2026-10-21/validation.json
deleted file mode 100644
index 3f25652..0000000
--- a/deploy/k3s/evidence/storage-2026-10-21/validation.json
+++ /dev/null
@@ -1,40 +0,0 @@
-{
- "base_main": "ba2bb86",
- "branch": "feat/simple-storage-ethernet-20261021",
- "go_api_tests": 43,
- "go_cli_tests": 6,
- "go_race": "passed",
- "go_vet": "passed",
- "react_tests": 16,
- "typescript": "passed",
- "frontend_lint": "passed",
- "frontend_build": "passed",
- "python_folder_tests": 6,
- "npm_audit_vulnerabilities": 0,
- "fake_backend_contract": "passed",
- "ethernet": {
- "main": "192.168.240.1/30",
- "main_interface": "enp6s0",
- "quietbox": "192.168.240.2/30",
- "quietbox_interface": "enp200s0",
- "link_mbps": 1000,
- "main_route": "192.168.240.2 dev enp6s0 src 192.168.240.1 uid 1000 \n cache",
- "small_probe_bytes": 16777216,
- "small_probe_write_MBps": 43.0,
- "small_probe_read_MBps": 101.0,
- "performance_note": "Single small direct-I/O NFS sample; not a sustained workload benchmark."
- },
- "auth_prerequisites": [
- "Require real unexpired database sessions; remove identity-cookie fallbacks",
- "Preserve Better Auth scrypt passwords, NOT NULL schema and deactivated accounts",
- "Existing administrator login verified against isolated database copy and live portal"
- ],
- "old_storage": "Retained unchanged for rollback; no migration",
- "large_storage_testing": false,
- "new_storage_mode": "ordinary directories with advisory team allowances",
- "remaining_boundaries": [
- "Single storage host and shared physical capacity",
- "No equal-space guarantee or per-team hard quota",
- "No automatic backup, local dataset cache, or resumable upload"
- ]
-}
diff --git a/deploy/k3s/evidence/storage-2026-10-21/workload-volumes.json b/deploy/k3s/evidence/storage-2026-10-21/workload-volumes.json
deleted file mode 100644
index a66be72..0000000
--- a/deploy/k3s/evidence/storage-2026-10-21/workload-volumes.json
+++ /dev/null
@@ -1,130 +0,0 @@
-[
- {
- "id": "mist-b5e74aec693e5111ae262449",
- "namespace": "mist-team-a91af070ba2a6bdc",
- "node": "utmist-z1opa08",
- "automount_service_account_token": false,
- "volumes": [
- {
- "claim": "team-results",
- "read_only": false,
- "nfs": {
- "path": "/srv/mist-data/teams/team-a91af070ba2a6bdc/results",
- "server": "192.168.240.2"
- },
- "mount_options": [
- "nfsvers=4.1",
- "hard",
- "timeo=100",
- "retrans=2"
- ]
- },
- {
- "claim": "team-datasets",
- "read_only": true,
- "nfs": {
- "path": "/srv/mist-data/teams/team-a91af070ba2a6bdc/datasets",
- "server": "192.168.240.2"
- },
- "mount_options": [
- "nfsvers=4.1",
- "hard",
- "timeo=100",
- "retrans=2"
- ]
- }
- ],
- "mounts": [
- {
- "mountPath": "/checkpoints/mist-b5e74aec693e5111ae262449",
- "name": "checkpoints",
- "subPath": "members/21008782734082c80dbd4eb4/jobs/mist-b5e74aec693e5111ae262449/outputs"
- },
- {
- "mountPath": "/outputs",
- "name": "checkpoints",
- "subPath": "members/21008782734082c80dbd4eb4/jobs/mist-b5e74aec693e5111ae262449/outputs"
- },
- {
- "mountPath": "/inputs",
- "name": "input",
- "readOnly": true,
- "subPath": "members/21008782734082c80dbd4eb4/datasets/dataset-3c4ddd41a5c75880df01796f/content"
- },
- {
- "mountPath": "/mist-script",
- "name": "script",
- "readOnly": true
- }
- ]
- },
- {
- "id": "mist-57208c18e561c8cef36e6697",
- "namespace": "mist-team-a91af070ba2a6bdc",
- "node": "utmist-tt",
- "automount_service_account_token": false,
- "volumes": [
- {
- "claim": "team-results",
- "read_only": false,
- "nfs": {
- "path": "/srv/mist-data/teams/team-a91af070ba2a6bdc/results",
- "server": "192.168.240.2"
- },
- "mount_options": [
- "nfsvers=4.1",
- "hard",
- "timeo=100",
- "retrans=2"
- ]
- },
- {
- "claim": "team-datasets",
- "read_only": true,
- "nfs": {
- "path": "/srv/mist-data/teams/team-a91af070ba2a6bdc/datasets",
- "server": "192.168.240.2"
- },
- "mount_options": [
- "nfsvers=4.1",
- "hard",
- "timeo=100",
- "retrans=2"
- ]
- }
- ],
- "mounts": [
- {
- "mountPath": "/checkpoints/mist-57208c18e561c8cef36e6697",
- "name": "checkpoints",
- "subPath": "members/21008782734082c80dbd4eb4/jobs/mist-57208c18e561c8cef36e6697/outputs"
- },
- {
- "mountPath": "/outputs",
- "name": "checkpoints",
- "subPath": "members/21008782734082c80dbd4eb4/jobs/mist-57208c18e561c8cef36e6697/outputs"
- },
- {
- "mountPath": "/home/utmist-tt/TT/tt-metal",
- "name": "metal",
- "readOnly": true
- },
- {
- "mountPath": "/home/utmist-tt/.local/share/uv/python",
- "name": "python",
- "readOnly": true
- },
- {
- "mountPath": "/inputs",
- "name": "input",
- "readOnly": true,
- "subPath": "members/21008782734082c80dbd4eb4/datasets/dataset-3c4ddd41a5c75880df01796f/content"
- },
- {
- "mountPath": "/mist-script",
- "name": "script",
- "readOnly": true
- }
- ]
- }
-]
diff --git a/deploy/k3s/examples/Dockerfile.training b/deploy/k3s/examples/Dockerfile.training
deleted file mode 100644
index 7c80e76..0000000
--- a/deploy/k3s/examples/Dockerfile.training
+++ /dev/null
@@ -1,8 +0,0 @@
-# CPU: docker build -f deploy/k3s/examples/Dockerfile.training -t mist-training:cpu-v1 deploy/k3s/examples
-# NVIDIA: add --build-arg BASE_IMAGE=pytorch/pytorch:2.5.1-cuda12.4-cudnn9-runtime
-ARG BASE_IMAGE=python:3.10-slim
-FROM ${BASE_IMAGE}
-WORKDIR /app
-COPY train.py dataset.csv /app/
-ENTRYPOINT ["python", "-u", "train.py"]
-CMD ["--backend", "cpu"]
diff --git a/deploy/k3s/examples/dataset.csv b/deploy/k3s/examples/dataset.csv
deleted file mode 100644
index e6fa537..0000000
--- a/deploy/k3s/examples/dataset.csv
+++ /dev/null
@@ -1,8 +0,0 @@
-x,y
--2,-5
--1,-2
--0.5,-0.5
-0,1
-0.5,2.5
-1,4
-2,7
diff --git a/deploy/k3s/examples/train.py b/deploy/k3s/examples/train.py
deleted file mode 100644
index 59f54cc..0000000
--- a/deploy/k3s/examples/train.py
+++ /dev/null
@@ -1,74 +0,0 @@
-"""Packaged regression example: CPU stdlib, or PyTorch CUDA on each GPU.
-
-The tiny dataset is baked into the image until Mist's dataset service exists.
-The default entrypoint and /app WORKDIR deliberately test image semantics.
-"""
-import argparse
-import csv
-import json
-import os
-from pathlib import Path
-
-
-def cpu(rows, output):
- weight, bias = 0.0, 0.0
- initial = sum(y * y for x, y in rows) / len(rows)
- for _ in range(200):
- errors = [(weight * x + bias - y, x) for x, y in rows]
- weight -= 0.05 * 2 * sum(error * x for error, x in errors) / len(rows)
- bias -= 0.05 * 2 * sum(error for error, x in errors) / len(rows)
- loss = sum((weight * x + bias - y) ** 2 for x, y in rows) / len(rows)
- checkpoint = output / 'model.json'
- checkpoint.write_text(json.dumps({'weight': weight, 'bias': bias}) + '\n')
- restored = json.loads(checkpoint.read_text())
- assert restored['weight'] == weight and restored['bias'] == bias
- return [{'initial_loss': initial, 'final_loss': loss, 'checkpoint': checkpoint.name}]
-
-
-def nvidia(rows, output, expected):
- import torch
- assert torch.cuda.is_available(), 'CUDA is unavailable; refusing CPU fallback'
- assert torch.cuda.device_count() == expected, 'Allocated GPU count differs'
- results = []
- for index in range(expected):
- device = torch.device(f'cuda:{index}')
- inputs = torch.tensor([[x] for x, y in rows], device=device)
- targets = torch.tensor([[y] for x, y in rows], device=device)
- model = torch.nn.Linear(1, 1, device=device)
- optimizer = torch.optim.SGD(model.parameters(), lr=0.05)
- initial = torch.nn.functional.mse_loss(model(inputs), targets).item()
- for _ in range(200):
- optimizer.zero_grad()
- loss = torch.nn.functional.mse_loss(model(inputs), targets)
- loss.backward()
- optimizer.step()
- final = torch.nn.functional.mse_loss(model(inputs), targets).item()
- checkpoint = output / f'nvidia-{index}.pt'
- torch.save(model.state_dict(), checkpoint)
- restored = torch.nn.Linear(1, 1, device=device)
- restored.load_state_dict(torch.load(checkpoint, weights_only=True, map_location=device))
- assert torch.allclose(model(inputs), restored(inputs))
- results.append({'device': torch.cuda.get_device_name(index), 'initial_loss': initial, 'final_loss': final, 'checkpoint': checkpoint.name})
- return results
-
-
-def main():
- parser = argparse.ArgumentParser()
- parser.add_argument('--backend', choices=['cpu', 'nvidia'], default='cpu')
- parser.add_argument('--devices', type=int, default=1)
- args = parser.parse_args()
- assert Path.cwd() == Path('/app'), 'Image WORKDIR was overridden'
- with Path('dataset.csv').open() as dataset:
- rows = [(float(row['x']), float(row['y'])) for row in csv.DictReader(dataset)]
- output = Path(os.environ.get('MIST_OUTPUT_DIR', '/outputs'))
- output.mkdir(parents=True, exist_ok=True)
- results = cpu(rows, output) if args.backend == 'cpu' else nvidia(rows, output, args.devices)
- assert all(result['final_loss'] < 0.001 for result in results), 'Training did not converge'
- evidence = {'backend': args.backend, 'cwd': str(Path.cwd()), 'results': results, 'greeting': os.environ.get('GREETING', '')}
- (output / 'results.json').write_text(json.dumps(evidence, indent=2) + '\n')
- print(json.dumps(evidence), flush=True)
- print(f'PACKAGED_TRAINING_PASSED backend={args.backend}', flush=True)
-
-
-if __name__ == '__main__':
- main()
diff --git a/deploy/k3s/examples/train_shared.py b/deploy/k3s/examples/train_shared.py
deleted file mode 100644
index 9ae65c8..0000000
--- a/deploy/k3s/examples/train_shared.py
+++ /dev/null
@@ -1,212 +0,0 @@
-"""Small accelerator training check; TT gradients are computed explicitly in TT-NN.
-
-CPU Torch creates the data, records metrics, and serializes checkpoints. All
-forward passes, gradients, and weight updates run on the selected accelerator.
-This is a linear-regression smoke test, not a model compatibility benchmark.
-"""
-
-import argparse
-import json
-import math
-import os
-import subprocess
-import sys
-import time
-from datetime import datetime, timezone
-from pathlib import Path
-
-import torch
-
-
-def dataset():
- # Uploaded dataset; deliberately no generated-data fallback.
- path = Path(os.environ.get("MIST_INPUT_DIR", "/inputs")) / "regression.json"
- data = json.loads(path.read_text())
- tensors = [torch.tensor(data[key], dtype=torch.float32) for key in
- ("x", "y", "validation_x", "validation_y")]
- assert [tuple(t.shape) for t in tensors] == [(128,32),(128,32),(64,32),(64,32)]
- print(f"DATASET_LOADED file={path}", flush=True)
- return tensors
-
-
-def await_start(output, start_file, metadata):
- if not start_file:
- return
- (output / "ready.json").write_text(json.dumps({**metadata,
- "ready_at": datetime.now(timezone.utc).isoformat()}) + "\n")
- print(f"BARRIER_READY {json.dumps(metadata)}", flush=True)
- deadline = time.monotonic() + 300
- while not start_file.exists():
- assert time.monotonic() < deadline, "Training start barrier timed out"
- time.sleep(0.2)
-
-
-def check_and_save(backend, device_id, weights, initial, final, validation, output):
- change = weights.abs().max().item()
- assert all(math.isfinite(v) for v in (initial, final, validation, change))
- assert final < initial * 0.02, f"Loss did not converge: {initial} -> {final}"
- assert validation < 5e-4, f"Held-out MSE too high: {validation}"
- assert change > 0.05, "Weights were not updated"
- path = output / f"{backend}-{device_id}.pt"
- torch.save({"weights": weights.cpu(), "initial_mse": initial,
- "final_mse": final, "validation_mse": validation}, path)
- loaded = torch.load(path, map_location="cpu", weights_only=True)
- torch.testing.assert_close(loaded["weights"], weights.cpu(), rtol=0, atol=0)
- result = {"backend": backend, "device_id": device_id,
- "initial_mse": initial, "final_mse": final,
- "validation_mse": validation, "max_weight_change": change,
- "checkpoint": str(path)}
- print(json.dumps(result), flush=True)
- return result, loaded["weights"]
-
-
-def nvidia(output, steps, expected_devices, start_file, board_pci):
- assert torch.cuda.is_available(), "CUDA unavailable; CPU fallback is forbidden"
- count = torch.cuda.device_count()
- assert count == expected_devices, f"Expected {expected_devices} allocated GPUs; saw {count}"
- print(f"CUDA devices={count}; torch={torch.__version__}", flush=True)
- nodes = sorted(str(path) for path in Path("/dev").glob("nvidia[0-9]*"))
- assert len(nodes) == count, f"Unexpected NVIDIA device nodes: {nodes}"
- (output / "allocation.json").write_text(json.dumps({"device_nodes": nodes,
- "gpu_count": count}, indent=2) + "\n")
- results = []
- for index in range(count):
- device = torch.device(f"cuda:{index}")
- print(f"Training on {device}: {torch.cuda.get_device_name(index)}", flush=True)
- x, y, validation_x, validation_y = (t.to(device) for t in dataset())
- model = torch.nn.Linear(32, 32, bias=False).to(device)
- torch.nn.init.zeros_(model.weight)
- optimizer = torch.optim.SGD(model.parameters(), lr=0.2)
- if index == 0:
- await_start(output, start_file, {"device_nodes": nodes, "gpu_count": count})
- started = datetime.now(timezone.utc).isoformat()
- initial = (model(x) - y).square().mean().item()
- for step in range(steps):
- optimizer.zero_grad(set_to_none=True)
- residual = model(x) - y
- loss = residual.square().sum() / (2 * x.shape[0])
- loss.backward()
- assert model.weight.grad.is_cuda
- optimizer.step()
- if step % 20 == 0:
- print(f"cuda:{index} step={step} mse={residual.square().mean().item():.8f}", flush=True)
- with torch.no_grad():
- final = (model(x) - y).square().mean().item()
- validation = (model(validation_x) - validation_y).square().mean().item()
- result, weights = check_and_save("nvidia", index, model.weight.T.detach(),
- initial, final, validation, output)
- restored_error = (validation_x @ weights.to(device) - validation_y).square().mean().item()
- assert abs(restored_error - validation) < 1e-7
- result["checkpoint_reload_mse"] = restored_error
- result["device_name"] = torch.cuda.get_device_name(index)
- result["started_at"] = started
- result["finished_at"] = datetime.now(timezone.utc).isoformat()
- result["device_nodes"] = nodes
- results.append(result)
- return results
-
-
-def tenstorrent(output, steps, expected_devices, start_file, board_pci):
- nodes = sorted(str(path) for path in Path("/dev/tenstorrent").iterdir())
-
- def pci_address(node):
- path = Path("/sys/class/tenstorrent") / ("tenstorrent!" + Path(node).name)
- assert path.exists(), f"Missing PCI identity for {node}"
- return path.resolve().parents[1].name
-
- if board_pci:
- nodes = [node for node in nodes if pci_address(node) == board_pci]
- assert len(nodes) == 1, f"PCI selector does not match one allocated board: {board_pci}"
- assert len(nodes) * 2 == expected_devices, f"Unexpected n300 device nodes: {nodes}"
- if len(nodes) > 1:
- # Arbitrary n300 subsets need not form a connected fabric mesh.
- # This test trains independently, so use one runtime per board.
- results = []
- for ordinal, node in enumerate(nodes):
- bdf = pci_address(node)
- board_output = output / ("board-" + Path(node).name)
- command = [sys.executable, __file__, "tenstorrent", "--steps", str(steps),
- "--expected-devices", "2", "--output", str(board_output), "--board-pci", bdf]
- if start_file:
- command += ["--start-file", str(start_file)]
- print(f"Training reserved board {node}, PCI {bdf}", flush=True)
- subprocess.run(command, env={**os.environ, "TT_VISIBLE_DEVICES": bdf}, check=True)
- for result in json.loads((board_output / "results.json").read_text()):
- result["board_local_device_id"] = result["device_id"]
- result["device_id"] = ordinal * 2 + result["device_id"]
- result["board_pci"] = bdf
- results.append(result)
- assert len(results) == expected_devices
- (output / "allocation.json").write_text(json.dumps({"device_nodes": nodes,
- "chip_count": expected_devices}, indent=2) + "\n")
- return results
-
- import ttnn
-
- count = ttnn.get_num_devices()
- assert count == expected_devices, f"Expected {expected_devices} allocated chips; saw {count}"
- (output / "allocation.json").write_text(json.dumps({"device_nodes": nodes,
- "chip_count": count}, indent=2) + "\n")
- print(f"Tenstorrent devices={count}; torch used for data/metrics={torch.__version__}", flush=True)
- results = []
- for index in range(count):
- print(f"Opening Tenstorrent device {index}", flush=True)
- device = ttnn.open_device(device_id=index)
- try:
- if index == 0 and start_file:
- await_start(output, start_file, {"device_nodes": nodes, "chip_count": count})
- started = datetime.now(timezone.utc).isoformat()
- x, y, validation_x, validation_y = dataset()
- options = dict(dtype=ttnn.bfloat16, layout=ttnn.TILE_LAYOUT,
- device=device, memory_config=ttnn.DRAM_MEMORY_CONFIG)
- dx, dy, dvx = (ttnn.from_torch(t, **options) for t in (x, y, validation_x))
- dxt = ttnn.from_torch(x.T.contiguous(), **options)
- weights = ttnn.from_torch(torch.zeros(32, 32), **options)
-
- def mse(inputs, target, matrix):
- prediction = ttnn.to_torch(ttnn.matmul(inputs, matrix)).float()
- return (prediction - target).square().mean().item()
-
- initial = mse(dx, y, weights)
- for step in range(steps):
- prediction = ttnn.matmul(dx, weights)
- residual = ttnn.subtract(prediction, dy)
- # Gradient of ||XW-Y||^2/(2N) is X.T@(XW-Y)/N.
- gradient = ttnn.matmul(dxt, residual)
- weights = ttnn.subtract(weights, ttnn.multiply(gradient, 0.2 / x.shape[0]))
- if step % 20 == 0:
- print(f"tt:{index} step={step} mse={mse(dx, y, weights):.8f}", flush=True)
- final = mse(dx, y, weights)
- validation = mse(dvx, validation_y, weights)
- host_weights = ttnn.to_torch(weights).float()
- result, loaded = check_and_save("tenstorrent", index, host_weights,
- initial, final, validation, output)
- restored = ttnn.from_torch(loaded, **options)
- restored_error = mse(dvx, validation_y, restored)
- assert abs(restored_error - validation) < 1e-7
- result["checkpoint_reload_mse"] = restored_error
- result["started_at"] = started
- result["finished_at"] = datetime.now(timezone.utc).isoformat()
- result["device_nodes"] = nodes
- results.append(result)
- finally:
- ttnn.close_device(device)
- return results
-
-
-if __name__ == "__main__":
- parser = argparse.ArgumentParser()
- parser.add_argument("backend", choices=("nvidia", "tenstorrent"))
- parser.add_argument("--steps", type=int, default=100)
- parser.add_argument("--output", type=Path, default=Path("/checkpoints"))
- parser.add_argument("--expected-devices", type=int,
- help="Expected allocated chip/GPU count; defaults to the original whole-node test")
- parser.add_argument("--start-file", type=Path,
- help="Optional shared-file barrier for the concurrent allocation test")
- parser.add_argument("--board-pci", help=argparse.SUPPRESS)
- args = parser.parse_args()
- args.output.mkdir(parents=True, exist_ok=True)
- expected = args.expected_devices or (2 if args.backend == "nvidia" else 8)
- results = globals()[args.backend](args.output, args.steps, expected, args.start_file, args.board_pci)
- (args.output / "results.json").write_text(json.dumps(results, indent=2) + "\n")
- print(f"TRAINING_PASSED backend={args.backend} devices={len(results)}", flush=True)
diff --git a/deploy/k3s/foundation-results.json b/deploy/k3s/foundation-results.json
deleted file mode 100644
index c13ecca..0000000
--- a/deploy/k3s/foundation-results.json
+++ /dev/null
@@ -1,133 +0,0 @@
-{
- "recorded_at": "2026-10-09T02:57:05.690026+00:00",
- "branch": "feat/local-job-foundation",
- "baseline_commit": "2f9b178",
- "scope": "Parts 1\u20133 plus node-local persistent outputs. Shared datasets/downloads (4), deployment/login (5) remain.",
- "api_image": "mist-api:foundation-20261008",
- "api_image_id": "sha256:57ed3a9eebe54610f7353f29bbd24d93259f0d664e171381a5023a2d7d5e4770",
- "checks": {
- "frontend_interactions": 5,
- "kubernetes_backend_tests": 12,
- "src_test_suite": "passed with isolated Redis and existing Docker test image",
- "cli_test_suite": "passed",
- "typescript": "passed",
- "production_build": "passed",
- "changed_frontend_lint": "passed",
- "browser_cpu_packaged_image": "passed",
- "browser_two_nvidia_gpus": "passed",
- "browser_tt_two_boards_four_chips": "passed",
- "browser_exit_17_and_cancel": "passed",
- "nvidia_occupied_queue_cancel_reuse": "passed",
- "tt_occupied_queue_cancel_reuse": "passed",
- "persistent_outputs_new_containers_both_nodes": "passed",
- "literal_arguments_working_directory_output_alias": "passed",
- "image_pull_reason_and_deadline_in_browser": "passed",
- "history_survives_api_restarts": "passed",
- "rbac_inventory_read_only": "passed"
- },
- "browser_jobs": {
- "packaged-cpu": {
- "id": "mist-261e82c2848fc79b234afc4a",
- "state": "Success",
- "node": "utmist-z1opa08",
- "device_count": 0
- },
- "tt-two-boards": {
- "id": "mist-f9a86752969598c182c3d79f",
- "state": "Success",
- "node": "utmist-tt",
- "device_count": 2
- },
- "packaged-two-gpus": {
- "id": "mist-d3756747c2e68bf76ea2e876",
- "state": "Success",
- "node": "utmist-z1opa08",
- "device_count": 2
- },
- "failure": {
- "id": "mist-aa022009f288d22836d04b85",
- "state": "Failure",
- "node": "utmist-z1opa08",
- "device_count": 0
- },
- "cancel": {
- "id": "mist-99d4a51c2f49d394a8796ba3",
- "state": "Cancelled",
- "node": "utmist-z1opa08",
- "device_count": 0
- }
- },
- "allocation_jobs": {
- "literal-arguments": "mist-2aee9108f5ef651e0b06eccc",
- "nvidia-holder": "mist-cce552238d50aaf3fc8ab895",
- "nvidia-queued": "mist-e0d28d07f23bd40653c65de0",
- "tenstorrent-holder": "mist-5737ba8da7abaccd3a45698d",
- "tenstorrent-queued": "mist-aa8e4101f47855166dfa4b36"
- },
- "image_pull_failure_job": "mist-2e6965771e3330e013df1fc3",
- "idle_hardware": {
- "observed_at": "2026-10-09T02:57:05.530844763Z",
- "machines": [
- {
- "name": "utmist-z1opa08",
- "address": "10.0.0.175",
- "ready": true,
- "schedulable": true,
- "cpu_allocatable": "56",
- "memory_allocatable": "164912728Ki"
- },
- {
- "name": "utmist-tt",
- "address": "10.0.0.112",
- "ready": true,
- "schedulable": true,
- "cpu_allocatable": "32",
- "memory_allocatable": "511127964Ki"
- }
- ],
- "pools": [
- {
- "accelerator": "nvidia",
- "node": "utmist-z1opa08",
- "unit": "GPU",
- "total": 2,
- "allocated": 0,
- "available": 2,
- "ready": true
- },
- {
- "accelerator": "tenstorrent",
- "node": "utmist-tt",
- "unit": "board",
- "total": 4,
- "allocated": 0,
- "available": 4,
- "ready": true,
- "chips_per_device": 2
- }
- ]
- },
- "job_history_count": 40,
- "job_history_read_seconds": 0.193,
- "evidence_directory": "/home/utmist/mist-foundation-results-2026-10-08",
- "evidence_files": [
- "browser.json",
- "foundation-acceptance.json",
- "image-pull-failure.json",
- "rbac.json",
- "jobs-browser.png",
- "machines-browser.png",
- "nvidia-training-browser.png",
- "image-pull-browser.png",
- "cpu-persisted-files.txt",
- "nvidia-persisted-files.txt",
- "tenstorrent-persisted-files.txt"
- ],
- "limits": [
- "Trusted local single-owner pilot; no authenticated tenant isolation",
- "Node-local output storage; no dataset uploads or browser downloads",
- "TT uses tested image and host runtime; arbitrary models/autograd/distributed training not validated",
- "Private registry secret wiring documented; private registry login not exercised",
- "Jobs/output files retained; no automatic cleanup, storage quota or backup"
- ]
-}
diff --git a/deploy/k3s/gpu-smoke-job.yaml b/deploy/k3s/gpu-smoke-job.yaml
deleted file mode 100644
index 0aa711a..0000000
--- a/deploy/k3s/gpu-smoke-job.yaml
+++ /dev/null
@@ -1,28 +0,0 @@
-apiVersion: batch/v1
-kind: Job
-metadata:
- name: gpu-smoke
- namespace: mist
- labels:
- app.kubernetes.io/name: mist-gpu-smoke
-spec:
- backoffLimit: 0
- activeDeadlineSeconds: 600
- template:
- metadata:
- labels:
- app.kubernetes.io/name: mist-gpu-smoke
- spec:
- restartPolicy: Never
- containers:
- - name: vectoradd
- image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0
- resources:
- requests:
- cpu: 250m
- memory: 256Mi
- nvidia.com/gpu: 1
- limits:
- cpu: '1'
- memory: 512Mi
- nvidia.com/gpu: 1
diff --git a/deploy/k3s/mist-api-forward.service b/deploy/k3s/mist-api-forward.service
deleted file mode 100644
index 648fa61..0000000
--- a/deploy/k3s/mist-api-forward.service
+++ /dev/null
@@ -1,11 +0,0 @@
-[Unit]
-Description=Mist Kubernetes API on localhost
-After=network-online.target
-
-[Service]
-ExecStart=/usr/local/bin/k3s kubectl --kubeconfig /home/utmist/.kube/config --namespace mist-system port-forward service/mist-api 3000:3000 --address 127.0.0.1
-Restart=always
-RestartSec=2
-
-[Install]
-WantedBy=default.target
diff --git a/deploy/k3s/mist-api-results.json b/deploy/k3s/mist-api-results.json
deleted file mode 100644
index aab9beb..0000000
--- a/deploy/k3s/mist-api-results.json
+++ /dev/null
@@ -1,191 +0,0 @@
-{
- "verified_at": "2026-10-03T20:42:24.107077+00:00",
- "status": "passed",
- "implementation": {
- "executor": "kubernetes (default)",
- "workload_namespace": "mist",
- "api_namespace": "mist-system",
- "owner": "utmist (configured single owner, not authenticated)",
- "api_url": "http://127.0.0.1:3000",
- "jobs_url": "http://127.0.0.1:3001/jobs",
- "api_image": "mist-api:k3s-pilot-20261003",
- "api_image_config_digest": "sha256:d33a5da4e1761641292340b2f607d4ec8ce95afb5f76c2797044b30747a81743",
- "api_image_index_digest": "sha256:54f36ea00c210f7d3c28a909404ae3b0eab4600a97e72deaee6db2b0b9642b6d"
- },
- "cluster": {
- "k3s_version": "v1.36.4+k3s1",
- "nodes": {
- "utmist-z1opa08": {
- "internal_ip": "10.0.0.175",
- "nvidia_rtx_a4000_gpus": 2
- },
- "utmist-tt": {
- "internal_ip": "10.0.0.112",
- "n300_boards": 4,
- "chips_per_board": 2
- }
- },
- "tt_operator": "0.3.0",
- "fabric_manager": "0.2.30",
- "tt_dra_driver": "0.0.59"
- },
- "api_acceptance": {
- "run": "api-1003-202246",
- "submitted_jobs": 17,
- "expected_terminal_states": {
- "Success": 13,
- "Failure": 2,
- "Cancelled": 2
- },
- "invalid_submissions_rejected_without_creating_jobs": true,
- "unmanaged_jobs_hidden_and_cancellation_denied": true,
- "cpu_execution_and_persisted_output": true,
- "actual_failure_exit_code": 17,
- "deadline_failure_reported": true,
- "cancelled_logs_retained": true,
- "cancellation_released_whole_tt_reservation": true,
- "gpu_queues_and_automatic_reuse": true,
- "tt_queues_and_automatic_reuse": true,
- "nvidia_gpu_counts_tested": [
- 1,
- 2
- ],
- "tt_board_counts_tested": [
- 1,
- 2,
- 3,
- 4
- ],
- "training": "Independent 32x32 regression per chip/GPU; NVIDIA autograd, TT explicit TT-NN gradients",
- "concurrency": {
- "nvidia": {
- "concurrent_jobs": 2,
- "first_device_training_overlap_seconds": 1.168433
- },
- "tenstorrent": {
- "concurrent_jobs": 4,
- "first_device_training_overlap_seconds": 9.202772
- }
- },
- "archived_checkpoints_from_initial_17_jobs": {
- "cpu": {
- "persisted_output": true
- },
- "nvidia": {
- "training_results": 5,
- "checkpoints": 5,
- "max_validation_mse": 9.320668823420419e-07,
- "convergence_and_checkpoint_reload_passed": true
- },
- "tenstorrent": {
- "training_results": 24,
- "checkpoints": 24,
- "max_validation_mse": 0.00017509252938907593,
- "convergence_and_checkpoint_reload_passed": true
- }
- },
- "additional_three_board_job": {
- "id": "mist-05fcb5a03c359c8a24890d84",
- "state": "Success",
- "chips_tested": 6,
- "exit_code": 0
- }
- },
- "clients": {
- "cli_submit_status_logs_cancel": "passed",
- "browser_cpu_submit_status_logs_cancel": "passed",
- "browser_two_tt_boards_four_chip_training": "passed",
- "browser_temporary_api_outage_recovery": "passed"
- },
- "restart": {
- "retained_jobs": 23,
- "states_exit_codes_and_logs_preserved": true,
- "local_port_forward_recovered": true
- },
- "access_checks": [
- {
- "verb": "create",
- "resource": "jobs",
- "namespace": "mist",
- "allowed": true,
- "expected": true
- },
- {
- "verb": "get",
- "resource": "pods/log",
- "namespace": "mist",
- "allowed": true,
- "expected": true
- },
- {
- "verb": "get",
- "resource": "resourceclaims.resource.k8s.io",
- "namespace": "mist",
- "allowed": true,
- "expected": true
- },
- {
- "verb": "get",
- "resource": "secrets",
- "namespace": "mist",
- "allowed": false,
- "expected": false
- },
- {
- "verb": "get",
- "resource": "nodes",
- "namespace": "mist",
- "allowed": false,
- "expected": false
- },
- {
- "verb": "create",
- "resource": "pods/exec",
- "namespace": "mist",
- "allowed": false,
- "expected": false
- },
- {
- "verb": "create",
- "resource": "jobs",
- "namespace": "gpu-operator",
- "allowed": false,
- "expected": false
- }
- ],
- "tests": {
- "backend_full_suite": "go -C src test -timeout 90s ./... passed with an isolated local Redis",
- "backend_kubernetes_suite": "go -C src test -run TestKubernetes ./... passed",
- "cli": "go -C cli test ./... passed",
- "web_component_tests": "3 tests passed",
- "web_production_build": "npm run build passed",
- "typescript": "npx tsc --noEmit passed",
- "default_executor_and_default_private_kubeconfig_loading": "passed"
- },
- "evidence_directory": "/home/utmist/mist-api-results-2026-10-03",
- "evidence_files": [
- "api-acceptance.json",
- "checkpoint-verification.json",
- "concurrency-verification.json",
- "cpu-checkpoints.tar.gz",
- "nvidia-checkpoints.tar.gz",
- "tenstorrent-checkpoints.tar.gz",
- "cli.json",
- "browser.json",
- "jobs-browser.png",
- "rbac.json",
- "restart.json",
- "default-executor.json",
- "three-boards.json",
- "browser-recovery.json"
- ],
- "limits": [
- "Local single-owner pilot; no per-request authentication, credits or team quotas.",
- "Node-local checkpoint PVCs shared by jobs of the same compute profile.",
- "History depends on retained Kubernetes Jobs; normal logs depend on retained Pods and kubelet log retention.",
- "Host-mounted TT-Metal/Python runtime; hugepage mounts are host-shared.",
- "Independent chip tests do not validate large models, TT autograd, distributed training or arbitrary mesh topology.",
- "Vendor ResourceSlice memory capacity is incorrect; claims do not use it for placement.",
- "Legacy Docker/Redis supervisor remains incomplete; other web pages and authentication commands retain prototype behavior."
- ]
-}
diff --git a/deploy/k3s/mist-api.yaml b/deploy/k3s/mist-api.yaml
deleted file mode 100644
index 19f8a2b..0000000
--- a/deploy/k3s/mist-api.yaml
+++ /dev/null
@@ -1,243 +0,0 @@
-apiVersion: v1
-kind: Namespace
-metadata:
- name: mist-system
----
-apiVersion: v1
-kind: Namespace
-metadata:
- name: mist
----
-apiVersion: v1
-kind: ServiceAccount
-metadata:
- name: mist-api
- namespace: mist-system
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: Role
-metadata:
- name: mist-api
- namespace: mist
-rules:
- - apiGroups: [batch]
- resources: [jobs]
- verbs: [get, list, create, update, delete]
- - apiGroups: [""]
- resources: [pods, events]
- verbs: [get, list]
- - apiGroups: [""]
- resources: [pods/log]
- verbs: [get]
- - apiGroups: [""]
- resources: [configmaps]
- verbs: [get, create, update, delete]
- - apiGroups: [resource.k8s.io]
- resources: [resourceclaims, resourceclaimtemplates]
- verbs: [get]
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: RoleBinding
-metadata:
- name: mist-api
- namespace: mist
-roleRef:
- apiGroup: rbac.authorization.k8s.io
- kind: Role
- name: mist-api
-subjects:
- - kind: ServiceAccount
- name: mist-api
- namespace: mist-system
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRole
-metadata:
- name: mist-hardware-reader
-rules:
- - apiGroups: [""]
- resources: [nodes, pods]
- verbs: [list]
- - apiGroups: [resource.k8s.io]
- resources: [resourceclaims, resourceslices]
- verbs: [list]
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRoleBinding
-metadata:
- name: mist-hardware-reader
-roleRef:
- apiGroup: rbac.authorization.k8s.io
- kind: ClusterRole
- name: mist-hardware-reader
-subjects:
- - kind: ServiceAccount
- name: mist-api
- namespace: mist-system
----
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
- name: mist-cpu-checkpoints
- namespace: mist
-spec:
- accessModes: [ReadWriteOnce]
- storageClassName: local-path
- resources:
- requests:
- storage: 256Mi
----
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
- name: training-nvidia-checkpoints
- namespace: mist
-spec:
- accessModes: [ReadWriteOnce]
- storageClassName: local-path
- resources:
- requests:
- storage: 256Mi
----
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
- name: training-tenstorrent-checkpoints
- namespace: mist
-spec:
- accessModes: [ReadWriteOnce]
- storageClassName: local-path
- resources:
- requests:
- storage: 256Mi
----
-apiVersion: apps/v1
-kind: Deployment
-metadata:
- name: mist-api
- namespace: mist-system
-spec:
- replicas: 1
- strategy:
- type: Recreate
- selector:
- matchLabels:
- app: mist-api
- template:
- metadata:
- labels:
- app: mist-api
- spec:
- serviceAccountName: mist-api
- nodeSelector:
- kubernetes.io/hostname: utmist-z1opa08
- securityContext:
- runAsNonRoot: true
- runAsUser: 65532
- seccompProfile:
- type: RuntimeDefault
- containers:
- - name: api
- image: mist-api:folders-20261021
- imagePullPolicy: Never
- ports:
- - name: http
- containerPort: 3000
- env:
- - name: MIST_TEAMS_ENABLED
- value: "true"
- - name: MIST_INTERNAL_TOKEN
- valueFrom:
- secretKeyRef:
- name: mist-internal-auth
- key: MIST_INTERNAL_TOKEN
- - name: MIST_EXECUTOR
- value: kubernetes
- - name: MIST_HTTP_ADDR
- value: 0.0.0.0:3000
- - name: MIST_NAMESPACE
- value: mist
- - name: MIST_PILOT_OWNER
- value: utmist
- # Local examples must be built/imported on the CPU/NVIDIA node.
- # Registry-backed images are approved here using exact references.
- - name: MIST_ALLOWED_IMAGES
- value: "python:3.10-slim,pytorch/pytorch:2.5.1-cuda12.4-cudnn9-runtime,ghcr.io/tenstorrent/tt-metal/tt-metalium/ubuntu-22.04-dev-amd64:6b765c0bc2b40ea02566d169d7b0ee902377ba04,busybox:1.36,alpine:3.16.3,nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0,mist-training:cpu-v1,mist-training:nvidia-v1"
- - name: MIST_AUTH_DATABASE
- value: /data/auth.sqlite
- - name: BETTER_AUTH_SECRET
- valueFrom:
- secretKeyRef:
- name: mist-auth-bootstrap
- key: BETTER_AUTH_SECRET
- - name: MIST_ADMIN_EMAIL
- valueFrom:
- secretKeyRef:
- name: mist-auth-bootstrap
- key: MIST_ADMIN_EMAIL
- - name: MIST_ADMIN_PASSWORD
- valueFrom:
- secretKeyRef:
- name: mist-auth-bootstrap
- key: MIST_ADMIN_PASSWORD
- - name: MIST_TRUSTED_PROXY_IPS
- value: "10.42.0.1,10.0.0.175,127.0.0.1"
- - name: MIST_LEGACY_OWNER_EMAIL
- value: utmist.infra@gmail.com
- - name: MIST_ALLOWED_ORIGINS
- value: http://100.73.139.66:8088,http://utmist.tail459b5e.ts.net:8088,http://127.0.0.1:8088,http://127.0.0.1:3001,http://localhost:3001,https://utmist.tail459b5e.ts.net
- - name: MIST_STORAGE_ROOT
- value: /storage
- - name: MIST_NFS_SERVER
- value: 192.168.240.2
- - name: MIST_NFS_ROOT
- value: /srv/mist-data
- - name: MIST_SHARED_PVC
- value: mist-folder-jobs
- volumeMounts:
- - name: storage
- mountPath: /storage
- - name: auth-data
- mountPath: /data
- securityContext:
- allowPrivilegeEscalation: false
- readOnlyRootFilesystem: true
- capabilities:
- drop: [ALL]
- resources:
- requests:
- cpu: 100m
- memory: 128Mi
- limits:
- cpu: "2"
- memory: 512Mi
- readinessProbe:
- httpGet:
- path: /healthz
- port: http
- initialDelaySeconds: 2
- livenessProbe:
- httpGet:
- path: /healthz
- port: http
- initialDelaySeconds: 15
- periodSeconds: 15
- volumes:
- - name: storage
- persistentVolumeClaim:
- claimName: mist-folder-api
- - name: auth-data
- persistentVolumeClaim:
- claimName: mist-auth-data
----
-apiVersion: v1
-kind: Service
-metadata:
- name: mist-api
- namespace: mist-system
-spec:
- selector:
- app: mist-api
- ports:
- - port: 3000
- targetPort: http
diff --git a/deploy/k3s/mist-tenstorrent-claims.yaml b/deploy/k3s/mist-tenstorrent-claims.yaml
deleted file mode 100644
index 6436108..0000000
--- a/deploy/k3s/mist-tenstorrent-claims.yaml
+++ /dev/null
@@ -1,72 +0,0 @@
-# One per-pod claim; n300 count is boards, not individual chips.
-apiVersion: resource.k8s.io/v1
-kind: ResourceClaimTemplate
-metadata:
- name: mist-tenstorrent-1-boards
- namespace: mist
-spec:
- spec:
- devices:
- requests:
- - name: boards
- exactly:
- deviceClassName: tenstorrent.com
- count: 1
- selectors:
- - cel:
- expression: device.attributes["tenstorrent.com"].boardName == "n300"
- && device.attributes["tenstorrent.com"].chipCount == 2
----
-apiVersion: resource.k8s.io/v1
-kind: ResourceClaimTemplate
-metadata:
- name: mist-tenstorrent-2-boards
- namespace: mist
-spec:
- spec:
- devices:
- requests:
- - name: boards
- exactly:
- deviceClassName: tenstorrent.com
- count: 2
- selectors:
- - cel:
- expression: device.attributes["tenstorrent.com"].boardName == "n300"
- && device.attributes["tenstorrent.com"].chipCount == 2
----
-apiVersion: resource.k8s.io/v1
-kind: ResourceClaimTemplate
-metadata:
- name: mist-tenstorrent-3-boards
- namespace: mist
-spec:
- spec:
- devices:
- requests:
- - name: boards
- exactly:
- deviceClassName: tenstorrent.com
- count: 3
- selectors:
- - cel:
- expression: device.attributes["tenstorrent.com"].boardName == "n300"
- && device.attributes["tenstorrent.com"].chipCount == 2
----
-apiVersion: resource.k8s.io/v1
-kind: ResourceClaimTemplate
-metadata:
- name: mist-tenstorrent-4-boards
- namespace: mist
-spec:
- spec:
- devices:
- requests:
- - name: boards
- exactly:
- deviceClassName: tenstorrent.com
- count: 4
- selectors:
- - cel:
- expression: device.attributes["tenstorrent.com"].boardName == "n300"
- && device.attributes["tenstorrent.com"].chipCount == 2
diff --git a/deploy/k3s/mist-web.service b/deploy/k3s/mist-web.service
deleted file mode 100644
index 9a136d7..0000000
--- a/deploy/k3s/mist-web.service
+++ /dev/null
@@ -1,13 +0,0 @@
-[Unit]
-Description=Mist web interface on localhost
-After=mist-api-forward.service
-
-[Service]
-WorkingDirectory=/home/utmist/Mist/web-interface
-ExecStart=/home/utmist/.local/share/mist-runtimes/node-22.16.0/bin/node /home/utmist/Mist/web-interface/node_modules/vite/bin/vite.js --host 127.0.0.1 --port 3001 --strictPort
-Environment=MIST_API_PROXY=http://127.0.0.1:3000
-Restart=always
-RestartSec=2
-
-[Install]
-WantedBy=default.target
diff --git a/deploy/k3s/nvidia-gpu-operator-values.yaml b/deploy/k3s/nvidia-gpu-operator-values.yaml
deleted file mode 100644
index 5d897e5..0000000
--- a/deploy/k3s/nvidia-gpu-operator-values.yaml
+++ /dev/null
@@ -1,10 +0,0 @@
-# This node already has a working NVIDIA host driver. Let the operator install
-# the container toolkit and device plugin without replacing that driver.
-driver:
- enabled: false
-
-# CDI handles allocated workload GPUs. Enable the NRI plugin for GPU management
-# containers without changing k3s's containerd runtime configuration. This is
-# intended for the pilot; NVIDIA documents the NRI plugin as not yet GA.
-cdi:
- nriPluginEnabled: true
diff --git a/deploy/k3s/setup-runbook.md b/deploy/k3s/setup-runbook.md
deleted file mode 100644
index 643eb3a..0000000
--- a/deploy/k3s/setup-runbook.md
+++ /dev/null
@@ -1,9 +0,0 @@
-# Current k3s setup runbook
-
-The maintained two-node runbook is [k3s setup and accelerator handoff](../../docs/content/k3s-setup.md).
-It covers current versions, LAN/Tailscale roles, fresh server/worker boundaries,
-NVIDIA/TT operators, diagnostics and troubleshooting.
-
-- [Tailscale and administrator SSH](../../docs/content/tailscale-ssh.md)
-- [Application deployment and service operations](../private/README.md)
-- [Original single-node installation and worker-join history](../../docs/content/archive/k3s-setup-history.md)
diff --git a/deploy/k3s/storage/README.md b/deploy/k3s/storage/README.md
deleted file mode 100644
index 9a15f42..0000000
--- a/deploy/k3s/storage/README.md
+++ /dev/null
@@ -1,144 +0,0 @@
-# Shared folders over direct Ethernet
-
-Storage implementation for the two-machine installation, October 21, 2026.
-The authoritative data for new submissions is an ordinary directory on QuietBox's
-existing disk. There are no new disk images, loop devices or per-team mounts.
-
-```text
-NVIDIA enp6s0 192.168.240.1/30
- |
- direct 1-Gbps Ethernet
- |
-QuietBox enp200s0 192.168.240.2/30
- /srv/mist-data/
- .mist-storage-layout.json
- metadata/provision-requests/ API-owned desired teams
- metadata/provision-status/ readiness and sampled usage
- teams//
- datasets/
- metadata/datasets/ published dataset records
- .uploads/ upload staging
- common/datasets//content/
- members//datasets//content/
- results/
- common/jobs//outputs/
- members//jobs//outputs/
-```
-
-## Access and limits
-
-- Accounts and sessions remain in the main-node local SQLite PVC, separate from
- shared files. Go handles authentication and team authorization.
-- All members can read their team's folders. A member writes their own folder;
- common writes require permission. Other teams require an explicit grant.
-- Kubernetes mounts only the selected dataset and the job's output directory.
- Datasets are read-only at `/inputs`; results are writable at `/outputs`.
-- `storage_gib` and `model_storage_gib` remain API fields, with defaults of
- 200 and 500 GiB. They are **advisory allowances**. They neither reserve physical
- space nor prevent job writes. They may be raised/lowered without migration.
-- Every team shares the underlying disk. No equal-space guarantee or automatic
- backup, replication, dataset cache or result deletion is provided.
-- Uploads use current disk free space minus 1 GiB, plus an optional
- `MIST_MAX_DATASET_GIB` ceiling. Jobs can still exhaust the shared disk.
- ZIP processing needs both archive and extraction space. Transfers stream;
- interrupted uploads restart. Existing CPU/device/runtime policies remain hard
- admission limits.
-
-## Provisioning and usage
-
-`team_folders.py` is a local root-owned service without a network listener or
-cluster credentials. It accepts bounded API-owned manifests, creates directories,
-rejects symlinks/unexpected mounts and publishes `shared-folders-v1` readiness.
-Usage counts logical regular-file bytes, excludes symlinks, and deduplicates hard
-links within each pool. The host samples every 30 seconds; active writes may make
-these approximate. The API does not walk the NFS directory tree for every poll.
-
-`GET /storage` reports:
-
-- `capacity_bytes`, `available_bytes`: actual shared filesystem space.
-- `dataset_used_bytes`, `model_used_bytes`: sampled team file usage.
-- `model_capacity_bytes`, `model_available_bytes`: compatibility aliases for the
- same shared disk, not a separate model filesystem or reservation.
-- `dataset_allocation_gib`, `model_allocation_gib`: advisory values.
-- `storage_limits_enforced: false`, `storage_mode: shared-folders`.
-- `usage_observed_at`, `usage_stale`, `storage_low`: freshness and disk warning.
-
-The old `GET /teams` field `storage_budget_gib` remains informational shared
-capacity for compatibility. Allocations are not summed into a reservation budget.
-
-## Host setup
-
-Run commands as root in the **owning host's** mount/network namespace. Scripts
-are installation-specific; verify interface names/addresses on another system.
-
-On NVIDIA:
-
-```bash
-bash deploy/k3s/storage/setup-storage-link.sh main
-```
-
-On QuietBox, from a checkout/copy of these files:
-
-```bash
-bash deploy/k3s/storage/setup-storage-link.sh quietbox
-bash deploy/k3s/storage/setup-quietbox.sh
-install -d -m 0755 /usr/local/lib/mist
-install -m 0755 deploy/k3s/storage/team_folders.py /usr/local/lib/mist/team_folders.py
-install -m 0644 deploy/k3s/storage/mist-team-storage.service /etc/systemd/system/
-systemctl daemon-reload
-systemctl enable --now mist-team-storage
-```
-
-The NetworkManager profile `mist-storage-direct` persists addresses without a
-gateway. Existing LAN/default routes, Tailscale and k3s node addresses stay in use.
-NFS permits only `192.168.240.1` and `192.168.240.2`; the firewall allows main's
-TCP 2049 on `enp200s0`. Root squash maps root requests to the storage API identity.
-
-New bindings use NFS **4.1**. During inspection, Linux reused an existing 4.2
-connection to QuietBox's old LAN address even when mounting its Ethernet alias.
-A separate 4.1 connection makes the new Ethernet transport verifiable without
-tearing down existing old mounts. Verify both the mount and TCP connection, not
-only the requested mount address.
-
-The initial cutover uses new `mist-folder-api`/`mist-folder-jobs` bindings and
-per-team `team-datasets`/`team-results` claims, leaving old bindings intact.
-Apply `shared.yaml` and the API deployment, then wait for new team readiness.
-`ensureVolume` refuses an existing binding pointing to a different NFS export.
-
-## Prelaunch reset and rollback
-
-The user chose fresh storage without data migration. Old files under
-`/srv/mist-storage` and its backing images remain untouched as rollback material.
-Old retained Jobs may still appear in history; their old files are not copied to
-the new root. New datasets and results use `/srv/mist-data`.
-
-Before cutover, record the API/auth deployment specifications, web release target
-and team policies. Create a consistent SQLite backup using its database backup
-API. Pause the old API/auth before the Go-auth API takes the shared auth PVC.
-
-Rollback restores the previous API/auth deployment and provisioning unit, and
-repoints Nginx's web release symlink. Old volumes were not replaced or deleted.
-An operational rollback after accepting new writes must account for those writes;
-this is a prelaunch cutover, not automatic bidirectional migration.
-
-## Verify
-
-```bash
-python3 -m unittest discover -s deploy/k3s/storage -p test_folders.py -v
-python3 deploy/k3s/verify_storage_folders.py --out /tmp/mist-storage-check
-```
-
-The live verifier creates two workspaces and a regular account, checks small
-uploads/downloads, archive rejection and team isolation, then trains on one
-NVIDIA GPU and one n300 board. It retains a protected account file in its output
-directory for optional browser review; never commit that file. Completed test
-jobs/results remain inspectable. It does not migrate or delete old research data.
-
-Optional browser review uses Playwright from `MIST_PLAYWRIGHT_MODULE` (the local
-installation defaults to `/tmp/mist-browser-check/node_modules/playwright/index.mjs`):
-
-```bash
-MIST_STORAGE_EVIDENCE_DIR=/tmp/mist-storage-check node deploy/k3s/verify_storage_folders_browser.mjs
-```
-
-Recorded live results: [October 21 evidence](../evidence/storage-2026-10-21/README.md).
diff --git a/deploy/k3s/storage/mist-team-storage.service b/deploy/k3s/storage/mist-team-storage.service
deleted file mode 100644
index 2ffc705..0000000
--- a/deploy/k3s/storage/mist-team-storage.service
+++ /dev/null
@@ -1,16 +0,0 @@
-[Unit]
-Description=Mist shared team folders and usage reporting
-After=local-fs.target nfs-server.service
-RequiresMountsFor=/srv/mist-data
-Requires=nfs-server.service
-
-[Service]
-Type=simple
-Environment=MIST_STORAGE_ROOT=/srv/mist-data
-ExecStart=/usr/bin/python3 /usr/local/lib/mist/team_folders.py
-Restart=on-failure
-RestartSec=5
-UMask=0077
-
-[Install]
-WantedBy=multi-user.target
diff --git a/deploy/k3s/storage/setup-quietbox.sh b/deploy/k3s/storage/setup-quietbox.sh
deleted file mode 100755
index 799941f..0000000
--- a/deploy/k3s/storage/setup-quietbox.sh
+++ /dev/null
@@ -1,41 +0,0 @@
-#!/usr/bin/env bash
-# Initialize ordinary storage folders on QuietBox. Never formats/mounts a disk.
-set -euo pipefail
-[ "$(id -u)" -eq 0 ] || { echo 'Run as root on QuietBox' >&2; exit 1; }
-root=/srv/mist-data
-[ ! -L "$root" ] || { echo 'Refusing a symlink storage root' >&2; exit 1; }
-if [ -d "$root" ] && [ -n "$(ls -A "$root")" ] && [ ! -f "$root/.mist-storage-layout.json" ]; then
- echo 'Refusing to adopt a nonempty unknown directory' >&2
- exit 1
-fi
-install -d -o 65532 -g 65532 -m 0750 "$root"
-if [ -e "$root/.mist-storage-layout.json" ]; then
- python3 - "$root" <<'PY'
-import json, sys
-from pathlib import Path
-marker = Path(sys.argv[1]) / '.mist-storage-layout.json'
-assert not marker.is_symlink()
-assert json.loads(marker.read_text())['layout'] == 'shared-folders-v1'
-PY
-else
- printf '{"layout":"shared-folders-v1"}\n' > "$root/.mist-storage-layout.json"
- chmod 0644 "$root/.mist-storage-layout.json"
-fi
-install -d -o 65532 -g 65532 -m 0700 "$root/metadata" "$root/metadata/provision-requests" "$root/metadata/provision-status"
-install -d -o 65532 -g 65532 -m 0755 "$root/teams"
-install -d /etc/exports.d /etc/nfs.conf.d
-cat > /etc/exports.d/mist-folders.exports <<'EXPORT'
-/srv/mist-data 192.168.240.1(rw,sync,no_subtree_check,root_squash,anonuid=65532,anongid=65532,fsid=8410) 192.168.240.2(rw,sync,no_subtree_check,root_squash,anonuid=65532,anongid=65532,fsid=8410)
-EXPORT
-cat > /etc/nfs.conf.d/mist.conf <<'CONF'
-[nfsd]
-vers3=n
-vers4=y
-CONF
-systemctl enable --now nfs-server
-exportfs -ra
-if ufw status | grep -q 'Status: active'; then
- ufw allow in on enp200s0 from 192.168.240.1 to any port 2049 proto tcp comment mist-direct-nfs
-fi
-findmnt -T "$root"
-df -h "$root"
diff --git a/deploy/k3s/storage/setup-storage-link.sh b/deploy/k3s/storage/setup-storage-link.sh
deleted file mode 100644
index 33ced6c..0000000
--- a/deploy/k3s/storage/setup-storage-link.sh
+++ /dev/null
@@ -1,35 +0,0 @@
-#!/usr/bin/env bash
-# Run on each owning host as root. No gateway or cluster-interface changes.
-set -euo pipefail
-case "${1:-}" in
- main) iface=enp6s0; address=192.168.240.1/30 ;;
- quietbox) iface=enp200s0; address=192.168.240.2/30 ;;
- *) echo 'Usage: setup-storage-link.sh main|quietbox' >&2; exit 2 ;;
-esac
-[ "$(id -u)" -eq 0 ] || { echo 'Run as root on the owning host' >&2; exit 1; }
-[ "$(cat "/sys/class/net/$iface/carrier")" = 1 ] || { echo "$iface has no cable link" >&2; exit 1; }
-[ -z "$(ip -4 route show default dev "$iface")" ] || { echo 'Refusing to replace a default-route connection' >&2; exit 1; }
-profile=mist-storage-direct
-current=$(nmcli -g GENERAL.CONNECTION device show "$iface")
-install -d -m 0700 /var/lib/mist
-if [ -n "$current" ] && [ "$current" != '--' ] && [ "$current" != "$profile" ]; then
- uuid=$(nmcli -g connection.uuid connection show "$current")
- auto=$(nmcli -g connection.autoconnect connection show "$uuid")
- record=/var/lib/mist/storage-link-previous.txt
- if [ ! -f "$record" ]; then
- printf '%s\n%s\n%s\n' "$iface" "$uuid" "$auto" > "$record"
- chmod 0600 "$record"
- fi
- nmcli connection modify "$uuid" connection.autoconnect no
-fi
-if nmcli connection show "$profile" >/dev/null 2>&1; then
- [ "$(nmcli -g connection.interface-name connection show "$profile")" = "$iface" ] || exit 1
-else
- nmcli connection add type ethernet ifname "$iface" con-name "$profile"
-fi
-nmcli connection modify "$profile" \
- connection.autoconnect yes connection.autoconnect-priority 100 \
- ipv4.method manual ipv4.addresses "$address" ipv4.gateway '' \
- ipv4.dns '' ipv4.never-default yes ipv6.method disabled
-nmcli --wait 15 connection up "$profile"
-ip -4 address show dev "$iface"
diff --git a/deploy/k3s/storage/shared.yaml b/deploy/k3s/storage/shared.yaml
deleted file mode 100644
index 5298e0d..0000000
--- a/deploy/k3s/storage/shared.yaml
+++ /dev/null
@@ -1,62 +0,0 @@
-apiVersion: v1
-kind: PersistentVolume
-metadata:
- name: mist-folder-api
-spec:
- capacity:
- storage: 1Gi
- accessModes: [ReadWriteMany]
- persistentVolumeReclaimPolicy: Retain
- storageClassName: ""
- claimRef:
- name: mist-folder-api
- namespace: mist-system
- mountOptions: [nfsvers=4.1, hard, timeo=100, retrans=2]
- nfs:
- server: 192.168.240.2
- path: /srv/mist-data
----
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
- name: mist-folder-api
- namespace: mist-system
-spec:
- storageClassName: ""
- volumeName: mist-folder-api
- accessModes: [ReadWriteMany]
- resources:
- requests:
- # Static NFS binding declaration, not a quota. API reports real disk space.
- storage: 1Gi
----
-apiVersion: v1
-kind: PersistentVolume
-metadata:
- name: mist-folder-jobs
-spec:
- capacity:
- storage: 1Gi
- accessModes: [ReadWriteMany]
- persistentVolumeReclaimPolicy: Retain
- storageClassName: ""
- claimRef:
- name: mist-folder-jobs
- namespace: mist
- mountOptions: [nfsvers=4.1, hard, timeo=100, retrans=2]
- nfs:
- server: 192.168.240.2
- path: /srv/mist-data
----
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
- name: mist-folder-jobs
- namespace: mist
-spec:
- storageClassName: ""
- volumeName: mist-folder-jobs
- accessModes: [ReadWriteMany]
- resources:
- requests:
- storage: 1Gi
diff --git a/deploy/k3s/storage/team_folders.py b/deploy/k3s/storage/team_folders.py
deleted file mode 100644
index 50c6efc..0000000
--- a/deploy/k3s/storage/team_folders.py
+++ /dev/null
@@ -1,151 +0,0 @@
-#!/usr/bin/env python3
-"""Prepare ordinary team directories and publish usage; never format or migrate.
-
-Only bounded API-owned requests outside workload mounts are accepted. Allowances
-are advisory. The filesystem's real free space is shared by all teams.
-"""
-import json
-import os
-from pathlib import Path
-import re
-import stat
-import tempfile
-import time
-
-LAYOUT = 'shared-folders-v1'
-TEAM_ID = re.compile(r'^team-[a-f0-9]{16}$')
-
-
-class FolderStore:
- def __init__(self, root, uid=65532, gid=65532):
- self.root = Path(root)
- self.uid, self.gid = uid, gid
- marker = self.root / '.mist-storage-layout.json'
- if self.root.is_symlink() or marker.is_symlink():
- raise ValueError('storage root and layout marker must not be symlinks')
- if json.loads(marker.read_text()).get('layout') != LAYOUT:
- raise ValueError('shared-folder storage has not been initialized')
- self.device = self.root.stat().st_dev
- self.requests = self.directory('metadata/provision-requests', 0o700)
- self.status = self.directory('metadata/provision-status', 0o700)
- self.directory('teams', 0o755)
-
- def directory(self, relative, mode):
- path = self.root
- parts = Path(relative).parts
- if not parts or any(part in ('.', '..', '/') for part in parts):
- raise ValueError('invalid storage directory')
- for part in parts:
- path = path / part
- if not path.exists() and not path.is_symlink():
- path.mkdir(mode=0o750)
- os.chown(path, self.uid, self.gid)
- entry = path.lstat()
- if not stat.S_ISDIR(entry.st_mode) or entry.st_dev != self.device:
- raise ValueError('storage directories must be ordinary directories on the shared disk')
- os.chmod(path, mode)
- os.chown(path, self.uid, self.gid)
- return path
-
- def request(self, path):
- entry = path.lstat()
- if (not TEAM_ID.fullmatch(path.stem) or not stat.S_ISREG(entry.st_mode)
- or entry.st_uid != self.uid or entry.st_size > 1024):
- raise ValueError('request must be a bounded API-owned regular file')
- body = json.loads(path.read_text())
- if set(body) != {'id', 'storage_gib', 'model_storage_gib'} or body['id'] != path.stem:
- raise ValueError('invalid team request schema')
- for key in ('storage_gib', 'model_storage_gib'):
- if type(body[key]) is not int or not 1 <= body[key] <= 2**20:
- raise ValueError('invalid advisory allowance')
- return body
-
- @staticmethod
- def usage(path):
- """Logical regular-file bytes. Never follow workload-created symlinks."""
- total = 0
- device = path.stat().st_dev
- seen = set()
- def failed(error):
- if not isinstance(error, FileNotFoundError):
- raise error
- for _, _, files, descriptor in os.fwalk(path, follow_symlinks=False, onerror=failed):
- if os.fstat(descriptor).st_dev != device:
- raise ValueError('unexpected nested filesystem in team storage')
- for name in files:
- try:
- info = os.stat(name, dir_fd=descriptor, follow_symlinks=False)
- except FileNotFoundError:
- continue # A live job may rename/delete files during sampling.
- if stat.S_ISREG(info.st_mode):
- identity = (info.st_dev, info.st_ino)
- if identity not in seen:
- total += info.st_size
- seen.add(identity)
- return total
-
- def provision(self, body, previous=None):
- team = body['id']
- if not TEAM_ID.fullmatch(team):
- raise ValueError('invalid team ID')
- datasets = self.directory(f'teams/{team}/datasets', 0o750)
- results = self.directory(f'teams/{team}/results', 0o750)
- for name in ('metadata', 'metadata/datasets', '.uploads'):
- self.directory(f'teams/{team}/datasets/{name}', 0o700)
- for pool in ('datasets', 'results'):
- for scope in ('common', 'members'):
- self.directory(f'teams/{team}/{pool}/{scope}', 0o755)
- usage = previous or {}
- if time.time() - usage.get('usage_observed_unix', 0) >= 30:
- usage = {'dataset_used_bytes': self.usage(datasets),
- 'model_used_bytes': self.usage(results),
- 'usage_observed_unix': int(time.time())}
- return {**body, 'state': 'Ready', 'layout': LAYOUT,
- **{key: usage[key] for key in
- ('dataset_used_bytes', 'model_used_bytes', 'usage_observed_unix')}}
-
- def publish(self, team, value):
- path = self.status / f'{team}.json'
- serialized = json.dumps(value, sort_keys=True)
- if path.exists() and not path.is_symlink() and path.read_text() == serialized:
- return
- fd, temporary = tempfile.mkstemp(prefix='.mist-', dir=self.status)
- try:
- with os.fdopen(fd, 'w') as stream:
- stream.write(serialized)
- stream.flush()
- os.fsync(stream.fileno())
- os.chown(temporary, self.uid, self.gid)
- os.replace(temporary, path)
- finally:
- if os.path.exists(temporary):
- os.unlink(temporary)
-
- def reconcile(self, cache):
- for path in sorted(self.requests.glob('team-*.json')):
- try:
- body = self.request(path)
- except Exception as error:
- print(f'Request rejected: {path.name}: {error}', flush=True)
- continue
- try:
- value = self.provision(body, cache.get(body['id']))
- cache[body['id']] = value
- except Exception as error:
- value = {'state': 'Error', 'layout': LAYOUT, 'error': str(error)}
- print(f"Provisioning failed: {body['id']}: {error}", flush=True)
- self.publish(body['id'], value)
-
-
-def main():
- if os.geteuid() != 0:
- raise SystemExit('Run as root on the storage host')
- store = FolderStore(os.environ.get('MIST_STORAGE_ROOT', '/srv/mist-data'))
- cache = {}
- while True:
- store.reconcile(cache)
- time.sleep(2)
-
-
-if __name__ == '__main__':
- main()
diff --git a/deploy/k3s/storage/test_folders.py b/deploy/k3s/storage/test_folders.py
deleted file mode 100644
index 714afd9..0000000
--- a/deploy/k3s/storage/test_folders.py
+++ /dev/null
@@ -1,95 +0,0 @@
-import json
-import os
-from pathlib import Path
-import tempfile
-import unittest
-
-from team_folders import FolderStore, LAYOUT
-
-
-class FolderTests(unittest.TestCase):
- def setUp(self):
- self.temporary = tempfile.TemporaryDirectory()
- self.addCleanup(self.temporary.cleanup)
- self.root = Path(self.temporary.name)
- (self.root / '.mist-storage-layout.json').write_text(json.dumps({'layout': LAYOUT}))
- self.store = FolderStore(self.root, os.getuid(), os.getgid())
- self.team = 'team-aaaaaaaaaaaaaaaa'
- self.body = {'id': self.team, 'storage_gib': 200, 'model_storage_gib': 500}
-
- def request(self, body):
- path = self.store.requests / f'{self.team}.json'
- path.write_text(json.dumps(body))
- return path
-
- def test_plain_folders_preserve_files_and_allow_lower_allowances(self):
- ready = self.store.provision(self.body)
- data = self.root / 'teams' / self.team / 'datasets'
- models = data.parent / 'results'
- (data / 'common' / 'tiny.csv').write_bytes(b'1,2\n')
- (models / 'common' / 'model.bin').write_bytes(b'weights')
- lower = {**self.body, 'storage_gib': 1, 'model_storage_gib': 1}
- updated = self.store.provision(lower)
- self.assertEqual(ready['layout'], LAYOUT)
- self.assertEqual(updated['dataset_used_bytes'], 4)
- self.assertEqual(updated['model_used_bytes'], 7)
- self.assertEqual(updated['storage_gib'], 1)
- self.assertEqual(data.stat().st_dev, models.stat().st_dev)
- self.assertFalse(list(self.root.rglob('*.img')))
-
- def test_usage_does_not_follow_links_or_double_count_hardlinks(self):
- self.store.provision(self.body)
- data = self.root / 'teams' / self.team / 'datasets'
- (data / 'common' / 'data').write_bytes(b'123')
- os.link(data / 'common' / 'data', data / 'common' / 'alias')
- (data / 'common' / 'outside').symlink_to('/etc')
- (data / 'common' / 'file-link').symlink_to('/etc/passwd')
- self.assertEqual(FolderStore.usage(data), 3)
-
- def test_unsafe_paths_fail_and_do_not_change_target(self):
- outside = self.root / 'outside'
- outside.mkdir(mode=0o700)
- (self.root / 'teams' / self.team).symlink_to(outside)
- with self.assertRaises(ValueError):
- self.store.provision(self.body)
- self.assertEqual(list(outside.iterdir()), [])
-
- def test_requests_reject_symlinks_invalid_schema_ids_and_allowances(self):
- path = self.request(self.body)
- self.assertEqual(self.store.request(path), self.body)
- for body in [dict(self.body, storage_gib=True), dict(self.body, model_storage_gib=0),
- dict(self.body, id='../outside'), dict(self.body, extra='unexpected')]:
- self.request(body)
- with self.assertRaises(ValueError):
- self.store.request(path)
- path.unlink()
- target = self.root / 'request.json'
- target.write_text(json.dumps(self.body))
- path.symlink_to(target)
- with self.assertRaises(ValueError):
- self.store.request(path)
-
- def test_reconciliation_publishes_errors_and_recovers(self):
- self.request(self.body)
- cache = {}
- self.store.reconcile(cache)
- path = self.store.status / f'{self.team}.json'
- self.assertEqual(json.loads(path.read_text())['state'], 'Ready')
- results = self.root / 'teams' / self.team / 'results'
- results.rename(results.with_name('retained'))
- results.symlink_to('/tmp')
- self.store.reconcile(cache)
- self.assertEqual(json.loads(path.read_text())['state'], 'Error')
- results.unlink()
- results.with_name('retained').rename(results)
- self.store.reconcile(cache)
- self.assertEqual(json.loads(path.read_text())['state'], 'Ready')
-
- def test_uninitialized_root_is_refused(self):
- (self.root / '.mist-storage-layout.json').write_text('{"layout":"old-image-layout"}')
- with self.assertRaises(ValueError):
- FolderStore(self.root, os.getuid(), os.getgid())
-
-
-if __name__ == '__main__':
- unittest.main()
diff --git a/deploy/k3s/submit_tenstorrent_smoke.py b/deploy/k3s/submit_tenstorrent_smoke.py
deleted file mode 100644
index bec956a..0000000
--- a/deploy/k3s/submit_tenstorrent_smoke.py
+++ /dev/null
@@ -1,59 +0,0 @@
-"""Submit a uniquely named training check reserving one to four n300 boards.
-
-Uses the admin KUBECONFIG, k3s kubectl, and Python PyYAML. This submits the
-small TT-NN regression test; application/model integration is separate.
-"""
-
-import argparse
-import copy
-from pathlib import Path
-import re
-import subprocess
-
-import yaml
-
-
-def manifests(name, boards, steps):
- docs = list(yaml.safe_load_all(Path(__file__).with_name("training-tenstorrent-job.yaml").read_text()))
- pvc, claim, job = copy.deepcopy(docs)
- claim["metadata"]["name"] = f"training-tenstorrent-{boards}-boards"
- claim["spec"]["spec"]["devices"]["requests"][0]["exactly"]["count"] = boards
- job["metadata"]["name"] = name
- pod = job["spec"]["template"]["spec"]
- pod["resourceClaims"][0]["resourceClaimTemplateName"] = claim["metadata"]["name"]
- train = pod["containers"][0]
- train["command"] = train["command"][:4] + ["--expected-devices", str(boards * 2),
- "--steps", str(steps), "--output", f"/checkpoints/{name}"]
- for resources in (train["resources"]["requests"], train["resources"]["limits"]):
- resources["hugepages-1Gi"] = f"{boards * 2}Gi"
- train["resources"]["requests"]["cpu"] = str(boards * 2)
- train["resources"]["requests"]["memory"] = f"{boards * 4}Gi"
- train["resources"]["limits"]["cpu"] = str(boards * 8)
- train["resources"]["limits"]["memory"] = f"{boards * 8}Gi"
- return pvc, claim, job
-
-
-if __name__ == "__main__":
- parser = argparse.ArgumentParser()
- parser.add_argument("name", help="Unique Kubernetes Job name; also used for the checkpoint directory")
- parser.add_argument("--boards", type=int, choices=range(1, 5), default=1)
- parser.add_argument("--steps", type=int, default=100)
- parser.add_argument("--dry-run", action="store_true", help="Print the manifests without changing the cluster")
- args = parser.parse_args()
- if len(args.name) > 63 or not re.fullmatch(r"[a-z0-9](?:[-a-z0-9]*[a-z0-9])?", args.name):
- parser.error("name must be a DNS label with at most 63 characters")
- if args.steps < 100:
- parser.error("at least 100 steps are required for the convergence check")
- pvc, claim, job = manifests(args.name, args.boards, args.steps)
- if args.dry_run:
- print(yaml.safe_dump_all([pvc, claim, job]), end="")
- else:
- subprocess.run(["k3s", "kubectl", "apply", "-f", "-"],
- input=yaml.safe_dump_all([pvc, claim]), text=True, check=True)
- configmap = subprocess.run(["k3s", "kubectl", "-n", "mist", "create", "configmap",
- "training-smoke-script", "--from-file=" + str(Path(__file__).with_name("training_smoke.py")),
- "--dry-run=client", "-o", "yaml"], text=True, capture_output=True, check=True).stdout
- subprocess.run(["k3s", "kubectl", "apply", "-f", "-"], input=configmap, text=True, check=True)
- subprocess.run(["k3s", "kubectl", "create", "-f", "-"],
- input=yaml.safe_dump(job), text=True, check=True)
- print(f"Requested {args.boards} board(s), {args.boards * 2} chips; checkpoints: /checkpoints/{args.name}")
diff --git a/deploy/k3s/team-rbac.yaml b/deploy/k3s/team-rbac.yaml
deleted file mode 100644
index 0a44d35..0000000
--- a/deploy/k3s/team-rbac.yaml
+++ /dev/null
@@ -1,39 +0,0 @@
-apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRole
-metadata:
- name: mist-team-controller
-rules:
- - apiGroups: [""]
- resources: [namespaces, configmaps, persistentvolumes, persistentvolumeclaims, resourcequotas]
- verbs: [get, list, create, update, patch]
- - apiGroups: [""]
- resources: [configmaps]
- verbs: [delete]
- - apiGroups: [batch]
- resources: [jobs]
- verbs: [get, list, create, update, patch, delete]
- - apiGroups: [""]
- resources: [pods, events]
- verbs: [get, list]
- - apiGroups: [""]
- resources: [pods/log]
- verbs: [get]
- - apiGroups: [resource.k8s.io]
- resources: [resourceclaims, resourceclaimtemplates]
- verbs: [get, list, create]
- - apiGroups: [networking.k8s.io]
- resources: [networkpolicies]
- verbs: [get, list, create, update]
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRoleBinding
-metadata:
- name: mist-team-controller
-roleRef:
- apiGroup: rbac.authorization.k8s.io
- kind: ClusterRole
- name: mist-team-controller
-subjects:
- - kind: ServiceAccount
- name: mist-api
- namespace: mist-system
diff --git a/deploy/k3s/team-workload-policy.yaml b/deploy/k3s/team-workload-policy.yaml
deleted file mode 100644
index f12757e..0000000
--- a/deploy/k3s/team-workload-policy.yaml
+++ /dev/null
@@ -1,38 +0,0 @@
-apiVersion: admissionregistration.k8s.io/v1
-kind: ValidatingAdmissionPolicy
-metadata:
- name: mist-team-boundary
-spec:
- failurePolicy: Fail
- matchConstraints:
- resourceRules:
- - apiGroups: [""]
- apiVersions: [v1]
- operations: [CREATE, UPDATE]
- resources: [pods, pods/ephemeralcontainers]
- variables:
- - name: containers
- expression: 'object.spec.containers + (has(object.spec.initContainers) ? object.spec.initContainers : []) + (has(object.spec.ephemeralContainers) ? object.spec.ephemeralContainers : [])'
- validations:
- - expression: '(!has(object.spec.hostNetwork) || !object.spec.hostNetwork) && (!has(object.spec.hostPID) || !object.spec.hostPID) && (!has(object.spec.hostIPC) || !object.spec.hostIPC)'
- message: Team workloads cannot join host namespaces.
- - expression: 'has(object.spec.automountServiceAccountToken) && !object.spec.automountServiceAccountToken'
- message: Team workloads must disable Kubernetes credentials.
- - expression: 'has(object.spec.securityContext) && has(object.spec.securityContext.seccompProfile) && object.spec.securityContext.seccompProfile.type == "RuntimeDefault"'
- message: Team workloads must use the default seccomp profile.
- - expression: 'variables.containers.all(c, has(c.securityContext) && has(c.securityContext.allowPrivilegeEscalation) && !c.securityContext.allowPrivilegeEscalation && (!has(c.securityContext.privileged) || !c.securityContext.privileged) && has(c.securityContext.capabilities) && has(c.securityContext.capabilities.drop) && "ALL" in c.securityContext.capabilities.drop && (!has(c.securityContext.capabilities.add) || size(c.securityContext.capabilities.add) == 0))'
- message: Team containers must drop capabilities and forbid privilege escalation.
- - expression: '!has(object.spec.volumes) || object.spec.volumes.all(v, has(v.persistentVolumeClaim) || has(v.configMap) || has(v.emptyDir) || (has(v.hostPath) && v.hostPath.path in ["/home/utmist-tt/TT/tt-metal", "/home/utmist-tt/.local/share/uv/python"] && variables.containers.all(c, !has(c.volumeMounts) || c.volumeMounts.all(m, m.name != v.name || (has(m.readOnly) && m.readOnly)))))'
- message: Team volumes allow PVCs, scripts, scratch space and read-only TT runtime mounts.
----
-apiVersion: admissionregistration.k8s.io/v1
-kind: ValidatingAdmissionPolicyBinding
-metadata:
- name: mist-team-boundary
-spec:
- policyName: mist-team-boundary
- validationActions: [Deny]
- matchResources:
- namespaceSelector:
- matchLabels:
- mist.io/workloads: "true"
diff --git a/deploy/k3s/tenstorrent-allocation-results.json b/deploy/k3s/tenstorrent-allocation-results.json
deleted file mode 100644
index c62a4a7..0000000
--- a/deploy/k3s/tenstorrent-allocation-results.json
+++ /dev/null
@@ -1,153 +0,0 @@
-{
- "verified_at": "2026-10-03T19:23:13.903202+00:00",
- "node": "utmist-tt",
- "k3s_version": "v1.36.4+k3s1",
- "hardware": {
- "boards": 4,
- "type": "n300",
- "chips_per_board": 2,
- "total_chips": 8
- },
- "operator": {
- "chart_version": "0.3.0",
- "fabric_manager_version": "0.2.30",
- "dra_driver_version": "0.0.59",
- "namespace": "tt-operator-system",
- "host_driver_managed": false,
- "image_ids": {
- "ghcr.io/tenstorrent/tt-fabric-manager:0.2.30": "ghcr.io/tenstorrent/tt-fabric-manager@sha256:82c9063a5182051f0980459eaacb3e2d878a197874dfe05c6a5eb9288f4cca33",
- "ghcr.io/tenstorrent/tt-dra-driver:0.0.59": "ghcr.io/tenstorrent/tt-dra-driver@sha256:ac0089e9b6939170300aa325c62605b48cf12313ca6cec8c9d8a04f416d12dfb"
- }
- },
- "allocation": {
- "device_class": "tenstorrent.com",
- "unit": "one n300 board including both chips",
- "claims": "one ResourceClaimTemplate-generated claim per pod",
- "hugepages_1Gi_per_board": "2Gi",
- "privileged_training": false
- },
- "concurrency_test": {
- "run": "tt-alloc-1003-190834",
- "simultaneous_jobs": 4,
- "boards_per_job": 1,
- "chips_tested_per_job": 2,
- "steps_per_chip": 500,
- "overlap_seconds": 3.924301,
- "reserved_devices": {
- "tt-alloc-1003-190834-0-wzcrc": "tt-3",
- "tt-alloc-1003-190834-1-b8fdc": "tt-1",
- "tt-alloc-1003-190834-2-5qqtz": "tt-2",
- "tt-alloc-1003-190834-3-f552p": "tt-0"
- },
- "device_access_probes": {
- "tt-alloc-1003-190834-0-wzcrc": {
- "allowed": [
- "0"
- ],
- "denied": [
- "1",
- "2",
- "3"
- ]
- },
- "tt-alloc-1003-190834-1-b8fdc": {
- "allowed": [
- "2"
- ],
- "denied": [
- "0",
- "1",
- "3"
- ]
- },
- "tt-alloc-1003-190834-2-5qqtz": {
- "allowed": [
- "3"
- ],
- "denied": [
- "0",
- "1",
- "2"
- ]
- },
- "tt-alloc-1003-190834-3-f552p": {
- "allowed": [
- "1"
- ],
- "denied": [
- "0",
- "2",
- "3"
- ]
- }
- },
- "fifth_job_pending": true,
- "queue_messages": [
- "0/2 nodes are available: 1 cannot allocate all claims, 1 node(s) didn't match Pod's node affinity/selector. still not schedulable, preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling."
- ],
- "fifth_job_completed_after_release": true,
- "all_jobs_exit_code": 0,
- "final_training_mse": 6.243230745894834e-05,
- "held_out_mse": 0.00017509252938907593,
- "checkpoint_reload_passed": true
- },
- "additional_checks": {
- "allocator_restart_recovered": true,
- "training_after_restart_chips": 2,
- "nvidia_cuda_regression": "Test PASSED",
- "multi_board_training": [
- {
- "boards": 2,
- "chips": 4,
- "passed": true,
- "checkpoint_reload_passed": true
- },
- {
- "boards": 4,
- "chips": 8,
- "passed": true,
- "checkpoint_reload_passed": true
- }
- ],
- "admission": {
- "allowed": {
- "accepted": true,
- "message": "pod/allocation-policy-check created (server dry run)\n"
- },
- "privileged": {
- "accepted": false,
- "message": "The Pod \"allocation-policy-check\" is invalid: spec.containers[0].securityContext: Invalid value: {\"Capabilities\":{\"Add\":null,\"Drop\":[\"ALL\"]},\"Privileged\":true,\"SELinuxOptions\":null,\"WindowsOptions\":null,\"RunAsUser\":null,\"RunAsGroup\":null,\"RunAsNonRoot\":null,\"ReadOnlyRootFilesystem\":null,\"AllowPrivilegeEscalation\":false,\"ProcMount\":null,\"SeccompProfile\":null,\"AppArmorProfile\":null}: cannot set `allowPrivilegeEscalation` to false and `privileged` to true\n"
- },
- "docker-socket": {
- "accepted": false,
- "message": "The pods \"allocation-policy-check\" is invalid: : ValidatingAdmissionPolicy 'mist-accelerator-isolation' with binding 'mist-accelerator-isolation' denied request: Use DRA/CDI for devices and PVCs for storage; arbitrary host mounts are forbidden in Mist.\n"
- },
- "writable-metal": {
- "accepted": false,
- "message": "The pods \"allocation-policy-check\" is invalid: : ValidatingAdmissionPolicy 'mist-accelerator-isolation' with binding 'mist-accelerator-isolation' denied request: The host TT-Metal and Python environment mounts must be read-only.\n"
- },
- "added-capability": {
- "accepted": false,
- "message": "The pods \"allocation-policy-check\" is invalid: : ValidatingAdmissionPolicy 'mist-accelerator-isolation' with binding 'mist-accelerator-isolation' denied request: Mist workloads cannot enable privileged access, privilege escalation, or additional Linux capabilities.\n"
- }
- }
- },
- "runtime": {
- "tt_metal_host_commit": "80ec10ab611570a6818fdebc1398c8c9141fb9df",
- "multi_board_smoke": "Separate runtime per reserved board, selected by PCI BDF in TT_VISIBLE_DEVICES; each chip trains independently."
- },
- "limitations": [
- "Training is a synthetic TTNN linear-regression test with explicit analytic gradients, not arbitrary-model autograd or distributed training.",
- "DRA board-count requests do not guarantee a connected fabric topology for multi-board models.",
- "Mist API integration still needs to create Kubernetes Jobs and device claims.",
- "Host runtime directories and hugepage filesystems remain shared; this is not a full untrusted-tenant security boundary.",
- "Vendor ResourceSlice memory capacity is incorrect (281453674972256 bytes per n300); claims intentionally do not use memory-capacity placement."
- ],
- "artifacts": {
- "directory": "/home/utmist/k3s-allocation-results-2026-10-03",
- "concurrent_checkpoints": "/home/utmist/k3s-allocation-results-2026-10-03/concurrent-checkpoints.tar.gz",
- "recovery_and_multi_board_checkpoints": "/home/utmist/k3s-allocation-results-2026-10-03/additional-checkpoints.tar.gz",
- "checkpoint_pvc": "mist/training-tenstorrent-checkpoints",
- "concurrent_pvc_directory": "/checkpoints/tt-alloc-1003-190834"
- }
-}
diff --git a/deploy/k3s/tenstorrent-operator-values.yaml b/deploy/k3s/tenstorrent-operator-values.yaml
deleted file mode 100644
index 63f75aa..0000000
--- a/deploy/k3s/tenstorrent-operator-values.yaml
+++ /dev/null
@@ -1,29 +0,0 @@
-# Pinned with tt-operator chart 0.3.0. Host TT driver/firmware are managed
-# separately. Reuse the cluster's existing NVIDIA-provided NFD installation.
-node-feature-discovery:
- enabled: false
-tt-k8s-driver-manager:
- enabled: false
-tt-telemetry:
- enabled: false
-jobset:
- enabled: false
-kubepmix:
- enabled: false
-tt-fabric-manager:
- enabled: true
- fullnameOverride: tt-fabric-manager
- imagePullSecrets: []
- controller:
- nodeSelector:
- kubernetes.io/hostname: utmist-z1opa08
- agent:
- nodeSelector:
- kubernetes.io/hostname: utmist-tt
-tt-dra-driver:
- enabled: true
- imagePullSecrets: []
- kubeletPlugin:
- nodeSelector:
- kubernetes.io/hostname: utmist-tt
- fabricManagerAgentAddress: tt-fabric-manager-agent.tt-operator-system.svc.cluster.local:50053
diff --git a/deploy/k3s/tenstorrent-workload-policy.yaml b/deploy/k3s/tenstorrent-workload-policy.yaml
deleted file mode 100644
index eb14ca1..0000000
--- a/deploy/k3s/tenstorrent-workload-policy.yaml
+++ /dev/null
@@ -1,40 +0,0 @@
-# Applies only to the managed workload namespace. Operator DaemonSets live
-# elsewhere and retain the privileges their vendor components require.
-# Direct single-node probes remain allowed: device cgroups deny unclaimed
-# character devices. Whole-device-directory and Docker-socket mounts are denied.
-apiVersion: admissionregistration.k8s.io/v1
-kind: ValidatingAdmissionPolicy
-metadata:
- name: mist-accelerator-isolation
-spec:
- failurePolicy: Fail
- matchConstraints:
- resourceRules:
- - apiGroups: [""]
- apiVersions: [v1]
- operations: [CREATE, UPDATE]
- resources: [pods, pods/ephemeralcontainers]
- variables:
- - name: containers
- expression: 'object.spec.containers + (has(object.spec.initContainers) ? object.spec.initContainers : []) + (has(object.spec.ephemeralContainers) ? object.spec.ephemeralContainers : [])'
- validations:
- - expression: 'variables.containers.all(c, !has(c.securityContext) || ((!has(c.securityContext.privileged) || !c.securityContext.privileged) && (!has(c.securityContext.allowPrivilegeEscalation) || !c.securityContext.allowPrivilegeEscalation) && (!has(c.securityContext.capabilities) || !has(c.securityContext.capabilities.add) || size(c.securityContext.capabilities.add) == 0)))'
- message: Mist workloads cannot enable privileged access, privilege escalation, or additional Linux capabilities.
- - expression: '(!has(object.spec.hostPID) || !object.spec.hostPID) && (!has(object.spec.hostIPC) || !object.spec.hostIPC)'
- message: Mist workloads cannot join the host PID or IPC namespace.
- - expression: '!has(object.spec.volumes) || object.spec.volumes.all(v, !has(v.hostPath) || v.hostPath.path in ["/home/utmist-tt/TT/tt-metal", "/home/utmist-tt/.local/share/uv/python", "/dev/tenstorrent/0", "/dev/tenstorrent/1", "/dev/tenstorrent/2", "/dev/tenstorrent/3"])'
- message: Use DRA/CDI for devices and PVCs for storage; arbitrary host mounts are forbidden in Mist.
- - expression: '!has(object.spec.volumes) || object.spec.volumes.all(v, !has(v.hostPath) || !(v.hostPath.path in ["/home/utmist-tt/TT/tt-metal", "/home/utmist-tt/.local/share/uv/python"]) || variables.containers.all(c, !has(c.volumeMounts) || c.volumeMounts.all(m, m.name != v.name || (has(m.readOnly) && m.readOnly))))'
- message: The host TT-Metal and Python environment mounts must be read-only.
----
-apiVersion: admissionregistration.k8s.io/v1
-kind: ValidatingAdmissionPolicyBinding
-metadata:
- name: mist-accelerator-isolation
-spec:
- policyName: mist-accelerator-isolation
- validationActions: [Deny]
- matchResources:
- namespaceSelector:
- matchLabels:
- kubernetes.io/metadata.name: mist
diff --git a/deploy/k3s/training-nvidia-job.yaml b/deploy/k3s/training-nvidia-job.yaml
deleted file mode 100644
index d6d3393..0000000
--- a/deploy/k3s/training-nvidia-job.yaml
+++ /dev/null
@@ -1,50 +0,0 @@
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
- name: training-nvidia-checkpoints
- namespace: mist
-spec:
- accessModes: [ReadWriteOnce]
- storageClassName: local-path
- resources:
- requests:
- storage: 256Mi
----
-apiVersion: batch/v1
-kind: Job
-metadata:
- name: training-nvidia
- namespace: mist
-spec:
- backoffLimit: 0
- activeDeadlineSeconds: 1800
- template:
- spec:
- nodeSelector:
- kubernetes.io/hostname: utmist-z1opa08
- restartPolicy: Never
- containers:
- - name: train
- image: pytorch/pytorch:2.5.1-cuda12.4-cudnn9-runtime
- command: [python, -u, /scripts/training_smoke.py, nvidia]
- resources:
- requests:
- cpu: "2"
- memory: 2Gi
- limits:
- cpu: "4"
- memory: 4Gi
- nvidia.com/gpu: "2"
- volumeMounts:
- - name: script
- mountPath: /scripts
- readOnly: true
- - name: checkpoints
- mountPath: /checkpoints
- volumes:
- - name: script
- configMap:
- name: training-smoke-script
- - name: checkpoints
- persistentVolumeClaim:
- claimName: training-nvidia-checkpoints
diff --git a/deploy/k3s/training-smoke-results.json b/deploy/k3s/training-smoke-results.json
deleted file mode 100644
index 6474e8f..0000000
--- a/deploy/k3s/training-smoke-results.json
+++ /dev/null
@@ -1,61 +0,0 @@
-{
- "date": "2026-10-03",
- "test": "32x32 linear regression, 100 full-batch gradient steps",
- "dataset": {
- "seed": 42,
- "training_samples": 128,
- "validation_samples": 64
- },
- "acceptance": {
- "maximum_final_to_initial_mse_ratio": 0.02,
- "maximum_validation_mse": 0.0005,
- "minimum_max_weight_change": 0.05,
- "checkpoint_reload_mse_tolerance": 1e-07
- },
- "scope": "Independent device tests; no distributed training or performance benchmark. TT uses direct host devices, explicit TT-NN gradients, and a host-mounted Python/TT-Metal build; DRA scheduling and TT autograd are not tested.",
- "tenstorrent_host_metal_commit": "80ec10ab611570a6818fdebc1398c8c9141fb9df",
- "backends": {
- "nvidia": {
- "status": "passed",
- "devices_tested": 2,
- "node": "utmist-z1opa08",
- "image_id": "docker.io/pytorch/pytorch@sha256:c8268a92a69bd500f8be0e665b2630ee006dadaf7bfbc24249141b15ff622755",
- "container_started_at": "2026-10-03T18:26:07Z",
- "container_finished_at": "2026-10-03T18:26:12Z",
- "training_mse_initial": 0.3193739950656891,
- "training_mse_final_range": [
- 2.018663423086764e-07,
- 2.018663423086764e-07
- ],
- "validation_mse_range": [
- 9.320668823420419e-07,
- 9.320668823420419e-07
- ],
- "checkpoint_files": 2,
- "pvc": "training-nvidia-checkpoints",
- "logs": "/home/utmist/k3s-training-results-2026-10-03/nvidia.log",
- "checkpoint_archive": "/home/utmist/k3s-training-results-2026-10-03/nvidia-checkpoints.tar.gz"
- },
- "tenstorrent": {
- "status": "passed",
- "devices_tested": 8,
- "node": "utmist-tt",
- "image_id": "ghcr.io/tenstorrent/tt-metal/tt-metalium/ubuntu-22.04-dev-amd64@sha256:14621babd1f9d1c97cc3ee9528e180a31f30cec2d6114ca281daf1a9477c7a74",
- "container_started_at": "2026-10-03T18:23:05Z",
- "container_finished_at": "2026-10-03T18:23:24Z",
- "training_mse_initial": 0.3193739652633667,
- "training_mse_final_range": [
- 6.238709465833381e-05,
- 6.238709465833381e-05
- ],
- "validation_mse_range": [
- 0.00017507212760392576,
- 0.00017507212760392576
- ],
- "checkpoint_files": 8,
- "pvc": "training-tenstorrent-checkpoints",
- "logs": "/home/utmist/k3s-training-results-2026-10-03/tenstorrent.log",
- "checkpoint_archive": "/home/utmist/k3s-training-results-2026-10-03/tenstorrent-checkpoints.tar.gz"
- }
- }
-}
diff --git a/deploy/k3s/training-tenstorrent-job.yaml b/deploy/k3s/training-tenstorrent-job.yaml
deleted file mode 100644
index 75b4f0b..0000000
--- a/deploy/k3s/training-tenstorrent-job.yaml
+++ /dev/null
@@ -1,106 +0,0 @@
-# One n300 board (two chips), reserved through Tenstorrent DRA/CDI.
-# Host driver, TT-Metal checkout, and Python environment remain host-managed.
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
- name: training-tenstorrent-checkpoints
- namespace: mist
-spec:
- accessModes: [ReadWriteOnce]
- storageClassName: local-path
- resources:
- requests:
- storage: 256Mi
----
-apiVersion: resource.k8s.io/v1
-kind: ResourceClaimTemplate
-metadata:
- name: training-tenstorrent-board
- namespace: mist
-spec:
- spec:
- devices:
- requests:
- - name: board
- exactly:
- deviceClassName: tenstorrent.com
- count: 1
- selectors:
- - cel:
- expression: 'device.attributes["tenstorrent.com"].boardName == "n300" && device.attributes["tenstorrent.com"].chipCount == 2'
----
-apiVersion: batch/v1
-kind: Job
-metadata:
- name: training-tenstorrent-dra
- namespace: mist
-spec:
- backoffLimit: 0
- activeDeadlineSeconds: 1800
- template:
- spec:
- nodeSelector:
- kubernetes.io/hostname: utmist-tt
- restartPolicy: Never
- automountServiceAccountToken: false
- resourceClaims:
- - name: board
- resourceClaimTemplateName: training-tenstorrent-board
- containers:
- - name: train
- image: ghcr.io/tenstorrent/tt-metal/tt-metalium/ubuntu-22.04-dev-amd64:6b765c0bc2b40ea02566d169d7b0ee902377ba04
- imagePullPolicy: IfNotPresent
- workingDir: /tmp
- command: [/home/utmist-tt/TT/tt-metal/python_env/bin/python, -u, /scripts/training_smoke.py, tenstorrent, --expected-devices, "2", --output, /checkpoints/dra-single]
- securityContext:
- privileged: false
- allowPrivilegeEscalation: false
- capabilities:
- drop: [ALL]
- env:
- - name: TT_METAL_HOME
- value: /home/utmist-tt/TT/tt-metal
- - name: PYTHONPATH
- value: /home/utmist-tt/TT/tt-metal
- - name: LD_LIBRARY_PATH
- value: /home/utmist-tt/TT/tt-metal/build/lib
- - name: TT_METAL_CACHE
- value: /tmp/tt-metal-cache
- resources:
- claims:
- - name: board
- requests:
- cpu: "2"
- memory: 4Gi
- hugepages-1Gi: 2Gi
- limits:
- cpu: "8"
- memory: 8Gi
- hugepages-1Gi: 2Gi
- volumeMounts:
- - name: script
- mountPath: /scripts
- readOnly: true
- - name: checkpoints
- mountPath: /checkpoints
- - name: metal
- mountPath: /home/utmist-tt/TT/tt-metal
- readOnly: true
- - name: python
- mountPath: /home/utmist-tt/.local/share/uv/python
- readOnly: true
- volumes:
- - name: script
- configMap:
- name: training-smoke-script
- - name: checkpoints
- persistentVolumeClaim:
- claimName: training-tenstorrent-checkpoints
- - name: metal
- hostPath:
- path: /home/utmist-tt/TT/tt-metal
- type: Directory
- - name: python
- hostPath:
- path: /home/utmist-tt/.local/share/uv/python
- type: Directory
diff --git a/deploy/k3s/training_smoke.py b/deploy/k3s/training_smoke.py
deleted file mode 100644
index a21a610..0000000
--- a/deploy/k3s/training_smoke.py
+++ /dev/null
@@ -1,209 +0,0 @@
-"""Small accelerator training check; TT gradients are computed explicitly in TT-NN.
-
-CPU Torch creates the data, records metrics, and serializes checkpoints. All
-forward passes, gradients, and weight updates run on the selected accelerator.
-This is a linear-regression smoke test, not a model compatibility benchmark.
-"""
-
-import argparse
-import json
-import math
-import os
-import subprocess
-import sys
-import time
-from datetime import datetime, timezone
-from pathlib import Path
-
-import torch
-
-
-def dataset():
- generator = torch.Generator().manual_seed(42)
- truth = torch.randn(32, 32, generator=generator) * 0.1
- x = torch.randn(128, 32, generator=generator)
- validation_x = torch.randn(64, 32, generator=generator)
- return x, x @ truth, validation_x, validation_x @ truth
-
-
-def await_start(output, start_file, metadata):
- if not start_file:
- return
- (output / "ready.json").write_text(json.dumps({**metadata,
- "ready_at": datetime.now(timezone.utc).isoformat()}) + "\n")
- print(f"BARRIER_READY {json.dumps(metadata)}", flush=True)
- deadline = time.monotonic() + 300
- while not start_file.exists():
- assert time.monotonic() < deadline, "Training start barrier timed out"
- time.sleep(0.2)
-
-
-def check_and_save(backend, device_id, weights, initial, final, validation, output):
- change = weights.abs().max().item()
- assert all(math.isfinite(v) for v in (initial, final, validation, change))
- assert final < initial * 0.02, f"Loss did not converge: {initial} -> {final}"
- assert validation < 5e-4, f"Held-out MSE too high: {validation}"
- assert change > 0.05, "Weights were not updated"
- path = output / f"{backend}-{device_id}.pt"
- torch.save({"weights": weights.cpu(), "initial_mse": initial,
- "final_mse": final, "validation_mse": validation}, path)
- loaded = torch.load(path, map_location="cpu", weights_only=True)
- torch.testing.assert_close(loaded["weights"], weights.cpu(), rtol=0, atol=0)
- result = {"backend": backend, "device_id": device_id,
- "initial_mse": initial, "final_mse": final,
- "validation_mse": validation, "max_weight_change": change,
- "checkpoint": str(path)}
- print(json.dumps(result), flush=True)
- return result, loaded["weights"]
-
-
-def nvidia(output, steps, expected_devices, start_file, board_pci):
- assert torch.cuda.is_available(), "CUDA unavailable; CPU fallback is forbidden"
- count = torch.cuda.device_count()
- assert count == expected_devices, f"Expected {expected_devices} allocated GPUs; saw {count}"
- print(f"CUDA devices={count}; torch={torch.__version__}", flush=True)
- nodes = sorted(str(path) for path in Path("/dev").glob("nvidia[0-9]*"))
- assert len(nodes) == count, f"Unexpected NVIDIA device nodes: {nodes}"
- (output / "allocation.json").write_text(json.dumps({"device_nodes": nodes,
- "gpu_count": count}, indent=2) + "\n")
- results = []
- for index in range(count):
- device = torch.device(f"cuda:{index}")
- print(f"Training on {device}: {torch.cuda.get_device_name(index)}", flush=True)
- x, y, validation_x, validation_y = (t.to(device) for t in dataset())
- model = torch.nn.Linear(32, 32, bias=False).to(device)
- torch.nn.init.zeros_(model.weight)
- optimizer = torch.optim.SGD(model.parameters(), lr=0.2)
- if index == 0:
- await_start(output, start_file, {"device_nodes": nodes, "gpu_count": count})
- started = datetime.now(timezone.utc).isoformat()
- initial = (model(x) - y).square().mean().item()
- for step in range(steps):
- optimizer.zero_grad(set_to_none=True)
- residual = model(x) - y
- loss = residual.square().sum() / (2 * x.shape[0])
- loss.backward()
- assert model.weight.grad.is_cuda
- optimizer.step()
- if step % 20 == 0:
- print(f"cuda:{index} step={step} mse={residual.square().mean().item():.8f}", flush=True)
- with torch.no_grad():
- final = (model(x) - y).square().mean().item()
- validation = (model(validation_x) - validation_y).square().mean().item()
- result, weights = check_and_save("nvidia", index, model.weight.T.detach(),
- initial, final, validation, output)
- restored_error = (validation_x @ weights.to(device) - validation_y).square().mean().item()
- assert abs(restored_error - validation) < 1e-7
- result["checkpoint_reload_mse"] = restored_error
- result["device_name"] = torch.cuda.get_device_name(index)
- result["started_at"] = started
- result["finished_at"] = datetime.now(timezone.utc).isoformat()
- result["device_nodes"] = nodes
- results.append(result)
- return results
-
-
-def tenstorrent(output, steps, expected_devices, start_file, board_pci):
- nodes = sorted(str(path) for path in Path("/dev/tenstorrent").iterdir())
-
- def pci_address(node):
- path = Path("/sys/class/tenstorrent") / ("tenstorrent!" + Path(node).name)
- assert path.exists(), f"Missing PCI identity for {node}"
- return path.resolve().parents[1].name
-
- if board_pci:
- nodes = [node for node in nodes if pci_address(node) == board_pci]
- assert len(nodes) == 1, f"PCI selector does not match one allocated board: {board_pci}"
- assert len(nodes) * 2 == expected_devices, f"Unexpected n300 device nodes: {nodes}"
- if len(nodes) > 1:
- # Arbitrary n300 subsets need not form a connected fabric mesh.
- # This test trains independently, so use one runtime per board.
- results = []
- for ordinal, node in enumerate(nodes):
- bdf = pci_address(node)
- board_output = output / ("board-" + Path(node).name)
- command = [sys.executable, __file__, "tenstorrent", "--steps", str(steps),
- "--expected-devices", "2", "--output", str(board_output), "--board-pci", bdf]
- if start_file:
- command += ["--start-file", str(start_file)]
- print(f"Training reserved board {node}, PCI {bdf}", flush=True)
- subprocess.run(command, env={**os.environ, "TT_VISIBLE_DEVICES": bdf}, check=True)
- for result in json.loads((board_output / "results.json").read_text()):
- result["board_local_device_id"] = result["device_id"]
- result["device_id"] = ordinal * 2 + result["device_id"]
- result["board_pci"] = bdf
- results.append(result)
- assert len(results) == expected_devices
- (output / "allocation.json").write_text(json.dumps({"device_nodes": nodes,
- "chip_count": expected_devices}, indent=2) + "\n")
- return results
-
- import ttnn
-
- count = ttnn.get_num_devices()
- assert count == expected_devices, f"Expected {expected_devices} allocated chips; saw {count}"
- (output / "allocation.json").write_text(json.dumps({"device_nodes": nodes,
- "chip_count": count}, indent=2) + "\n")
- print(f"Tenstorrent devices={count}; torch used for data/metrics={torch.__version__}", flush=True)
- results = []
- for index in range(count):
- print(f"Opening Tenstorrent device {index}", flush=True)
- device = ttnn.open_device(device_id=index)
- try:
- if index == 0 and start_file:
- await_start(output, start_file, {"device_nodes": nodes, "chip_count": count})
- started = datetime.now(timezone.utc).isoformat()
- x, y, validation_x, validation_y = dataset()
- options = dict(dtype=ttnn.bfloat16, layout=ttnn.TILE_LAYOUT,
- device=device, memory_config=ttnn.DRAM_MEMORY_CONFIG)
- dx, dy, dvx = (ttnn.from_torch(t, **options) for t in (x, y, validation_x))
- dxt = ttnn.from_torch(x.T.contiguous(), **options)
- weights = ttnn.from_torch(torch.zeros(32, 32), **options)
-
- def mse(inputs, target, matrix):
- prediction = ttnn.to_torch(ttnn.matmul(inputs, matrix)).float()
- return (prediction - target).square().mean().item()
-
- initial = mse(dx, y, weights)
- for step in range(steps):
- prediction = ttnn.matmul(dx, weights)
- residual = ttnn.subtract(prediction, dy)
- # Gradient of ||XW-Y||^2/(2N) is X.T@(XW-Y)/N.
- gradient = ttnn.matmul(dxt, residual)
- weights = ttnn.subtract(weights, ttnn.multiply(gradient, 0.2 / x.shape[0]))
- if step % 20 == 0:
- print(f"tt:{index} step={step} mse={mse(dx, y, weights):.8f}", flush=True)
- final = mse(dx, y, weights)
- validation = mse(dvx, validation_y, weights)
- host_weights = ttnn.to_torch(weights).float()
- result, loaded = check_and_save("tenstorrent", index, host_weights,
- initial, final, validation, output)
- restored = ttnn.from_torch(loaded, **options)
- restored_error = mse(dvx, validation_y, restored)
- assert abs(restored_error - validation) < 1e-7
- result["checkpoint_reload_mse"] = restored_error
- result["started_at"] = started
- result["finished_at"] = datetime.now(timezone.utc).isoformat()
- result["device_nodes"] = nodes
- results.append(result)
- finally:
- ttnn.close_device(device)
- return results
-
-
-if __name__ == "__main__":
- parser = argparse.ArgumentParser()
- parser.add_argument("backend", choices=("nvidia", "tenstorrent"))
- parser.add_argument("--steps", type=int, default=100)
- parser.add_argument("--output", type=Path, default=Path("/checkpoints"))
- parser.add_argument("--expected-devices", type=int,
- help="Expected allocated chip/GPU count; defaults to the original whole-node test")
- parser.add_argument("--start-file", type=Path,
- help="Optional shared-file barrier for the concurrent allocation test")
- parser.add_argument("--board-pci", help=argparse.SUPPRESS)
- args = parser.parse_args()
- args.output.mkdir(parents=True, exist_ok=True)
- expected = args.expected_devices or (2 if args.backend == "nvidia" else 8)
- results = globals()[args.backend](args.output, args.steps, expected, args.start_file, args.board_pci)
- (args.output / "results.json").write_text(json.dumps(results, indent=2) + "\n")
- print(f"TRAINING_PASSED backend={args.backend} devices={len(results)}", flush=True)
diff --git a/deploy/k3s/verify_department.mjs b/deploy/k3s/verify_department.mjs
deleted file mode 100644
index 06ba8e7..0000000
--- a/deploy/k3s/verify_department.mjs
+++ /dev/null
@@ -1,562 +0,0 @@
-// Run from the repo root. Playwright is supplied externally; credentials and
-// cookies stay in a protected local state file, never in the public report.
-import {
- readFileSync,
- writeFileSync,
- mkdirSync,
- existsSync,
- chmodSync,
-} from "node:fs";
-import { randomBytes } from "node:crypto";
-import { execFileSync } from "node:child_process";
-const { chromium } = await import(
- process.env.MIST_PLAYWRIGHT_MODULE ??
- "/tmp/mist-browser-check/node_modules/playwright/index.mjs"
-);
-const base = process.env.MIST_SITE_URL ?? "http://100.73.139.66:8088";
-const out =
- process.env.MIST_EVIDENCE_DIR ??
- "/home/utmist/mist-department-results-2026-10-09";
-mkdirSync(out, { recursive: true, mode: 0o700 });
-const statePath = out + "/private-state.json";
-const credentials = JSON.parse(
- readFileSync("/home/utmist/.config/mist/admin-bootstrap.json"),
-);
-const browser = await chromium.launch({ headless: true });
-const admin = await browser.newContext({
- baseURL: base,
- viewport: { width: 1440, height: 1000 },
-});
-const errors = [];
-const state = existsSync(statePath) ? JSON.parse(readFileSync(statePath)) : {};
-const report = {
- started: new Date().toISOString(),
- site: base,
- checks: {},
- jobs: [],
- teams: [],
-};
-function save() {
- writeFileSync(statePath, JSON.stringify(state, null, 2), { mode: 0o600 });
- chmodSync(statePath, 0o600);
-}
-function check(ok, message) {
- if (!ok) throw new Error(message);
-}
-function passed(name) {
- report.checks[name] = true;
- writeFileSync(out + "/results.json", JSON.stringify(report, null, 2));
- console.log("PASS " + name);
-}
-async function call(
- ctx,
- path,
- method = "GET",
- data,
- team = "",
- expected = 200,
-) {
- let r;
- for (let attempt = 0; attempt < (method === "GET" ? 4 : 1); attempt++) {
- try {
- r = await ctx.request.fetch(base + path, {
- method,
- data,
- timeout: 15000,
- headers: { Origin: base, ...(team ? { "X-Mist-Team": team } : {}) },
- });
- if (method === "GET" && [502, 503].includes(r.status()) && attempt < 3) {
- await new Promise((resolve) => setTimeout(resolve, 1500));
- continue;
- }
- break;
- } catch {
- if (method === "GET" && attempt < 3) {
- await new Promise((resolve) => setTimeout(resolve, 1500));
- continue;
- }
- throw new Error(
- `${method} ${path}: transport interrupted; no credentials logged`,
- );
- }
- }
-
- check(
- r.status() === expected,
- `${method} ${path} expected ${expected}, got ${r.status()}: ${(await r.text()).slice(0, 1500)}`,
- );
- return path.includes("/files/download")
- ? r.body()
- : r.headers()["content-type"]?.includes("json")
- ? r.json()
- : r.body();
-}
-async function pick(target, label, optionsOrValue, maybeValue) {
- const value = maybeValue === undefined ? optionsOrValue : maybeValue
- await target.getByLabel(label, {exact:true}).click()
- await target.locator('[role=option][data-value='+JSON.stringify(String(value))+']').click()
-}
-async function login(ctx, email, password) {
- await call(ctx, "/auth/sign-in/email", "POST", { email, password });
-}
-async function waitReady(team, ctx = admin) {
- for (let i = 0; i < 60; i++) {
- const r = await ctx.request.get(base + "/api/storage", {
- headers: { "X-Mist-Team": team },
- });
- if (r.ok()) return r.json();
- await new Promise((r) => setTimeout(r, 1000));
- }
- throw new Error("Team storage never ready " + team);
-}
-async function submit(ctx, team, name, data = {}) {
- const result = await call(
- ctx,
- "/api/jobs",
- "POST",
- {
- name,
- type: "command",
- accelerator: "cpu",
- command: ["python", "-c", "print('JOB_PASSED',flush=True)"],
- timeout_seconds: 180,
- ...data,
- },
- team,
- 201,
- );
- report.jobs.push({ id: result.job_id, name, team });
- return result.job_id;
-}
-async function waitJob(ctx, team, id, want = "Success", seconds = 600) {
- for (let i = 0; i < seconds; i += 2) {
- const j = await call(ctx, "/api/jobs/" + id, "GET", undefined, team);
- if (j.job_state === want) return j;
- if (["Success", "Failure", "Cancelled"].includes(j.job_state)) {
- const logs = await call(
- ctx,
- "/api/jobs/" + id + "/logs",
- "GET",
- undefined,
- team,
- );
- throw new Error(
- `${id} ${j.job_state}: ${JSON.stringify(logs).slice(-3000)}`,
- );
- }
- await new Promise((r) => setTimeout(r, 2000));
- }
- throw new Error("Job timed out " + id);
-}
-async function terminalJob(ctx, team, id) {
- const j = await waitJob(ctx, team, id);
- const logs = await call(
- ctx,
- "/api/jobs/" + id + "/logs",
- "GET",
- undefined,
- team,
- );
- writeFileSync(out + "/" + id + ".log", logs.logs);
- return { j, logs: logs.logs };
-}
-async function upload(ctx, team, name, text, scope) {
- return call(
- ctx,
- `/api/datasets?filename=${encodeURIComponent(name)}${scope ? "&scope=" + scope : ""}`,
- "POST",
- Buffer.from(text),
- team,
- 201,
- );
-}
-const kube = (...args) =>
- execFileSync("k3s", ["kubectl", ...args], {
- env: { ...process.env, KUBECONFIG: "/home/utmist/.kube/config" },
- encoding: "utf8",
- });
-try {
- await login(admin, credentials.email, credentials.password);
- const account = (await call(admin, "/api/session")).user;
- state.adminID = account.id;
- const legacyBefore = await call(admin, "/api/jobs");
- report.legacy_job_count_before = legacyBefore.count;
- if (!state.a) {
- state.a = await call(
- admin,
- "/api/teams",
- "POST",
- { name: "Research pilot", storage_gib: 1 },
- "",
- 201,
- );
- save();
- }
- if (!state.b) {
- state.b = await call(
- admin,
- "/api/teams",
- "POST",
- { name: "Verification isolation", storage_gib: 1 },
- "",
- 201,
- );
- save();
- }
- const A = state.a.id,
- B = state.b.id;
- report.teams = [A, B];
- // Only unfinished fixtures created by this verifier are cancelled on rerun.
- for (const team of [A, B]) {
- const jobs = JSON.parse(
- kube("get", "jobs", "-n", "mist-" + team, "-o", "json"),
- ).items;
- for (const job of jobs) {
- if (
- job.metadata.annotations?.["mist.io/cancelled-at"] ||
- job.status?.conditions?.some(
- (c) => ["Complete", "Failed"].includes(c.type) && c.status === "True",
- )
- )
- continue;
- kube(
- "patch",
- "job",
- job.metadata.name,
- "-n",
- job.metadata.namespace,
- "--type=merge",
- "-p",
- JSON.stringify({
- spec: { suspend: true },
- metadata: {
- annotations: {
- "mist.io/cancelled-at": new Date().toISOString(),
- "mist.io/stop-reason": "Previous verification fixture cancelled",
- },
- },
- }),
- );
- }
- }
-
- await call(
- admin,
- `/api/teams/${A}`,
- "PATCH",
- { disabled: false, policy: { ...state.a.policy, concurrent: 1 } },
- "",
- );
- await call(admin, `/api/teams/${B}`, "PATCH", { disabled: false }, "");
- await call(admin, `/api/teams/${A}/members`, "POST", {
- user_id: account.id,
- common_writer: true,
- });
- for (const [key, name] of [
- ["alice", "Researcher A"],
- ["mate", "Teammate A"],
- ["bob", "Researcher B"],
- ]) {
- if (!state[key]) {
- const password = randomBytes(20).toString("base64url"),
- email = `${key}-${Date.now()}@mist.invalid`;
- const result = await call(admin, "/auth/admin/create-user", "POST", {
- name,
- email,
- password,
- role: "user",
- });
- state[key] = { id: result.user.id, name, email, password };
- save();
- }
- await call(admin, `/api/teams/${key === "bob" ? B : A}/members`, "POST", {
- user_id: state[key].id,
- common_writer: false,
- });
- }
- const alice = await browser.newContext({ baseURL: base }),
- mate = await browser.newContext({ baseURL: base }),
- bob = await browser.newContext({ baseURL: base });
- await Promise.all([
- login(alice, state.alice.email, state.alice.password),
- login(mate, state.mate.email, state.mate.password),
- login(bob, state.bob.email, state.bob.password),
- ]);
- await Promise.all([waitReady(A), waitReady(B, bob)]);
- passed("team_filesystems_ready");
- const capacity = await call(admin, "/api/storage", "GET", undefined, A);
- check(
- capacity.capacity_bytes < 1.1 * 1024 ** 3,
- "Team sees parent disk capacity",
- );
- report.storage = capacity;
- const ds = await upload(alice, A, "member.txt", "TEAM_A_MEMBER_DATA");
- const common = await upload(
- admin,
- A,
- "common.txt",
- "TEAM_A_COMMON_DATA",
- "common",
- );
- check(
- (await call(mate, "/api/datasets", "GET", undefined, A)).datasets.some(
- (d) => d.id === ds.id,
- ),
- "Teammate cannot read member data",
- );
- await call(mate, `/api/datasets/${ds.id}`, "DELETE", undefined, A, 403);
- await call(
- alice,
- "/api/datasets?filename=no.txt&scope=common",
- "POST",
- Buffer.from("bad"),
- A,
- 403,
- );
- await call(bob, `/api/datasets/${ds.id}`, "GET", undefined, B, 404);
- await call(bob, "/api/datasets", "GET", undefined, A, 403);
- await call(bob, "/api/teams", "POST", { name: "unauthorized" }, "", 403);
- passed("team_read_write_and_cross_team_denial");
- await call(admin, `/api/teams/${A}/grants`, "POST", {
- target_team: B,
- scope: state.alice.id,
- });
- const grant = (await call(admin, "/api/teams")).teams
- .find((t) => t.id === A)
- .grants.find((g) => g.scope === state.alice.id && g.target_team === B);
- const shared = await call(
- bob,
- `/api/datasets/${ds.id}/files/download?path=member.txt`,
- "GET",
- undefined,
- B,
- );
- check(shared.toString() === "TEAM_A_MEMBER_DATA", "Shared download mismatch");
- await call(bob, `/api/datasets/${common.id}`, "GET", undefined, B, 404);
- await call(bob, `/api/datasets/${ds.id}`, "DELETE", undefined, B, 403);
- const sharedID = await submit(bob, B, "Read-only shared input", {
- dataset_id: ds.id,
- script:
- "from pathlib import Path\nassert Path('/inputs/member.txt').read_text()=='TEAM_A_MEMBER_DATA'\ntry:\n Path('/inputs/bad').write_text('bad')\n raise AssertionError('writable shared input')\nexcept OSError: pass\nPath('/outputs/shared.json').write_text('ok')\nprint('SHARED_INPUT_PASSED',flush=True)",
- command: undefined,
- });
- await terminalJob(bob, B, sharedID);
- const revokedID = await submit(bob, B, "Revoke shared input", {
- dataset_id: ds.id,
- command: ["python", "-c", "import time; time.sleep(180)"],
- });
- await waitJob(bob, B, revokedID, "InProgress", 90);
- await call(admin, `/api/teams/${A}/grants/${grant.id}`, "DELETE");
- await waitJob(bob, B, revokedID, "Cancelled", 60);
- await call(bob, `/api/datasets/${ds.id}/files`, "GET", undefined, B, 404);
- passed("scoped_readonly_sharing_and_running_job_revocation");
- // Durable FIFO queue survives an API/controller rollout, and teams each get a turn.
- const running = await submit(alice, A, "Queue holds capacity", {
- command: [
- "python",
- "-c",
- 'import time; print("RUNNING",flush=True); time.sleep(180)',
- ],
- });
- await waitJob(alice, A, running, "InProgress", 90);
- const queued = await submit(alice, A, "Queue survives restart");
- const other = await submit(bob, B, "Other team gets a turn");
- await terminalJob(bob, B, other);
- check(
- (await call(alice, "/api/jobs/" + queued, "GET", undefined, A))
- .job_state === "Scheduled",
- "Team concurrency bypassed",
- );
- kube("rollout", "restart", "deployment/mist-api", "-n", "mist-system");
- kube(
- "rollout",
- "status",
- "deployment/mist-api",
- "-n",
- "mist-system",
- "--timeout=120s",
- );
- check(
- (await call(alice, "/api/jobs/" + queued, "GET", undefined, A))
- .job_state === "Scheduled",
- "Queue lost on restart",
- );
- await call(alice, `/api/jobs/${running}/cancel`, "POST", undefined, A);
- await terminalJob(alice, A, queued);
- passed("durable_fifo_team_concurrency_fair_turns_and_cancel");
- for (const body of [
- { cpu: "9" },
- { memory: "64Gi" },
- { device_count: 3, accelerator: "nvidia" },
- { timeout_seconds: 86401 },
- { image: "evil.invalid/app:v1" },
- { storage_scope: state.mate.id },
- ]) {
- await call(
- alice,
- "/api/jobs",
- "POST",
- { command: ["true"], ...body },
- A,
- body.storage_scope ? 403 : 400,
- );
- }
- await call(alice, "/api/jobs", "POST", { command: ["true"] }, "", 403);
- passed("policy_limits_and_legacy_bypass_denied");
- // Network probes run inside an ordinary job, under the actual k3s policy.
- const isolation = await submit(alice, A, "Workload network isolation", {
- script: `import socket,os\nfrom pathlib import Path\nassert not Path('/var/run/secrets/kubernetes.io/serviceaccount/token').exists()\nfor host,port in [('10.43.1.116',3000),('10.0.0.112',2049),('10.0.0.175',6443),('100.95.175.37',22)]:\n s=socket.socket();s.settimeout(2)\n try:\n s.connect((host,port))\n raise AssertionError('private network reachable '+host)\n except (TimeoutError,ConnectionRefusedError,OSError): pass\n finally: s.close()\nassert socket.gethostbyname('github.com')\nPath('/outputs/isolation.txt').write_text('private network and credentials denied')\nprint('ISOLATION_PASSED',flush=True)`,
- command: undefined,
- });
- check(
- (await terminalJob(alice, A, isolation)).logs.includes("ISOLATION_PASSED"),
- "Missing isolation marker",
- );
- passed("live_container_private_network_and_credentials_denied");
- // The full-volume writer cleans its own test file; it never touches real data.
- const quota = await submit(bob, B, "Filesystem capacity enforcement", {
- script: `import errno,json\nfrom pathlib import Path\np=Path('/outputs/quota-probe.bin');size=0\ntry:\n with p.open('wb',buffering=0) as f:\n for i in range(2048):\n f.write(b'x'*1048576);size+=1048576\n raise AssertionError('2GiB fit in a 1GiB team filesystem')\nexcept OSError as e:\n assert e.errno in (errno.ENOSPC,errno.EDQUOT),e\n print('STORAGE_QUOTA_PASSED bytes='+str(size),flush=True)\nfinally:\n p.unlink(missing_ok=True)\nPath('/outputs/quota.json').write_text(json.dumps({'bytes_before_limit':size}))`,
- command: undefined,
- });
- check(
- (await terminalJob(bob, B, quota)).logs.includes("STORAGE_QUOTA_PASSED"),
- "Quota not enforced",
- );
- passed("training_writes_obey_filesystem_capacity");
- const matrix = (n, m, p) =>
- Array.from({ length: n }, (_, i) =>
- Array.from({ length: m }, (_, j) => Math.sin((i + 1) * (j + 2) + p)),
- );
- const multiply = (x, w) =>
- x.map((row) =>
- w[0].map((_, k) => row.reduce((sum, v, j) => sum + v * w[j][k], 0)),
- );
- const x = matrix(128, 32, 0.1),
- vx = matrix(64, 32, 1.2),
- truth = matrix(32, 32, 2.3).map((row) => row.map((v) => v * 0.1));
- const regression = await upload(
- admin,
- A,
- "regression.json",
- JSON.stringify({
- x,
- y: multiply(x, truth),
- validation_x: vx,
- validation_y: multiply(vx, truth),
- }),
- "common",
- );
- state.dataset = regression;
- save();
- const training = readFileSync("deploy/k3s/examples/train_shared.py", "utf8");
- for (const backend of ["nvidia", "tenstorrent"]) {
- const script = training.replace(
- "args = parser.parse_args()",
- `args = parser.parse_args(['${backend}','--expected-devices','2','--output','/outputs'])`,
- );
- const id = await submit(admin, A, "Department training " + backend, {
- accelerator: backend,
- device_count: backend === "nvidia" ? 2 : 1,
- script,
- command: undefined,
- dataset_id: regression.id,
- storage_scope: "common",
- timeout_seconds: 600,
- });
- const { j, logs } = await terminalJob(admin, A, id);
- check(
- logs.includes("TRAINING_PASSED") &&
- logs.includes("DATASET_LOADED file=/inputs/regression.json"),
- "Training did not use uploaded data",
- );
- const files = (
- await call(admin, `/api/jobs/${id}/files`, "GET", undefined, A)
- ).files;
- check(files.length > 0, "No persisted model outputs");
- for (const f of files) {
- const data = await call(
- admin,
- `/api/jobs/${id}/files/download?path=${encodeURIComponent(f.path)}`,
- "GET",
- undefined,
- A,
- );
- check(data.length === f.size, "Downloaded model size mismatch");
- writeFileSync(out + `/${backend}-${f.path.replaceAll("/", "_")}`, data);
- }
- report.jobs.find((e) => e.id === id).node = j.node;
- passed(backend + "_real_training_uploaded_data_and_downloaded_weights");
- }
- const page = await admin.newPage();
- page.on("pageerror", (e) => errors.push(e.message));
- await page.goto("/teams");
- await page.getByRole("heading", { name: "Teams", exact: true }).waitFor();
- await pick(page, "Workspace", { exact: true }, A);
- await page.screenshot({ path: out + "/teams-admin.png", fullPage: true });
- await page.goto("/jobs");
- await page
- .getByRole("heading", { name: "Job history", exact: true })
- .waitFor();
- await page
- .getByRole("button", { name: /Show details for Department training/ })
- .first()
- .click();
- await page
- .getByRole("button", { name: "Files", exact: true })
- .first()
- .click();
- await page.getByRole("heading", { name: "Saved files" }).waitFor();
- await page.screenshot({ path: out + "/jobs-expanded.png", fullPage: true });
- passed("browser_rows_expand_and_saved_files_open");
- await page
- .getByRole("button", { name: /Hide details for Department training/ })
- .first()
- .click();
- await page.screenshot({ path: out + "/jobs-rows.png", fullPage: true });
- await page.goto("/datasets");
- await page.getByRole("heading", { name: /Team folders/ }).waitFor();
- await page.screenshot({ path: out + "/team-folders.png", fullPage: true });
- passed("browser_admin_workspace_and_shared_folders");
- const before = state.a.policy.storage_gib;
- await call(
- admin,
- `/api/teams/${A}`,
- "PATCH",
- { policy: { ...state.a.policy, storage_gib: before + 1 } },
- "",
- );
- state.a.policy.storage_gib = before + 1;
- save();
- const grown = await waitReady(A);
- check(
- grown.capacity_bytes > capacity.capacity_bytes,
- "Storage growth did not resize filesystem",
- );
- passed("online_team_storage_growth");
- await call(
- admin,
- `/api/teams/${A}`,
- "PATCH",
- { policy: { ...state.a.policy, storage_gib: 1 } },
- "",
- 400,
- );
- await call(admin, `/api/teams/${B}`, "PATCH", { disabled: true });
- for (const member of [state.alice, state.mate])
- await call(admin, `/api/teams/${A}/members/${member.id}`, "DELETE");
- const legacyAfter = await call(admin, "/api/jobs");
- check(legacyAfter.count === legacyBefore.count, "Legacy job history changed");
- passed("legacy_history_preserved");
- check(errors.length === 0, errors.join("; "));
- report.browser_errors = errors;
- report.finished = new Date().toISOString();
- writeFileSync(out + "/results.json", JSON.stringify(report, null, 2));
- console.log(JSON.stringify(report, null, 2));
-} catch (e) {
- report.error = e.message;
- writeFileSync(out + "/results.json", JSON.stringify(report, null, 2));
- throw e;
-} finally {
- await browser.close();
-}
-
diff --git a/deploy/k3s/verify_department_admission.py b/deploy/k3s/verify_department_admission.py
deleted file mode 100644
index 49c1f89..0000000
--- a/deploy/k3s/verify_department_admission.py
+++ /dev/null
@@ -1,24 +0,0 @@
-# Verification for this recorded two-machine pilot; never prints credentials.
-import copy,json,os,pathlib,subprocess
-out=pathlib.Path('/home/utmist/mist-department-results-2026-10-09')
-env={**os.environ,'KUBECONFIG':'/home/utmist/.kube/config'}
-ns='mist-team-db3933031a6c2437'
-def k(*args, data=None):return subprocess.run(['k3s','kubectl',*args],input=data,text=True,capture_output=True,env=env)
-jobs=json.loads(k('get','jobs','-n',ns,'-o','json').stdout)['items']
-source=next(j for j in jobs if j['metadata']['name']=='mist-7d426bc89696c01e19474c00')
-base={'apiVersion':'v1','kind':'Pod','metadata':{'name':'admission-probe','namespace':ns},'spec':source['spec']['template']['spec']}
-probes={
-'host_network':lambda p:p['spec'].update(hostNetwork=True),
-'host_pid':lambda p:p['spec'].update(hostPID=True),
-'service_account_token':lambda p:p['spec'].update(automountServiceAccountToken=True),
-'privileged_container':lambda p:p['spec']['containers'][0]['securityContext'].update(privileged=True,allowPrivilegeEscalation=True),
-'root_host_path':lambda p:p['spec']['volumes'].append({'name':'forbidden','hostPath':{'path':'/'}}),
-'capability_add':lambda p:p['spec']['containers'][0]['securityContext']['capabilities'].update(add=['SYS_ADMIN'])}
-report={}
-for name,edit in probes.items():
- p=copy.deepcopy(base);edit(p)
- r=k('create','--dry-run=server','-f','-',data=json.dumps(p))
- assert r.returncode and 'mist-team-boundary' in r.stderr,(name,r.stderr)
- report[name]={'denied':True,'reason':r.stderr.strip()}
- print('PASS ADMISSION_DENIED_'+name)
-(out/'admission-results.json').write_text(json.dumps(report,indent=2)+'\n')
diff --git a/deploy/k3s/verify_department_browser.mjs b/deploy/k3s/verify_department_browser.mjs
deleted file mode 100644
index bc3c5ec..0000000
--- a/deploy/k3s/verify_department_browser.mjs
+++ /dev/null
@@ -1,419 +0,0 @@
-// Real admin/researcher browser flow. Keep passwords only in protected state.
-import {
- readFileSync,
- writeFileSync,
- mkdirSync,
- existsSync,
- chmodSync,
-} from "node:fs";
-import { randomBytes } from "node:crypto";
-const { chromium } = await import(
- process.env.MIST_PLAYWRIGHT_MODULE ??
- "/tmp/mist-browser-check/node_modules/playwright/index.mjs"
-);
-const base = process.env.MIST_SITE_URL ?? "http://100.73.139.66:8088";
-const out =
- process.env.MIST_EVIDENCE_DIR ??
- "/home/utmist/mist-department-results-2026-10-09";
-mkdirSync(out, { recursive: true, mode: 0o700 });
-const creds = JSON.parse(
- readFileSync("/home/utmist/.config/mist/admin-bootstrap.json"),
-);
-const file = out + "/browser-private-state.json";
-const state = existsSync(file) ? JSON.parse(readFileSync(file)) : {};
-const report = {
- checks: {},
- browser_errors: [],
- started: new Date().toISOString(),
-};
-function save() {
- writeFileSync(file, JSON.stringify(state, null, 2), { mode: 0o600 });
- chmodSync(file, 0o600);
-}
-function check(ok, message) {
- if (!ok) throw new Error(message);
-}
-function pass(name) {
- report.checks[name] = true;
- writeFileSync(out + "/browser-results.json", JSON.stringify(report, null, 2));
- console.log("PASS " + name);
-}
-const browser = await chromium.launch({ headless: true });
-const ctx = await browser.newContext({
- baseURL: base,
- viewport: { width: 1600, height: 1100 },
-});
-const page = await ctx.newPage();
-page.on("pageerror", (e) => report.browser_errors.push(e.message));
-async function pick(target, label, optionsOrValue, maybeValue) {
- const value = maybeValue === undefined ? optionsOrValue : maybeValue;
- await target.getByLabel(label, { exact: true }).click();
- await target
- .locator("[role=option][data-value=" + JSON.stringify(String(value)) + "]")
- .click();
-}
-async function login(target, email, password) {
- await target.goto("/jobs");
- await target.getByLabel("Email", { exact: true }).fill(email);
- await target.getByLabel("Password", { exact: true }).fill(password);
- await target.getByRole("button", { name: "Sign in", exact: true }).click();
- await target.getByRole("heading", { name: "Jobs", exact: true }).waitFor();
-}
-async function api(context, path, method = "GET", data, team, expected = 200) {
- let r;
- try {
- r = await context.request.fetch(base + path, {
- method,
- data,
- headers: { Origin: base, ...(team ? { "X-Mist-Team": team } : {}) },
- });
- } catch {
- throw new Error(
- method + " " + path + " transport failure; no credentials logged",
- );
- }
- check(
- r.status() === expected,
- `${path}: expected ${expected}, got ${r.status()}`,
- );
- return r.headers()["content-type"]?.includes("json") ? r.json() : r.body();
-}
-try {
- await login(page, creds.email, creds.password);
- pass("admin_browser_login");
- const account = (await api(ctx, "/api/session")).user;
- if (state.team && process.env.MIST_VERIFY_NEW_TEAM === "1") {
- await api(ctx, `/api/teams/${state.team.id}`, "PATCH", {
- disabled: true,
- name: "Previous browser verification",
- });
- delete state.team;
- save();
- }
- if (state.member && process.env.MIST_VERIFY_NEW_MEMBER === "1") {
- delete state.member;
- save();
- }
- if (!state.member) {
- const m = {
- name: "Browser verification researcher",
- email: `browser-${Date.now()}@mist.invalid`,
- password: randomBytes(20).toString("base64url"),
- };
- await page.goto("/profile");
- await page.getByRole("tab", { name: "Members", exact: true }).click();
- await page.getByRole("button", { name: "New member", exact: true }).click();
- await page.getByLabel("Name", { exact: true }).fill(m.name);
- await page.getByLabel("Email", { exact: true }).fill(m.email);
- await page.getByLabel("Initial password").fill(m.password);
- const response = page.waitForResponse((r) =>
- r.url().endsWith("/auth/admin/create-user"),
- );
- await page.getByRole("button", { name: "Add member", exact: true }).click();
- const created = await response;
- check(created.ok(), "UI account creation failed");
- m.id = (await created.json()).user.id;
- state.member = m;
- save();
- }
- pass("browser_account_creation");
- if (!state.team) {
- await page.goto("/teams");
- await page.getByRole("button", { name: "New team", exact: true }).click();
- await page.getByLabel("New team name").fill("Browser verification");
- await page.getByLabel("Dataset storage (GiB)", { exact: true }).fill("1");
- await page.getByLabel("Model storage (GiB)", { exact: true }).fill("1");
- const response = page.waitForResponse(
- (r) => r.url().endsWith("/api/teams") && r.request().method() === "POST",
- );
- await page
- .getByRole("button", { name: "Create team", exact: true })
- .click();
- const created = await response;
- check(created.ok(), "UI team creation failed");
- state.team = await created.json();
- save();
- }
- const team = state.team.id;
- await api(ctx, `/api/teams/${team}`, "PATCH", { disabled: false });
- await api(ctx, `/api/teams/${team}/members`, "POST", {
- user_id: account.id,
- common_writer: true,
- });
- await page.goto("/teams");
- await pick(page, "Workspace", { exact: true }, team);
- await page
- .getByRole("button", { name: "Manage Browser verification", exact: true })
- .click();
- const section = page.getByRole("dialog", {
- name: "Team settings",
- exact: true,
- });
- await pick(section, "Existing Mist account", state.member.id);
- await section.getByLabel("Can write common folder").check();
- await section
- .getByRole("button", { name: "Add/update teammate", exact: true })
- .click();
- await section.locator("li").filter({ hasText: state.member.email }).waitFor();
- await page.getByRole("tab", { name: "Limits", exact: true }).click();
- await section.getByLabel("Total CPU cores").fill("4");
- await section.getByLabel("Total memory (for example 16Gi)").fill("8Gi");
- await section
- .getByLabel("Maximum job runtime (seconds)", { exact: true })
- .fill("120");
- await section
- .getByRole("button", { name: "Save limits", exact: true })
- .click();
- await section
- .getByRole("status")
- .filter({ hasText: "Team limits saved." })
- .waitFor();
- pass("browser_team_creation_membership_common_write_and_policy");
- await page
- .getByRole("button", { name: "Close Team settings", exact: true })
- .click();
- await page.goto("/datasets");
- await page.getByRole("button", { name: "Add dataset", exact: true }).click();
- await page
- .getByLabel("Dataset name (optional)")
- .fill("Browser-uploaded dataset");
- await pick(page, "Upload folder", "common");
- await page.getByLabel("File or ZIP archive").setInputFiles({
- name: "ui.txt",
- mimeType: "text/plain",
- buffer: Buffer.from("BROWSER_UPLOADED_DATA"),
- });
- await page
- .getByRole("button", { name: "Upload dataset", exact: true })
- .click({ timeout: 120000 });
- await page
- .getByRole("status")
- .filter({ hasText: "is ready to use" })
- .waitFor({ timeout: 60000 });
- const ds = (
- await api(ctx, "/api/datasets", "GET", undefined, team)
- ).datasets.find((d) => d.name === "Browser-uploaded dataset");
- check(ds, "Browser dataset missing");
- state.dataset = ds;
- save();
- await page.locator(".mist-folder-details > summary").click();
- await pick(page, "Browse common or teammate files", team + ":common");
- await page
- .getByRole("link", { name: "Download", exact: true })
- .first()
- .waitFor();
- const downloaded = page.waitForEvent("download");
- await page
- .getByRole("link", { name: "Download", exact: true })
- .first()
- .click();
- check(
- (await (await downloaded).failure()) === null,
- "Browser dataset download failed",
- );
- await page.screenshot({
- path: out + "/final-datasets-desktop.png",
- fullPage: true,
- });
- pass("browser_upload_common_folder_browse_and_download");
- const memberCtx = await browser.newContext({
- baseURL: base,
- viewport: { width: 1440, height: 1000 },
- }),
- member = await memberCtx.newPage();
- member.on("pageerror", (e) => report.browser_errors.push(e.message));
- await login(member, state.member.email, state.member.password);
- await pick(member, "Workspace", { exact: true }, team);
- await member.getByRole("button", { name: "New job", exact: true }).click();
- const jobName = "Browser container experiment " + Date.now();
- await member.waitForFunction(
- () => document.querySelector("#job-timeout")?.value === "120",
- );
- pass("job_defaults_follow_loaded_team_policy");
- await member.getByLabel("Name", { exact: true }).fill(jobName);
- await member.getByLabel("Container image").fill("python:3.11-slim");
- await pick(member, "Dataset (optional)", ds.id);
- await member
- .getByLabel("Python script", { exact: true })
- .fill(
- "from pathlib import Path\nassert Path('/inputs/ui.txt').read_text()=='BROWSER_UPLOADED_DATA'\nPath('/outputs/browser-result.txt').write_text('BROWSER_JOB_PASSED')\nprint('BROWSER_JOB_PASSED',flush=True)\n",
- );
- await member.getByText("Resources and environment", { exact: true }).click();
- await member.getByLabel("Deadline (seconds)").fill("120");
- const submission = member.waitForResponse(
- (r) => r.url().endsWith("/api/jobs") && r.request().method() === "POST",
- );
- await member.getByRole("button", { name: "Submit job", exact: true }).click();
- const submitted = await submission;
- check(submitted.ok(), "Browser job submission failed: " + submitted.status() + " " + (submitted.ok() ? "" : await submitted.text()));
- const id = (await submitted.json()).job_id;
- report.job_id = id;
- for (let i = 0; i < 100; i++) {
- const j = await api(memberCtx, `/api/jobs/${id}`, "GET", undefined, team);
- if (j.job_state === "Success") break;
- check(
- !["Failure", "Cancelled"].includes(j.job_state),
- "Browser container experiment failed",
- );
- await new Promise((r) => setTimeout(r, 2000));
- }
- check(
- (await api(memberCtx, `/api/jobs/${id}`, "GET", undefined, team))
- .job_state === "Success",
- "Browser job did not finish",
- );
- await member
- .getByRole("button", { name: "Show details for " + jobName, exact: true })
- .click();
- await member.getByRole("button", { name: "Logs", exact: true }).click();
- await member.getByText("BROWSER_JOB_PASSED", { exact: false }).waitFor();
- await member.getByRole("button", { name: "Files", exact: true }).click();
- const output = member.waitForEvent("download");
- await member
- .getByRole("link", { name: "browser-result.txt", exact: true })
- .click();
- await (await output).saveAs(out + "/browser-result.txt");
- pass("regular_browser_selfservice_image_submit_expand_logs_result_download");
- // A regular account has no administrator controls and cannot reset accounts.
- await member.goto("/teams");
- check(
- (await member
- .getByRole("button", { name: "Create team", exact: true })
- .count()) === 0,
- "Regular user sees team administration",
- );
- await api(
- memberCtx,
- "/auth/admin/set-user-password",
- "POST",
- { userId: account.id, newPassword: randomBytes(20).toString("base64url") },
- undefined,
- 403,
- );
- pass("regular_browser_admin_controls_denied");
- // Admin resets a test account's password in the same portal.
- await page.goto("/profile");
- await page.getByRole("tab", { name: "Members", exact: true }).click();
- await page
- .getByRole("button", { name: "Reset password", exact: true })
- .click();
- await pick(page, "Account to reset", state.member.id);
- const next = randomBytes(20).toString("base64url");
- await page.getByLabel("New temporary password").fill(next);
- await page
- .getByRole("button", {
- name: "Reset password and sign out sessions",
- exact: true,
- })
- .click();
- await page
- .getByRole("status")
- .filter({ hasText: "Password reset and existing sessions signed out." })
- .waitFor();
- await api(memberCtx, "/api/jobs", "GET", undefined, team, 401);
- state.member.password = next;
- save();
- await memberCtx.close();
- const reloginCtx = await browser.newContext({ baseURL: base }),
- relogin = await reloginCtx.newPage();
- await login(relogin, state.member.email, next);
- await reloginCtx.close();
- pass("browser_password_reset_revokes_sessions_new_login_works");
- await page
- .getByRole("button", { name: "Close Reset password", exact: true })
- .click();
- await page.goto("/jobs");
- await pick(page, "Workspace", { exact: true }, team);
- await page
- .getByRole("button", { name: "Show details for " + jobName, exact: true })
- .click();
- await page.screenshot({
- path: out + "/final-jobs-desktop.png",
- fullPage: true,
- });
- await page
- .getByRole("button", { name: "Hide details for " + jobName, exact: true })
- .click();
- await page.setViewportSize({ width: 390, height: 844 });
- await page.screenshot({
- path: out + "/final-jobs-mobile.png",
- fullPage: true,
- });
- check(
- await page.evaluate(
- () => document.documentElement.scrollWidth <= window.innerWidth,
- ),
- "Mobile viewport overflows horizontally",
- );
- await page.setViewportSize({ width: 1600, height: 1100 });
- await page.getByRole("button", { name: "New job", exact: true }).click();
- await page.getByRole("radio", { name: /Tenstorrent/ }).check();
- await page.waitForFunction(() =>
- document
- .querySelector('input[name="compute"]:checked')
- ?.closest("label")
- ?.classList.contains("is-selected"),
- );
- await page
- .getByRole("radio", { name: "Training check", exact: true })
- .check();
- await page.screenshot({
- path: out + "/final-job-composer.png",
- fullPage: true,
- });
- await page.setViewportSize({ width: 390, height: 844 });
- check(
- await page
- .getByRole("dialog")
- .evaluate((el) => el.scrollWidth <= el.clientWidth),
- "Mobile job composer overflows horizontally",
- );
- await page.screenshot({
- path: out + "/final-job-composer-mobile.png",
- fullPage: false,
- });
- await page
- .getByRole("button", { name: "Close New job", exact: true })
- .click();
- await page.setViewportSize({ width: 1600, height: 1100 });
- pass("responsive_job_dialog_and_device_slider");
- await pick(page, "Workspace", { exact: true }, "");
- await page.waitForFunction(
- () => !document.querySelector('[aria-label="Next page"]')?.disabled,
- );
- check(
- (await page.getByRole("button", { name: /Show details for/ }).count()) <=
- 10,
- "History exceeds page size",
- );
- check(
- await page
- .getByRole("button", { name: "Next page", exact: true })
- .isEnabled(),
- "Preserved history has no next page",
- );
- await page.getByRole("button", { name: "Next page", exact: true }).click();
- await page.getByRole("status").filter({ hasText: "Page 2 of" }).waitFor();
- await page
- .getByRole("button", { name: "Previous page", exact: true })
- .click();
- await page.getByRole("status").filter({ hasText: "Page 1 of" }).waitFor();
- pass("history_pagination_next_previous_real_legacy_jobs");
- await pick(page, "Workspace", { exact: true }, team);
- await page.goto("/dashboard");
- await page.screenshot({ path: out + "/final-overview.png", fullPage: true });
- pass("desktop_mobile_layout_no_overflow_or_browser_errors");
- await api(ctx, `/api/teams/${team}`, "PATCH", { disabled: true });
- const pilot = JSON.parse(readFileSync(out + "/private-state.json")).a.id;
- await pick(page, "Workspace", { exact: true }, pilot);
- check(report.browser_errors.length === 0, report.browser_errors.join("; "));
- report.finished = new Date().toISOString();
- writeFileSync(out + "/browser-results.json", JSON.stringify(report, null, 2));
- console.log(JSON.stringify(report, null, 2));
-} catch (e) {
- report.error = e.message;
- writeFileSync(out + "/browser-results.json", JSON.stringify(report, null, 2));
- throw e;
-} finally {
- await browser.close();
-}
diff --git a/deploy/k3s/verify_department_remote_cli.py b/deploy/k3s/verify_department_remote_cli.py
deleted file mode 100644
index 38ec76a..0000000
--- a/deploy/k3s/verify_department_remote_cli.py
+++ /dev/null
@@ -1,47 +0,0 @@
-# Verification for this recorded two-machine pilot; never prints credentials.
-import json, pathlib, subprocess, time, shlex
-root=pathlib.Path('/home/utmist/mist-department-results-2026-10-09')
-credentials=json.loads(pathlib.Path('/home/utmist/.config/mist/admin-bootstrap.json').read_text())
-ssh=['ssh','-i','/home/utmist/.ssh/id_ed25519_quietbox_codex','-o','IdentitiesOnly=yes','-o','BatchMode=yes','utmist-tt@100.95.175.37']
-base='~/mist-department-verification/mist-cli --api-url http://100.73.139.66:8088/api --config ~/mist-department-verification/verification-config.json '
-def run(cmd, data=None):
- p=subprocess.run(ssh+[cmd], input=data, text=True,capture_output=True,timeout=40)
- if p.returncode: raise RuntimeError('Remote command failed: '+p.stderr[:2000])
- return p.stdout
-probe="""from pathlib import Path
-import torch
-assert torch.cuda.is_available(), 'CUDA unavailable'
-assert torch.cuda.device_count()==1, torch.cuda.device_count()
-x=torch.randn(128,128,device='cuda',requires_grad=True)
-loss=(x@x).square().mean()
-loss.backward()
-assert torch.isfinite(x.grad).all()
-Path('/outputs/remote-cli-proof.txt').write_text('REMOTE_CLI_NVIDIA_PASSED devices=1')
-print('REMOTE_CLI_NVIDIA_PASSED devices=1',flush=True)
-"""
-run("cat > ~/mist-department-verification/nvidia.py",probe)
-report={'origin_node':'utmist-tt','destination_node':'utmist-z1opa08','team_id':'team-db3933031a6c2437'}
-try:
- run(base+'auth login --email '+shlex.quote(credentials['email'])+' --password-stdin',credentials['password']+'\n')
- report['login']=True
- run(base+'team use team-db3933031a6c2437')
- report['team_selection']=True
- submitted=run(base+'job submit ~/mist-department-verification/nvidia.py --compute NVIDIA --devices 1 --timeout 120 --name remote-cli-nvidia')
- job=submitted.strip().split()[-1]; report['job_id']=job
- for _ in range(90):
- status=run(base+'job status '+job)
- if 'Success' in status: break
- if 'Failure' in status or 'Cancelled' in status: raise RuntimeError(status)
- time.sleep(2)
- else: raise RuntimeError('Remote submitted job did not finish')
- logs=run(base+'job logs '+job)
- assert 'utmist-z1opa08' in status and 'REMOTE_CLI_NVIDIA_PASSED' in logs
- report['job_succeeded']=True
- (root/'remote-cli-status.txt').write_text(status)
- (root/'remote-cli-job.log').write_text(logs)
- print(status,logs)
-finally:
- run(base+'auth logout')
- run("python3 -c \"from pathlib import Path; Path.home().joinpath('mist-department-verification/verification-config.json').unlink(missing_ok=True)\"")
-(root/'remote-cli-results.json').write_text(json.dumps(report,indent=2)+'\n')
-print('PASS SSH_TENSTORRENT_CLI_SUBMISSION_NVIDIA_ON_MAIN')
diff --git a/deploy/k3s/verify_department_runtime.py b/deploy/k3s/verify_department_runtime.py
deleted file mode 100644
index 57e35ce..0000000
--- a/deploy/k3s/verify_department_runtime.py
+++ /dev/null
@@ -1,81 +0,0 @@
-"""Live execution checks for the recorded pilot; credentials remain local."""
-import http.cookiejar
-import json
-import pathlib
-import time
-import urllib.error
-import urllib.request
-
-ROOT = pathlib.Path('/home/utmist/mist-department-results-2026-10-09')
-BASE = 'http://100.73.139.66:8088'
-TEAM = 'team-db3933031a6c2437'
-credentials = json.loads(pathlib.Path('/home/utmist/.config/mist/admin-bootstrap.json').read_text())
-client = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(http.cookiejar.CookieJar()))
-report = {'checks': {}, 'jobs': []}
-
-
-def api(path, body=None):
- request = urllib.request.Request(BASE + path, data=json.dumps(body).encode() if body is not None else None,
- headers={'Origin': BASE, 'Content-Type': 'application/json', 'X-Mist-Team': TEAM})
- with client.open(request, timeout=30) as response:
- return json.load(response)
-
-
-def submit(name, **data):
- job = api('/api/jobs', {'name': name, 'type': 'command', 'accelerator': 'cpu', 'timeout_seconds': 120,
- 'command': ['python', '-c', "print('PASSED',flush=True)"], **data})['job_id']
- report['jobs'].append({'id': job, 'name': name})
- return job
-
-
-def terminal(job, expected):
- for _ in range(120):
- result = api('/api/jobs/' + job)
- if result['job_state'] in ['Success', 'Failure', 'Cancelled']:
- assert result['job_state'] == expected, result
- log = api('/api/jobs/' + job + '/logs')['logs']
- (ROOT / (job + '.log')).write_text(log)
- return result, log
- time.sleep(2)
- raise RuntimeError('Job timed out ' + job)
-
-
-try:
- api('/auth/sign-in/email', {'email': credentials['email'], 'password': credentials['password']})
- first = submit('Concurrent TT board A', type='training-smoke', accelerator='tenstorrent', device_count=1, command=None)
- second = submit('Concurrent TT board B', type='training-smoke', accelerator='tenstorrent', device_count=1, command=None)
- allocations = {}
- concurrent = False
- for _ in range(120):
- jobs = [api('/api/jobs/' + job) for job in [first, second]]
- if all(j['job_state'] == 'InProgress' for j in jobs):
- concurrent = True
- for job in jobs:
- if job.get('devices'):
- allocations[job['id']] = job['devices']
- if all(j['job_state'] in ['Success', 'Failure', 'Cancelled'] for j in jobs):
- break
- time.sleep(1)
- assert concurrent, 'TT jobs did not overlap'
- for job in [first, second]:
- result, log = terminal(job, 'Success')
- assert result['node'] == 'utmist-tt' and 'TRAINING_PASSED backend=tenstorrent devices=2' in log, log
- assert len(allocations) == 2, allocations
- left = {(d['pool'], d['device']) for d in allocations[first]}
- right = {(d['pool'], d['device']) for d in allocations[second]}
- assert len(left) == len(right) == 1 and left.isdisjoint(right), allocations
- report['allocations'] = allocations
- report['checks']['concurrent_tt_jobs_distinct_boards_real_training'] = True
- print('PASS CONCURRENT_TT_JOBS_DISTINCT_BOARDS_TRAINING')
- failure = submit('Intentional failure', command=['python', '-c', "print('EXPECTED_EXIT_7',flush=True); raise SystemExit(7)"])
- result, log = terminal(failure, 'Failure')
- assert result.get('exit_code') == 7 and 'EXPECTED_EXIT_7' in log
- report['checks']['nonzero_exit_visible_no_retry'] = True
- deadline = submit('Execution deadline', timeout_seconds=10, command=['python', '-c', "import time; print('DEADLINE_STARTED',flush=True); time.sleep(120)"])
- result, log = terminal(deadline, 'Failure')
- assert 'Deadline' in result.get('message', '') or 'deadline' in result.get('message', '').lower(), result
- report['checks']['execution_deadline_failure_visible'] = True
- print('PASS EXIT_CODE_AND_DEADLINE_FAILURES')
-finally:
- api('/auth/sign-out', {})
-(ROOT / 'runtime-results.json').write_text(json.dumps(report, indent=2) + '\n')
diff --git a/deploy/k3s/verify_large_dataset.py b/deploy/k3s/verify_large_dataset.py
deleted file mode 100644
index ed8fcee..0000000
--- a/deploy/k3s/verify_large_dataset.py
+++ /dev/null
@@ -1,83 +0,0 @@
-"""Verify a streamed dataset exceeds 2 GiB and can be mounted by a real Job."""
-import hashlib
-import http.cookiejar
-import json
-import pathlib
-import tempfile
-import time
-import urllib.request
-
-OUT = pathlib.Path('/home/utmist/mist-department-results-2026-10-09')
-BASE = 'http://100.73.139.66:8088'
-credentials = json.loads(pathlib.Path('/home/utmist/.config/mist/admin-bootstrap.json').read_text())
-client = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(http.cookiejar.CookieJar()))
-report = {'checks': {}}
-team = ''
-
-def call(path, body=None):
- request = urllib.request.Request(BASE + path, data=json.dumps(body).encode() if body is not None else None,
- headers={'Origin': BASE, 'Content-Type': 'application/json', **({'X-Mist-Team': team} if team else {})})
- with client.open(request, timeout=30) as response:
- return json.load(response)
-
-try:
- call('/auth/sign-in/email', {'email': credentials['email'], 'password': credentials['password']})
- admin = call('/api/session')['user']
- teams = call('/api/teams')['teams']
- existing = next((t for t in teams if t['name'] == 'Large dataset verification'), None)
- if existing is None:
- existing = call('/api/teams', {'name': 'Large dataset verification', 'storage_gib': 4})
- team = existing['id']
- request = urllib.request.Request(BASE + '/api/teams/' + team, method='PATCH', data=b'{"disabled":false}', headers={'Origin': BASE, 'Content-Type': 'application/json'})
- client.open(request, timeout=30).close()
- call('/api/teams/' + team + '/members', {'user_id': admin['id'], 'common_writer': True})
- for _ in range(60):
- try:
- storage = call('/api/storage')
- if storage['upload_limit_bytes'] > 2**31: break
- except Exception: pass
- time.sleep(1)
- else: raise RuntimeError('Large team filesystem not ready')
- # Sparse local source avoids allocating a second local multi-GiB copy;
- # all bytes are nevertheless streamed over the actual portal/NFS path.
- size = 2**31 + 4 * 1024**2
- with tempfile.TemporaryFile() as source:
- source.write(b'LARGE_DATASET_START')
- source.truncate(size)
- source.seek(size - len(b'LARGE_DATASET_END!')); source.write(b'LARGE_DATASET_END!')
- source.seek(0)
- digest = hashlib.sha256()
- while chunk := source.read(1024 * 1024): digest.update(chunk)
- checksum = digest.hexdigest()
- source.seek(0)
- request = urllib.request.Request(BASE + '/api/datasets?filename=large.bin&name=Large%20dataset%20probe', data=source,
- headers={'Origin': BASE, 'Content-Type': 'application/octet-stream', 'Content-Length': str(size), 'X-Mist-Team': team})
- started = time.monotonic()
- with client.open(request, timeout=14400) as response:
- dataset = json.load(response)
- assert dataset['size'] == size and dataset['sha256'] == checksum, dataset
- report.update(team_id=team, dataset_id=dataset['id'], bytes=size, upload_seconds=round(time.monotonic()-started, 1))
- report['checks']['actual_upload_over_2_gib_checksum_verified'] = True
- print('PASS STREAMED_UPLOAD_OVER_2_GIB_CHECKSUM', flush=True)
- script = f"from pathlib import Path\np=Path('/inputs/large.bin')\nassert p.stat().st_size=={size}\nwith p.open('rb') as f:\n assert f.read(19)==b'LARGE_DATASET_START'\n f.seek(-len(b'LARGE_DATASET_END!'),2)\n assert f.read()==b'LARGE_DATASET_END!'\nprint('LARGE_DATASET_JOB_PASSED',flush=True)\n"
- job = call('/api/jobs', {'name':'Large dataset input verification', 'type':'command', 'accelerator':'cpu', 'dataset_id': dataset['id'], 'script':script, 'script_name':'verify.py', 'timeout_seconds':120})['job_id']
- report['job_id'] = job
- for _ in range(90):
- result = call('/api/jobs/' + job)
- if result['job_state'] in ['Success','Failure','Cancelled']: break
- time.sleep(2)
- assert result['job_state'] == 'Success', result
- logs = call('/api/jobs/' + job + '/logs')['logs']
- assert 'LARGE_DATASET_JOB_PASSED' in logs
- (OUT/'large-dataset-job.log').write_text(logs)
- report['checks']['large_file_mounted_readonly_real_job'] = True
- # Remove only this verifier's large dataset after its sole Job has finished.
- request = urllib.request.Request(BASE + '/api/datasets/' + dataset['id'], method='DELETE', headers={'X-Mist-Team':team})
- client.open(request, timeout=30).close()
- report['checks']['own_probe_removed_after_job_no_user_files_deleted'] = True
- request = urllib.request.Request(BASE + '/api/teams/' + team, method='PATCH', data=b'{"disabled":true}', headers={'Origin': BASE, 'Content-Type': 'application/json'})
- client.open(request, timeout=30).close()
- print('PASS LARGE_FILE_MOUNT_AND_OWN_PROBE_CLEANUP', flush=True)
-finally:
- call('/auth/sign-out', {})
- (OUT/'large-dataset-results.json').write_text(json.dumps(report, indent=2)+'\n')
diff --git a/deploy/k3s/verify_storage_folders.py b/deploy/k3s/verify_storage_folders.py
deleted file mode 100644
index 5e8c431..0000000
--- a/deploy/k3s/verify_storage_folders.py
+++ /dev/null
@@ -1,176 +0,0 @@
-#!/usr/bin/env python3
-"""Small live checks: regular-member uploads, isolation, and NVIDIA/TT training.
-
-Creates two small verification workspaces and one regular account. No large files,
-quota exhaustion, migration or old-data deletion. Credentials stay outside Git.
-"""
-import argparse
-import hashlib
-import http.cookiejar
-import io
-import json
-import os
-from pathlib import Path
-import random
-import secrets
-import time
-import urllib.error
-import urllib.parse
-import urllib.request
-import zipfile
-
-
-class Client:
- def __init__(self, base):
- self.base = base.rstrip('/')
- self.opener = urllib.request.build_opener(
- urllib.request.HTTPCookieProcessor(http.cookiejar.CookieJar()))
-
- def call(self, path, body=None, team=None, method=None, expected=200, raw=False):
- payload = body if isinstance(body, bytes) else json.dumps(body).encode() if body is not None else None
- headers = {'Origin': self.base, 'Content-Type': 'application/octet-stream' if isinstance(body, bytes) else 'application/json'}
- if team:
- headers['X-Mist-Team'] = team
- request = urllib.request.Request(self.base + path, data=payload, headers=headers, method=method)
- try:
- response = self.opener.open(request, timeout=30)
- except urllib.error.HTTPError as error:
- if error.code != expected:
- raise RuntimeError(f'{path}: unexpected HTTP {error.code}') from None
- response = error
- with response:
- assert response.code == expected, (path, response.code)
- return response.read() if raw else json.load(response)
-
-
-def main():
- parser = argparse.ArgumentParser(description=__doc__)
- parser.add_argument('--base', default='http://100.73.139.66:8088')
- parser.add_argument('--credentials', type=Path, default=Path('/home/utmist/.config/mist/admin-bootstrap.json'))
- parser.add_argument('--out', required=True, type=Path)
- args = parser.parse_args()
- args.out.mkdir(parents=True, exist_ok=True, mode=0o700)
- os.chmod(args.out, 0o700)
- report = {'checks': [], 'jobs': [], 'bulk_testing': False}
- admin, member = Client(args.base), Client(args.base)
- jobs = []
- stamp = secrets.token_hex(4)
-
- def passed(name):
- report['checks'].append(name)
- print('PASS ' + name, flush=True)
-
- try:
- creds = json.loads(args.credentials.read_text())
- identity = admin.call('/auth/sign-in/email', {'email': creds['email'], 'password': creds['password']})['user']
- teams = [admin.call('/api/teams', {'name': 'Storage verification ' + stamp + suffix,
- 'storage_gib': 1, 'model_storage_gib': 1}, expected=201) for suffix in (' A', ' B')]
- for team in teams:
- admin.call('/api/teams/' + team['id'] + '/members', {'user_id': identity['id'], 'common_writer': True})
- email, password = 'storage-check-' + stamp + '@mist.local', secrets.token_urlsafe(24)
- user = admin.call('/auth/admin/create-user', {'name': 'Storage check ' + stamp, 'email': email,
- 'password': password, 'role': 'user'})['user']
- admin.call('/api/teams/' + teams[0]['id'] + '/members', {'user_id': user['id'], 'common_writer': False})
- member.call('/auth/sign-in/email', {'email': email, 'password': password})
- team = teams[0]['id']
- state = args.out / 'member-private.json'
- state.write_text(json.dumps({'email': email, 'password': password, 'team_id': team}))
- state.chmod(0o600)
- for _ in range(60):
- try:
- storage = member.call('/api/storage', team=team)
- break
- except RuntimeError:
- time.sleep(1)
- else:
- raise AssertionError('team folders did not become ready')
- assert storage['storage_mode'] == 'shared-folders' and storage['storage_limits_enforced'] is False
- assert storage['upload_limit_bytes'] > 0
- passed('regular_member_login_and_shared_folder_readiness')
- member.call('/api/jobs', team=teams[1]['id'], expected=403)
- member.call('/api/datasets?filename=forbidden.txt&scope=common', b'no', team=team, expected=403)
- passed('membership_and_common_write_permissions_enforced')
-
- archive = io.BytesIO()
- with zipfile.ZipFile(archive, 'w') as z:
- z.writestr('../escape.txt', 'no')
- member.call('/api/datasets?filename=unsafe.zip&format=zip', archive.getvalue(), team=team, expected=400)
- passed('unsafe_archive_rejected')
-
- rng = random.Random(42)
- weights = [[rng.gauss(0, .1) for _ in range(32)] for _ in range(32)]
- x = [[rng.gauss(0, 1) for _ in range(32)] for _ in range(128)]
- vx = [[rng.gauss(0, 1) for _ in range(32)] for _ in range(64)]
- def targets(rows):
- return [[sum(row[k] * weights[k][j] for k in range(32)) for j in range(32)] for row in rows]
- payload = json.dumps({'x': x, 'y': targets(x), 'validation_x': vx, 'validation_y': targets(vx)}).encode()
- assert len(payload) < 400000
- dataset = member.call('/api/datasets?filename=regression.json&name=Ethernet%20storage%20check', payload, team=team, expected=201)
- report['dataset_bytes'] = len(payload)
- assert dataset['sha256'] == hashlib.sha256(payload).hexdigest()
- download = '/api/datasets/' + dataset['id'] + '/files/download?path=regression.json'
- assert member.call(download, team=team, raw=True) == payload
- admin.call('/api/datasets/' + dataset['id'], team=teams[1]['id'], expected=404)
- admin.call('/api/storage/files?' + urllib.parse.urlencode({'source_team': team, 'scope': user['id']}), team=teams[1]['id'], expected=404)
- passed('small_upload_checksum_download_and_cross_team_isolation')
-
- source = (Path(__file__).parent / 'examples/train_shared.py').read_text()
- for backend, expected in [('nvidia', 1), ('tenstorrent', 2)]:
- script = source.replace('args = parser.parse_args()',
- f"args = parser.parse_args(['{backend}','--expected-devices','{expected}','--output','/outputs'])")
- job = member.call('/api/jobs', {'name': 'Ethernet folder training ' + backend, 'type': 'command',
- 'accelerator': backend, 'device_count': 1, 'dataset_id': dataset['id'], 'script': script,
- 'script_name': 'train.py', 'timeout_seconds': 600}, team=team, expected=201)['job_id']
- jobs.append(job)
- report['jobs'].append({'id': job, 'backend': backend})
- member.call('/api/datasets/' + dataset['id'], team=team, method='DELETE', expected=409)
- passed('active_dataset_deletion_blocked')
-
- deadline = time.monotonic() + 660
- for job in report['jobs']:
- while time.monotonic() < deadline:
- status = member.call('/api/jobs/' + job['id'], team=team)
- if status['job_state'] in ('Success', 'Failure', 'Cancelled'):
- break
- time.sleep(2)
- log = member.call('/api/jobs/' + job['id'] + '/logs', team=team)['logs']
- (args.out / (job['backend'] + '.log')).write_text(log)
- assert status['job_state'] == 'Success', (job['backend'], status.get('message'))
- assert 'DATASET_LOADED file=/inputs/regression.json' in log and 'TRAINING_PASSED' in log
- files = member.call('/api/jobs/' + job['id'] + '/files', team=team)['files']
- assert any(f['path'].endswith('.pt') for f in files)
- for file in files:
- content = member.call('/api/jobs/' + job['id'] + '/files/download?' + urllib.parse.urlencode({'path': file['path']}), team=team, raw=True)
- assert len(content) == file['size']
- if file['path'] == 'results.json':
- job['metrics'] = json.loads(content)
- job['node'], job['status'] = status['node'], status['job_state']
- passed(job['backend'] + '_real_training_and_output_downloads')
- for _ in range(45):
- storage = member.call('/api/storage', team=team)
- if storage['dataset_used_bytes'] >= len(payload) and storage['model_used_bytes'] > 0:
- break
- time.sleep(1)
- assert storage['dataset_used_bytes'] >= len(payload) and storage['model_used_bytes'] > 0
- assert storage['usage_stale'] is False
- report['storage'] = storage
- passed('host_sampled_dataset_and_model_usage')
- report['teams'] = [t['id'] for t in teams]
- except Exception:
- for job in jobs:
- try:
- member.call('/api/jobs/' + job + '/cancel', team=team, method='POST')
- except Exception:
- pass
- raise
- finally:
- for client in (member, admin):
- try:
- client.call('/auth/sign-out', {}, method='POST')
- except Exception:
- pass
- (args.out / 'results.json').write_text(json.dumps(report, indent=2) + '\n')
-
-
-if __name__ == '__main__':
- main()
diff --git a/deploy/k3s/verify_storage_folders_browser.mjs b/deploy/k3s/verify_storage_folders_browser.mjs
deleted file mode 100644
index 5efbd03..0000000
--- a/deploy/k3s/verify_storage_folders_browser.mjs
+++ /dev/null
@@ -1,59 +0,0 @@
-// Review the real portal using the regular account from verify_storage_folders.py.
-// State and screenshots are written only to the protected external evidence dir.
-import { readFileSync, writeFileSync } from 'node:fs'
-import { createHash } from 'node:crypto'
-
-const { chromium } = await import(
- process.env.MIST_PLAYWRIGHT_MODULE ??
- '/tmp/mist-browser-check/node_modules/playwright/index.mjs'
-)
-const base = process.env.MIST_SITE_URL ?? 'http://100.73.139.66:8088'
-const out = process.env.MIST_STORAGE_EVIDENCE_DIR
-if (!out) throw new Error('Set MIST_STORAGE_EVIDENCE_DIR to the live verifier output directory')
-const state = JSON.parse(readFileSync(`${out}/member-private.json`, 'utf8'))
-const browser = await chromium.launch({ headless: true })
-const context = await browser.newContext({ viewport: { width: 1440, height: 1000 } })
-const page = await context.newPage()
-const report = { checks: [], browser_errors: [] }
-page.on('pageerror', (error) => report.browser_errors.push(error.message))
-try {
- await page.goto(`${base}/datasets`)
- await page.getByLabel('Email', { exact: true }).fill(state.email)
- await page.getByLabel('Password', { exact: true }).fill(state.password)
- await page.getByRole('button', { name: 'Sign in', exact: true }).click()
- await page.getByRole('heading', { name: 'Datasets', exact: true }).waitFor()
- await page.getByLabel('Workspace', { exact: true }).click()
- await page.locator(`[role=option][data-value="${state.team_id}"]`).click()
- await page.getByText(/Shared disk:/).waitFor()
- await page.getByText('Advisory allowance — shared disk space is not reserved.').first().waitFor()
- report.checks.push('regular_member_login_and_advisory_storage_display')
-
- await page.getByRole('button', { name: 'Add dataset', exact: true }).click()
- await page.getByLabel('Dataset name (optional)').fill('Browser Ethernet check')
- const payload = Buffer.from('x,y\n1,2\n')
- await page.getByLabel('File or ZIP archive', { exact: true }).setInputFiles({
- name: 'browser.csv', mimeType: 'text/csv', buffer: payload,
- })
- const response = page.waitForResponse((r) => r.url().includes('/api/datasets?') && r.request().method() === 'POST')
- await page.getByRole('button', { name: 'Upload dataset', exact: true }).click()
- const uploaded = await response
- if (uploaded.status() !== 201) throw new Error(`Upload HTTP ${uploaded.status()}`)
- const dataset = await uploaded.json()
- if (dataset.sha256 !== createHash('sha256').update(payload).digest('hex')) throw new Error('Browser upload checksum mismatch')
- await page.getByText('Browser Ethernet check is ready to use.', { exact: true }).waitFor()
- report.checks.push('real_browser_upload_with_verified_checksum')
- await page.screenshot({ path: `${out}/datasets-browser.png`, fullPage: true })
-
- await page.goto(`${base}/jobs`)
- await page.getByText('Ethernet folder training nvidia', { exact: true }).waitFor()
- await page.getByText('Ethernet folder training tenstorrent', { exact: true }).waitFor()
- report.checks.push('both_real_training_jobs_visible_in_member_history')
- await page.screenshot({ path: `${out}/jobs-browser.png`, fullPage: true })
- if (report.browser_errors.length) throw new Error('Browser reported JavaScript errors')
- report.checks.push('no_browser_javascript_errors')
- console.log('PASS ' + report.checks.join(', '))
-} finally {
- await context.request.post(`${base}/auth/sign-out`, { data: {}, headers: { Origin: base } }).catch(() => {})
- await browser.close()
- writeFileSync(`${out}/browser-results.json`, JSON.stringify(report, null, 2))
-}
diff --git a/deploy/k3s/verify_tenstorrent_allocation.py b/deploy/k3s/verify_tenstorrent_allocation.py
deleted file mode 100644
index 4762791..0000000
--- a/deploy/k3s/verify_tenstorrent_allocation.py
+++ /dev/null
@@ -1,225 +0,0 @@
-"""Live acceptance test: four isolated n300 jobs, a fifth queued, then reuse.
-
-Run from the repo root with an admin KUBECONFIG and Python PyYAML installed.
-Test jobs and checkpoint subdirectories are unique per run and retained.
-The device probes mount character nodes solely to check cgroup enforcement;
-normal training jobs receive their devices exclusively through DRA/CDI.
-"""
-
-import argparse
-import copy
-from datetime import datetime, timezone
-import json
-from pathlib import Path
-import subprocess
-import time
-
-import yaml
-
-
-def kubectl(*args, data=None):
- result = subprocess.run(["k3s", "kubectl", *args], input=data, text=True,
- capture_output=True, timeout=45)
- if result.returncode:
- raise RuntimeError(result.stderr or result.stdout)
- return result.stdout
-
-
-def get(kind, *args):
- return json.loads(kubectl("-n", "mist", "get", kind, *args, "-o", "json"))
-
-
-def poll(check, description, timeout=240):
- deadline = time.monotonic() + timeout
- while time.monotonic() < deadline:
- result = check()
- if result:
- return result
- time.sleep(2)
- raise AssertionError(f"Timed out: {description}")
-
-
-def pod_for(job):
- pods = get("pods", "-l", f"job-name={job}")["items"]
- if not pods:
- return None
- assert len(pods) == 1, f"Expected one pod for {job}"
- pod = pods[0]
- assert pod["status"]["phase"] != "Failed", kubectl("-n", "mist", "logs", pod["metadata"]["name"])
- return pod
-
-
-def collect_results(run, jobs, pods, artifacts):
- """Read retained PVC data after training pods exit, then remove the reader."""
- reader_name = run + "-results"
- reader = {"apiVersion": "v1", "kind": "Pod", "metadata": {
- "name": reader_name, "namespace": "mist"}, "spec": {
- "restartPolicy": "Never", "automountServiceAccountToken": False,
- "activeDeadlineSeconds": 180,
- "nodeSelector": {"kubernetes.io/hostname": "utmist-tt"},
- "containers": [{"name": "reader", "image": pods[0]["spec"]["containers"][0]["image"],
- "imagePullPolicy": "IfNotPresent", "command": ["sleep", "180"],
- "securityContext": {"allowPrivilegeEscalation": False, "capabilities": {"drop": ["ALL"]}},
- "resources": {"requests": {"cpu": "10m", "memory": "32Mi"},
- "limits": {"cpu": "1", "memory": "256Mi"}},
- "volumeMounts": [{"name": "checkpoints", "mountPath": "/checkpoints", "readOnly": True}]}],
- "volumes": [{"name": "checkpoints", "persistentVolumeClaim": {
- "claimName": "training-tenstorrent-checkpoints", "readOnly": True}}]}}
- kubectl("create", "-f", "-", data=json.dumps(reader))
- try:
- poll(lambda: get("pod", reader_name)["status"]["phase"] == "Running", "PVC reader running", timeout=90)
- script = '''import json, pathlib, sys
-root=pathlib.Path("/checkpoints")/sys.argv[1]
-data=[]
-for i in range(5):
- folder=root/str(i)
- results=json.loads((folder/"results.json").read_text())
- for result in results:
- assert (folder/("tenstorrent-"+str(result["device_id"])+".pt")).stat().st_size>1000
- data.append({"allocation":json.loads((folder/"allocation.json").read_text()), "results":results})
-print(json.dumps(data))
-'''
- summaries = json.loads(kubectl("-n", "mist", "exec", reader_name, "--", "python3", "-c", script, run))
- for index, summary in enumerate(summaries):
- summary["job"] = jobs[index]["metadata"]["name"]
- assert len(summary["results"]) == 2 and summary["allocation"]["chip_count"] == 2
- assert len(summary["allocation"]["device_nodes"]) == 1
- for result in summary["results"]:
- assert result["final_mse"] < result["initial_mse"] * 0.02
- assert result["validation_mse"] < 0.0005
- assert result["max_weight_change"] > 0.05
- assert abs(result["checkpoint_reload_mse"] - result["validation_mse"]) < 1e-7
- starts = [datetime.fromisoformat(s["results"][0]["started_at"]) for s in summaries[:4]]
- ends = [datetime.fromisoformat(s["results"][0]["finished_at"]) for s in summaries[:4]]
- overlap = (min(ends) - max(starts)).total_seconds()
- assert overlap > 0, "First-chip training must overlap across all four jobs"
- released = min(datetime.fromisoformat(p["status"]["containerStatuses"][0]["state"]["terminated"]["finishedAt"].replace("Z", "+00:00"))
- for p in pods[:4])
- assert datetime.fromisoformat(summaries[4]["results"][0]["started_at"]) > released
- data = {"four_job_training_overlap_seconds": overlap, "jobs": summaries}
- (artifacts / "training-results.json").write_text(json.dumps(data, indent=2) + "\n")
- with (artifacts / "concurrent-checkpoints.tar.gz").open("wb") as output:
- subprocess.run(["k3s", "kubectl", "-n", "mist", "exec", reader_name, "--", "tar",
- "-C", "/checkpoints", "-czf", "-", run], stdout=output, check=True, timeout=45)
- return data
- finally:
- kubectl("-n", "mist", "delete", "pod", reader_name, "--wait=false", "--ignore-not-found")
-
-
-def main():
- parser = argparse.ArgumentParser()
- parser.add_argument("--artifacts", type=Path, required=True)
- args = parser.parse_args()
- args.artifacts.mkdir(parents=True, exist_ok=True)
- run = "tt-alloc-" + datetime.now(timezone.utc).strftime("%m%d-%H%M%S")
- checkpoint_root = f"/checkpoints/{run}"
- docs = list(yaml.safe_load_all(Path("deploy/k3s/training-tenstorrent-job.yaml").read_text()))
- base = next(doc for doc in docs if doc["kind"] == "Job")
- kubectl("apply", "-f", "-", data=yaml.safe_dump_all(doc for doc in docs if doc["kind"] != "Job"))
- configmap = kubectl("-n", "mist", "create", "configmap", "training-smoke-script",
- "--from-file=deploy/k3s/training_smoke.py", "--dry-run=client", "-o", "yaml")
- kubectl("apply", "-f", "-", data=configmap)
- names = [f"{run}-{i}" for i in range(5)]
-
- def job(index):
- manifest = copy.deepcopy(base)
- manifest["metadata"]["name"] = names[index]
- manifest["metadata"]["labels"] = {"mist.io/allocation-test": run}
- spec = manifest["spec"]["template"]["spec"]
- spec["automountServiceAccountToken"] = False
- container = spec["containers"][0]
- container["command"] = container["command"][:4] + [
- "--expected-devices", "2", "--steps", "500", "--output", f"{checkpoint_root}/{index}",
- "--start-file", f"{checkpoint_root}/start"]
- for number in range(4):
- spec["volumes"].append({"name": f"probe-{number}", "hostPath": {
- "path": f"/dev/tenstorrent/{number}", "type": "CharDevice"}})
- container["volumeMounts"].append({"name": f"probe-{number}",
- "mountPath": f"/isolation/{number}"})
- return manifest
-
- kubectl("apply", "-f", "-", data=yaml.safe_dump_all(job(i) for i in range(4)))
- print(f"RUN={run}: waiting for four initialized training jobs", flush=True)
-
- def ready():
- pods = [pod_for(name) for name in names[:4]]
- if not all(pod and pod["status"]["phase"] == "Running" for pod in pods):
- return None
- logs = [kubectl("-n", "mist", "logs", pod["metadata"]["name"]) for pod in pods]
- return pods if all("BARRIER_READY" in log for log in logs) else None
-
- pods = poll(ready, "four jobs ready")
- claims = get("resourceclaims")
- allocations = {}
- for pod in pods:
- claim_name = pod["status"]["resourceClaimStatuses"][0]["resourceClaimName"]
- claim = next(c for c in claims["items"] if c["metadata"]["name"] == claim_name)
- result = claim["status"]["allocation"]["devices"]["results"]
- assert len(result) == 1
- allocations[pod["metadata"]["name"]] = result[0]["device"]
- assert len(set(allocations.values())) == 4, allocations
- (args.artifacts / "claims-while-full.json").write_text(json.dumps(claims, indent=2) + "\n")
- (args.artifacts / "pods-while-full.json").write_text(json.dumps(pods, indent=2) + "\n")
-
- probe = '''import json,os,pathlib
-visible=list(pathlib.Path("/dev/tenstorrent").iterdir())
-assert len(visible)==1, visible
-allowed=[]; denied=[]
-for i in range(4):
- try:
- fd=os.open("/isolation/"+str(i),os.O_RDWR); os.close(fd); allowed.append(str(i))
- except PermissionError:
- denied.append(str(i))
-assert allowed==[visible[0].name], (allowed,visible)
-assert len(denied)==3, denied
-print(json.dumps({"allowed":allowed,"denied":denied}))
-'''
- isolation = {}
- for pod in pods:
- name = pod["metadata"]["name"]
- isolation[name] = json.loads(kubectl("-n", "mist", "exec", name, "--", "python3", "-c", probe))
- print(f"Four distinct reservations, device access isolated: {allocations}", flush=True)
-
- kubectl("apply", "-f", "-", data=yaml.safe_dump(job(4)))
-
- def queued():
- pod = pod_for(names[4])
- if not pod:
- return None
- events = get("events", "--field-selector", f"involvedObject.name={pod['metadata']['name']}")["items"]
- failed = [event for event in events if event["reason"] == "FailedScheduling"]
- if not failed:
- return None
- assert pod["status"]["phase"] == "Pending", pod
- assert not pod["spec"].get("nodeName"), pod
- assert any("resourceclaim" in e["message"].lower() or
- "cannot allocate" in e["message"].lower() for e in failed), failed
- return {"pod": pod, "events": failed}
-
- queue_evidence = poll(queued, "fifth job blocked by device allocation", timeout=60)
- (args.artifacts / "fifth-job-queued.json").write_text(json.dumps(queue_evidence, indent=2) + "\n")
- print("Fifth job is pending because all boards are reserved; releasing training barrier", flush=True)
- kubectl("-n", "mist", "exec", pods[0]["metadata"]["name"], "--", "touch", f"{checkpoint_root}/start")
-
- def complete():
- jobs = [get("jobs", name) for name in names]
- assert not any(j["status"].get("failed") for j in jobs), "A training job failed"
- return jobs if all(j["status"].get("succeeded") == 1 for j in jobs) else None
-
- jobs = poll(complete, "all five jobs successfully trained", timeout=300)
- final_pods = [pod_for(name) for name in names]
- for index, pod in enumerate(final_pods):
- log = kubectl("-n", "mist", "logs", pod["metadata"]["name"])
- assert "TRAINING_PASSED backend=tenstorrent devices=2" in log
- assert pod["status"]["containerStatuses"][0]["state"]["terminated"]["exitCode"] == 0
- (args.artifacts / f"job-{index}.log").write_text(log)
- summary = {"run": run, "allocations": allocations, "isolation": isolation,
- "fifth_job_queued": True, "all_five_jobs_complete": True,
- "checkpoint_root": checkpoint_root, "jobs": jobs, "pods": final_pods}
- summary["training"] = collect_results(run, jobs, final_pods, args.artifacts)
- (args.artifacts / "acceptance.json").write_text(json.dumps(summary, indent=2) + "\n")
- print(f"ALLOCATION_PASSED run={run}; all five jobs completed; checkpoints at {checkpoint_root}", flush=True)
-
-
-if __name__ == "__main__":
- main()
diff --git a/deploy/private/README.md b/deploy/private/README.md
deleted file mode 100644
index 8ba6b6f..0000000
--- a/deploy/private/README.md
+++ /dev/null
@@ -1,171 +0,0 @@
-# Private deployment operations
-
-Current user/admin and architecture guide: [department rollout](../../docs/content/department-rollout.md).
-Execution/evidence: [rollout checklist](../../docs/content/department-rollout-execution.md).
-Setup: [k3s](../../docs/content/k3s-setup.md), [Tailscale/SSH](../../docs/content/tailscale-ssh.md),
-[storage](../../docs/content/storage.md), and [developer setup](../../docs/content/development.md).
-
-## Services and persistent state
-
-| Component | Location | Persistent state |
-|---|---|---|
-| Nginx production portal | Main, localhost and `100.73.139.66:8088` | `/srv/mist-web/releases`, atomic `current` symlink |
-| Go API, auth, and admission controller | `mist-system`, main, one Recreate replica | Team/queue ConfigMaps; Kubernetes Jobs; Local `mist-auth-data` SQLite WAL PVC |
-| k3s server / agent | Main / QuietBox | Existing cluster and device operators |
-| NFSv4 storage | QuietBox | Ordinary `/srv/mist-data/teams` dataset/result folders over direct Ethernet |
-| Team provisioning agent | QuietBox `mist-team-storage.service` | Validated requests, ordinary directories and sampled usage |
-| Portal host firewall | Both `mist-portal-firewall.service` | Owned INPUT/FORWARD chains and host config |
-
-API image tag is `mist-api:folders-20261021`, imported into main's k3s containerd.
-New jobs run in `mist-team-` namespaces. Historical `mist` workloads/PVCs remain.
-Never scale the current API above one replica: admission requires one controller.
-
-## Build and redeploy on main
-
-Run from a checkout of the reviewed current main on the NVIDIA host. Required:
-Go 1.26.0, Node 22.16+, working Docker access, the private kubeconfig, existing
-auth Secrets, GPU/TT operators and mounted QuietBox storage. The script is for
-this existing installation; it is not a fresh-cluster/bootstrap installer.
-Checking out/merging Git does not restart or deploy services automatically.
-The API is Kubernetes-only; Redis and the retired Docker supervisor are unnecessary.
-
-
-```bash
-bash deploy/private/build-and-deploy.sh
-```
-
-The script builds API/CLI and the API container image, imports it, applies
-RBAC/admission/network manifests, waits for API rollout, installs web
-packages, type-checks/builds the SPA and atomically installs its production
-release. It requires the existing Docker and Kubernetes access. Host imports
-and web installation use authorized Docker bind/chroot/nsenter access; the
-process must enter the actual host mount/network namespace for host changes.
-
-Bootstrap credentials remain outside Git in mode-0600 local files and Kubernetes
-Secrets. `prepare-internal-secret.py` reuses the internal credential on redeploy.
-Never recreate the auth database, change its signing secret or print Secret YAML
-as part of an ordinary deploy. The script fails if bootstrap credentials are
-missing. Fresh-host installation needs deliberate address/node configuration,
-storage preparation and protected Secret creation; this is a deployment of the
-existing two-machine installation.
-
-Frontend-only release:
-
-```bash
-npm --prefix web-interface run build
-# As root in the actual main host namespace:
-bash deploy/private/install-web.sh /home/utmist/Mist
-```
-
-Use the pinned Node/Go runtime locations from the build script. A frontend
-rollback can repoint `/srv/mist-web/current` atomically to a previous complete
-release. Keep its API contract compatible. For a backend regression, rebuild a
-known department version with the same manifests/Secrets/state. The earlier
-account-only API does not enforce team policies and is not a safe ordinary
-rollback for research users. The pre-cleanup department release `3431ffd` contains
-the verified team architecture; rebuild a compatible department version rather
-than restoring the archived pre-platform main for normal users.
-
-## QuietBox host setup and storage cutover
-
-Follow [shared-folder storage operations](../k3s/storage/README.md). This supersedes
-image-backed setup, online filesystem growth and model-pool migration instructions.
-New storage uses `/srv/mist-data` over `192.168.240.2` and new NFS volume bindings.
-Existing image files and bindings are retained for rollback; fresh storage was
-requested without data migration. No physical disk is formatted.
-
-Team dataset/model values are advisory. Track actual shared-disk free space and
-sampled team file usage. Uploads preserve 1 GiB of disk headroom; running jobs can
-still exhaust storage. New Kubernetes PV/PVC sizes are binding declarations,
-not capacity reservations or write limits.
-
-The October 21 rollout also preserves existing Better Auth password hashes and
-deactivated accounts in Go auth, and requires real unexpired sessions. Session
-cookies based only on user IDs are rejected. Auth data remains on its existing
-local PVC; a consistent pre-cutover database snapshot is retained privately.
-
-## Portal network rules
-
-Install `portal-firewall.sh` under `/usr/local/lib/mist` and the accompanying
-systemd unit on both hosts. `/etc/mist-portal-firewall.conf` contains
-`MIST_PORTAL_PORT=8088` on main and `MIST_PORTAL_PORT=0` on QuietBox. Enable/restart
-`mist-portal-firewall` after installation. The unit follows Tailscale/k3s service
-restarts; manually reapply after UFW/netfilter resets.
-
-The script restricts ordinary sources on `tailscale0` to main TCP 8088 and denies
-forwarded cluster traffic. The explicit administrator/infrastructure IPv4 and
-IPv6 allowlist is in the script. Update it before relying on a new administrator
-device. LAN rules and cloud Tailscale ACL/sharing settings are unchanged.
-`verify-portal-firewall.sh` checks the actual installed chains using an isolated
-synthetic untrusted source and removes its temporary namespace/rules afterward.
-It requires root in the main host network namespace.
-
-## Services to restart and inspect
-
-Use a root-capable administrator session for service mutations. Logs are available
-through `journalctl -u ` on the owning host and `kubectl logs` for pods.
-Ordinary application redeployment should use the build script; restart k3s/NFS
-only when resolving their own infrastructure problem because workloads depend on them.
-
-| Component | Owner | Status command |
-|---|---|---|
-| Portal | Main | `systemctl status nginx --no-pager` |
-| API, auth, and controller | Cluster | `k3s kubectl -n mist-system rollout status deployment/mist-api` |
-| Control plane | Main | `systemctl status k3s --no-pager` |
-| Worker | QuietBox | `systemctl status k3s-agent --no-pager` |
-| Shared storage | QuietBox | `systemctl status nfs-server mist-team-storage --no-pager` |
-| Private network | Both | `systemctl status tailscaled --no-pager` |
-| Portal restrictions | Both | `systemctl status mist-portal-firewall --no-pager` |
-
-The `mist-api-forward` and `mist-web` user services are local development helpers,
-not requirements for the production portal. The default-namespace test Nginx and
-k3s's Traefik also do not serve the current portal; it uses host Nginx on 8088.
-
-## Check after startup
-
-Main:
-
-```bash
-systemctl is-active k3s tailscaled nginx mist-portal-firewall
-KUBECONFIG=/home/utmist/.kube/config k3s kubectl get nodes
-KUBECONFIG=/home/utmist/.kube/config k3s kubectl get pods -n mist-system
-curl -fsS http://127.0.0.1:8088/api/session
-```
-
-QuietBox:
-
-```bash
-systemctl is-active k3s-agent tailscaled nfs-server mist-team-storage mist-portal-firewall
-findmnt -T /srv/mist-data
-df -h /srv/mist-data
-```
-
-With root access, inspect `exportfs -v` and the provisioning service journal if a
-team is not ready. Anonymous `/api/jobs` must return 401; the website must allow
-real login and show two NVIDIA GPUs/four TT boards. Both nodes must be Ready.
-
-## Troubleshooting
-
-- **401:** Sign in again. SSH/Tailscale identity is separate from Mist identity.
-- **403:** Confirm active team membership, scope writing authority and current
- policy. Administrators also need team membership to use files/jobs.
-- **503:** Check auth and API pods, DNS, shared directory readiness and provisioning
- status. Access and missing team mounts fail closed.
-- **Queued:** Check per-team concurrency/resources, node health and device
- availability. Aged large requests drain their target node; no preemption.
-- **507 / ENOSPC:** Team filesystem is full. Check actual QuietBox disk space; advisory allowances do not add capacity.
- Delete only unneeded datasets that have no active job references. No automatic result cleanup was added.
-- **Image rejected:** Use an explicit tag/digest from an allowed registry. No
- per-image approval is needed. Private pull-credential management is excluded.
-- **Image pull/runtime failure:** Verify the image is publicly accessible and
- includes its dependencies. Custom TT images need compatible TT libraries;
- select container runtime. The tested default TT profile remains available.
-- **Revocation pending:** An admin mutation returns 503 if job suspension could
- not be completed; correct cluster connectivity and retry.
-- **Login rate limited:** Retry after the indicated delay. Session checks are
- authoritative and do not share the restrictive password-login rate limit.
-
-Existing `backup-auth.sh` is a manual foundation utility; no automatic backup,
-recovery, HTTPS/Serve, monitoring or cleanup service was added. HTTP remains
-private over Tailscale. There is one control plane, one auth database instance
-and one NFS storage host; no high-availability failover is claimed.
diff --git a/deploy/private/backup-auth.sh b/deploy/private/backup-auth.sh
deleted file mode 100755
index bf49162..0000000
--- a/deploy/private/backup-auth.sh
+++ /dev/null
@@ -1,15 +0,0 @@
-#!/bin/bash
-# Run on main node. SQLite online backup produces a consistent database snapshot.
-# Output is protected and contains member credentials/session data.
-set -euo pipefail
-export KUBECONFIG=${KUBECONFIG:-/home/utmist/.kube/config}
-backup_dir=${1:-/home/utmist/.local/share/mist-backups}
-mkdir -p "$backup_dir"
-chmod 0700 "$backup_dir"
-umask 077
-file="$backup_dir/auth-$(date -u +%Y%m%dT%H%M%S).sqlite"
-trap 'rm -f "$file.partial"' EXIT
-k3s kubectl exec -n mist-system deployment/mist-api -- cat /data/auth.sqlite > "$file.partial"
-test -s "$file.partial"
-mv "$file.partial" "$file"
-echo "Auth backup saved: $file"
diff --git a/deploy/private/build-and-deploy.sh b/deploy/private/build-and-deploy.sh
deleted file mode 100755
index 880c4ba..0000000
--- a/deploy/private/build-and-deploy.sh
+++ /dev/null
@@ -1,31 +0,0 @@
-#!/bin/bash
-# API/auth images are built/imported on the main node; no registry push is needed.
-# Requires Kubernetes credentials and root privileges for image import/web install.
-set -euo pipefail
-cd "$(dirname "$0")/../.."
-export PATH=/home/utmist/.local/share/mist-runtimes/go-1.25.1/bin:/home/utmist/.local/share/mist-runtimes/node-22.16.0/bin:$PATH
-export KUBECONFIG=${KUBECONFIG:-/home/utmist/.kube/config}
-# Never silently deploy an unauthenticated service if its bootstrap secret is missing.
-k3s kubectl get secret mist-auth-bootstrap -n mist-system >/dev/null
-python3 deploy/private/prepare-internal-secret.py
-CGO_ENABLED=0 go -C src build -o ../bin/mist-api .
-CGO_ENABLED=0 go -C cli build -o ../bin/mist-cli .
-docker build -f deploy/k3s/Dockerfile.api -t mist-api:folders-20261021 .
-archive=$(mktemp /tmp/mist-deploy.XXXXXX.tar)
-trap 'python3 -c "import os,sys; os.unlink(sys.argv[1])" "$archive"' EXIT
-docker save -o "$archive" mist-api:folders-20261021
-# Existing approved Docker access permits host import, without changing sudo policy.
-docker run --rm --mount type=bind,src=/,dst=/host alpine:3.16.3 chroot /host /usr/local/bin/k3s ctr images import "$archive"
-k3s kubectl apply -f deploy/k3s/storage/shared.yaml -f deploy/k3s/team-rbac.yaml -f deploy/k3s/team-workload-policy.yaml -f deploy/k3s/mist-api.yaml
-k3s kubectl rollout restart deployment/mist-api -n mist-system
-k3s kubectl rollout status deployment/mist-api -n mist-system --timeout=120s
-npm --prefix web-interface ci
-npx --prefix web-interface tsc --noEmit -p web-interface/tsconfig.json
-npm --prefix web-interface run build
-docker run --rm --privileged --pid host --network host --mount type=bind,src=/,dst=/host alpine:3.16.3 chroot /host /usr/bin/nsenter -t 1 -m -n -p -r -w /bin/bash "$PWD/deploy/private/install-web.sh" "$PWD"
-for attempt in $(seq 1 15); do
- if curl --fail --silent http://127.0.0.1:8088/api/session; then exit 0; fi
- sleep 2
-done
-echo 'Private website session check failed' >&2
-exit 1
diff --git a/deploy/private/install-web.sh b/deploy/private/install-web.sh
deleted file mode 100755
index 9d26d22..0000000
--- a/deploy/private/install-web.sh
+++ /dev/null
@@ -1,17 +0,0 @@
-#!/bin/bash
-# Run as root in the main node's host mount namespace, after npm run build.
-set -euo pipefail
-repo=${1:-/home/utmist/Mist}
-release=/srv/mist-web/releases/$(date -u +%Y%m%dT%H%M%S)
-install -d -m 0755 "$release"
-cp -r "$repo/web-interface/dist/." "$release/"
-chmod -R a+rX "$release"
-ln -s "$release" /srv/mist-web/current.new
-mv -Tf /srv/mist-web/current.new /srv/mist-web/current
-install -m 0644 "$repo/deploy/private/nginx.conf" /etc/nginx/conf.d/mist.conf
-install -d -m 0755 /etc/systemd/system/nginx.service.d
-install -m 0644 "$repo/deploy/private/nginx-mist-startup.conf" /etc/systemd/system/nginx.service.d/mist.conf
-systemctl daemon-reload
-nginx -t
-systemctl enable nginx
-systemctl reload nginx
diff --git a/deploy/private/mist-portal-firewall.service b/deploy/private/mist-portal-firewall.service
deleted file mode 100644
index 2c6fa97..0000000
--- a/deploy/private/mist-portal-firewall.service
+++ /dev/null
@@ -1,15 +0,0 @@
-[Unit]
-Description=Mist portal access boundary for shared Tailscale researchers
-After=network-pre.target tailscaled.service ufw.service k3s.service k3s-agent.service
-Wants=tailscaled.service
-PartOf=tailscaled.service k3s.service k3s-agent.service
-Before=nginx.service
-
-[Service]
-Type=oneshot
-EnvironmentFile=/etc/mist-portal-firewall.conf
-ExecStart=/usr/local/lib/mist/portal-firewall.sh
-RemainAfterExit=yes
-
-[Install]
-WantedBy=multi-user.target
diff --git a/deploy/private/nginx-mist-startup.conf b/deploy/private/nginx-mist-startup.conf
deleted file mode 100644
index 53a50e0..0000000
--- a/deploy/private/nginx-mist-startup.conf
+++ /dev/null
@@ -1,9 +0,0 @@
-# Install as /etc/systemd/system/nginx.service.d/mist.conf.
-[Unit]
-StartLimitIntervalSec=0
-Wants=network-online.target tailscaled.service k3s.service
-After=network-online.target tailscaled.service k3s.service
-[Service]
-# Tailscale may need a few seconds to restore its address before bind().
-Restart=on-failure
-RestartSec=5
diff --git a/deploy/private/nginx.conf b/deploy/private/nginx.conf
deleted file mode 100644
index fae10b9..0000000
--- a/deploy/private/nginx.conf
+++ /dev/null
@@ -1,36 +0,0 @@
-server {
- listen 127.0.0.1:8088;
- listen 100.73.139.66:8088;
- server_name utmist.tail459b5e.ts.net 100.73.139.66 localhost;
- root /srv/mist-web/current;
- index index.html;
- client_max_body_size 0;
- client_body_timeout 86400s;
- add_header X-Content-Type-Options nosniff always;
- add_header Referrer-Policy same-origin always;
- add_header X-Frame-Options DENY always;
- location /api/ {
- proxy_pass http://10.43.1.116:3000/;
- proxy_http_version 1.1;
- proxy_set_header Host $host;
- proxy_set_header X-Real-IP $remote_addr;
- proxy_request_buffering off;
- proxy_read_timeout 86400s;
- proxy_send_timeout 86400s;
- }
- location /auth/ {
- proxy_pass http://10.43.1.116:3000;
- proxy_set_header Host $host;
- proxy_set_header X-Real-IP $remote_addr;
- client_max_body_size 64k;
- }
- location /assets/ {
- try_files $uri =404;
- expires 1y;
- add_header Cache-Control "public, immutable";
- }
- location / {
- add_header Cache-Control "no-cache";
- try_files $uri $uri/ /index.html;
- }
-}
diff --git a/deploy/private/portal-firewall.sh b/deploy/private/portal-firewall.sh
deleted file mode 100644
index 366a776..0000000
--- a/deploy/private/portal-firewall.sh
+++ /dev/null
@@ -1,31 +0,0 @@
-#!/bin/bash
-# Root, actual host network namespace. Replace only Mist's chains atomically;
-# preserve all other firewall tables/chains and existing LAN access.
-set -euo pipefail
-portal_port=${MIST_PORTAL_PORT:-8088}
-for family in iptables ip6tables; do
- restore="${family}-restore"
- "$family" -w -N MIST-TAIL-INPUT 2>/dev/null || true
- "$family" -w -N MIST-TAIL-FORWARD 2>/dev/null || true
- {
- printf '*filter\n-F MIST-TAIL-INPUT\n-F MIST-TAIL-FORWARD\n'
- if [ "$family" = iptables ]; then
- for trusted in 100.73.139.66 100.95.175.37 100.99.194.52; do
- printf -- '-A MIST-TAIL-INPUT -s %s/32 -j RETURN\n' "$trusted"
- printf -- '-A MIST-TAIL-FORWARD -s %s/32 -j RETURN\n' "$trusted"
- done
- if [ "$portal_port" != 0 ]; then
- printf -- '-A MIST-TAIL-INPUT -p tcp --dport %s -j RETURN\n' "$portal_port"
- fi
- else
- for trusted in fd7a:115c:a1e0::d401:8bda fd7a:115c:a1e0::f72a:af26 fd7a:115c:a1e0::2901:c2d7; do
- printf -- '-A MIST-TAIL-INPUT -s %s/128 -j RETURN\n' "$trusted"
- printf -- '-A MIST-TAIL-FORWARD -s %s/128 -j RETURN\n' "$trusted"
- done
- fi
- # No IPv6 portal listener is enabled; do not permit an alternate bypass.
- printf -- '-A MIST-TAIL-INPUT -j DROP\n-A MIST-TAIL-FORWARD -j DROP\nCOMMIT\n'
- } | "$restore" --wait 10 --noflush
- "$family" -w -C INPUT -i tailscale0 -j MIST-TAIL-INPUT 2>/dev/null || "$family" -w -I INPUT 1 -i tailscale0 -j MIST-TAIL-INPUT
- "$family" -w -C FORWARD -i tailscale0 -j MIST-TAIL-FORWARD 2>/dev/null || "$family" -w -I FORWARD 1 -i tailscale0 -j MIST-TAIL-FORWARD
-done
diff --git a/deploy/private/prepare-internal-secret.py b/deploy/private/prepare-internal-secret.py
deleted file mode 100644
index 4c5c528..0000000
--- a/deploy/private/prepare-internal-secret.py
+++ /dev/null
@@ -1,29 +0,0 @@
-#!/usr/bin/python3
-"""Reuse the internal auth token on ordinary deployments. Never print its value."""
-import base64
-import json
-import os
-from pathlib import Path
-import secrets
-import subprocess
-
-env = {**os.environ, 'KUBECONFIG': os.environ.get('KUBECONFIG', '/home/utmist/.kube/config')}
-command = ['k3s', 'kubectl']
-token_file = Path('/home/utmist/.config/mist/internal-token')
-existing = subprocess.run(command + ['get', 'secret', 'mist-internal-auth', '-n', 'mist-system', '-o', 'json'], env=env, capture_output=True, text=True)
-if existing.returncode == 0:
- token = base64.b64decode(json.loads(existing.stdout)['data']['MIST_INTERNAL_TOKEN']).decode()
-else:
- if 'NotFound' not in existing.stderr:
- raise SystemExit('Cannot inspect internal Secret; refusing to replace credentials')
- token = token_file.read_text() if token_file.exists() else secrets.token_urlsafe(48)
-if len(token) < 32:
- raise SystemExit('Internal token is invalid')
-token_file.parent.mkdir(parents=True, exist_ok=True)
-token_file.write_text(token)
-token_file.chmod(0o600)
-body = {'apiVersion': 'v1', 'kind': 'Secret', 'metadata': {'name': 'mist-internal-auth', 'namespace': 'mist-system'}, 'stringData': {'MIST_INTERNAL_TOKEN': token}}
-result = subprocess.run(command + ['apply', '-f', '-'], input=json.dumps(body), env=env, capture_output=True, text=True)
-if result.returncode:
- raise SystemExit('Cannot apply internal Secret')
-print('Internal auth Secret ready; credentials preserved')
diff --git a/deploy/private/verify-portal-firewall.sh b/deploy/private/verify-portal-firewall.sh
deleted file mode 100644
index 3bf6417..0000000
--- a/deploy/private/verify-portal-firewall.sh
+++ /dev/null
@@ -1,52 +0,0 @@
-#!/bin/bash
-# Root in the host namespace. Exercise the installed enforcement chains with
-# an isolated, simulated untrusted Tailnet source; no real peer is impersonated.
-set -euo pipefail
-namespace=mist-firewall-check
-source_ip=100.127.254.253
-if ip netns list | awk '{print $1}' | grep -qx "$namespace"; then
- echo 'Verification namespace already exists; refusing to replace it' >&2; exit 1
-fi
-if ip route show "$source_ip/32" | grep -q .; then
- echo 'Verification source already routed; refusing to replace it' >&2; exit 1
-fi
-iptables -S MIST-TAIL-INPUT >/dev/null
-cleanup() {
- iptables -w -D INPUT -i mist-fw-host -j MIST-TAIL-INPUT 2>/dev/null || true
- iptables -w -D INPUT -i mist-fw-host -j ACCEPT 2>/dev/null || true
- iptables -w -D FORWARD -i mist-fw-host -j MIST-TAIL-FORWARD 2>/dev/null || true
- ip route del "$source_ip/32" dev mist-fw-host 2>/dev/null || true
- ip netns del "$namespace" 2>/dev/null || true
- ip link del mist-fw-host 2>/dev/null || true
-}
-trap cleanup EXIT
-ip netns add "$namespace"
-ip link add mist-fw-host type veth peer name mist-fw-guest
-ip link set mist-fw-guest netns "$namespace"
-ip addr add 198.18.0.1/30 dev mist-fw-host
-ip link set mist-fw-host up
-ip route add "$source_ip/32" dev mist-fw-host
-ip netns exec "$namespace" ip addr add 198.18.0.2/30 dev mist-fw-guest
-ip netns exec "$namespace" ip addr add "$source_ip/32" dev lo
-ip netns exec "$namespace" ip link set lo up
-ip netns exec "$namespace" ip link set mist-fw-guest up
-ip netns exec "$namespace" ip route add default via 198.18.0.1
-# Emulate the real tailscale0 dispatch with a test-only interface. Accept after
-# RETURN avoids Tailscale's deliberate rejection of 100/10 on non-tail devices.
-iptables -w -I INPUT 1 -i mist-fw-host -j ACCEPT
-iptables -w -I INPUT 1 -i mist-fw-host -j MIST-TAIL-INPUT
-iptables -w -I FORWARD 1 -i mist-fw-host -j MIST-TAIL-FORWARD
-code=$(ip netns exec "$namespace" curl --interface "$source_ip" --max-time 4 -sS -o /dev/null -w '%{http_code}' http://100.73.139.66:8088/api/session)
-[ "$code" = 200 ]
-echo 'PASS untrusted source reaches portal'
-for target in 100.73.139.66:22 100.73.139.66:6443 100.73.139.66:30491 10.43.1.116:3000; do
- if ! ip netns exec "$namespace" python3 -c 'import socket,sys; host,port=sys.argv[1].split(":"); s=socket.socket(); s.settimeout(2); s.bind((sys.argv[2],0))
-try:
- s.connect((host,int(port)))
-except OSError:
- sys.exit(0)
-sys.exit(1)' "$target" "$source_ip"; then
- echo "FAIL untrusted source reached $target" >&2; exit 1
- fi
- echo "PASS untrusted source denied $target"
-done
diff --git a/docs/README.md b/docs/README.md
deleted file mode 100644
index 2cad2d2..0000000
--- a/docs/README.md
+++ /dev/null
@@ -1,40 +0,0 @@
-# Mist Documentation
-
-This directory contains the documentation site for Mist, powered by **Hugo** and the **Hextra** documentation theme.
-
-## Viewing Documentation Locally
-
-To preview the documentation locally with live-reload:
-
-```bash
-cd docs
-hugo server
-```
-
-The site will be available at `http://localhost:1313/docs/`.
-
-To build the static site (outputting to `web-interface/public/docs` for the web portal):
-
-```bash
-cd docs
-hugo
-```
-
-## Documentation Guides
-
-Documentation source files are located in `content/`:
-
-| Guide | Description |
-|---|---|
-| [Department operating guide](content/department-rollout.md) | Sign in, manage teams, upload data, submit/download jobs |
-| [Architecture](content/architecture.md) | Service topologies, boundaries, and codebase layout |
-| [k3s setup](content/k3s-setup.md) | Cluster setup, worker nodes, and hardware acceleration |
-| [Tailscale and SSH](content/tailscale-ssh.md) | Remote secure networking and administrator SSH |
-| [Storage](content/storage.md) | Dataset and model shared storage pools |
-| [Developer setup](content/development.md) | Local frontend and backend development |
-| [Testing](content/testing.md) | Automated tests and evidence runs |
-| [Design system](content/design-system.md) | Portal design principles and components |
-| [Frontend architecture](content/frontend-architecture.md) | UI layout and backend offloading plan |
-| [Execution record](content/department-rollout-execution.md) | Department rollout status and validation |
-| [Cleanup record](content/repository-cleanup.md) | Repository maintenance history |
-| [Historical archive](content/archive/_index.md) | Archived previous-generation guides and plans |
diff --git a/docs/agents/architecture/api.md b/docs/agents/architecture/api.md
new file mode 100644
index 0000000..74698ad
--- /dev/null
+++ b/docs/agents/architecture/api.md
@@ -0,0 +1,239 @@
+# API
+
+`mistd api` is the only way clients and agents reach Mist state. This document
+fixes the conventions every endpoint follows, how callers authenticate, and the
+job wire types shared by the CLI, portal and scheduler. Each component document
+lists the endpoints it owns. Overview: [ARCHITECTURE.md](../../../ARCHITECTURE.md).
+
+## How it fits
+
+Every request passes the same middleware, then reaches the handlers of the
+component that owns it:
+
+```mermaid
+flowchart LR
+ portal["Portal
session cookie"] --> mw
+ relay["mist-relay
relay token + asserted user"] --> mw
+ agents["Provisioner, key lookup
host token"] --> mw
+ mw["Auth middleware
resolves principal"] --> authz{identity.Authorize}
+ authz --> idh["/v1/me, /v1/teams
Identity"]
+ authz --> sth["/v1/teams/slug/usage
Storage"]
+ authz --> jh["/v1/jobs, /v1/partitions, /v1/queue
Scheduler"]
+ authz --> adm["/v1/admin/...
Identity + Scheduler"]
+ mw --> inth["/internal/v1/...
Identity"]
+ idh & sth & jh & adm & inth --> db[(Postgres)]
+ jh -->|"logs, attach"| ex[Executor]
+```
+
+## Conventions
+
+- **Format.** JSON over HTTP. Paths are versioned: `/v1/...` for users and
+ admins, `/internal/v1/...` for host agents.
+- **Placement.** One `mistd api` process on the control node (NVIDIA box). The
+ portal's Nginx forwards `/api/*` to it and strips the prefix. Agents reach it
+ on the direct link (`192.168.240.1`).
+- **Logic lives in the API.** Clients don't compute permissions, defaults or
+ states; the API returns ready-to-display fields (for example `pending_reason`
+ and `can_cancel`). Clients may poll: every 5–10 s is fine.
+- **Errors** use one shape and a stable `code`:
+
+ ```json
+ { "error": { "code": "limit_exceeded", "message": "Team vision already has 2 GPUs running", "details": {} } }
+ ```
+
+ Codes: `unauthenticated`, `forbidden`, `not_found`, `invalid`,
+ `limit_exceeded`, `conflict`, `unavailable`, `internal`.
+- **Lists** take `?limit=` and `?cursor=` and return `{ "items": [...], "next_cursor": "..." }`.
+- **Times** are RFC 3339 UTC strings. **Durations** are integer seconds.
+ **Sizes** are integer bytes.
+- **Streams** (log follow, interactive attach) use WebSockets on the same paths
+ with `?follow=1` or an `Upgrade` header.
+
+## Authentication
+
+| Caller | Credential | Principal |
+|---|---|---|
+| Portal | Session cookie set at sign-in | The signed-in account |
+| CLI on the login gateway | Relay token + `X-Mist-Asserted-User` (see [login-gateway.md](login-gateway.md)) | The asserted account |
+| Provisioner, relay key lookup | Per-host agent token | The host; `/internal/*` only |
+| Laptop CLI (later) | Personal API token | The token's account |
+
+One middleware resolves the principal, and authorization goes through
+`identity.Authorize` (see [identity-and-accounts.md](identity-and-accounts.md)).
+Handlers never read identity headers directly. Check active account and current
+membership on every request and stream establishment; revoke existing streams
+when the authorization generation changes. The relay cannot assert disabled
+accounts. Agent tokens are scoped by host and operation, not interchangeable
+administrator credentials.
+
+Production uses HTTPS, `Secure`, `HttpOnly`, `SameSite=Lax` session cookies,
+CSRF protection on mutations, and an Origin allowlist for browser WebSockets.
+Nginx must strip user-supplied identity headers. Keep dev principal headers out
+of production binaries. See Identity for bounded directory freshness/leases.
+
+Authorization matrix: owners may view/cancel their currently authorized jobs and
+attach to their own interactive jobs; current team members may view team job
+metadata/scripts/logs; team leads may cancel team jobs; admins may view/cancel
+all jobs but cannot attach as the user. Former membership or ownership alone
+never grants continued team access. Non-team callers receive `not_found` on job
+and log endpoints; `/queue` provides only the explicitly public summary.
+
+`POST /v1/jobs` accepts a required `Idempotency-Key` from CLI clients. Store
+(account, key, canonical client request hash, job ID, resolved submission spec)
+atomically with submission. Look up the key before applying current defaults:
+same key/body returns the original job even if defaults changed; differing body
+returns `conflict`. Reauthorize access before returning a cached result. Retain keys
+for at least the job metadata lifetime. Cancellation is intrinsically idempotent
+and returns the current job after recording stop intent; it does not wait for
+termination or release resources.
+
+## Endpoint index
+
+| Area | Endpoints | Owner doc |
+|---|---|---|
+| Account | `GET/PATCH /v1/me`, `/v1/me/ssh-keys` | identity |
+| Teams | `GET /v1/teams`, `GET /v1/teams/{slug}` | identity |
+| Storage | `GET /v1/teams/{slug}/usage`, `GET /v1/volumes` (admin) | storage |
+| Jobs | `POST /v1/jobs`, `GET /v1/jobs`, `GET /v1/jobs/{id}`, `POST /v1/jobs/{id}/cancel`, `GET /v1/jobs/{id}/log`, `GET /v1/jobs/{id}/attach` (WebSocket) | scheduler / executors |
+| Cluster | `GET /v1/partitions`, `GET /v1/queue` | scheduler |
+| Admin | `/v1/admin/accounts`, `/v1/admin/partitions`, `/v1/admin/qos`, `/v1/admin/limits`, `/v1/admin/nodes` | identity, scheduler |
+| Agents | `/internal/v1/provision/*`, `/internal/v1/ssh-keys/{username}` | identity |
+| Cleanup | `POST /internal/v1/executions/{execution_id}/cleanup` | executor; node-scoped authenticated receipt |
+
+## Job wire types
+
+### Submitting: `POST /v1/jobs`
+
+```json
+{
+ "team": "vision",
+ "partition": "gpu",
+ "qos": "normal",
+ "name": "resnet-sweep-3",
+ "kind": "batch",
+ "script": "#!/bin/bash\n#MIST --gpus=1\npython train.py --lr 3e-4\n",
+ "args": [],
+ "workdir": "/mist/teams/vision/alice/resnet",
+ "output": "mist-%j.out",
+ "env": { "WANDB_PROJECT": "resnet" },
+ "resources": {
+ "cpus": 8,
+ "memory_bytes": 34359738368,
+ "gpus": 1,
+ "tt_boards": 0,
+ "scratch_bytes": 107374182400
+ },
+ "time_limit_seconds": 14400,
+ "signal_grace_seconds": 60
+}
+```
+
+- `kind`:
+ - `batch` needs `script`.
+ - `command` (from `mist run`) takes `argv` instead of `script`.
+ - `interactive` (from `mist shell`) takes neither.
+- **The API fills in missing fields** from partition and QoS defaults and
+ returns the complete, normalised spec. Defaults live only in the database,
+ never in clients.
+- **Validation:**
+ - The caller must be active and a current member of `team`; directory state
+ must be fresh. Admission rechecks this atomically, not just at submission.
+ - `workdir` must be inside the charged team. Omitted workdir uses the personal
+ home for an explicit team; otherwise derive team from the submitted canonical
+ path. Never accept a login landing folder as a job workdir.
+ - Resolve relative output under workdir and expand only `%j`. Require output
+ inside the charged team's durable tree. Executor revalidates inside the
+ restricted mount namespace as the user, handling symlinks/path races without
+ privileged traversal. Missing/unwritable paths produce a clear launch error.
+ - CPU, memory, scratch and time are positive bounded integers; accelerators
+ are nonnegative whole counts. API `cpus` is a whole-core count, converted
+ once to executor millicores (1 CPU = 1000 mCPU). Reject overflow and incompatible partition
+ resources. Enforce partition/QoS maximums, including signal grace 1–120 s.
+ - `gpu` needs GPUs >= 1 and TT boards = 0; `cpu` requires both = 0; `tt` needs
+ boards >= 1 and GPUs = 0. A request must fit at least one configured node.
+ - `interactive` kind requires interactive QoS; batch/command use normal QoS.
+ Users cannot obtain interactive priority for unattached batch jobs.
+ - Optional `runtime` selects only an approved retained partition profile;
+ resolve it to a digest/profile version at admission and freeze it.
+ - Reject unknown/mutually exclusive command fields. Bound script size (1 MiB),
+ env total (64 KiB), argument total (128 KiB) and name length (128 characters).
+ Preserve batch `args` separately; command uses `argv`, without a shell.
+ - User env cannot override identity, allocation, device visibility, internal
+ attach socket or timing fields. Runtime documents the protected names.
+- **Reply:** `201` with the job object, in state `PENDING`.
+
+### Job object: `GET /v1/jobs/{id}`
+
+```json
+{
+ "id": 412,
+ "name": "resnet-sweep-3",
+ "user": "alice",
+ "team": "vision",
+ "partition": "gpu",
+ "qos": "normal",
+ "kind": "batch",
+ "state": "PENDING",
+ "pending_reason": "RESOURCES",
+ "pending_detail": "Waiting for 1 GPU; estimated start after job 409 ends",
+ "exit_reason": null,
+ "exit_code": null,
+ "resources": { "cpus": 8, "memory_bytes": 34359738368, "gpus": 1, "tt_boards": 0, "scratch_bytes": 107374182400 },
+ "time_limit_seconds": 14400,
+ "node": null,
+ "workdir": "/mist/teams/vision/alice/resnet",
+ "output_path": "/mist/teams/vision/alice/resnet/mist-412.out",
+ "submitted_at": "2026-11-02T15:04:05Z",
+ "started_at": null,
+ "ended_at": null,
+ "can_cancel": true,
+ "can_attach": false
+}
+```
+
+**States and reasons** are defined in [scheduler.md](scheduler.md#job-states).
+The object also exposes `cleanup_pending`, `cleanup_detail`, `output_incomplete`,
+`authorization_generation`, and (interactive only) `attached`, `detached_at`,
+`reattach_deadline`. The API computes `can_cancel` and `can_attach` using state
+and current authorization. `CLEANING` is nonterminal; a `NODE_FAIL` job may be
+terminal with cleanup still pending and capacity quarantined.
+
+Other teams' jobs appear only in `GET /v1/queue` as an allowlisted summary:
+job ID, partition, QoS, state, resource counts, age and a public pending-reason
+code. Set `user`, `team`, `name`, `workdir` and `output_path` to null. Exclude
+scripts, env, arbitrary error text, event actors and detailed pending messages;
+redacting a few top-level names does not sanitize embedded paths or secrets.
+
+### Logs: `GET /v1/jobs/{id}/log`
+
+- **Running job:** returns the live output stream from the executor. Add
+ `?follow=1` to stream it over a WebSocket.
+- **Finished job:** returns the stored tail (last 64 KiB) from the database.
+- **The full log normally resides in `/mist`.** Include `output_path` and
+ `output_incomplete`. Node failure, OOM/SIGKILL, or storage errors can prevent a
+ final footer or file creation; the DB retains the observed outcome and last
+ available tail. API log viewing never reads arbitrary user-supplied paths.
+
+### Attach: `GET /v1/jobs/{id}/attach` (WebSocket)
+
+Binary frames carry terminal bytes. Text frames carry control messages:
+
+```jsonl
+{ "type": "resize", "cols": 120, "rows": 40 }
+{ "type": "signal", "name": "SIGINT" }
+{ "type": "exit", "code": 0 }
+```
+
+Only the currently authorized job owner can attach, while the interactive job
+is `RUNNING`. The holder allows one attachment; a second receives `conflict`.
+Network closure detaches, explicit `POST .../cancel` cancels, and shell exit ends
+the job. Resize and allowed terminal signals target the existing PTY/foreground
+process group; callers cannot specify a PID, executable or alternate command.
+The holder owns grace timing, so API restart cannot reset it. The CLI's `mist
+shell` and later browser terminal use this same endpoint.
+
+## Versioning
+
+Additive changes (new fields, endpoints or enum values) don't change the
+version. Clients must ignore unknown fields and show unknown enum values as
+plain text. Breaking changes go to `/v2`.
diff --git a/docs/agents/architecture/decisions.md b/docs/agents/architecture/decisions.md
new file mode 100644
index 0000000..1f31dff
--- /dev/null
+++ b/docs/agents/architecture/decisions.md
@@ -0,0 +1,279 @@
+# Architecture decisions
+
+Each entry records the context, the decision, and what follows from it. To
+change a decision, add a new entry that supersedes the old one rather than
+editing it. The system overview is [ARCHITECTURE.md](../../../ARCHITECTURE.md).
+
+## D1. Mist has its own scheduler; Kubernetes only runs jobs
+
+**Context.** The previous platform made every job a container image with
+declared mounts, and that friction is the reason for the redesign. An existing
+HPC scheduler (Slurm) was considered. The team chose to build the scheduler as
+part of Mist, and the cluster is expected to grow beyond two nodes.
+
+**Decision.** Mist's scheduler owns the queue, policy and placement. A pluggable
+**executor** turns an admitted job into running processes. The first executor
+uses k3s; the Kubernetes scheduler never decides when or where a job runs.
+
+**Consequences.** Mist implements queueing, limits, priority and accounting.
+Kubernetes provides process supervision, device allocation (NVIDIA device plugin,
+Tenstorrent DRA), cgroup limits and node membership. If both systems made
+placement decisions, jobs could sit pending with no visible reason; pinning pods
+to the scheduler's chosen node prevents that.
+
+## D2. Executors sit behind one interface, and each node has exactly one executor
+
+**Decision.** Executors implement the interface in [executors.md](executors.md).
+The K3s executor ships first. A native node executor (systemd units and cgroups,
+no containers) follows behind the same interface. Each partition maps to one
+executor, and each node belongs to one executor, because two systems cannot both
+own a node's devices.
+
+**Consequences.** The scheduler and API never import Kubernetes types. A shared
+conformance test suite defines executor behaviour. Moving a node between
+executors means draining it first.
+
+## D3. Postgres is the source of truth
+
+**Context.** The previous platform kept queue and policy state in ConfigMaps and
+suspended Jobs.
+
+**Decision.** All Mist state lives in Postgres. Pods, Linux accounts and folders
+are derived and reconciled from it.
+
+**Consequences.** Restarting `mistd` or wiping execution objects loses nothing;
+the executor rebuilds its view by comparing the database with what is actually
+running. Usage history for fairshare and accounting has a natural home.
+
+## D4. One Linux UID per person
+
+**Context.** Creating a Linux user per job was considered, so that only jobs
+would get privileges.
+
+**Decision.** Each person has one stable UID, the same on every host. Jobs run
+as that UID. Privileges (devices, write access to the right folders) come from
+the job's cgroup, mounts and supplementary groups, not from a different user.
+
+**Consequences.** Files keep one owner across login, jobs and teammates. There
+is no ownership mess between jobs. UID and GID allocation is centralised in the
+Mist DB, in fixed ranges that don't collide with local system accounts.
+
+## D5. Each job is scoped to one charged team
+
+**Decision.** Every job names a charged team; by default it is the team whose
+folder the job was submitted from. The job is given only that team's group and
+that team's folder, plus shared read-only software.
+
+**Consequences.** Someone in two teams cannot accidentally mix IP-bound data
+between them inside a job. They can still copy files by hand, since they can
+read both teams; no technical control prevents a person from moving data they
+can read. A person's home is per-team for the same reason (D8).
+
+## D6. Login moves files and manages jobs; code runs only in jobs
+
+**Decision.** Researcher SSH sessions are restricted by `ForceCommand` to:
+SFTP/scp, `rsync`, a small set of built-in file viewing commands, and the `mist`
+CLI. Anything that executes user code is a job.
+
+**Consequences.** The login host (also a storage and compute node) is protected
+from stray compute. Internet downloads become jobs (D11). This is stricter than
+typical HPC login nodes, and that is deliberate on a cluster with no dedicated
+login hardware.
+
+## D7. Login and storage on the TT box; control plane on the NVIDIA box
+
+**Decision.**
+
+- **TT box:** the durable storage (`/mist`), the login gateway and the
+ provisioner.
+- **NVIDIA box:** the k3s server, `mistd`, Postgres and the portal.
+
+**Consequences.** Uploads land on local disk with no network hop. The TT box is
+the single point of failure for logins and file access. The control plane and
+the database sit on separate hardware from the bulk storage.
+
+## D8. A researcher's team folder is their home
+
+**Decision.** `/mist/teams///` is both `$HOME` and the default
+working directory for jobs charged to that team. It is team-readable (`2750`).
+`.cache`, `.config`, `.local` and `.ssh` inside it are private (`0700`): the
+provisioner creates them that way, and the job entry wrapper resets them on
+every job. A few large shareable caches are mapped to the team's `.cache/` by
+environment variables.
+
+**Consequences.** Every tool's cache and config lands somewhere sensible without
+per-tool mapping. Credentials stored in the usual hidden folders stay private.
+**Accepted risk:** a tool that writes a secret as a top-level file without
+restricting its permissions exposes it to teammates (never to other teams).
+Someone in two teams keeps two homes. Researchers need this explained, so it
+appears under "What researchers see" in [storage.md](storage.md).
+
+## D9. Durable storage plus local scratch; growth by adding volumes
+
+**Decision.** `/mist` is the only durable storage. Node-local scratch is
+temporary and wiped after each job. Durable capacity grows by adding **volumes**
+(separate NFS exports), and each team folder lives entirely on one volume. No
+distributed filesystem is used.
+
+**Consequences.** Researchers never have to work out which machine holds a file.
+Adding storage is a database record plus an export. Spare disk on compute nodes
+goes to backups first, then scratch, and only then to new volumes.
+
+## D10. The direct link runs at 10 GbE on existing network cards
+
+**Context.** The direct cable uses Intel I210 ports, which max out at 1 Gb/s
+(about 110 MB/s). Loading a 14 GB model took around two minutes.
+
+**Decision.** Move the link to the 10GBASE-T ports already installed: the
+Aquantia AQC107 (`ens6`) on the NVIDIA box and the Intel X710
+(`enp10s0f0np0`) on the TT box. Use Cat6a cable.
+
+**Consequences.** NFS reaches roughly 700–1000 MB/s, so the separate node-local
+model cache is dropped from the design. The second X710 port can take one more
+direct-connected node. A fourth node needs a 10 GbE switch.
+
+## D11. Downloads are ordinary jobs on a `cpu` partition
+
+**Decision.** The TT box's CPUs form a `cpu` partition, which sits next to the
+storage. `mist run -p cpu -- ` and `mist shell -p cpu` cover every download
+tool. There is no dedicated download service.
+
+**Consequences.** Any source works (Hugging Face, S3, Kaggle, `git lfs`, `wget`),
+using the researcher's own tokens. GPUs are never held idle during downloads.
+
+## D12. Job outputs go to `/mist`; scratch is opt-in
+
+**Decision.**
+
+- **Output.** Stdout and stderr stream to `mist-.out` in the submit folder.
+ Mist appends its own messages there too (node, cancellation, time limit,
+ out-of-memory).
+- **Job record.** The database keeps the job's metadata, a copy of the script,
+ and a tail of the log.
+- **Scratch.** `$TMPDIR` points at node-local scratch, which is deleted when the
+ job ends.
+- **Time limits.** At the limit the job receives SIGTERM, then SIGKILL after a
+ grace period (default 60 s).
+
+**Consequences.** Nothing a researcher writes by default is ever lost. The
+portal can show logs without reading team files.
+
+## D13. Accelerator device files stay world-accessible; isolation comes from cgroups
+
+**Context.** `/dev/nvidia*` and `/dev/tenstorrent/*` are mode `0666` on both
+hosts.
+
+**Decision.** Keep the host modes. Inside a job, the executor's device cgroup
+(device plugin, or `DeviceAllow` in the node executor) exposes only the
+allocated devices. Host-level access is prevented by D6 and by removing
+unmanaged GPU consumers (for example an `ollama` service).
+
+**Consequences.** This matches Slurm's model. If the device files were tightened
+to `0660`, containers could no longer open the devices they were given, because
+non-root jobs inherit the host permissions.
+
+## D14. Misty owns people and teams; Mist owns accounts
+
+**Decision.** Misty is authoritative for who exists, which teams exist, and who
+belongs to them. Mist caches that data and adds what is specific to Mist:
+usernames, UIDs and GIDs, SSH keys, limits and roles. Until misty's API is
+available, a stub backed by a file stands in for it.
+
+**Consequences.** Removing someone from a team in misty eventually removes their
+Mist access to that team. Mist never edits people or teams.
+
+## D15. Other teams' jobs are anonymised in queue views
+
+**Decision.** Queue and partition views show every job's state, resources and
+age, so researchers can see why they are waiting. The user, team, job name and
+paths are shown only for the viewer's own teams.
+
+**Consequences.** Job names and paths can't leak IP-bound project details
+between teams.
+
+## D16. One Go binary, plain Kubernetes objects
+
+**Decision.** `mistd` is one Go binary with subcommands (`api`, `scheduler`,
+`provision`). It talks to Kubernetes through `client-go` and creates plain Pods.
+It uses no CRDs, operators, `kubectl` shell-outs or Kubernetes Jobs.
+
+**Consequences.** Mist's own state machine stays in Mist's code and database,
+which keeps it easy to understand and to swap executors.
+
+## D17. Accelerators are whole devices; CPU and memory are capped per GPU
+
+**Decision.**
+
+- **Whole devices.** Jobs request whole GPUs or TT boards; two 1-GPU jobs share
+ the A4000 box side by side.
+- **Per-GPU caps.** Each GPU partition caps CPUs and memory per allocated GPU,
+ so one job cannot leave another GPU unusable.
+- **No GPU-less jobs on GPU nodes.** `gpu` partition jobs must request at least
+ one GPU.
+- **No GPU sharing.** Fractional GPUs (time-slicing) are not offered, because
+ the A4000 has no hardware partitioning and time-slicing gives no memory
+ isolation.
+
+**Consequences.** Placement only counts devices, CPUs and memory. Time-slicing
+could later be added as an opt-in for small debug work.
+
+## D18. Mist admits jobs; Kubernetes may allocate DRA claims
+
+**Supersedes:** D1's unconditional scheduler bypass. Mist alone owns queue order,
+policy, admission and the target node. GPU/CPU Pods use `nodeName`. TT Pods omit
+it and use an exact unique node selector so Kubernetes can allocate/reserve DRA
+claims on the chosen node. A claim stall is a bounded launch failure, never an
+invitation to move the job. Pin and test the installed DRA driver and board reset
+scope before M4. Source: [Kubernetes DRA pre-scheduled Pods](https://kubernetes.io/docs/concepts/resource-management/dynamic-resource-allocation/how-dra-works/).
+
+## D19. Personal work defaults to read-only for teammates
+
+**Clarifies D8 and supersedes conflicting implementation defaults.** Jobs and
+uploads use `0027`; ordinary personal files are `0640`, directories inherit
+setgid and exclude group write. Only shared/cache areas have group-write default
+ACLs. Private dot-directories remain `0700`. Owners can explicitly change modes;
+provisioning never recursively rewrites researcher content. Verify read/write
+behavior as another UID, not merely the mode of the top-level directory.
+
+## D20. Access revocation includes live sessions and queued work
+
+**Refines D6 and D14.** `mist-users` is permanent; disabled accounts never lose
+the marker enforcing restricted SSH. Directory snapshots and gateway leases have
+a maximum 5 min validity, with no stale-cache extension. Known removal invalidates
+API sessions, stops affected pending/launching/running jobs and kills gateway
+sessions before updating groups. Existing process credentials do not change when
+membership files change. A root local deny store and watchdog enforce expiry even
+if the API is down. Revocation completion is acknowledged, not assumed while a
+node is unreachable. Sources: [Linux credentials](https://man7.org/linux/man-pages/man7/credentials.7.html),
+[usermod password locking](https://man7.org/linux/man-pages/man8/usermod.8.html).
+
+## D21. Execution intent, process outcome and cleanup are separate durable facts
+
+**Refines D3 and D12.** Commit immutable admission/launch intent before invoking an
+executor; recover the same execution ID after ambiguous responses. Never rerun
+started work automatically. A finished process enters `CLEANING`; resources stay
+charged until node cleanup is proven. Lost nodes retain quarantined allocations.
+Store result/log tail before object deletion and retain receipts/tombstones.
+Postgres preserves control state, not running processes or user files: wiping
+Pods can lose execution and scratch. OOM/SIGKILL can prevent an output footer;
+the API reports authoritative status and incomplete output explicitly.
+
+## D22. Interactive jobs own a persistent terminal
+
+**Decision.** One holder owns one shell/PTY for the execution lifetime. Attach is
+transport to that PTY, never a new shell. One client at a time, bounded replay,
+10 min local detach grace, and unchanged walltime apply under both executors.
+API/relay restart may detach but cannot reset grace or shell state. Holder death
+fails the job. User code stays inside the one allocation/cgroup.
+
+## D23. Isolation must hold on the host as well as in the job spec
+
+**Refines D4, D5 and D13.** NetworkPolicy is paired with host filtering, including
+job traffic to its own node, explicit DNS exceptions and tested IPv6 behavior.
+Native execution explicitly replaces supplementary groups and drops credentials
+before user code; systemd `SupplementaryGroups=` alone only extends NSS groups.
+The same real-node conformance suite gates K3s and native execution. Adding nodes
+uses a switch unless a separate bridge/routing design has been verified; D10's
+second-port suggestion is not a complete network design. Sources:
+[Kubernetes NetworkPolicy](https://kubernetes.io/docs/concepts/services-networking/network-policies/),
+[systemd.exec](https://www.freedesktop.org/software/systemd/man/258/systemd.exec.html).
diff --git a/docs/agents/architecture/deployment-and-network.md b/docs/agents/architecture/deployment-and-network.md
new file mode 100644
index 0000000..56a6951
--- /dev/null
+++ b/docs/agents/architecture/deployment-and-network.md
@@ -0,0 +1,179 @@
+# Deployment and network
+
+Covers which host runs what, how hosts and users connect, where configuration
+and secrets live, admin access, and how nodes are added. Overview:
+[ARCHITECTURE.md](../../../ARCHITECTURE.md). Related decisions: D7, D10, D13.
+
+## Topology
+
+```mermaid
+flowchart LR
+ Users([Researchers + admins]) -->|Tailscale| TSN
+ Users -->|Tailscale| TST
+ subgraph NV["utmist-z1opa08 · 56 CPU · 157 GiB · 2× A4000"]
+ TSN["tailscale0
100.73.139.66"] -->|"portal HTTPS :443"| NVsvc["k3s server · mistd api + scheduler
Postgres · Nginx portal"]
+ NVgpu[gpu partition]
+ ens6["ens6 · AQC107 10GBASE-T
192.168.240.1"]
+ end
+ subgraph TT["utmist-tt · 32 CPU · 503 GiB · 4× n300"]
+ TST["tailscale0
100.95.175.37"] -->|"ssh :22"| TTsvc["k3s agent · NFS /mist · sshd + mist-gate
mist-relay · provisioner"]
+ TTpart[cpu + tt partitions]
+ x710["enp10s0f0np0 · X710 10GBASE-T
192.168.240.2"]
+ end
+ ens6 <-->|"Cat6a direct link, 10 GbE
k3s, NFS, agents to API"| x710
+ LAN[("Home LAN / internet
10.0.0.0/24")]
+ NV --- LAN
+ TT --- LAN
+```
+
+## Hosts
+
+| | `utmist-z1opa08` (NVIDIA) | `utmist-tt` (QuietBox) |
+|---|---|---|
+| Hardware | 56 CPUs, 157 GiB RAM, 2× RTX A4000 16 GB, 2× 447 GB SATA SSD | 32 CPUs, 503 GiB RAM, 4× n300 (8 chips), 3.6 TB NVMe |
+| OS | Ubuntu 22.04, kernel 6.8, cgroup v2 | same |
+| Roles | k3s server; `mistd api` + `mistd scheduler`; Postgres; portal (Nginx); `gpu` partition; backup target (M3) | k3s agent; `/mist` volume 1 + NFS server; login gateway (sshd, `mist-gate`, `mist-relay`); `mistd provision`; `cpu` and `tt` partitions |
+
+## Networks
+
+| Network | Addresses | Carries |
+|---|---|---|
+| **Direct link (10 GbE)** | NVIDIA `ens6` `192.168.240.1/24`, TT `enp10s0f0np0` `192.168.240.2/24` | k3s node traffic and Flannel, NFS, relay and provisioner → API, backups |
+| LAN | NVIDIA `10.0.0.175` (currently over Wi-Fi), TT `10.0.0.112` | Internet access only |
+| Tailscale | NVIDIA `100.73.139.66`, TT `100.95.175.37` | Users: portal (NVIDIA HTTPS :443), SSH login (TT :22); admin SSH |
+
+**Moving the direct link to 10 GbE (M1):**
+
+- **Cable:** Cat6a (or Cat6 under about 50 m) between the NVIDIA box's `ens6`
+ (Aquantia AQC107) and the TT box's `enp10s0f0np0` (Intel X710). Both are
+ 10GBASE-T (RJ45) ports.
+- **Addresses:** move `192.168.240.1/.2` from the 1 GbE I210 ports to those two
+ ports, using a `/24` so more nodes can join.
+- **MTU:** jumbo frames (9000) on both ends are optional, but help NFS.
+- **Check:** `ethtool ` shows `Speed: 10000Mb/s`. An `iperf3` run should
+ reach above 9 Gb/s.
+
+**Wired LAN.** The NVIDIA box should also reach the LAN over wired Ethernet
+rather than Wi-Fi; one of its I210 ports frees up once the direct link moves.
+
+**k3s** is installed fresh, with `--node-ip` and `--flannel-iface` on the direct
+link, so cluster traffic never uses the LAN or Wi-Fi.
+
+## Services
+
+| Service | Host | Runs as |
+|---|---|---|
+| k3s server and agent | both | systemd |
+| NVIDIA GPU Operator (device plugin + container toolkit) | NVIDIA | k3s |
+| Tenstorrent DRA driver (M4) | TT | k3s |
+| Postgres | NVIDIA | k3s StatefulSet, data on local SSD (never NFS) |
+| `mistd api`, `mistd scheduler` | NVIDIA | k3s Deployments (1 replica each), reachable on `192.168.240.1` for agents |
+| Portal (Nginx + static build) | NVIDIA | systemd, HTTPS on Tailscale (production), proxies `/api`; `:8088` development only |
+| NFS server | TT | systemd |
+| sshd + `mist-gate` | TT | systemd |
+| `mist-relay` | TT | systemd (root) |
+| `mistd provision` | TT (later every node) | systemd (root), includes gateway revocation/watchdog integration |
+| `mistd cleanup-report` | both compute nodes | root-owned node service; narrowly scoped cleanup evidence |
+| `mist-noded` (M5) | nodes moved to the node executor | systemd (root) |
+
+## Configuration and secrets
+
+- **Configuration in Git:** `deploy/` (manifests, systemd units, sshd and
+ exports snippets, host setup scripts).
+- **Settings in the database:** partitions, QoS, limits, volumes. Admins change
+ them through the API.
+- **Secrets never go in Git:**
+ - Postgres credentials, the relay token and per-host agent tokens.
+ - Kubernetes Secrets for in-cluster use; `/etc/mist/*.token` (0600 root) on
+ hosts.
+ - Rotate any secret that has been pasted into a chat or document.
+
+## Admin access
+
+- **Separate admin accounts** for each admin, with SSH keys only, `sudo`, and
+ not in `mist-users`. Researcher accounts never have `sudo` or `docker`.
+- **Password SSH disabled** on both hosts. The shared `utmist` and `utmist-tt`
+ passwords are retired once admin accounts exist.
+- **Clean up existing host state** after checking with whoever created it:
+ - leftover local accounts (`dev`, `terrymist`, `labadmin`, `labmember`)
+ - the `ollama` service and group
+ - Docker
+- **Device files stay `0666`** (D13). This is safe only because researcher
+ logins can't run code and unmanaged GPU consumers are removed.
+
+## Starting fresh
+
+Before M1 goes live:
+
+1. Stop and remove the previous platform's API, namespaces (`mist-team-*`),
+ ConfigMaps and Jobs.
+2. Keep the old storage folders untouched until a human confirms they can be
+ deleted.
+3. Reinstall k3s on the direct link.
+
+## Adding a node
+
+1. **Connect it to the direct-link network.** For three or more nodes, use a
+ 10 GbE switch with all nodes on `192.168.240.0/24`. A second direct cable is
+ not automatically the same L2 network: an alternative needs an explicitly
+ designed bridge or routed subnet and routing/firewall tests before use.
+2. **Install Ubuntu** with the drivers for its accelerators.
+3. **K3s executor:** join it as a k3s agent and install its device plugin. Node
+ executor: install `mist-noded` and `mistd provision`.
+4. **Register it in Mist:** add it to a partition (new or existing) and set its
+ reserved host share.
+5. **Run the executor conformance suite** on it, then mark it `up`.
+
+## Work by milestone
+
+- **M1:**
+ - 10 GbE link and wired LAN
+ - fresh k3s
+ - Postgres
+ - deploy `mistd` and the portal
+ - gateway host setup
+ - admin accounts and secret rotation
+ - remove the old platform
+ - job/host firewall enforcement, authorization lease watchdog and cleanup reporter
+ - basic monitoring: disk headroom, stale directory/provisioning generations,
+ failed cleanup, quarantined nodes, scheduler leadership and launch timeouts
+ - pin k3s/kernel/runtime/plugins and record tested feature support
+- **M3:** backup target, retention and restore exercises; richer monitoring UI.
+
+## Open questions
+
+- Monitoring and alerting stack.
+- Whether the NVIDIA box's `sdb` is free for scratch and backups.
+- A stable login hostname.
+
+## Job network boundary (M1)
+
+Deployment owns host firewall rules; Executors owns job namespace/NetworkPolicy
+configuration. Both must pass the same conformance checks before a node is up.
+
+- Tag job traffic by its network interface/source before NAT. Filter both host
+ INPUT and routed/FORWARD paths; a pod's own host is not implicitly trusted.
+ Deny new job connections to every host address (direct-link, LAN, Tailscale,
+ pod gateway), Kubernetes service/pod CIDRs, RFC1918, CGNAT and link-local ranges.
+ Include locally bound public addresses, if added later. Allow only approved
+ DNS resolvers on TCP/UDP 53 and public internet egress; allow established
+ response traffic. Deny unsolicited inbound traffic to jobs.
+- Host-origin NFS mounts, control-plane traffic, cleanup reporting and admin SSH
+ are separate trusted flows. Never open NFS from the job network to make a
+ mount work; mounts are performed by the host. Block job access to host runtime
+ sockets and Unix sockets through the filesystem too.
+- M1 job networking is IPv4-only; disable IPv6 in job namespaces/routing. Enable
+ it only after equivalent IPv6/link-local rules pass tests. Startup must fail
+ closed if policy installation fails; rule reload and k3s restart must not
+ create an allow-all interval. Pin/test the actual CNI and host firewall ordering.
+- Test from CPU jobs on the storage host and GPU jobs on the control host: own
+ host and peer SSH/NFS/API/Postgres/kubelet ports, service IP aliases, pod peers,
+ LAN/Tailscale peers, DNS, internet HTTPS, and IPv6 bypass attempts. Also verify
+ legitimate host NFS mounts and admin access remain functional.
+
+Use authenticated TLS for relay/provisioner/cleanup traffic on the direct link;
+per-host bearer tokens authorize operations but do not encrypt them. Expose the
+production portal through a stable Tailscale HTTPS name with a trusted certificate.
+Record the selected hostname/certificate provisioning in deployment configuration.
+Restrict internal API listeners to trusted hosts; user-facing proxy routes never
+expose `/internal`. A clean restart must restore policies before admitting jobs.
diff --git a/docs/agents/architecture/executors.md b/docs/agents/architecture/executors.md
new file mode 100644
index 0000000..1e13f03
--- /dev/null
+++ b/docs/agents/architecture/executors.md
@@ -0,0 +1,469 @@
+# Executors
+
+An executor turns an admitted job into processes on a node, streams their I/O,
+and reports what happened. The scheduler decides; the executor carries the
+decision out. Overview: [ARCHITECTURE.md](../../../ARCHITECTURE.md). Related
+decisions: D1, D2, D13, D16, D17.
+
+There are two executors:
+
+- the **K3s executor**, used from M1
+- the **node executor** (M5), which runs jobs as systemd units without
+ containers
+
+Both implement the same interface and pass the same conformance tests. Each node
+is managed by exactly one executor.
+
+## How it fits
+
+Both executors take the same `LaunchSpec` and end at the same `mist-job-entry`.
+Only the middle differs:
+
+```mermaid
+flowchart LR
+ S[Scheduler] -->|LaunchSpec| I{{Executor interface}}
+ I --> K["K3s executor (M1)"]
+ I -.-> NE["Node executor (M5)"]
+
+ K -->|client-go| KAPI[k3s API server]
+ KAPI --> Pod["Pod job-ID
uid, team gid, chosen node,
requests = limits, no token"]
+ KAPI --> CM["ConfigMap job-ID
script + job.json"]
+ Pod --> Vol["Volumes: team folder, software,
scratch, /dev/shm, mist-nss"]
+ Pod --> Dev["Device plugin / DRA
allocated GPUs or boards"]
+ Pod --> Entry[mist-job-entry]
+
+ NE -.->|gRPC + mTLS| Agent[mist-noded]
+ Agent -.->|D-Bus| Unit["systemd unit mist-job-ID
User, DeviceAllow, MemoryMax, BindPaths"]
+ Unit -.-> Entry
+
+ Entry --> Work[Script, command or shell]
+ K -->|"Started, Exited, Lost"| S
+```
+
+## Interface
+
+```go
+package executor
+
+type Executor interface {
+ Name() string
+
+ // Nodes reports schedulable capacity and health for nodes this executor manages.
+ Nodes(ctx context.Context) ([]Node, error)
+
+ // Launch ensures the one execution exists on spec.Node. Same ID/spec is
+ // repeat-safe; never recreate an execution with a recorded native object
+ // UID or evidence that it has started/ended after that object disappears.
+ Launch(ctx context.Context, spec LaunchSpec) error
+
+ // Terminate sends SIGTERM, then SIGKILL after grace. Idempotent.
+ Terminate(ctx context.Context, jobID int64, grace time.Duration, reason string) error
+
+ // Attach connects a terminal to a running interactive job; returns when the session ends.
+ Attach(ctx context.Context, jobID int64, term Terminal) error
+
+ // Logs streams the job's live output (stdout+stderr as written to the .out file).
+ Logs(ctx context.Context, jobID int64, follow bool) (io.ReadCloser, error)
+
+ // List returns every execution this executor knows about, for reconciliation.
+ List(ctx context.Context) ([]Execution, error)
+
+ // Events is an at-least-once notification stream. Persist and dedupe by
+ // EventID; List/observation receipts recover missed notifications.
+ Events() <-chan Event
+}
+
+type LaunchSpec struct {
+ JobID int64
+ ExecutionID string // immutable UUID; one execution per job
+ SpecHash string // immutable normalized spec digest
+ AuthGeneration int64 // authorization used at admission
+ LaunchDeadline time.Time // admitted_at + launch timeout
+ Kind Kind // Batch | Command | Interactive
+ Node string
+ User User // Name, UID, PersonalGID
+ TeamSlug string
+ TeamGID int
+ ExtraGIDs []int // mist-users
+ Resources Resources // CPUs, MemoryBytes, GPUs, TTBoards, ScratchBytes
+ DeviceIDs []string // optional; set when the scheduler assigns specific devices
+ Mounts []Mount // resolved team folder + software (volume, source, target, readOnly)
+ Workdir string
+ OutputPath string
+ Script string // Batch
+ Args []string // Batch arguments (never shell-concatenated)
+ Argv []string // Command
+ Env map[string]string // frozen base; work-start/deadline fields derived at startup
+ Runtime string // partition runtime image / host profile, pinned
+ TimeLimit time.Duration // executor sets a backstop slightly beyond this
+ SignalGrace time.Duration
+}
+
+type Event struct {
+ EventID string // stable across redelivery
+ JobID int64
+ ExecutionID string
+ Kind EventKind // Started | Finished | Exited | LaunchFailed | Lost
+ Node string
+ ExitCode *int
+ Reason ExitReason // Normal | OutOfMemory | ScratchExceeded | Deadline | Signaled | ...
+ Detail string // human-readable, shown to users
+ At time.Time
+}
+
+type Node struct {
+ Name string
+ Partitions []string
+ Capacity Resources
+ Devices []Device // optional per-device inventory
+ Healthy bool
+ Detail string
+}
+```
+
+The scheduler and API depend only on this package. Kubernetes and systemd types
+never leak outside their executor packages. Define the supporting wire types in
+this package before component implementations:
+
+- `Execution`: JobID, ExecutionID, SpecHash, Node, observed phase, native object
+ UID, started/finished timestamps, exit result, cleanup status and evidence.
+- `Resources`: CPU millicores, memory/scratch bytes, whole GPU/board counts,
+ hugepage bytes (M4). `Mount`: volume ID, source, absolute target, read-only.
+- `Terminal`: byte input/output, resize and allowed signal controls, plus a
+ cancellation context. Stream cancellation detaches; it does not cancel a job.
+- `List` must distinguish unavailable/incomplete observations from authoritative
+ absence. Failed calls are errors, never an empty successful list. Merge live
+ objects with durable execution receipts; a watch stream is not a history store.
+
+The executor persists create intent, object UID, start/finish observations and
+cleanup receipts in executor-owned database tables through its package. Persist
+finish metadata and bounded log tail before deleting execution objects. Scheduler
+state changes and event consumption are transactional; events carry stable IDs.
+Only the scheduler issues Launch/Terminate; the API uses read/log/attach methods.
+A mismatched SpecHash for an existing ExecutionID is a hard conflict. M1 never
+reruns an execution that has started, even if the process ended before a watch
+observed it. A recorded native object UID also forbids recreating a missing Pod:
+it may have run before deletion without a start notification. Without evidence
+that an ambiguous creation never ran, report loss/quarantine rather than replay.
+Durable tombstones prevent ID reuse after cleanup.
+
+## Guarantees every executor must provide
+
+These guarantees are tested by a shared **conformance suite**
+(`internal/executor/conformance`) that runs against each implementation on real
+nodes.
+
+1. **Identity.** Processes run as `User.UID`. The primary group is `TeamGID`.
+ Supplementary groups may contain only `ExtraGIDs`, `PersonalGID` and a
+ redundant `TeamGID` (some runtimes include the primary group there). No other
+ team/system groups are allowed. There is no privilege escalation or capability.
+2. **Filesystem view.**
+ - **Visible:** only the given mounts, at `/mist/teams/` and
+ `/mist/software` (read-only), plus `/scratch`. Other teams' folders are
+ absent, not merely denied.
+ - **Writable:** only the team folder (as permissions allow), `/scratch`,
+ `/tmp` and memory-backed `/dev/shm`. `/tmp` lives inside scratch.
+3. **Devices.** Only allocated GPUs/TT boards can be opened. CUDA presents the
+ allocated GPU set with logical indices from 0; host device filenames need not
+ be renumbered. Do not hard-code CUDA_VISIBLE_DEVICES=0 for every physical GPU;
+ verify the plugin/runtime's UUID or index mapping for both A4000s. Native
+ execution uses allocated physical IDs/UUIDs; CUDA remaps that visible set.
+4. **Limits.** CPU and memory limits apply, swap is off, and scratch is capped.
+ Hitting the memory limit gives `OutOfMemory`; detected scratch excess gives
+ `ScratchExceeded`. K3s ephemeral storage limits are eviction thresholds, not
+ synchronous write quotas. Reserve disk headroom and reject admissions under
+ disk pressure. Include `/dev/shm` in the memory budget.
+5. **Time limits.** `Terminate` delivers SIGTERM to the job's processes, then
+ SIGKILL after the grace period. A backstop kills the job even if the
+ scheduler is down.
+6. **Network.**
+ - **Allowed:** outbound internet and DNS.
+ - **Blocked:** the LAN, the direct link, the Tailscale range, cluster
+ services and the Kubernetes API.
+ - **Inbound:** none.
+7. **Output.** Output goes to `OutputPath` through `mist-job-entry` (below), and
+ live output is also available from `Logs`.
+8. **Cleanup.** When the job ends, all of its processes are gone, scratch is
+ deleted and devices are free. `Finished` reports process exit; `Exited`
+ acknowledges completed cleanup. Allocation release requires `Exited` or an
+ equivalent durable cleanup receipt. Failed cleanup keeps capacity reserved;
+ node loss quarantines it until reconciliation proves cleanup.
+9. **Repeat-safe and recoverable.** Repeating `Launch` or `Terminate` is safe.
+ After a restart, `List` reports what is actually running.
+
+## `mist-job-entry`
+
+A small static Go binary that is the first process of every job, under any
+executor. It is the single place that implements the job contract in
+[runtime-environment.md](runtime-environment.md):
+
+1. **Set up:**
+ - set `umask 0027`
+ - create, or reset to `0700`, `$HOME/{.cache,.config,.local,.ssh}`
+ - create the output file and write a header (job ID, node, resources, start
+ time)
+2. **Run the work:**
+ - **Batch:** `bash
-