Simplify team storage and use direct Ethernet for NFS

README.md

@@ -7,6 +7,14 @@ collect persistent results through the same website or CLI.
 **Private website:** [http://100.73.139.66:8088](http://100.73.139.66:8088), accessible through authorized
 Tailscale connectivity. A Mist account and team membership are also required.
 
+## Storage update — October 21
+
+New submissions use ordinary team dataset/result folders on QuietBox, accessed
+through the direct Ethernet link. Storage allowances are advisory; team access
+is enforced. See [current storage operations](deploy/k3s/storage/README.md) for
+layout, setup, API semantics, verification and rollback. The earlier operating
+snapshots describe the retired image-backed layout.
+
 ## Current installation
 
 

deploy/k3s/evidence/storage-2026-10-21/README.md

@@ -0,0 +1,57 @@
+# Shared-folder storage release verification
+
+Verified on the real two-machine installation October 21, 2026 (America/New_York).
+Source branch: `feat/simple-storage-ethernet-20261021`, based on main `ba2bb86`.
+
+## Evidence
+
+- [Validation summary](validation.json): source checks, small Ethernet sample,
+  authentication prerequisites and operating boundaries.
+- [Live API/training results](results.json): eight checks including regular-member
+  access, upload checksum/download, ZIP traversal rejection, team isolation,
+  active-dataset deletion protection, both accelerators and sampled usage.
+- [Actual workload mounts](workload-volumes.json): direct Ethernet NFS exports,
+  selected read-only inputs, scoped outputs and disabled service-account tokens.
+- [Deployed API artifact](deployed-api.json): image identity and binary checksum,
+  matched to the built candidate.
+- [QuietBox audit](host-storage.json): ordinary ext4 directories, zero image files
+  in the new root, service state, shared capacity and persistent Ethernet profile.
+- [Browser checks](browser-results.json), [dataset screen](datasets-browser.png),
+  [job screen](jobs-browser.png): real regular-member login, advisory display,
+  an actual browser upload and visible completed jobs; no JavaScript errors.
+- Training logs: [NVIDIA](nvidia.log), [Tenstorrent](tenstorrent.log).
+
+Training uploaded **255,854 bytes**, requested **one NVIDIA GPU** and **one n300
+board (two chips)**, read `/inputs/regression.json`, trained actual accelerator
+operations and saved/reloaded/downloaded checkpoints from `/outputs`. Loss and
+held-out validation checks passed. The browser added an 8-byte CSV afterward.
+
+All **43 API + 6 CLI Go tests** pass with race detection, alongside vet,
+**16 React tests**, TypeScript, lint/build and **6 Python folder tests**. The CLI
+entry package has no test files; no actual test was skipped. Vite/launch-editor
+were patched within the existing major version and npm reports zero vulnerabilities.
+
+The host is now configured with the new API/web release and local folder service.
+The separate old auth deployment remains scaled to zero for rollback. Existing
+auth credentials and deactivated-account state were preserved. No physical disk
+was formatted, old storage data deleted, large quota allocated, or bulk upload run.
+
+## Failures resolved during verification
+
+- Go-auth main previously accepted user-ID cookies, could not verify existing
+  Better Auth password hashes, omitted an old schema's required `emailVerified`
+  value, and did not honor old deactivated accounts. Targeted fixes and regression
+  tests were required to deploy latest main safely.
+- The first dynamic team bindings retained the old export prefix. Verification
+  caught the mount failure; the new test jobs were cancelled, only the new wrong
+  bindings were replaced, and the correct Ethernet/folder exports were tested.
+  The successful results linked above are from the final rerun.
+- NFS 4.2 reused an existing transport to the old LAN alias. New bindings use a
+  separate NFS 4.1 connection, verified on the Ethernet path.
+
+Private credentials, sessions, database backups and rollback manifests are kept
+outside Git in `/home/utmist/mist-storage-validation-20261021`. The reusable
+[live verifier](../../verify_storage_folders.py) creates its own test account and
+workspaces; never publish its `member-private.json` file.
+
+Operations and rollback: [storage guide](../../storage/README.md).

deploy/k3s/evidence/storage-2026-10-21/browser-results.json

@@ -0,0 +1,9 @@
+{
+  "checks": [
+    "regular_member_login_and_advisory_storage_display",
+    "real_browser_upload_with_verified_checksum",
+    "both_real_training_jobs_visible_in_member_history",
+    "no_browser_javascript_errors"
+  ],
+  "browser_errors": []
+}

deploy/k3s/evidence/storage-2026-10-21/deployed-api.json

@@ -0,0 +1,6 @@
+{
+  "image": "mist-api:folders-20261021",
+  "image_id": "sha256:350ced027c45a35e388f4ec5ef34f818074e856ff100c5e12c8f058fbbe8a9ef",
+  "binary_sha256": "ef980d2e624a3f015a3f45d87f49563861475e6c56aa8f808fb85f76990e3204",
+  "node": "utmist-z1opa08"
+}

deploy/k3s/evidence/storage-2026-10-21/host-storage.json

@@ -0,0 +1,11 @@
+{
+  "root": "/srv/mist-data",
+  "filesystem": "/dev/nvme0n1p2 ext4",
+  "image_files_in_new_root": 0,
+  "logical_regular_file_bytes": 535003,
+  "capacity_bytes": 3936290357248,
+  "available_bytes": 2010709856256,
+  "storage_service": "active",
+  "ethernet_speed_mbps": "1000",
+  "ethernet_profile_autoconnect": "yes"
+}

deploy/k3s/evidence/storage-2026-10-21/nvidia.log

@@ -0,0 +1,10 @@
+CUDA devices=1; torch=2.5.1+cu124
+Training on cuda:0: NVIDIA RTX A4000
+DATASET_LOADED file=/inputs/regression.json
+cuda:0 step=0 mse=0.32162201
+cuda:0 step=20 mse=0.00133983
+cuda:0 step=40 mse=0.00006271
+cuda:0 step=60 mse=0.00000431
+cuda:0 step=80 mse=0.00000035
+{"backend": "nvidia", "device_id": 0, "initial_mse": 0.32162201404571533, "final_mse": 3.100358014762605e-08, "validation_mse": 9.109567145060282e-08, "max_weight_change": 0.37507763504981995, "checkpoint": "/outputs/nvidia-0.pt"}
+TRAINING_PASSED backend=nvidia devices=1

deploy/k3s/evidence/storage-2026-10-21/results.json

@@ -0,0 +1,98 @@
+{
+  "checks": [
+    "regular_member_login_and_shared_folder_readiness",
+    "membership_and_common_write_permissions_enforced",
+    "unsafe_archive_rejected",
+    "small_upload_checksum_download_and_cross_team_isolation",
+    "active_dataset_deletion_blocked",
+    "nvidia_real_training_and_output_downloads",
+    "tenstorrent_real_training_and_output_downloads",
+    "host_sampled_dataset_and_model_usage"
+  ],
+  "jobs": [
+    {
+      "id": "mist-b5e74aec693e5111ae262449",
+      "backend": "nvidia",
+      "metrics": [
+        {
+          "backend": "nvidia",
+          "device_id": 0,
+          "initial_mse": 0.32162201404571533,
+          "final_mse": 3.100358014762605e-08,
+          "validation_mse": 9.109567145060282e-08,
+          "max_weight_change": 0.37507763504981995,
+          "checkpoint": "/outputs/nvidia-0.pt",
+          "checkpoint_reload_mse": 9.109567145060282e-08,
+          "device_name": "NVIDIA RTX A4000",
+          "started_at": "2026-10-22T00:28:43.814031+00:00",
+          "finished_at": "2026-10-22T00:28:44.148257+00:00",
+          "device_nodes": [
+            "/dev/nvidia1"
+          ]
+        }
+      ],
+      "node": "utmist-z1opa08",
+      "status": "Success"
+    },
+    {
+      "id": "mist-57208c18e561c8cef36e6697",
+      "backend": "tenstorrent",
+      "metrics": [
+        {
+          "backend": "tenstorrent",
+          "device_id": 0,
+          "initial_mse": 0.32162201404571533,
+          "final_mse": 5.1175324188079685e-05,
+          "validation_mse": 0.00011455668573034927,
+          "max_weight_change": 0.369140625,
+          "checkpoint": "/outputs/tenstorrent-0.pt",
+          "checkpoint_reload_mse": 0.00011455668573034927,
+          "started_at": "2026-10-22T00:28:49.539632+00:00",
+          "finished_at": "2026-10-22T00:28:59.859064+00:00",
+          "device_nodes": [
+            "/dev/tenstorrent/1"
+          ]
+        },
+        {
+          "backend": "tenstorrent",
+          "device_id": 1,
+          "initial_mse": 0.32162201404571533,
+          "final_mse": 5.1175324188079685e-05,
+          "validation_mse": 0.00011455668573034927,
+          "max_weight_change": 0.369140625,
+          "checkpoint": "/outputs/tenstorrent-1.pt",
+          "checkpoint_reload_mse": 0.00011455668573034927,
+          "started_at": "2026-10-22T00:29:11.964071+00:00",
+          "finished_at": "2026-10-22T00:29:15.617335+00:00",
+          "device_nodes": [
+            "/dev/tenstorrent/1"
+          ]
+        }
+      ],
+      "node": "utmist-tt",
+      "status": "Success"
+    }
+  ],
+  "bulk_testing": false,
+  "dataset_bytes": 255854,
+  "storage": {
+    "available_bytes": 2010710540288,
+    "capacity_bytes": 3936291389440,
+    "dataset_allocation_gib": 1,
+    "dataset_used_bytes": 256360,
+    "enabled": true,
+    "max_dataset_bytes": 1125899906842624,
+    "model_allocation_gib": 1,
+    "model_used_bytes": 6065,
+    "storage_limits_enforced": false,
+    "storage_low": false,
+    "storage_mode": "shared-folders",
+    "upload_limit_bytes": 2009636798464,
+    "usage_observed_at": "2026-10-22T00:28:54Z",
+    "usage_stale": false
+  },
+  "teams": [
+    "team-a91af070ba2a6bdc",
+    "team-71d2f7b638c597e1"
+  ]
+}

deploy/k3s/evidence/storage-2026-10-21/tenstorrent.log

@@ -0,0 +1,58 @@
+2026-10-22 00:28:44.905 | DEBUG    | ttnn:<module>:78 - Initial ttnn.CONFIG:
+Config{cache_path=/root/.cache/ttnn,model_cache_path=/root/.cache/ttnn/models,tmp_dir=/tmp/ttnn,enable_model_cache=false,enable_fast_runtime_mode=true,validate_program_args=false,throw_exception_on_fallback=false,enable_logging=false,enable_graph_report=false,enable_graph_python_stack_traces=false,enable_detailed_buffer_report=false,enable_detailed_tensor_report=false,enable_comparison_mode=false,comparison_mode_should_raise_exception=false,comparison_mode_pcc=0.9999,root_report_path=generated/ttnn/reports,report_name=std::nullopt,std::nullopt}
+2026-10-22 00:28:45.104 | info     |          Device | Opening user mode device driver (tt_cluster.cpp:228)
+2026-10-22 00:28:45.104 | info     |             UMD | Cluster constructor started. (cluster.cpp:389)
+2026-10-22 00:28:45.104 | info     |             UMD | Creating TopologyDiscovery for architecture: wormhole_b0 (topology_discovery.cpp:96)
+2026-10-22 00:28:45.104 | info     |             UMD | Starting topology discovery. (topology_discovery.cpp:115)
+2026-10-22 00:28:45.106 | info     |             UMD | Established firmware bundle version: 19.4.2 (topology_discovery.cpp:575)
+2026-10-22 00:28:45.114 | info     |             UMD | Completed topology discovery. (topology_discovery.cpp:119)
+2026-10-22 00:28:45.151 | warning  |             UMD | hwloc_set_area_membind(): failed for physical_device_id: 1 on NodeSet: [0] with errno: Operation not permitted (pid: 1 tid: 137514765571904) (cpuset_lib.cpp:440)
+2026-10-22 00:28:45.151 | warning  |             UMD | ---- ttSiliconDevice::init_hugepage: bind_area_to_memory_nodeset() failed (physical_device_id: 1 ch: 0). Hugepage allocation is not on NumaNode matching TT Device. Side-Effect is decreased Device->Host perf (Issue #893). (silicon_sysmem_manager.cpp:271)
+2026-10-22 00:28:45.151 | warning  |             UMD | hwloc_set_area_membind(): failed for physical_device_id: 1 on NodeSet: [0] with errno: Operation not permitted (pid: 1 tid: 137514765571904) (cpuset_lib.cpp:440)
+2026-10-22 00:28:45.151 | warning  |             UMD | ---- ttSiliconDevice::init_hugepage: bind_area_to_memory_nodeset() failed (physical_device_id: 1 ch: 1). Hugepage allocation is not on NumaNode matching TT Device. Side-Effect is decreased Device->Host perf (Issue #893). (silicon_sysmem_manager.cpp:271)
+2026-10-22 00:28:45.154 | info     |             UMD | Opening local chip ids/PCIe ids: {0}/[1] and remote chip ids {1} (cluster.cpp:173)
+2026-10-22 00:28:45.154 | info     |             UMD | IOMMU: disabled (cluster.cpp:147)
+2026-10-22 00:28:45.154 | info     |             UMD | KMD version: 2.5.0 (cluster.cpp:150)
+2026-10-22 00:28:45.154 | info     |             UMD | Cluster constructor completed. (cluster.cpp:648)
+2026-10-22 00:28:45.155 | info     |             UMD | Starting devices in cluster (cluster.cpp:1188)
+2026-10-22 00:28:45.158 | info     |             UMD | Starting devices in cluster completed. (cluster.cpp:1196)
+Tenstorrent devices=2; torch used for data/metrics=2.11.0+cpu
+Opening Tenstorrent device 0
+2026-10-22 00:28:45.209 | info     |     Distributed | Using auto discovery to generate mesh graph. (metal_env.cpp:437)
+2026-10-22 00:28:45.209 | info     |     Distributed | Constructing control plane using auto-discovery (no mesh graph descriptor). (metal_env.cpp:499)
+2026-10-22 00:28:45.210 | info     |          Fabric | Logical multi-mesh adjacency: intermesh degree histogram {0:1}; intra-mesh degree histograms mesh0 {1:2} (topology_mapper_utils.cpp:3329)
+2026-10-22 00:28:45.210 | info     |          Fabric | Physical multi-mesh adjacency: intermesh degree histogram {0:1}; intra-mesh degree histograms mesh0 {1:2} (topology_mapper_utils.cpp:3337)
+DATASET_LOADED file=/inputs/regression.json
+tt:0 step=0 mse=0.18711002
+tt:0 step=20 mse=0.00110440
+tt:0 step=40 mse=0.00008407
+tt:0 step=60 mse=0.00005286
+tt:0 step=80 mse=0.00005125
+{"backend": "tenstorrent", "device_id": 0, "initial_mse": 0.32162201404571533, "final_mse": 5.1175324188079685e-05, "validation_mse": 0.00011455668573034927, "max_weight_change": 0.369140625, "checkpoint": "/outputs/tenstorrent-0.pt"}
+Opening Tenstorrent device 1
+2026-10-22 00:28:59.905 | info     |           Metal | Fabric config changed from FabricConfig::DISABLED to FabricConfig::FABRIC_1D, reinitializing control plane (metal_env.cpp:337)
+2026-10-22 00:28:59.905 | info     |     Distributed | Using auto discovery to generate mesh graph. (metal_env.cpp:437)
+2026-10-22 00:28:59.905 | info     |     Distributed | Constructing control plane using auto-discovery (no mesh graph descriptor). (metal_env.cpp:499)
+2026-10-22 00:28:59.906 | info     |          Fabric | Logical multi-mesh adjacency: intermesh degree histogram {0:1}; intra-mesh degree histograms mesh0 {1:2} (topology_mapper_utils.cpp:3329)
+2026-10-22 00:28:59.906 | info     |          Fabric | Physical multi-mesh adjacency: intermesh degree histogram {0:1}; intra-mesh degree histograms mesh0 {1:2} (topology_mapper_utils.cpp:3337)
+2026-10-22 00:28:59.907 | info     |           Metal | Enabling FabricConfig::FABRIC_1D only for dispatch. If your workload requires fabric, please set the fabric config accordingly. (device_manager.cpp:291)
+2026-10-22 00:28:59.907 | info     |           Metal | Dispatch on FabricConfig::FABRIC_1D with 1 Command Queues
+ (device_manager.cpp:299)
+2026-10-22 00:28:59.909 | info     |           Metal | Initializing Fabric (fabric_firmware_initializer.cpp:296)
+2026-10-22 00:29:01.929 | info     |           Metal | Fabric initialized on 2 devices (fabric_firmware_initializer.cpp:444)
+2026-10-22 00:29:01.929 | info     |           Metal | Fabric Initialized with config FabricConfig::FABRIC_1D (fabric_firmware_initializer.cpp:313)
+DATASET_LOADED file=/inputs/regression.json
+tt:1 step=0 mse=0.18711002
+tt:1 step=20 mse=0.00110440
+tt:1 step=40 mse=0.00008407
+tt:1 step=60 mse=0.00005286
+tt:1 step=80 mse=0.00005125
+{"backend": "tenstorrent", "device_id": 1, "initial_mse": 0.32162201404571533, "final_mse": 5.1175324188079685e-05, "validation_mse": 0.00011455668573034927, "max_weight_change": 0.369140625, "checkpoint": "/outputs/tenstorrent-1.pt"}
+TRAINING_PASSED backend=tenstorrent devices=2
+2026-10-22 00:29:16.207 | info     |    BuildKernels | JIT cache stats: 0/58 hits (0.0%) [0 cached, 10 build-once dedup, 0 merged artifacts, 0 merged genfiles] (build_cache_telemetry.cpp:207)
+2026-10-22 00:29:16.207 | info     |    BuildKernels | JIT telemetry: 2 registered TelemetryTokens (build_cache_telemetry.cpp:230)
+2026-10-22 00:29:16.212 | info     |          Device | Closing user mode device drivers (tt_cluster.cpp:513)
+2026-10-22 00:29:16.212 | info     |             UMD | Closing devices in cluster (cluster.cpp:1201)
+2026-10-22 00:29:16.220 | info     |             UMD | Closing devices in cluster completed. (cluster.cpp:1210)
+2026-10-22 00:29:16.220 | info     |             UMD | Cluster destructor started. (cluster.cpp:916)
+2026-10-22 00:29:16.220 | info     |             UMD | Cluster destructor completed. (cluster.cpp:919)

deploy/k3s/evidence/storage-2026-10-21/validation.json

@@ -0,0 +1,40 @@
+{
+  "base_main": "ba2bb86",
+  "branch": "feat/simple-storage-ethernet-20261021",
+  "go_api_tests": 43,
+  "go_cli_tests": 6,
+  "go_race": "passed",
+  "go_vet": "passed",
+  "react_tests": 16,
+  "typescript": "passed",
+  "frontend_lint": "passed",
+  "frontend_build": "passed",
+  "python_folder_tests": 6,
+  "npm_audit_vulnerabilities": 0,
+  "fake_backend_contract": "passed",
+  "ethernet": {
+    "main": "192.168.240.1/30",
+    "main_interface": "enp6s0",
+    "quietbox": "192.168.240.2/30",
+    "quietbox_interface": "enp200s0",
+    "link_mbps": 1000,
+    "main_route": "192.168.240.2 dev enp6s0 src 192.168.240.1 uid 1000 \n    cache",
+    "small_probe_bytes": 16777216,
+    "small_probe_write_MBps": 43.0,
+    "small_probe_read_MBps": 101.0,
+    "performance_note": "Single small direct-I/O NFS sample; not a sustained workload benchmark."
+  },
+  "auth_prerequisites": [
+    "Require real unexpired database sessions; remove identity-cookie fallbacks",
+    "Preserve Better Auth scrypt passwords, NOT NULL schema and deactivated accounts",
+    "Existing administrator login verified against isolated database copy and live portal"
+  ],
+  "old_storage": "Retained unchanged for rollback; no migration",
+  "large_storage_testing": false,
+  "new_storage_mode": "ordinary directories with advisory team allowances",
+  "remaining_boundaries": [
+    "Single storage host and shared physical capacity",
+    "No equal-space guarantee or per-team hard quota",
+    "No automatic backup, local dataset cache, or resumable upload"
+  ]
+}