diff --git a/README.md b/README.md index b34c67a..aec78c8 100644 --- a/README.md +++ b/README.md @@ -7,6 +7,14 @@ collect persistent results through the same website or CLI. **Private website:** [http://100.73.139.66:8088](http://100.73.139.66:8088), accessible through authorized Tailscale connectivity. A Mist account and team membership are also required. +## Storage update — October 21 + +New submissions use ordinary team dataset/result folders on QuietBox, accessed +through the direct Ethernet link. Storage allowances are advisory; team access +is enforced. See [current storage operations](deploy/k3s/storage/README.md) for +layout, setup, API semantics, verification and rollback. The earlier operating +snapshots describe the retired image-backed layout. + ## Current installation diff --git a/deploy/k3s/evidence/storage-2026-10-21/README.md b/deploy/k3s/evidence/storage-2026-10-21/README.md new file mode 100644 index 0000000..4868077 --- /dev/null +++ b/deploy/k3s/evidence/storage-2026-10-21/README.md @@ -0,0 +1,57 @@ +# Shared-folder storage release verification + +Verified on the real two-machine installation October 21, 2026 (America/New_York). +Source branch: `feat/simple-storage-ethernet-20261021`, based on main `ba2bb86`. + +## Evidence + +- [Validation summary](validation.json): source checks, small Ethernet sample, + authentication prerequisites and operating boundaries. +- [Live API/training results](results.json): eight checks including regular-member + access, upload checksum/download, ZIP traversal rejection, team isolation, + active-dataset deletion protection, both accelerators and sampled usage. +- [Actual workload mounts](workload-volumes.json): direct Ethernet NFS exports, + selected read-only inputs, scoped outputs and disabled service-account tokens. +- [Deployed API artifact](deployed-api.json): image identity and binary checksum, + matched to the built candidate. +- [QuietBox audit](host-storage.json): ordinary ext4 directories, zero image files + in the new root, service state, shared capacity and persistent Ethernet profile. +- [Browser checks](browser-results.json), [dataset screen](datasets-browser.png), + [job screen](jobs-browser.png): real regular-member login, advisory display, + an actual browser upload and visible completed jobs; no JavaScript errors. +- Training logs: [NVIDIA](nvidia.log), [Tenstorrent](tenstorrent.log). + +Training uploaded **255,854 bytes**, requested **one NVIDIA GPU** and **one n300 +board (two chips)**, read `/inputs/regression.json`, trained actual accelerator +operations and saved/reloaded/downloaded checkpoints from `/outputs`. Loss and +held-out validation checks passed. The browser added an 8-byte CSV afterward. + +All **43 API + 6 CLI Go tests** pass with race detection, alongside vet, +**16 React tests**, TypeScript, lint/build and **6 Python folder tests**. The CLI +entry package has no test files; no actual test was skipped. Vite/launch-editor +were patched within the existing major version and npm reports zero vulnerabilities. + +The host is now configured with the new API/web release and local folder service. +The separate old auth deployment remains scaled to zero for rollback. Existing +auth credentials and deactivated-account state were preserved. No physical disk +was formatted, old storage data deleted, large quota allocated, or bulk upload run. + +## Failures resolved during verification + +- Go-auth main previously accepted user-ID cookies, could not verify existing + Better Auth password hashes, omitted an old schema's required `emailVerified` + value, and did not honor old deactivated accounts. Targeted fixes and regression + tests were required to deploy latest main safely. +- The first dynamic team bindings retained the old export prefix. Verification + caught the mount failure; the new test jobs were cancelled, only the new wrong + bindings were replaced, and the correct Ethernet/folder exports were tested. + The successful results linked above are from the final rerun. +- NFS 4.2 reused an existing transport to the old LAN alias. New bindings use a + separate NFS 4.1 connection, verified on the Ethernet path. + +Private credentials, sessions, database backups and rollback manifests are kept +outside Git in `/home/utmist/mist-storage-validation-20261021`. The reusable +[live verifier](../../verify_storage_folders.py) creates its own test account and +workspaces; never publish its `member-private.json` file. + +Operations and rollback: [storage guide](../../storage/README.md). diff --git a/deploy/k3s/evidence/storage-2026-10-21/browser-results.json b/deploy/k3s/evidence/storage-2026-10-21/browser-results.json new file mode 100644 index 0000000..5a600f2 --- /dev/null +++ b/deploy/k3s/evidence/storage-2026-10-21/browser-results.json @@ -0,0 +1,9 @@ +{ + "checks": [ + "regular_member_login_and_advisory_storage_display", + "real_browser_upload_with_verified_checksum", + "both_real_training_jobs_visible_in_member_history", + "no_browser_javascript_errors" + ], + "browser_errors": [] +} \ No newline at end of file diff --git a/deploy/k3s/evidence/storage-2026-10-21/datasets-browser.png b/deploy/k3s/evidence/storage-2026-10-21/datasets-browser.png new file mode 100644 index 0000000..45ace93 Binary files /dev/null and b/deploy/k3s/evidence/storage-2026-10-21/datasets-browser.png differ diff --git a/deploy/k3s/evidence/storage-2026-10-21/deployed-api.json b/deploy/k3s/evidence/storage-2026-10-21/deployed-api.json new file mode 100644 index 0000000..66853f1 --- /dev/null +++ b/deploy/k3s/evidence/storage-2026-10-21/deployed-api.json @@ -0,0 +1,6 @@ +{ + "image": "mist-api:folders-20261021", + "image_id": "sha256:350ced027c45a35e388f4ec5ef34f818074e856ff100c5e12c8f058fbbe8a9ef", + "binary_sha256": "ef980d2e624a3f015a3f45d87f49563861475e6c56aa8f808fb85f76990e3204", + "node": "utmist-z1opa08" +} diff --git a/deploy/k3s/evidence/storage-2026-10-21/host-storage.json b/deploy/k3s/evidence/storage-2026-10-21/host-storage.json new file mode 100644 index 0000000..69b3477 --- /dev/null +++ b/deploy/k3s/evidence/storage-2026-10-21/host-storage.json @@ -0,0 +1,11 @@ +{ + "root": "/srv/mist-data", + "filesystem": "/dev/nvme0n1p2 ext4", + "image_files_in_new_root": 0, + "logical_regular_file_bytes": 535003, + "capacity_bytes": 3936290357248, + "available_bytes": 2010709856256, + "storage_service": "active", + "ethernet_speed_mbps": "1000", + "ethernet_profile_autoconnect": "yes" +} diff --git a/deploy/k3s/evidence/storage-2026-10-21/jobs-browser.png b/deploy/k3s/evidence/storage-2026-10-21/jobs-browser.png new file mode 100644 index 0000000..32aa044 Binary files /dev/null and b/deploy/k3s/evidence/storage-2026-10-21/jobs-browser.png differ diff --git a/deploy/k3s/evidence/storage-2026-10-21/nvidia.log b/deploy/k3s/evidence/storage-2026-10-21/nvidia.log new file mode 100644 index 0000000..e447011 --- /dev/null +++ b/deploy/k3s/evidence/storage-2026-10-21/nvidia.log @@ -0,0 +1,10 @@ +CUDA devices=1; torch=2.5.1+cu124 +Training on cuda:0: NVIDIA RTX A4000 +DATASET_LOADED file=/inputs/regression.json +cuda:0 step=0 mse=0.32162201 +cuda:0 step=20 mse=0.00133983 +cuda:0 step=40 mse=0.00006271 +cuda:0 step=60 mse=0.00000431 +cuda:0 step=80 mse=0.00000035 +{"backend": "nvidia", "device_id": 0, "initial_mse": 0.32162201404571533, "final_mse": 3.100358014762605e-08, "validation_mse": 9.109567145060282e-08, "max_weight_change": 0.37507763504981995, "checkpoint": "/outputs/nvidia-0.pt"} +TRAINING_PASSED backend=nvidia devices=1 diff --git a/deploy/k3s/evidence/storage-2026-10-21/results.json b/deploy/k3s/evidence/storage-2026-10-21/results.json new file mode 100644 index 0000000..0a2e84e --- /dev/null +++ b/deploy/k3s/evidence/storage-2026-10-21/results.json @@ -0,0 +1,98 @@ +{ + "checks": [ + "regular_member_login_and_shared_folder_readiness", + "membership_and_common_write_permissions_enforced", + "unsafe_archive_rejected", + "small_upload_checksum_download_and_cross_team_isolation", + "active_dataset_deletion_blocked", + "nvidia_real_training_and_output_downloads", + "tenstorrent_real_training_and_output_downloads", + "host_sampled_dataset_and_model_usage" + ], + "jobs": [ + { + "id": "mist-b5e74aec693e5111ae262449", + "backend": "nvidia", + "metrics": [ + { + "backend": "nvidia", + "device_id": 0, + "initial_mse": 0.32162201404571533, + "final_mse": 3.100358014762605e-08, + "validation_mse": 9.109567145060282e-08, + "max_weight_change": 0.37507763504981995, + "checkpoint": "/outputs/nvidia-0.pt", + "checkpoint_reload_mse": 9.109567145060282e-08, + "device_name": "NVIDIA RTX A4000", + "started_at": "2026-10-22T00:28:43.814031+00:00", + "finished_at": "2026-10-22T00:28:44.148257+00:00", + "device_nodes": [ + "/dev/nvidia1" + ] + } + ], + "node": "utmist-z1opa08", + "status": "Success" + }, + { + "id": "mist-57208c18e561c8cef36e6697", + "backend": "tenstorrent", + "metrics": [ + { + "backend": "tenstorrent", + "device_id": 0, + "initial_mse": 0.32162201404571533, + "final_mse": 5.1175324188079685e-05, + "validation_mse": 0.00011455668573034927, + "max_weight_change": 0.369140625, + "checkpoint": "/outputs/tenstorrent-0.pt", + "checkpoint_reload_mse": 0.00011455668573034927, + "started_at": "2026-10-22T00:28:49.539632+00:00", + "finished_at": "2026-10-22T00:28:59.859064+00:00", + "device_nodes": [ + "/dev/tenstorrent/1" + ] + }, + { + "backend": "tenstorrent", + "device_id": 1, + "initial_mse": 0.32162201404571533, + "final_mse": 5.1175324188079685e-05, + "validation_mse": 0.00011455668573034927, + "max_weight_change": 0.369140625, + "checkpoint": "/outputs/tenstorrent-1.pt", + "checkpoint_reload_mse": 0.00011455668573034927, + "started_at": "2026-10-22T00:29:11.964071+00:00", + "finished_at": "2026-10-22T00:29:15.617335+00:00", + "device_nodes": [ + "/dev/tenstorrent/1" + ] + } + ], + "node": "utmist-tt", + "status": "Success" + } + ], + "bulk_testing": false, + "dataset_bytes": 255854, + "storage": { + "available_bytes": 2010710540288, + "capacity_bytes": 3936291389440, + "dataset_allocation_gib": 1, + "dataset_used_bytes": 256360, + "enabled": true, + "max_dataset_bytes": 1125899906842624, + "model_allocation_gib": 1, + "model_used_bytes": 6065, + "storage_limits_enforced": false, + "storage_low": false, + "storage_mode": "shared-folders", + "upload_limit_bytes": 2009636798464, + "usage_observed_at": "2026-10-22T00:28:54Z", + "usage_stale": false + }, + "teams": [ + "team-a91af070ba2a6bdc", + "team-71d2f7b638c597e1" + ] +} diff --git a/deploy/k3s/evidence/storage-2026-10-21/tenstorrent.log b/deploy/k3s/evidence/storage-2026-10-21/tenstorrent.log new file mode 100644 index 0000000..989051e --- /dev/null +++ b/deploy/k3s/evidence/storage-2026-10-21/tenstorrent.log @@ -0,0 +1,58 @@ +2026-10-22 00:28:44.905 | DEBUG | ttnn::78 - Initial ttnn.CONFIG: +Config{cache_path=/root/.cache/ttnn,model_cache_path=/root/.cache/ttnn/models,tmp_dir=/tmp/ttnn,enable_model_cache=false,enable_fast_runtime_mode=true,validate_program_args=false,throw_exception_on_fallback=false,enable_logging=false,enable_graph_report=false,enable_graph_python_stack_traces=false,enable_detailed_buffer_report=false,enable_detailed_tensor_report=false,enable_comparison_mode=false,comparison_mode_should_raise_exception=false,comparison_mode_pcc=0.9999,root_report_path=generated/ttnn/reports,report_name=std::nullopt,std::nullopt} +2026-10-22 00:28:45.104 | info | Device | Opening user mode device driver (tt_cluster.cpp:228) +2026-10-22 00:28:45.104 | info | UMD | Cluster constructor started. (cluster.cpp:389) +2026-10-22 00:28:45.104 | info | UMD | Creating TopologyDiscovery for architecture: wormhole_b0 (topology_discovery.cpp:96) +2026-10-22 00:28:45.104 | info | UMD | Starting topology discovery. (topology_discovery.cpp:115) +2026-10-22 00:28:45.106 | info | UMD | Established firmware bundle version: 19.4.2 (topology_discovery.cpp:575) +2026-10-22 00:28:45.114 | info | UMD | Completed topology discovery. (topology_discovery.cpp:119) +2026-10-22 00:28:45.151 | warning | UMD | hwloc_set_area_membind(): failed for physical_device_id: 1 on NodeSet: [0] with errno: Operation not permitted (pid: 1 tid: 137514765571904) (cpuset_lib.cpp:440) +2026-10-22 00:28:45.151 | warning | UMD | ---- ttSiliconDevice::init_hugepage: bind_area_to_memory_nodeset() failed (physical_device_id: 1 ch: 0). Hugepage allocation is not on NumaNode matching TT Device. Side-Effect is decreased Device->Host perf (Issue #893). (silicon_sysmem_manager.cpp:271) +2026-10-22 00:28:45.151 | warning | UMD | hwloc_set_area_membind(): failed for physical_device_id: 1 on NodeSet: [0] with errno: Operation not permitted (pid: 1 tid: 137514765571904) (cpuset_lib.cpp:440) +2026-10-22 00:28:45.151 | warning | UMD | ---- ttSiliconDevice::init_hugepage: bind_area_to_memory_nodeset() failed (physical_device_id: 1 ch: 1). Hugepage allocation is not on NumaNode matching TT Device. Side-Effect is decreased Device->Host perf (Issue #893). (silicon_sysmem_manager.cpp:271) +2026-10-22 00:28:45.154 | info | UMD | Opening local chip ids/PCIe ids: {0}/[1] and remote chip ids {1} (cluster.cpp:173) +2026-10-22 00:28:45.154 | info | UMD | IOMMU: disabled (cluster.cpp:147) +2026-10-22 00:28:45.154 | info | UMD | KMD version: 2.5.0 (cluster.cpp:150) +2026-10-22 00:28:45.154 | info | UMD | Cluster constructor completed. (cluster.cpp:648) +2026-10-22 00:28:45.155 | info | UMD | Starting devices in cluster (cluster.cpp:1188) +2026-10-22 00:28:45.158 | info | UMD | Starting devices in cluster completed. (cluster.cpp:1196) +Tenstorrent devices=2; torch used for data/metrics=2.11.0+cpu +Opening Tenstorrent device 0 +2026-10-22 00:28:45.209 | info | Distributed | Using auto discovery to generate mesh graph. (metal_env.cpp:437) +2026-10-22 00:28:45.209 | info | Distributed | Constructing control plane using auto-discovery (no mesh graph descriptor). (metal_env.cpp:499) +2026-10-22 00:28:45.210 | info | Fabric | Logical multi-mesh adjacency: intermesh degree histogram {0:1}; intra-mesh degree histograms mesh0 {1:2} (topology_mapper_utils.cpp:3329) +2026-10-22 00:28:45.210 | info | Fabric | Physical multi-mesh adjacency: intermesh degree histogram {0:1}; intra-mesh degree histograms mesh0 {1:2} (topology_mapper_utils.cpp:3337) +DATASET_LOADED file=/inputs/regression.json +tt:0 step=0 mse=0.18711002 +tt:0 step=20 mse=0.00110440 +tt:0 step=40 mse=0.00008407 +tt:0 step=60 mse=0.00005286 +tt:0 step=80 mse=0.00005125 +{"backend": "tenstorrent", "device_id": 0, "initial_mse": 0.32162201404571533, "final_mse": 5.1175324188079685e-05, "validation_mse": 0.00011455668573034927, "max_weight_change": 0.369140625, "checkpoint": "/outputs/tenstorrent-0.pt"} +Opening Tenstorrent device 1 +2026-10-22 00:28:59.905 | info | Metal | Fabric config changed from FabricConfig::DISABLED to FabricConfig::FABRIC_1D, reinitializing control plane (metal_env.cpp:337) +2026-10-22 00:28:59.905 | info | Distributed | Using auto discovery to generate mesh graph. (metal_env.cpp:437) +2026-10-22 00:28:59.905 | info | Distributed | Constructing control plane using auto-discovery (no mesh graph descriptor). (metal_env.cpp:499) +2026-10-22 00:28:59.906 | info | Fabric | Logical multi-mesh adjacency: intermesh degree histogram {0:1}; intra-mesh degree histograms mesh0 {1:2} (topology_mapper_utils.cpp:3329) +2026-10-22 00:28:59.906 | info | Fabric | Physical multi-mesh adjacency: intermesh degree histogram {0:1}; intra-mesh degree histograms mesh0 {1:2} (topology_mapper_utils.cpp:3337) +2026-10-22 00:28:59.907 | info | Metal | Enabling FabricConfig::FABRIC_1D only for dispatch. If your workload requires fabric, please set the fabric config accordingly. (device_manager.cpp:291) +2026-10-22 00:28:59.907 | info | Metal | Dispatch on FabricConfig::FABRIC_1D with 1 Command Queues + (device_manager.cpp:299) +2026-10-22 00:28:59.909 | info | Metal | Initializing Fabric (fabric_firmware_initializer.cpp:296) +2026-10-22 00:29:01.929 | info | Metal | Fabric initialized on 2 devices (fabric_firmware_initializer.cpp:444) +2026-10-22 00:29:01.929 | info | Metal | Fabric Initialized with config FabricConfig::FABRIC_1D (fabric_firmware_initializer.cpp:313) +DATASET_LOADED file=/inputs/regression.json +tt:1 step=0 mse=0.18711002 +tt:1 step=20 mse=0.00110440 +tt:1 step=40 mse=0.00008407 +tt:1 step=60 mse=0.00005286 +tt:1 step=80 mse=0.00005125 +{"backend": "tenstorrent", "device_id": 1, "initial_mse": 0.32162201404571533, "final_mse": 5.1175324188079685e-05, "validation_mse": 0.00011455668573034927, "max_weight_change": 0.369140625, "checkpoint": "/outputs/tenstorrent-1.pt"} +TRAINING_PASSED backend=tenstorrent devices=2 +2026-10-22 00:29:16.207 | info | BuildKernels | JIT cache stats: 0/58 hits (0.0%) [0 cached, 10 build-once dedup, 0 merged artifacts, 0 merged genfiles] (build_cache_telemetry.cpp:207) +2026-10-22 00:29:16.207 | info | BuildKernels | JIT telemetry: 2 registered TelemetryTokens (build_cache_telemetry.cpp:230) +2026-10-22 00:29:16.212 | info | Device | Closing user mode device drivers (tt_cluster.cpp:513) +2026-10-22 00:29:16.212 | info | UMD | Closing devices in cluster (cluster.cpp:1201) +2026-10-22 00:29:16.220 | info | UMD | Closing devices in cluster completed. (cluster.cpp:1210) +2026-10-22 00:29:16.220 | info | UMD | Cluster destructor started. (cluster.cpp:916) +2026-10-22 00:29:16.220 | info | UMD | Cluster destructor completed. (cluster.cpp:919) diff --git a/deploy/k3s/evidence/storage-2026-10-21/validation.json b/deploy/k3s/evidence/storage-2026-10-21/validation.json new file mode 100644 index 0000000..3f25652 --- /dev/null +++ b/deploy/k3s/evidence/storage-2026-10-21/validation.json @@ -0,0 +1,40 @@ +{ + "base_main": "ba2bb86", + "branch": "feat/simple-storage-ethernet-20261021", + "go_api_tests": 43, + "go_cli_tests": 6, + "go_race": "passed", + "go_vet": "passed", + "react_tests": 16, + "typescript": "passed", + "frontend_lint": "passed", + "frontend_build": "passed", + "python_folder_tests": 6, + "npm_audit_vulnerabilities": 0, + "fake_backend_contract": "passed", + "ethernet": { + "main": "192.168.240.1/30", + "main_interface": "enp6s0", + "quietbox": "192.168.240.2/30", + "quietbox_interface": "enp200s0", + "link_mbps": 1000, + "main_route": "192.168.240.2 dev enp6s0 src 192.168.240.1 uid 1000 \n cache", + "small_probe_bytes": 16777216, + "small_probe_write_MBps": 43.0, + "small_probe_read_MBps": 101.0, + "performance_note": "Single small direct-I/O NFS sample; not a sustained workload benchmark." + }, + "auth_prerequisites": [ + "Require real unexpired database sessions; remove identity-cookie fallbacks", + "Preserve Better Auth scrypt passwords, NOT NULL schema and deactivated accounts", + "Existing administrator login verified against isolated database copy and live portal" + ], + "old_storage": "Retained unchanged for rollback; no migration", + "large_storage_testing": false, + "new_storage_mode": "ordinary directories with advisory team allowances", + "remaining_boundaries": [ + "Single storage host and shared physical capacity", + "No equal-space guarantee or per-team hard quota", + "No automatic backup, local dataset cache, or resumable upload" + ] +} diff --git a/deploy/k3s/evidence/storage-2026-10-21/workload-volumes.json b/deploy/k3s/evidence/storage-2026-10-21/workload-volumes.json new file mode 100644 index 0000000..a66be72 --- /dev/null +++ b/deploy/k3s/evidence/storage-2026-10-21/workload-volumes.json @@ -0,0 +1,130 @@ +[ + { + "id": "mist-b5e74aec693e5111ae262449", + "namespace": "mist-team-a91af070ba2a6bdc", + "node": "utmist-z1opa08", + "automount_service_account_token": false, + "volumes": [ + { + "claim": "team-results", + "read_only": false, + "nfs": { + "path": "/srv/mist-data/teams/team-a91af070ba2a6bdc/results", + "server": "192.168.240.2" + }, + "mount_options": [ + "nfsvers=4.1", + "hard", + "timeo=100", + "retrans=2" + ] + }, + { + "claim": "team-datasets", + "read_only": true, + "nfs": { + "path": "/srv/mist-data/teams/team-a91af070ba2a6bdc/datasets", + "server": "192.168.240.2" + }, + "mount_options": [ + "nfsvers=4.1", + "hard", + "timeo=100", + "retrans=2" + ] + } + ], + "mounts": [ + { + "mountPath": "/checkpoints/mist-b5e74aec693e5111ae262449", + "name": "checkpoints", + "subPath": "members/21008782734082c80dbd4eb4/jobs/mist-b5e74aec693e5111ae262449/outputs" + }, + { + "mountPath": "/outputs", + "name": "checkpoints", + "subPath": "members/21008782734082c80dbd4eb4/jobs/mist-b5e74aec693e5111ae262449/outputs" + }, + { + "mountPath": "/inputs", + "name": "input", + "readOnly": true, + "subPath": "members/21008782734082c80dbd4eb4/datasets/dataset-3c4ddd41a5c75880df01796f/content" + }, + { + "mountPath": "/mist-script", + "name": "script", + "readOnly": true + } + ] + }, + { + "id": "mist-57208c18e561c8cef36e6697", + "namespace": "mist-team-a91af070ba2a6bdc", + "node": "utmist-tt", + "automount_service_account_token": false, + "volumes": [ + { + "claim": "team-results", + "read_only": false, + "nfs": { + "path": "/srv/mist-data/teams/team-a91af070ba2a6bdc/results", + "server": "192.168.240.2" + }, + "mount_options": [ + "nfsvers=4.1", + "hard", + "timeo=100", + "retrans=2" + ] + }, + { + "claim": "team-datasets", + "read_only": true, + "nfs": { + "path": "/srv/mist-data/teams/team-a91af070ba2a6bdc/datasets", + "server": "192.168.240.2" + }, + "mount_options": [ + "nfsvers=4.1", + "hard", + "timeo=100", + "retrans=2" + ] + } + ], + "mounts": [ + { + "mountPath": "/checkpoints/mist-57208c18e561c8cef36e6697", + "name": "checkpoints", + "subPath": "members/21008782734082c80dbd4eb4/jobs/mist-57208c18e561c8cef36e6697/outputs" + }, + { + "mountPath": "/outputs", + "name": "checkpoints", + "subPath": "members/21008782734082c80dbd4eb4/jobs/mist-57208c18e561c8cef36e6697/outputs" + }, + { + "mountPath": "/home/utmist-tt/TT/tt-metal", + "name": "metal", + "readOnly": true + }, + { + "mountPath": "/home/utmist-tt/.local/share/uv/python", + "name": "python", + "readOnly": true + }, + { + "mountPath": "/inputs", + "name": "input", + "readOnly": true, + "subPath": "members/21008782734082c80dbd4eb4/datasets/dataset-3c4ddd41a5c75880df01796f/content" + }, + { + "mountPath": "/mist-script", + "name": "script", + "readOnly": true + } + ] + } +] diff --git a/deploy/k3s/mist-api.yaml b/deploy/k3s/mist-api.yaml index f41c783..19f8a2b 100644 --- a/deploy/k3s/mist-api.yaml +++ b/deploy/k3s/mist-api.yaml @@ -138,7 +138,7 @@ spec: type: RuntimeDefault containers: - name: api - image: mist-api:departments-20261009 + image: mist-api:folders-20261021 imagePullPolicy: Never ports: - name: http @@ -188,8 +188,12 @@ spec: value: http://100.73.139.66:8088,http://utmist.tail459b5e.ts.net:8088,http://127.0.0.1:8088,http://127.0.0.1:3001,http://localhost:3001,https://utmist.tail459b5e.ts.net - name: MIST_STORAGE_ROOT value: /storage + - name: MIST_NFS_SERVER + value: 192.168.240.2 + - name: MIST_NFS_ROOT + value: /srv/mist-data - name: MIST_SHARED_PVC - value: mist-shared-jobs + value: mist-folder-jobs volumeMounts: - name: storage mountPath: /storage @@ -221,7 +225,7 @@ spec: volumes: - name: storage persistentVolumeClaim: - claimName: mist-shared-api + claimName: mist-folder-api - name: auth-data persistentVolumeClaim: claimName: mist-auth-data diff --git a/deploy/k3s/storage/README.md b/deploy/k3s/storage/README.md new file mode 100644 index 0000000..9a15f42 --- /dev/null +++ b/deploy/k3s/storage/README.md @@ -0,0 +1,144 @@ +# Shared folders over direct Ethernet + +Storage implementation for the two-machine installation, October 21, 2026. +The authoritative data for new submissions is an ordinary directory on QuietBox's +existing disk. There are no new disk images, loop devices or per-team mounts. + +```text +NVIDIA enp6s0 192.168.240.1/30 + | + direct 1-Gbps Ethernet + | +QuietBox enp200s0 192.168.240.2/30 + /srv/mist-data/ + .mist-storage-layout.json + metadata/provision-requests/ API-owned desired teams + metadata/provision-status/ readiness and sampled usage + teams// + datasets/ + metadata/datasets/ published dataset records + .uploads/ upload staging + common/datasets//content/ + members//datasets//content/ + results/ + common/jobs//outputs/ + members//jobs//outputs/ +``` + +## Access and limits + +- Accounts and sessions remain in the main-node local SQLite PVC, separate from + shared files. Go handles authentication and team authorization. +- All members can read their team's folders. A member writes their own folder; + common writes require permission. Other teams require an explicit grant. +- Kubernetes mounts only the selected dataset and the job's output directory. + Datasets are read-only at `/inputs`; results are writable at `/outputs`. +- `storage_gib` and `model_storage_gib` remain API fields, with defaults of + 200 and 500 GiB. They are **advisory allowances**. They neither reserve physical + space nor prevent job writes. They may be raised/lowered without migration. +- Every team shares the underlying disk. No equal-space guarantee or automatic + backup, replication, dataset cache or result deletion is provided. +- Uploads use current disk free space minus 1 GiB, plus an optional + `MIST_MAX_DATASET_GIB` ceiling. Jobs can still exhaust the shared disk. + ZIP processing needs both archive and extraction space. Transfers stream; + interrupted uploads restart. Existing CPU/device/runtime policies remain hard + admission limits. + +## Provisioning and usage + +`team_folders.py` is a local root-owned service without a network listener or +cluster credentials. It accepts bounded API-owned manifests, creates directories, +rejects symlinks/unexpected mounts and publishes `shared-folders-v1` readiness. +Usage counts logical regular-file bytes, excludes symlinks, and deduplicates hard +links within each pool. The host samples every 30 seconds; active writes may make +these approximate. The API does not walk the NFS directory tree for every poll. + +`GET /storage` reports: + +- `capacity_bytes`, `available_bytes`: actual shared filesystem space. +- `dataset_used_bytes`, `model_used_bytes`: sampled team file usage. +- `model_capacity_bytes`, `model_available_bytes`: compatibility aliases for the + same shared disk, not a separate model filesystem or reservation. +- `dataset_allocation_gib`, `model_allocation_gib`: advisory values. +- `storage_limits_enforced: false`, `storage_mode: shared-folders`. +- `usage_observed_at`, `usage_stale`, `storage_low`: freshness and disk warning. + +The old `GET /teams` field `storage_budget_gib` remains informational shared +capacity for compatibility. Allocations are not summed into a reservation budget. + +## Host setup + +Run commands as root in the **owning host's** mount/network namespace. Scripts +are installation-specific; verify interface names/addresses on another system. + +On NVIDIA: + +```bash +bash deploy/k3s/storage/setup-storage-link.sh main +``` + +On QuietBox, from a checkout/copy of these files: + +```bash +bash deploy/k3s/storage/setup-storage-link.sh quietbox +bash deploy/k3s/storage/setup-quietbox.sh +install -d -m 0755 /usr/local/lib/mist +install -m 0755 deploy/k3s/storage/team_folders.py /usr/local/lib/mist/team_folders.py +install -m 0644 deploy/k3s/storage/mist-team-storage.service /etc/systemd/system/ +systemctl daemon-reload +systemctl enable --now mist-team-storage +``` + +The NetworkManager profile `mist-storage-direct` persists addresses without a +gateway. Existing LAN/default routes, Tailscale and k3s node addresses stay in use. +NFS permits only `192.168.240.1` and `192.168.240.2`; the firewall allows main's +TCP 2049 on `enp200s0`. Root squash maps root requests to the storage API identity. + +New bindings use NFS **4.1**. During inspection, Linux reused an existing 4.2 +connection to QuietBox's old LAN address even when mounting its Ethernet alias. +A separate 4.1 connection makes the new Ethernet transport verifiable without +tearing down existing old mounts. Verify both the mount and TCP connection, not +only the requested mount address. + +The initial cutover uses new `mist-folder-api`/`mist-folder-jobs` bindings and +per-team `team-datasets`/`team-results` claims, leaving old bindings intact. +Apply `shared.yaml` and the API deployment, then wait for new team readiness. +`ensureVolume` refuses an existing binding pointing to a different NFS export. + +## Prelaunch reset and rollback + +The user chose fresh storage without data migration. Old files under +`/srv/mist-storage` and its backing images remain untouched as rollback material. +Old retained Jobs may still appear in history; their old files are not copied to +the new root. New datasets and results use `/srv/mist-data`. + +Before cutover, record the API/auth deployment specifications, web release target +and team policies. Create a consistent SQLite backup using its database backup +API. Pause the old API/auth before the Go-auth API takes the shared auth PVC. + +Rollback restores the previous API/auth deployment and provisioning unit, and +repoints Nginx's web release symlink. Old volumes were not replaced or deleted. +An operational rollback after accepting new writes must account for those writes; +this is a prelaunch cutover, not automatic bidirectional migration. + +## Verify + +```bash +python3 -m unittest discover -s deploy/k3s/storage -p test_folders.py -v +python3 deploy/k3s/verify_storage_folders.py --out /tmp/mist-storage-check +``` + +The live verifier creates two workspaces and a regular account, checks small +uploads/downloads, archive rejection and team isolation, then trains on one +NVIDIA GPU and one n300 board. It retains a protected account file in its output +directory for optional browser review; never commit that file. Completed test +jobs/results remain inspectable. It does not migrate or delete old research data. + +Optional browser review uses Playwright from `MIST_PLAYWRIGHT_MODULE` (the local +installation defaults to `/tmp/mist-browser-check/node_modules/playwright/index.mjs`): + +```bash +MIST_STORAGE_EVIDENCE_DIR=/tmp/mist-storage-check node deploy/k3s/verify_storage_folders_browser.mjs +``` + +Recorded live results: [October 21 evidence](../evidence/storage-2026-10-21/README.md). diff --git a/deploy/k3s/storage/grow-quietbox-store.py b/deploy/k3s/storage/grow-quietbox-store.py deleted file mode 100644 index 4b9997a..0000000 --- a/deploy/k3s/storage/grow-quietbox-store.py +++ /dev/null @@ -1,34 +0,0 @@ -#!/usr/bin/python3 -"""Online sparse expansion of the known Mist loop image; preserves existing data.""" -import argparse -import os -from pathlib import Path -import stat -import subprocess - -GIB = 1024**3 -image = Path('/srv/mist-storage-volume.img') -root = Path('/srv/mist-storage') -def run(*args): return subprocess.check_output(args,text=True,stderr=subprocess.STDOUT).strip() - -parser = argparse.ArgumentParser() -parser.add_argument('--gib',type=int,default=1600) -args = parser.parse_args() -if os.geteuid()!=0: raise SystemExit('Requires the actual QuietBox host namespace') -if image.is_symlink() or not stat.S_ISREG(image.stat().st_mode): raise SystemExit('Unknown image') -if run('blkid','-s','LABEL','-o','value',str(image))!='mist-shared': raise SystemExit('Unknown filesystem label') -device = run('findmnt','-n','-o','SOURCE','--mountpoint',str(root)) -if Path(run('losetup','-n','-O','BACK-FILE',device))!=image: raise SystemExit('Unknown backing device') -size=args.gib*GIB -if not 100<=args.gib<=2**20 or size fs.f_bavail*fs.f_frsize-200*GIB: raise SystemExit('Insufficient host space after 200 GiB reserve') -# Capacity is sparse: reserve no large amount of host space in advance. -# The guard above checks that the requested capacity could fit with host headroom. -with image.open('r+b') as backing: - backing.truncate(size) -subprocess.run(['losetup','-c',device],check=True) -subprocess.run(['resize2fs',device],check=True) -print('Known shared image expanded without bulk preallocation:',args.gib,'GiB') -subprocess.run(['df','-h',str(root),'/'],check=True) diff --git a/deploy/k3s/storage/mist-team-storage.service b/deploy/k3s/storage/mist-team-storage.service index 8981428..2ffc705 100644 --- a/deploy/k3s/storage/mist-team-storage.service +++ b/deploy/k3s/storage/mist-team-storage.service @@ -1,12 +1,13 @@ [Unit] -Description=Mist bounded team filesystem provisioning +Description=Mist shared team folders and usage reporting After=local-fs.target nfs-server.service -RequiresMountsFor=/srv/mist-storage +RequiresMountsFor=/srv/mist-data Requires=nfs-server.service [Service] Type=simple -ExecStart=/usr/bin/python3 /usr/local/lib/mist/team_volumes.py +Environment=MIST_STORAGE_ROOT=/srv/mist-data +ExecStart=/usr/bin/python3 /usr/local/lib/mist/team_folders.py Restart=on-failure RestartSec=5 UMask=0077 diff --git a/deploy/k3s/storage/setup-quietbox.sh b/deploy/k3s/storage/setup-quietbox.sh index e9484b7..799941f 100755 --- a/deploy/k3s/storage/setup-quietbox.sh +++ b/deploy/k3s/storage/setup-quietbox.sh @@ -1,50 +1,41 @@ -#!/bin/bash -# Run as root in QuietBox's host mount namespace. Never formats a physical disk. +#!/usr/bin/env bash +# Initialize ordinary storage folders on QuietBox. Never formats/mounts a disk. set -euo pipefail -image=/srv/mist-storage-volume.img -root=/srv/mist-storage -mkdir -p "$root" -if mountpoint -q "$root"; then - [ -f "$image" ] && [ "$(blkid -s LABEL -o value "$image")" = mist-shared ] - source=$(findmnt -n -o SOURCE --target "$root") - [ "$(losetup -n -O BACK-FILE "$source")" = "$image" ] -else - if [ -n "$(ls -A "$root")" ]; then echo 'Refusing to cover nonempty storage directory' >&2; exit 1; fi - if [ ! -e "$image" ]; then - (set -o noclobber; : > "$image") - chmod 0600 "$image" - truncate -s 100G "$image" - mkfs.ext4 -m 1 -L mist-shared "$image" - fi - # Check this is our ext4 image before mounting an existing file. - [ "$(blkid -s LABEL -o value "$image")" = mist-shared ] - mount -o loop,nodev,nosuid "$image" "$root" +[ "$(id -u)" -eq 0 ] || { echo 'Run as root on QuietBox' >&2; exit 1; } +root=/srv/mist-data +[ ! -L "$root" ] || { echo 'Refusing a symlink storage root' >&2; exit 1; } +if [ -d "$root" ] && [ -n "$(ls -A "$root")" ] && [ ! -f "$root/.mist-storage-layout.json" ]; then + echo 'Refusing to adopt a nonempty unknown directory' >&2 + exit 1 fi -if ! grep -q '^/srv/mist-storage-volume.img ' /etc/fstab; then - echo '/srv/mist-storage-volume.img /srv/mist-storage ext4 loop,nodev,nosuid 0 0' >> /etc/fstab +install -d -o 65532 -g 65532 -m 0750 "$root" +if [ -e "$root/.mist-storage-layout.json" ]; then + python3 - "$root" <<'PY' +import json, sys +from pathlib import Path +marker = Path(sys.argv[1]) / '.mist-storage-layout.json' +assert not marker.is_symlink() +assert json.loads(marker.read_text())['layout'] == 'shared-folders-v1' +PY +else + printf '{"layout":"shared-folders-v1"}\n' > "$root/.mist-storage-layout.json" + chmod 0644 "$root/.mist-storage-layout.json" fi -chown 65532:65532 "$root" -chmod 0750 "$root" -install -d -o 65532 -g 65532 -m 0700 "$root/metadata" "$root/metadata/datasets" "$root/.uploads" -install -d -o 65532 -g 65532 -m 0755 "$root/datasets" "$root/jobs" "$root/legacy" -mkdir -p /etc/exports.d /etc/nfs.conf.d -cat > /etc/exports.d/mist.exports <<'EXPORT' -/srv/mist-storage 10.0.0.175(rw,sync,crossmnt,no_subtree_check,root_squash,anonuid=65532,anongid=65532) 10.0.0.112(rw,sync,crossmnt,no_subtree_check,root_squash,anonuid=65532,anongid=65532) +install -d -o 65532 -g 65532 -m 0700 "$root/metadata" "$root/metadata/provision-requests" "$root/metadata/provision-status" +install -d -o 65532 -g 65532 -m 0755 "$root/teams" +install -d /etc/exports.d /etc/nfs.conf.d +cat > /etc/exports.d/mist-folders.exports <<'EXPORT' +/srv/mist-data 192.168.240.1(rw,sync,no_subtree_check,root_squash,anonuid=65532,anongid=65532,fsid=8410) 192.168.240.2(rw,sync,no_subtree_check,root_squash,anonuid=65532,anongid=65532,fsid=8410) EXPORT cat > /etc/nfs.conf.d/mist.conf <<'CONF' [nfsd] vers3=n vers4=y CONF -mkdir -p /etc/systemd/system/nfs-server.service.d -printf '[Unit]\nRequiresMountsFor=/srv/mist-storage\n' > /etc/systemd/system/nfs-server.service.d/mist-storage.conf -systemctl daemon-reload systemctl enable --now nfs-server exportfs -ra -if ufw status | grep -q "Status: active"; then - ufw allow from 10.0.0.175 to any port 2049 proto tcp comment mist-nfs - ufw allow from 10.0.0.112 to any port 2049 proto tcp comment mist-nfs +if ufw status | grep -q 'Status: active'; then + ufw allow in on enp200s0 from 192.168.240.1 to any port 2049 proto tcp comment mist-direct-nfs fi -mountpoint "$root" +findmnt -T "$root" df -h "$root" -exportfs -v diff --git a/deploy/k3s/storage/setup-storage-link.sh b/deploy/k3s/storage/setup-storage-link.sh new file mode 100644 index 0000000..33ced6c --- /dev/null +++ b/deploy/k3s/storage/setup-storage-link.sh @@ -0,0 +1,35 @@ +#!/usr/bin/env bash +# Run on each owning host as root. No gateway or cluster-interface changes. +set -euo pipefail +case "${1:-}" in + main) iface=enp6s0; address=192.168.240.1/30 ;; + quietbox) iface=enp200s0; address=192.168.240.2/30 ;; + *) echo 'Usage: setup-storage-link.sh main|quietbox' >&2; exit 2 ;; +esac +[ "$(id -u)" -eq 0 ] || { echo 'Run as root on the owning host' >&2; exit 1; } +[ "$(cat "/sys/class/net/$iface/carrier")" = 1 ] || { echo "$iface has no cable link" >&2; exit 1; } +[ -z "$(ip -4 route show default dev "$iface")" ] || { echo 'Refusing to replace a default-route connection' >&2; exit 1; } +profile=mist-storage-direct +current=$(nmcli -g GENERAL.CONNECTION device show "$iface") +install -d -m 0700 /var/lib/mist +if [ -n "$current" ] && [ "$current" != '--' ] && [ "$current" != "$profile" ]; then + uuid=$(nmcli -g connection.uuid connection show "$current") + auto=$(nmcli -g connection.autoconnect connection show "$uuid") + record=/var/lib/mist/storage-link-previous.txt + if [ ! -f "$record" ]; then + printf '%s\n%s\n%s\n' "$iface" "$uuid" "$auto" > "$record" + chmod 0600 "$record" + fi + nmcli connection modify "$uuid" connection.autoconnect no +fi +if nmcli connection show "$profile" >/dev/null 2>&1; then + [ "$(nmcli -g connection.interface-name connection show "$profile")" = "$iface" ] || exit 1 +else + nmcli connection add type ethernet ifname "$iface" con-name "$profile" +fi +nmcli connection modify "$profile" \ + connection.autoconnect yes connection.autoconnect-priority 100 \ + ipv4.method manual ipv4.addresses "$address" ipv4.gateway '' \ + ipv4.dns '' ipv4.never-default yes ipv6.method disabled +nmcli --wait 15 connection up "$profile" +ip -4 address show dev "$iface" diff --git a/deploy/k3s/storage/shared.yaml b/deploy/k3s/storage/shared.yaml index fb9bee0..5298e0d 100644 --- a/deploy/k3s/storage/shared.yaml +++ b/deploy/k3s/storage/shared.yaml @@ -1,62 +1,62 @@ apiVersion: v1 kind: PersistentVolume metadata: - name: mist-shared-api + name: mist-folder-api spec: capacity: - storage: 1600Gi + storage: 1Gi accessModes: [ReadWriteMany] persistentVolumeReclaimPolicy: Retain storageClassName: "" claimRef: - name: mist-shared-api + name: mist-folder-api namespace: mist-system - mountOptions: [nfsvers=4.2, hard, timeo=100, retrans=2] + mountOptions: [nfsvers=4.1, hard, timeo=100, retrans=2] nfs: - server: 10.0.0.112 - path: /srv/mist-storage + server: 192.168.240.2 + path: /srv/mist-data --- apiVersion: v1 kind: PersistentVolumeClaim metadata: - name: mist-shared-api + name: mist-folder-api namespace: mist-system spec: storageClassName: "" - volumeName: mist-shared-api + volumeName: mist-folder-api accessModes: [ReadWriteMany] resources: requests: - # Existing static NFS claim size stays unchanged; actual capacity is in PV/statfs. - storage: 100Gi + # Static NFS binding declaration, not a quota. API reports real disk space. + storage: 1Gi --- apiVersion: v1 kind: PersistentVolume metadata: - name: mist-shared-jobs + name: mist-folder-jobs spec: capacity: - storage: 100Gi + storage: 1Gi accessModes: [ReadWriteMany] persistentVolumeReclaimPolicy: Retain storageClassName: "" claimRef: - name: mist-shared-jobs + name: mist-folder-jobs namespace: mist - mountOptions: [nfsvers=4.2, hard, timeo=100, retrans=2] + mountOptions: [nfsvers=4.1, hard, timeo=100, retrans=2] nfs: - server: 10.0.0.112 - path: /srv/mist-storage + server: 192.168.240.2 + path: /srv/mist-data --- apiVersion: v1 kind: PersistentVolumeClaim metadata: - name: mist-shared-jobs + name: mist-folder-jobs namespace: mist spec: storageClassName: "" - volumeName: mist-shared-jobs + volumeName: mist-folder-jobs accessModes: [ReadWriteMany] resources: requests: - storage: 100Gi + storage: 1Gi diff --git a/deploy/k3s/storage/team_folders.py b/deploy/k3s/storage/team_folders.py new file mode 100644 index 0000000..50c6efc --- /dev/null +++ b/deploy/k3s/storage/team_folders.py @@ -0,0 +1,151 @@ +#!/usr/bin/env python3 +"""Prepare ordinary team directories and publish usage; never format or migrate. + +Only bounded API-owned requests outside workload mounts are accepted. Allowances +are advisory. The filesystem's real free space is shared by all teams. +""" +import json +import os +from pathlib import Path +import re +import stat +import tempfile +import time + +LAYOUT = 'shared-folders-v1' +TEAM_ID = re.compile(r'^team-[a-f0-9]{16}$') + + +class FolderStore: + def __init__(self, root, uid=65532, gid=65532): + self.root = Path(root) + self.uid, self.gid = uid, gid + marker = self.root / '.mist-storage-layout.json' + if self.root.is_symlink() or marker.is_symlink(): + raise ValueError('storage root and layout marker must not be symlinks') + if json.loads(marker.read_text()).get('layout') != LAYOUT: + raise ValueError('shared-folder storage has not been initialized') + self.device = self.root.stat().st_dev + self.requests = self.directory('metadata/provision-requests', 0o700) + self.status = self.directory('metadata/provision-status', 0o700) + self.directory('teams', 0o755) + + def directory(self, relative, mode): + path = self.root + parts = Path(relative).parts + if not parts or any(part in ('.', '..', '/') for part in parts): + raise ValueError('invalid storage directory') + for part in parts: + path = path / part + if not path.exists() and not path.is_symlink(): + path.mkdir(mode=0o750) + os.chown(path, self.uid, self.gid) + entry = path.lstat() + if not stat.S_ISDIR(entry.st_mode) or entry.st_dev != self.device: + raise ValueError('storage directories must be ordinary directories on the shared disk') + os.chmod(path, mode) + os.chown(path, self.uid, self.gid) + return path + + def request(self, path): + entry = path.lstat() + if (not TEAM_ID.fullmatch(path.stem) or not stat.S_ISREG(entry.st_mode) + or entry.st_uid != self.uid or entry.st_size > 1024): + raise ValueError('request must be a bounded API-owned regular file') + body = json.loads(path.read_text()) + if set(body) != {'id', 'storage_gib', 'model_storage_gib'} or body['id'] != path.stem: + raise ValueError('invalid team request schema') + for key in ('storage_gib', 'model_storage_gib'): + if type(body[key]) is not int or not 1 <= body[key] <= 2**20: + raise ValueError('invalid advisory allowance') + return body + + @staticmethod + def usage(path): + """Logical regular-file bytes. Never follow workload-created symlinks.""" + total = 0 + device = path.stat().st_dev + seen = set() + def failed(error): + if not isinstance(error, FileNotFoundError): + raise error + for _, _, files, descriptor in os.fwalk(path, follow_symlinks=False, onerror=failed): + if os.fstat(descriptor).st_dev != device: + raise ValueError('unexpected nested filesystem in team storage') + for name in files: + try: + info = os.stat(name, dir_fd=descriptor, follow_symlinks=False) + except FileNotFoundError: + continue # A live job may rename/delete files during sampling. + if stat.S_ISREG(info.st_mode): + identity = (info.st_dev, info.st_ino) + if identity not in seen: + total += info.st_size + seen.add(identity) + return total + + def provision(self, body, previous=None): + team = body['id'] + if not TEAM_ID.fullmatch(team): + raise ValueError('invalid team ID') + datasets = self.directory(f'teams/{team}/datasets', 0o750) + results = self.directory(f'teams/{team}/results', 0o750) + for name in ('metadata', 'metadata/datasets', '.uploads'): + self.directory(f'teams/{team}/datasets/{name}', 0o700) + for pool in ('datasets', 'results'): + for scope in ('common', 'members'): + self.directory(f'teams/{team}/{pool}/{scope}', 0o755) + usage = previous or {} + if time.time() - usage.get('usage_observed_unix', 0) >= 30: + usage = {'dataset_used_bytes': self.usage(datasets), + 'model_used_bytes': self.usage(results), + 'usage_observed_unix': int(time.time())} + return {**body, 'state': 'Ready', 'layout': LAYOUT, + **{key: usage[key] for key in + ('dataset_used_bytes', 'model_used_bytes', 'usage_observed_unix')}} + + def publish(self, team, value): + path = self.status / f'{team}.json' + serialized = json.dumps(value, sort_keys=True) + if path.exists() and not path.is_symlink() and path.read_text() == serialized: + return + fd, temporary = tempfile.mkstemp(prefix='.mist-', dir=self.status) + try: + with os.fdopen(fd, 'w') as stream: + stream.write(serialized) + stream.flush() + os.fsync(stream.fileno()) + os.chown(temporary, self.uid, self.gid) + os.replace(temporary, path) + finally: + if os.path.exists(temporary): + os.unlink(temporary) + + def reconcile(self, cache): + for path in sorted(self.requests.glob('team-*.json')): + try: + body = self.request(path) + except Exception as error: + print(f'Request rejected: {path.name}: {error}', flush=True) + continue + try: + value = self.provision(body, cache.get(body['id'])) + cache[body['id']] = value + except Exception as error: + value = {'state': 'Error', 'layout': LAYOUT, 'error': str(error)} + print(f"Provisioning failed: {body['id']}: {error}", flush=True) + self.publish(body['id'], value) + + +def main(): + if os.geteuid() != 0: + raise SystemExit('Run as root on the storage host') + store = FolderStore(os.environ.get('MIST_STORAGE_ROOT', '/srv/mist-data')) + cache = {} + while True: + store.reconcile(cache) + time.sleep(2) + + +if __name__ == '__main__': + main() diff --git a/deploy/k3s/storage/team_volumes.py b/deploy/k3s/storage/team_volumes.py deleted file mode 100644 index 9ea777a..0000000 --- a/deploy/k3s/storage/team_volumes.py +++ /dev/null @@ -1,252 +0,0 @@ -#!/usr/bin/python3 -"""QuietBox host provisioning agent. No network listener or cluster credentials. - -Only API-owned manifests outside all workload mounts request capacity. Filesystem -images enforce capacity on every writer, including jobs. Never shrink/delete or -format an existing unknown image; preserve mounts and original storage data. -""" -import json -import hashlib -import shutil -import os -from pathlib import Path -import re -import stat -import subprocess -import tempfile -import time - -ROOT = Path('/srv/mist-storage') -REQUESTS = ROOT / 'metadata/provision-requests' -STATUS = ROOT / 'metadata/provision-status' -IMAGES = ROOT / '.team-volumes' -PATTERN = re.compile(r'^team-[a-f0-9]{16}$') -GIB = 1024 ** 3 - - -def run(*args): - return subprocess.check_output(args, text=True, stderr=subprocess.STDOUT).strip() - - -def atomic(path, data, mode=0o600, owner=None): - path.parent.mkdir(parents=True, exist_ok=True) - fd, tmp = tempfile.mkstemp(prefix='.mist-', dir=path.parent) - try: - with os.fdopen(fd, 'w') as output: - output.write(data) - output.flush() - os.fsync(output.fileno()) - os.chmod(tmp, mode) - if owner: - os.chown(tmp, *owner) - os.replace(tmp, path) - finally: - if os.path.exists(tmp): - os.unlink(tmp) - - -def manifest(path): - if not PATTERN.fullmatch(path.stem) or path.is_symlink(): - raise ValueError('invalid manifest path') - st = path.stat() - if not stat.S_ISREG(st.st_mode) or st.st_uid != 65532 or st.st_size > 1024: - raise ValueError('manifest must be a bounded API-owned regular file') - body = json.loads(path.read_text()) - if set(body) not in ({'id','storage_gib'},{'id','storage_gib','model_storage_gib'}) or body['id'] != path.stem: - raise ValueError('invalid manifest schema') - body.setdefault('model_storage_gib', body['storage_gib']) - for key in ['storage_gib','model_storage_gib']: - size = body[key] - if type(size) is not int or not 1 <= size <= budget(): - raise ValueError('storage outside the configured allocation budget') - return body - - -def provision_volume(team, gib, model=False): - image = IMAGES / (team + ('-models' if model else '') + '.img') - mount = ROOT / 'teams' / team - if model: mount = mount / '.models' - label = ('model-' if model else 'mist-') + team[-10:] - size = gib * GIB - created = False - if not image.exists(): - fd = os.open(image, os.O_CREAT | os.O_EXCL | os.O_WRONLY | os.O_NOFOLLOW, 0o600) - with os.fdopen(fd, 'wb') as file: - file.truncate(size) - run('mkfs.ext4', '-q', '-m', '1', '-L', label, str(image)) - created = True - if image.is_symlink() or not stat.S_ISREG(image.stat().st_mode): - raise ValueError('unknown image type') - if run('blkid', '-s', 'LABEL', '-o', 'value', str(image)) != label: - raise ValueError('refusing an unknown filesystem image') - old_size = image.stat().st_size - if old_size > size: - raise ValueError('refusing to shrink storage') - mount.mkdir(parents=True, exist_ok=True) - mounted = subprocess.run(['mountpoint', '-q', str(mount)]).returncode == 0 - if mounted: - device = run('findmnt', '-n', '-o', 'SOURCE', '--mountpoint', str(mount)) - if Path(run('losetup', '-n', '-O', 'BACK-FILE', device)) != image: - raise ValueError('mount is backed by an unknown device') - else: - if any(mount.iterdir()): - raise ValueError('refusing to cover a nonempty directory') - run('mount', '-o', 'loop,nodev,nosuid', str(image), str(mount)) - device = run('findmnt', '-n', '-o', 'SOURCE', '--mountpoint', str(mount)) - if old_size < size: - with image.open('r+b') as file: - file.truncate(size) - run('losetup', '-c', device) - # Retry online growth even if the preceding run resized the image but - # crashed before resizing ext4. - filesystem_bytes = os.statvfs(mount).f_blocks * os.statvfs(mount).f_frsize - if filesystem_bytes < size - 64 * 1024 ** 2: - run('losetup', '-c', device) - run('resize2fs', device) - os.chown(mount, 65532, 65532) - os.chmod(mount, 0o750) - # The filesystem root and metadata stay outside selected workload subpaths. - for name in ([] if model else ['metadata', 'metadata/datasets', '.uploads']): - directory = mount / name - directory.mkdir(parents=True, exist_ok=True) - os.chown(directory, 65532, 65532) - os.chmod(directory, 0o700) - for name in (['common','members'] if model else ['datasets', 'jobs', 'legacy', 'common', 'members']): - directory = mount / name - directory.mkdir(exist_ok=True) - os.chown(directory, 65532, 65532) - os.chmod(directory, 0o755) - fstab = Path('/etc/fstab') - entry = f'{image} {mount} ext4 loop,nodev,nosuid,x-systemd.requires-mounts-for=/srv/mist-storage 0 0\n' - contents = fstab.read_text() - if not any(line.startswith(str(image) + ' ') for line in contents.splitlines()): - with fstab.open('a') as file: - file.write(entry) - uuid = run('blkid', '-s', 'UUID', '-o', 'value', str(image)) - opts = f'rw,sync,no_subtree_check,root_squash,anonuid=65532,anongid=65532,fsid={uuid}' - export = f'{mount} 10.0.0.175({opts}) 10.0.0.112({opts})\n' - export_path = Path('/etc/exports.d') / (team + ('-models' if model else '') + '.exports') - changed = not export_path.exists() or export_path.read_text() != export - if changed: - atomic(export_path, export, 0o644) - if changed or created: - run('exportfs', '-ra') - return {'state': 'Ready', 'storage_gib': gib, 'filesystem_uuid': uuid} - - -def budget(): - configured = int(os.environ.get('MIST_TEAM_STORAGE_BUDGET_GIB', '1500')) - if not 2 <= configured <= 2**20: raise ValueError('invalid allocation budget') - fs = os.statvfs(ROOT) - return max(0, min(configured, fs.f_blocks * fs.f_frsize // GIB - 20)) - - -def fingerprint(path): - result = {} - for current, directories, files in os.walk(path, followlinks=False): - for entry in [Path(current), *[Path(current)/name for name in directories+files]]: - relative = str(entry.relative_to(path)) - if relative in result: continue - st = entry.lstat() - if stat.S_ISLNK(st.st_mode): data = ('link', os.readlink(entry)) - elif stat.S_ISDIR(st.st_mode): data = ('directory',) - elif stat.S_ISREG(st.st_mode): - digest = hashlib.sha256() - with entry.open('rb') as source: - while chunk := source.read(1024*1024): digest.update(chunk) - data = ('file', st.st_size, digest.hexdigest()) - else: raise ValueError('cannot migrate special output files') - result[relative] = (data, stat.S_IMODE(st.st_mode), st.st_uid, st.st_gid) - return result - - -def migrate_outputs(team): - root = ROOT/'teams'/team - scopes = [root/'common'] - scopes += [path for path in (root/'members').iterdir() if path.is_dir() and not path.is_symlink()] - for scope in scopes: - source = scope/'jobs' - if not source.exists(): continue - if source.is_symlink(): raise ValueError('unsafe legacy output root') - scope_target = root/'.models'/scope.relative_to(root) - if scope_target.is_symlink(): raise ValueError('unsafe model scope') - scope_target.mkdir(parents=True, exist_ok=True) - os.chown(scope_target,65532,65532); os.chmod(scope_target,0o755) - target = scope_target/'jobs' - target.mkdir(parents=True, exist_ok=True) - os.chown(target,65532,65532); os.chmod(target,0o755) - for job in source.iterdir(): - if job.is_symlink() or not job.is_dir(): raise ValueError('unsafe legacy job directory') - destination = target/job.name - before = fingerprint(job) - if destination.exists(): - if destination.is_symlink() or fingerprint(destination) != before: - raise ValueError('refusing to overwrite conflicting model results') - else: - stage = target/('.mist-migration-'+job.name) - if stage.exists(): - if stage.is_symlink(): raise ValueError('unsafe migration stage') - shutil.rmtree(stage) # Only a controlled duplicate; original still exists. - shutil.copytree(job,stage,symlinks=True) - # copytree preserves modes but not ownership; retain both for API access. - for relative, (_,mode,uid,gid) in before.items(): - item = stage/relative - os.chown(item,uid,gid,follow_symlinks=False) - if not item.is_symlink(): os.chmod(item,mode) - if fingerprint(stage) != before: raise ValueError('output migration checksum mismatch') - os.rename(stage,destination) - shutil.rmtree(job) # Complete, matching copy is published before removal. - - -def provision(body): - team = body['id'] - dataset = provision_volume(team,body['storage_gib']) - models = provision_volume(team,body['model_storage_gib'],model=True) - migrate_outputs(team) - return {**dataset, 'model_storage_gib':body['model_storage_gib'], 'model_filesystem_uuid':models['filesystem_uuid'], 'models_migrated':True} - - -def main(): - if os.geteuid() != 0: - raise SystemExit('Run as root in the host mount namespace') - if subprocess.run(['mountpoint', '-q', str(ROOT)]).returncode: - raise SystemExit('Base Mist storage is not mounted') - for directory in [REQUESTS, STATUS]: - directory.mkdir(parents=True, exist_ok=True) - os.chown(directory, 65532, 65532) - os.chmod(directory, 0o700) - IMAGES.mkdir(exist_ok=True) - os.chmod(IMAGES, 0o700) - os.chown(IMAGES, 0, 0) - (ROOT / 'teams').mkdir(exist_ok=True) - os.chown(ROOT / 'teams', 65532, 65532) - os.chmod(ROOT / 'teams', 0o755) - # Ready records from a previous boot must not authorize an unmounted path. - for path in STATUS.glob('team-*.json'): - atomic(path, json.dumps({'state': 'Provisioning'}), 0o600, (65532, 65532)) - while True: - desired = [] - for path in sorted(REQUESTS.glob('team-*.json')): - try: - desired.append(manifest(path)) - except Exception as error: - print(f'Manifest rejected: {path.name}: {error}', flush=True) - if sum(body['storage_gib']+body['model_storage_gib'] for body in desired) > budget(): - print('Allocation budget exceeded; provisioning refused', flush=True) - time.sleep(2) - continue - for body in desired: - try: - status = provision(body) - except Exception as error: - status = {'state': 'Error', 'storage_gib': body['storage_gib'], 'error': str(error)} - print(f"Provisioning failed: {body['id']}: {error}", flush=True) - path = STATUS / (body['id'] + '.json') - serialized = json.dumps(status, sort_keys=True) - if not path.exists() or path.read_text() != serialized: - atomic(path, serialized, 0o600, (65532, 65532)) - time.sleep(2) - - -if __name__ == '__main__': - main() diff --git a/deploy/k3s/storage/test_folders.py b/deploy/k3s/storage/test_folders.py new file mode 100644 index 0000000..714afd9 --- /dev/null +++ b/deploy/k3s/storage/test_folders.py @@ -0,0 +1,95 @@ +import json +import os +from pathlib import Path +import tempfile +import unittest + +from team_folders import FolderStore, LAYOUT + + +class FolderTests(unittest.TestCase): + def setUp(self): + self.temporary = tempfile.TemporaryDirectory() + self.addCleanup(self.temporary.cleanup) + self.root = Path(self.temporary.name) + (self.root / '.mist-storage-layout.json').write_text(json.dumps({'layout': LAYOUT})) + self.store = FolderStore(self.root, os.getuid(), os.getgid()) + self.team = 'team-aaaaaaaaaaaaaaaa' + self.body = {'id': self.team, 'storage_gib': 200, 'model_storage_gib': 500} + + def request(self, body): + path = self.store.requests / f'{self.team}.json' + path.write_text(json.dumps(body)) + return path + + def test_plain_folders_preserve_files_and_allow_lower_allowances(self): + ready = self.store.provision(self.body) + data = self.root / 'teams' / self.team / 'datasets' + models = data.parent / 'results' + (data / 'common' / 'tiny.csv').write_bytes(b'1,2\n') + (models / 'common' / 'model.bin').write_bytes(b'weights') + lower = {**self.body, 'storage_gib': 1, 'model_storage_gib': 1} + updated = self.store.provision(lower) + self.assertEqual(ready['layout'], LAYOUT) + self.assertEqual(updated['dataset_used_bytes'], 4) + self.assertEqual(updated['model_used_bytes'], 7) + self.assertEqual(updated['storage_gib'], 1) + self.assertEqual(data.stat().st_dev, models.stat().st_dev) + self.assertFalse(list(self.root.rglob('*.img'))) + + def test_usage_does_not_follow_links_or_double_count_hardlinks(self): + self.store.provision(self.body) + data = self.root / 'teams' / self.team / 'datasets' + (data / 'common' / 'data').write_bytes(b'123') + os.link(data / 'common' / 'data', data / 'common' / 'alias') + (data / 'common' / 'outside').symlink_to('/etc') + (data / 'common' / 'file-link').symlink_to('/etc/passwd') + self.assertEqual(FolderStore.usage(data), 3) + + def test_unsafe_paths_fail_and_do_not_change_target(self): + outside = self.root / 'outside' + outside.mkdir(mode=0o700) + (self.root / 'teams' / self.team).symlink_to(outside) + with self.assertRaises(ValueError): + self.store.provision(self.body) + self.assertEqual(list(outside.iterdir()), []) + + def test_requests_reject_symlinks_invalid_schema_ids_and_allowances(self): + path = self.request(self.body) + self.assertEqual(self.store.request(path), self.body) + for body in [dict(self.body, storage_gib=True), dict(self.body, model_storage_gib=0), + dict(self.body, id='../outside'), dict(self.body, extra='unexpected')]: + self.request(body) + with self.assertRaises(ValueError): + self.store.request(path) + path.unlink() + target = self.root / 'request.json' + target.write_text(json.dumps(self.body)) + path.symlink_to(target) + with self.assertRaises(ValueError): + self.store.request(path) + + def test_reconciliation_publishes_errors_and_recovers(self): + self.request(self.body) + cache = {} + self.store.reconcile(cache) + path = self.store.status / f'{self.team}.json' + self.assertEqual(json.loads(path.read_text())['state'], 'Ready') + results = self.root / 'teams' / self.team / 'results' + results.rename(results.with_name('retained')) + results.symlink_to('/tmp') + self.store.reconcile(cache) + self.assertEqual(json.loads(path.read_text())['state'], 'Error') + results.unlink() + results.with_name('retained').rename(results) + self.store.reconcile(cache) + self.assertEqual(json.loads(path.read_text())['state'], 'Ready') + + def test_uninitialized_root_is_refused(self): + (self.root / '.mist-storage-layout.json').write_text('{"layout":"old-image-layout"}') + with self.assertRaises(ValueError): + FolderStore(self.root, os.getuid(), os.getgid()) + + +if __name__ == '__main__': + unittest.main() diff --git a/deploy/k3s/storage/test_migration.py b/deploy/k3s/storage/test_migration.py deleted file mode 100644 index e6f7403..0000000 --- a/deploy/k3s/storage/test_migration.py +++ /dev/null @@ -1,82 +0,0 @@ -"""Run as host root in an isolated temporary directory; never uses live stores.""" -import importlib.util -import json -import os -from pathlib import Path -import shutil -import tempfile -import unittest - -spec=importlib.util.spec_from_file_location('volumes',Path(__file__).with_name('team_volumes.py')) -volumes=importlib.util.module_from_spec(spec);spec.loader.exec_module(volumes) - -@unittest.skipUnless(os.geteuid()==0,'Ownership preservation requires root') -class MigrationTests(unittest.TestCase): - def setUp(self): - self.tmp=tempfile.TemporaryDirectory(prefix='mist-migration-test-') - self.previous=volumes.ROOT - volumes.ROOT=Path(self.tmp.name) - self.team='team-aaaaaaaaaaaaaaaa' - self.root=volumes.ROOT/'teams'/self.team - (self.root/'members').mkdir(parents=True) - self.source=self.root/'common'/'jobs'/'mist-test' - self.source.mkdir(parents=True) - (self.source/'weights.bin').write_bytes(b'EXISTING_WEIGHTS') - (self.source/'weights.bin').chmod(0o600) - os.symlink('/missing-outside-file',self.source/'escape') - self.target=self.root/'.models'/'common'/'jobs'/'mist-test' - def tearDown(self): - volumes.ROOT=self.previous - self.tmp.cleanup() - def test_preserve_content_modes_ownership_symlinks_and_repeat(self): - before=volumes.fingerprint(self.source) - volumes.migrate_outputs(self.team) - self.assertEqual(volumes.fingerprint(self.target),before) - self.assertFalse(self.source.exists()) - volumes.migrate_outputs(self.team) - self.assertEqual(volumes.fingerprint(self.target),before) - def test_retry_after_atomic_publish_before_original_removal(self): - shutil.copytree(self.source,self.target,symlinks=True) - volumes.migrate_outputs(self.team) - self.assertFalse(self.source.exists()) - self.assertEqual((self.target/'weights.bin').read_bytes(),b'EXISTING_WEIGHTS') - def test_conflict_does_not_overwrite_either_copy(self): - self.target.mkdir(parents=True) - (self.target/'weights.bin').write_bytes(b'DIFFERENT_MODEL') - with self.assertRaisesRegex(ValueError,'conflicting'): volumes.migrate_outputs(self.team) - self.assertEqual((self.source/'weights.bin').read_bytes(),b'EXISTING_WEIGHTS') - self.assertEqual((self.target/'weights.bin').read_bytes(),b'DIFFERENT_MODEL') - def test_partial_stage_and_symlink_job_directory(self): - stage=self.target.parent/('.mist-migration-'+self.source.name) - stage.mkdir(parents=True) - (stage/'partial').write_bytes(b'partial') - volumes.migrate_outputs(self.team) - self.assertFalse(stage.exists()) - self.assertEqual((self.target/'weights.bin').read_bytes(),b'EXISTING_WEIGHTS') - os.symlink('/outside',self.source) - with self.assertRaisesRegex(ValueError,'unsafe legacy job'): volumes.migrate_outputs(self.team) - def test_member_scope_remains_writable_by_api_owner(self): - member=self.root/'members'/'member-hash' - old=member/'jobs'/'mist-member' - old.mkdir(parents=True) - (old/'weights.bin').write_bytes(b'MEMBER_WEIGHTS') - before=volumes.fingerprint(old) - volumes.migrate_outputs(self.team) - scope=self.root/'.models'/'members'/'member-hash' - self.assertEqual(scope.stat().st_uid,65532) - self.assertEqual(scope.stat().st_mode & 0o777,0o755) - self.assertEqual(volumes.fingerprint(scope/'jobs'/'mist-member'),before) - # Restart repairs a formerly root-owned, inaccessible parent as well. - os.chown(scope,0,0);os.chmod(scope,0o700) - volumes.migrate_outputs(self.team) - self.assertEqual(scope.stat().st_uid,65532) - self.assertEqual(scope.stat().st_mode & 0o777,0o755) - def test_manifest_schema_owner_and_numeric_bounds(self): - request=volumes.ROOT/(self.team+'.json') - for size in [False,0,-1,volumes.budget()+1]: - request.write_text(json.dumps({'id':self.team,'storage_gib':1,'model_storage_gib':size}));os.chown(request,65532,65532) - with self.assertRaises(ValueError):volumes.manifest(request) - request.write_text(json.dumps({'id':self.team,'storage_gib':1,'model_storage_gib':2}));os.chown(request,65532,65532) - self.assertEqual(volumes.manifest(request)['model_storage_gib'],2) - -if __name__=='__main__':unittest.main() diff --git a/deploy/k3s/verify_split_storage.py b/deploy/k3s/verify_split_storage.py deleted file mode 100644 index 81ae113..0000000 --- a/deploy/k3s/verify_split_storage.py +++ /dev/null @@ -1,123 +0,0 @@ -"""Verify migrated/split storage with small data and real accelerator training. - -No quota exhaustion or bulk upload. Credentials stay in the local protected file. -""" -import hashlib -import http.cookiejar -import json -import math -import os -from pathlib import Path -import subprocess -import time -import urllib.error -import urllib.parse -import urllib.request - -BASE = 'http://100.73.139.66:8088' -TEAM = 'team-db3933031a6c2437' -OUT = Path('/home/utmist/mist-department-results-2026-10-09') -REPO = Path(__file__).resolve().parents[2] -credentials = json.loads(Path('/home/utmist/.config/mist/admin-bootstrap.json').read_text()) -client = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(http.cookiejar.CookieJar())) -report = {'checks': {}, 'jobs': [], 'bulk_testing': False} - -def api(path, data=None, method=None, expected=200, raw=False): - body = data if isinstance(data, bytes) else json.dumps(data).encode() if data is not None else None - r = urllib.request.Request(BASE + path, data=body, method=method, - headers={'Origin': BASE, 'Content-Type': 'application/octet-stream' if isinstance(data, bytes) else 'application/json', 'X-Mist-Team': TEAM}) - try: - response = client.open(r, timeout=30) - except urllib.error.HTTPError as e: - if e.code != expected: raise RuntimeError(f'{path}: HTTP {e.code}: {e.read().decode()}') from None - return json.load(e) - with response: - assert response.status == expected, (path, response.status) - return response.read() if raw else json.load(response) - -def passed(name): - report['checks'][name] = True - print('PASS ' + name, flush=True) - -def download(job, path): - return api(f'/api/jobs/{job}/files/download?path=' + urllib.parse.quote(path, safe=''), raw=True) - -try: - api('/auth/sign-in/email', {'email': credentials['email'], 'password': credentials['password']}) - teams = api('/api/teams') - team = next(t for t in teams['teams'] if t['id'] == TEAM) - storage = api('/api/storage') - assert storage['model_allocation_gib'] == team['policy']['model_storage_gib'] - assert storage['dataset_allocation_gib'] == team['policy']['storage_gib'] - assert storage['model_capacity_bytes'] > 0 and storage['upload_limit_bytes'] > 0 - report['storage'] = storage - passed('both_pools_ready_with_independent_capacities') - for field in ['storage_gib', 'model_storage_gib']: - policy = dict(team['policy']); policy[field] -= 1 - api('/api/teams/' + TEAM, {'policy': policy}, method='PATCH', expected=400) - policy = dict(team['policy']); policy['storage_gib'] = teams['storage_budget_gib'] - api('/api/teams/' + TEAM, {'policy': policy}, method='PATCH', expected=400) - after = next(t for t in api('/api/teams')['teams'] if t['id'] == TEAM) - assert after['policy'] == team['policy'] - passed('both_shrinks_and_combined_overcommit_rejected_without_mutation') - # Local evidence retains the exact small checkpoint bytes downloaded before migration. - evidence = OUT - old_jobs = {'nvidia':'mist-8644eccafaf0b42c3d4add05', 'tenstorrent':'mist-e5c04ea2cc126a786d6fe783'} - count = 0 - for backend, job in old_jobs.items(): - files = api('/api/jobs/' + job + '/files')['files'] - for f in files: - saved = evidence / (backend + '-' + f['path'].replace('/', '_')) - if saved.exists(): - assert hashlib.sha256(saved.read_bytes()).digest() == hashlib.sha256(download(job, f['path'])).digest() - count += 1 - assert any(f['path'].endswith('.pt') for f in files) - assert count >= 4, count - report['preserved_checkpoint_files'] = count - passed('old_accelerator_checkpoints_preserved_byte_for_byte') - matrix = lambda n,m,p: [[math.sin((i+1)*(j+2)+p) for j in range(m)] for i in range(n)] - x, vx = matrix(128,32,.1), matrix(64,32,1.2) - w = [[v*.1 for v in row] for row in matrix(32,32,2.3)] - multiply = lambda a: [[sum(row[j]*w[j][k] for j in range(32)) for k in range(32)] for row in a] - payload = json.dumps({'x':x,'y':multiply(x),'validation_x':vx,'validation_y':multiply(vx)}).encode() - assert len(payload) < 300000 - ds = api('/api/datasets?filename=regression.json&name=Small%20split%20storage%20check&scope=common', payload, expected=201) - assert ds['size'] == len(payload) and ds['sha256'] == hashlib.sha256(payload).hexdigest() - report['dataset_bytes'] = len(payload) - passed('small_dataset_uploaded_with_verified_checksum') - for backend in ['nvidia', 'tenstorrent']: - training = (REPO / 'deploy/k3s/examples/train_shared.py').read_text().replace('args = parser.parse_args()', - f"args = parser.parse_args(['{backend}','--expected-devices','2','--output','/outputs'])") - job = api('/api/jobs', {'name':'Split storage ' + backend, 'type':'command', 'accelerator':backend, - 'device_count':2 if backend == 'nvidia' else 1, 'script':training, 'script_name':'train.py', - 'dataset_id':ds['id'], 'storage_scope':'common', 'timeout_seconds':600}, expected=201)['job_id'] - report['jobs'].append({'id':job,'backend':backend}) - for _ in range(240): - result = api('/api/jobs/' + job) - if result['job_state'] in ['Success','Failure','Cancelled']: break - time.sleep(2) - assert result['job_state'] == 'Success', result - logs = api('/api/jobs/' + job + '/logs')['logs'] - assert 'DATASET_LOADED file=/inputs/regression.json' in logs and 'TRAINING_PASSED' in logs - files = api('/api/jobs/' + job + '/files')['files'] - assert any(f['path'].endswith('.pt') for f in files) - for f in files: assert len(download(job, f['path'])) == f['size'] - (OUT / ('split-' + backend + '.log')).write_text(logs) - report['jobs'][-1]['node'] = result['node'] - # Inspect actual Kubernetes volumes, rather than inferring them from status. - detail = json.loads(subprocess.check_output(['k3s','kubectl','-n','mist-' + TEAM,'get','job',job,'-o','json'], - env={**os.environ,'KUBECONFIG':'/home/utmist/.kube/config'})) - claims = {v['name']:v['persistentVolumeClaim']['claimName'] for v in detail['spec']['template']['spec']['volumes'] if 'persistentVolumeClaim' in v} - assert claims['input'] == 'team-storage' and claims['checkpoints'] == 'team-models', claims - pod = detail['spec']['template']['spec'] - assert next(v for v in pod['volumes'] if v['name'] == 'input')['persistentVolumeClaim']['readOnly'] - mounts = pod['containers'][0]['volumeMounts'] - assert next(m for m in mounts if m['mountPath'] == '/inputs')['readOnly'] - assert next(m for m in mounts if m['mountPath'] == '/outputs')['subPath'].endswith('/jobs/' + job + '/outputs') - passed(backend + '_uploaded_input_readonly_and_models_saved_in_separate_pvc') - # Preserve job outputs, remove only this small verifier dataset after its jobs finish. - api('/api/datasets/' + ds['id'], method='DELETE') - passed('own_small_probe_removed_after_terminal_jobs') -finally: - api('/auth/sign-out', {}) - (OUT / 'split-storage-results.json').write_text(json.dumps(report, indent=2) + '\n') diff --git a/deploy/k3s/verify_storage_folders.py b/deploy/k3s/verify_storage_folders.py new file mode 100644 index 0000000..5e8c431 --- /dev/null +++ b/deploy/k3s/verify_storage_folders.py @@ -0,0 +1,176 @@ +#!/usr/bin/env python3 +"""Small live checks: regular-member uploads, isolation, and NVIDIA/TT training. + +Creates two small verification workspaces and one regular account. No large files, +quota exhaustion, migration or old-data deletion. Credentials stay outside Git. +""" +import argparse +import hashlib +import http.cookiejar +import io +import json +import os +from pathlib import Path +import random +import secrets +import time +import urllib.error +import urllib.parse +import urllib.request +import zipfile + + +class Client: + def __init__(self, base): + self.base = base.rstrip('/') + self.opener = urllib.request.build_opener( + urllib.request.HTTPCookieProcessor(http.cookiejar.CookieJar())) + + def call(self, path, body=None, team=None, method=None, expected=200, raw=False): + payload = body if isinstance(body, bytes) else json.dumps(body).encode() if body is not None else None + headers = {'Origin': self.base, 'Content-Type': 'application/octet-stream' if isinstance(body, bytes) else 'application/json'} + if team: + headers['X-Mist-Team'] = team + request = urllib.request.Request(self.base + path, data=payload, headers=headers, method=method) + try: + response = self.opener.open(request, timeout=30) + except urllib.error.HTTPError as error: + if error.code != expected: + raise RuntimeError(f'{path}: unexpected HTTP {error.code}') from None + response = error + with response: + assert response.code == expected, (path, response.code) + return response.read() if raw else json.load(response) + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--base', default='http://100.73.139.66:8088') + parser.add_argument('--credentials', type=Path, default=Path('/home/utmist/.config/mist/admin-bootstrap.json')) + parser.add_argument('--out', required=True, type=Path) + args = parser.parse_args() + args.out.mkdir(parents=True, exist_ok=True, mode=0o700) + os.chmod(args.out, 0o700) + report = {'checks': [], 'jobs': [], 'bulk_testing': False} + admin, member = Client(args.base), Client(args.base) + jobs = [] + stamp = secrets.token_hex(4) + + def passed(name): + report['checks'].append(name) + print('PASS ' + name, flush=True) + + try: + creds = json.loads(args.credentials.read_text()) + identity = admin.call('/auth/sign-in/email', {'email': creds['email'], 'password': creds['password']})['user'] + teams = [admin.call('/api/teams', {'name': 'Storage verification ' + stamp + suffix, + 'storage_gib': 1, 'model_storage_gib': 1}, expected=201) for suffix in (' A', ' B')] + for team in teams: + admin.call('/api/teams/' + team['id'] + '/members', {'user_id': identity['id'], 'common_writer': True}) + email, password = 'storage-check-' + stamp + '@mist.local', secrets.token_urlsafe(24) + user = admin.call('/auth/admin/create-user', {'name': 'Storage check ' + stamp, 'email': email, + 'password': password, 'role': 'user'})['user'] + admin.call('/api/teams/' + teams[0]['id'] + '/members', {'user_id': user['id'], 'common_writer': False}) + member.call('/auth/sign-in/email', {'email': email, 'password': password}) + team = teams[0]['id'] + state = args.out / 'member-private.json' + state.write_text(json.dumps({'email': email, 'password': password, 'team_id': team})) + state.chmod(0o600) + for _ in range(60): + try: + storage = member.call('/api/storage', team=team) + break + except RuntimeError: + time.sleep(1) + else: + raise AssertionError('team folders did not become ready') + assert storage['storage_mode'] == 'shared-folders' and storage['storage_limits_enforced'] is False + assert storage['upload_limit_bytes'] > 0 + passed('regular_member_login_and_shared_folder_readiness') + member.call('/api/jobs', team=teams[1]['id'], expected=403) + member.call('/api/datasets?filename=forbidden.txt&scope=common', b'no', team=team, expected=403) + passed('membership_and_common_write_permissions_enforced') + + archive = io.BytesIO() + with zipfile.ZipFile(archive, 'w') as z: + z.writestr('../escape.txt', 'no') + member.call('/api/datasets?filename=unsafe.zip&format=zip', archive.getvalue(), team=team, expected=400) + passed('unsafe_archive_rejected') + + rng = random.Random(42) + weights = [[rng.gauss(0, .1) for _ in range(32)] for _ in range(32)] + x = [[rng.gauss(0, 1) for _ in range(32)] for _ in range(128)] + vx = [[rng.gauss(0, 1) for _ in range(32)] for _ in range(64)] + def targets(rows): + return [[sum(row[k] * weights[k][j] for k in range(32)) for j in range(32)] for row in rows] + payload = json.dumps({'x': x, 'y': targets(x), 'validation_x': vx, 'validation_y': targets(vx)}).encode() + assert len(payload) < 400000 + dataset = member.call('/api/datasets?filename=regression.json&name=Ethernet%20storage%20check', payload, team=team, expected=201) + report['dataset_bytes'] = len(payload) + assert dataset['sha256'] == hashlib.sha256(payload).hexdigest() + download = '/api/datasets/' + dataset['id'] + '/files/download?path=regression.json' + assert member.call(download, team=team, raw=True) == payload + admin.call('/api/datasets/' + dataset['id'], team=teams[1]['id'], expected=404) + admin.call('/api/storage/files?' + urllib.parse.urlencode({'source_team': team, 'scope': user['id']}), team=teams[1]['id'], expected=404) + passed('small_upload_checksum_download_and_cross_team_isolation') + + source = (Path(__file__).parent / 'examples/train_shared.py').read_text() + for backend, expected in [('nvidia', 1), ('tenstorrent', 2)]: + script = source.replace('args = parser.parse_args()', + f"args = parser.parse_args(['{backend}','--expected-devices','{expected}','--output','/outputs'])") + job = member.call('/api/jobs', {'name': 'Ethernet folder training ' + backend, 'type': 'command', + 'accelerator': backend, 'device_count': 1, 'dataset_id': dataset['id'], 'script': script, + 'script_name': 'train.py', 'timeout_seconds': 600}, team=team, expected=201)['job_id'] + jobs.append(job) + report['jobs'].append({'id': job, 'backend': backend}) + member.call('/api/datasets/' + dataset['id'], team=team, method='DELETE', expected=409) + passed('active_dataset_deletion_blocked') + + deadline = time.monotonic() + 660 + for job in report['jobs']: + while time.monotonic() < deadline: + status = member.call('/api/jobs/' + job['id'], team=team) + if status['job_state'] in ('Success', 'Failure', 'Cancelled'): + break + time.sleep(2) + log = member.call('/api/jobs/' + job['id'] + '/logs', team=team)['logs'] + (args.out / (job['backend'] + '.log')).write_text(log) + assert status['job_state'] == 'Success', (job['backend'], status.get('message')) + assert 'DATASET_LOADED file=/inputs/regression.json' in log and 'TRAINING_PASSED' in log + files = member.call('/api/jobs/' + job['id'] + '/files', team=team)['files'] + assert any(f['path'].endswith('.pt') for f in files) + for file in files: + content = member.call('/api/jobs/' + job['id'] + '/files/download?' + urllib.parse.urlencode({'path': file['path']}), team=team, raw=True) + assert len(content) == file['size'] + if file['path'] == 'results.json': + job['metrics'] = json.loads(content) + job['node'], job['status'] = status['node'], status['job_state'] + passed(job['backend'] + '_real_training_and_output_downloads') + for _ in range(45): + storage = member.call('/api/storage', team=team) + if storage['dataset_used_bytes'] >= len(payload) and storage['model_used_bytes'] > 0: + break + time.sleep(1) + assert storage['dataset_used_bytes'] >= len(payload) and storage['model_used_bytes'] > 0 + assert storage['usage_stale'] is False + report['storage'] = storage + passed('host_sampled_dataset_and_model_usage') + report['teams'] = [t['id'] for t in teams] + except Exception: + for job in jobs: + try: + member.call('/api/jobs/' + job + '/cancel', team=team, method='POST') + except Exception: + pass + raise + finally: + for client in (member, admin): + try: + client.call('/auth/sign-out', {}, method='POST') + except Exception: + pass + (args.out / 'results.json').write_text(json.dumps(report, indent=2) + '\n') + + +if __name__ == '__main__': + main() diff --git a/deploy/k3s/verify_storage_folders_browser.mjs b/deploy/k3s/verify_storage_folders_browser.mjs new file mode 100644 index 0000000..5efbd03 --- /dev/null +++ b/deploy/k3s/verify_storage_folders_browser.mjs @@ -0,0 +1,59 @@ +// Review the real portal using the regular account from verify_storage_folders.py. +// State and screenshots are written only to the protected external evidence dir. +import { readFileSync, writeFileSync } from 'node:fs' +import { createHash } from 'node:crypto' + +const { chromium } = await import( + process.env.MIST_PLAYWRIGHT_MODULE ?? + '/tmp/mist-browser-check/node_modules/playwright/index.mjs' +) +const base = process.env.MIST_SITE_URL ?? 'http://100.73.139.66:8088' +const out = process.env.MIST_STORAGE_EVIDENCE_DIR +if (!out) throw new Error('Set MIST_STORAGE_EVIDENCE_DIR to the live verifier output directory') +const state = JSON.parse(readFileSync(`${out}/member-private.json`, 'utf8')) +const browser = await chromium.launch({ headless: true }) +const context = await browser.newContext({ viewport: { width: 1440, height: 1000 } }) +const page = await context.newPage() +const report = { checks: [], browser_errors: [] } +page.on('pageerror', (error) => report.browser_errors.push(error.message)) +try { + await page.goto(`${base}/datasets`) + await page.getByLabel('Email', { exact: true }).fill(state.email) + await page.getByLabel('Password', { exact: true }).fill(state.password) + await page.getByRole('button', { name: 'Sign in', exact: true }).click() + await page.getByRole('heading', { name: 'Datasets', exact: true }).waitFor() + await page.getByLabel('Workspace', { exact: true }).click() + await page.locator(`[role=option][data-value="${state.team_id}"]`).click() + await page.getByText(/Shared disk:/).waitFor() + await page.getByText('Advisory allowance — shared disk space is not reserved.').first().waitFor() + report.checks.push('regular_member_login_and_advisory_storage_display') + + await page.getByRole('button', { name: 'Add dataset', exact: true }).click() + await page.getByLabel('Dataset name (optional)').fill('Browser Ethernet check') + const payload = Buffer.from('x,y\n1,2\n') + await page.getByLabel('File or ZIP archive', { exact: true }).setInputFiles({ + name: 'browser.csv', mimeType: 'text/csv', buffer: payload, + }) + const response = page.waitForResponse((r) => r.url().includes('/api/datasets?') && r.request().method() === 'POST') + await page.getByRole('button', { name: 'Upload dataset', exact: true }).click() + const uploaded = await response + if (uploaded.status() !== 201) throw new Error(`Upload HTTP ${uploaded.status()}`) + const dataset = await uploaded.json() + if (dataset.sha256 !== createHash('sha256').update(payload).digest('hex')) throw new Error('Browser upload checksum mismatch') + await page.getByText('Browser Ethernet check is ready to use.', { exact: true }).waitFor() + report.checks.push('real_browser_upload_with_verified_checksum') + await page.screenshot({ path: `${out}/datasets-browser.png`, fullPage: true }) + + await page.goto(`${base}/jobs`) + await page.getByText('Ethernet folder training nvidia', { exact: true }).waitFor() + await page.getByText('Ethernet folder training tenstorrent', { exact: true }).waitFor() + report.checks.push('both_real_training_jobs_visible_in_member_history') + await page.screenshot({ path: `${out}/jobs-browser.png`, fullPage: true }) + if (report.browser_errors.length) throw new Error('Browser reported JavaScript errors') + report.checks.push('no_browser_javascript_errors') + console.log('PASS ' + report.checks.join(', ')) +} finally { + await context.request.post(`${base}/auth/sign-out`, { data: {}, headers: { Origin: base } }).catch(() => {}) + await browser.close() + writeFileSync(`${out}/browser-results.json`, JSON.stringify(report, null, 2)) +} diff --git a/deploy/private/README.md b/deploy/private/README.md index 5a8fe9e..8ba6b6f 100644 --- a/deploy/private/README.md +++ b/deploy/private/README.md @@ -12,18 +12,18 @@ Setup: [k3s](../../docs/content/k3s-setup.md), [Tailscale/SSH](../../docs/conten | Nginx production portal | Main, localhost and `100.73.139.66:8088` | `/srv/mist-web/releases`, atomic `current` symlink | | Go API, auth, and admission controller | `mist-system`, main, one Recreate replica | Team/queue ConfigMaps; Kubernetes Jobs; Local `mist-auth-data` SQLite WAL PVC | | k3s server / agent | Main / QuietBox | Existing cluster and device operators | -| NFSv4 storage | QuietBox | 1,600 GiB sparse image; separate bounded dataset/model filesystems | -| Team provisioning agent | QuietBox `mist-team-storage.service` | Validated manifests, fstab mounts and NFS exports | +| NFSv4 storage | QuietBox | Ordinary `/srv/mist-data/teams` dataset/result folders over direct Ethernet | +| Team provisioning agent | QuietBox `mist-team-storage.service` | Validated requests, ordinary directories and sampled usage | | Portal host firewall | Both `mist-portal-firewall.service` | Owned INPUT/FORWARD chains and host config | -API image tag is `mist-api:departments-20261009`, imported into main's k3s containerd. +API image tag is `mist-api:folders-20261021`, imported into main's k3s containerd. New jobs run in `mist-team-` namespaces. Historical `mist` workloads/PVCs remain. Never scale the current API above one replica: admission requires one controller. ## Build and redeploy on main Run from a checkout of the reviewed current main on the NVIDIA host. Required: -Go 1.25.1, Node 22.16+, working Docker access, the private kubeconfig, existing +Go 1.26.0, Node 22.16+, working Docker access, the private kubeconfig, existing auth Secrets, GPU/TT operators and mounted QuietBox storage. The script is for this existing installation; it is not a fresh-cluster/bootstrap installer. Checking out/merging Git does not restart or deploy services automatically. @@ -66,56 +66,23 @@ rollback for research users. The pre-cleanup department release `3431ffd` contai the verified team architecture; rebuild a compatible department version rather than restoring the archived pre-platform main for normal users. -## QuietBox host setup +## QuietBox host setup and storage cutover -The existing `deploy/k3s/storage/setup-quietbox.sh` creates only the bounded base -image when missing; it never formats a physical disk. Base exports use `crossmnt` -to expose explicitly mounted child filesystems. Before changing storage, inspect -mounts and existing exports. Do not replace a live filesystem with an empty image. +Follow [shared-folder storage operations](../k3s/storage/README.md). This supersedes +image-backed setup, online filesystem growth and model-pool migration instructions. +New storage uses `/srv/mist-data` over `192.168.240.2` and new NFS volume bindings. +Existing image files and bindings are retained for rollback; fresh storage was +requested without data migration. No physical disk is formatted. -Install the new provisioning service as root on QuietBox: +Team dataset/model values are advisory. Track actual shared-disk free space and +sampled team file usage. Uploads preserve 1 GiB of disk headroom; running jobs can +still exhaust storage. New Kubernetes PV/PVC sizes are binding declarations, +not capacity reservations or write limits. -```bash -install -d -m 0755 /usr/local/lib/mist -install -m 0755 deploy/k3s/storage/team_volumes.py /usr/local/lib/mist/team_volumes.py -install -m 0644 deploy/k3s/storage/mist-team-storage.service /etc/systemd/system/ -systemctl daemon-reload -systemctl enable --now mist-team-storage -``` - -The agent reads API-owned requests outside workload mounts, enforces the combined -1,500 GiB dataset + model allocation budget, refuses unknown images/nonempty covered directories, -and supports online growth. Team disable/removal never deletes files or frees an -allocation. Shrinking is rejected. New teams default to 200 GiB datasets and 500 GiB models/results. Per-member -folders share the applicable team pool's hard filesystem capacity. PV/PVC sizes are declarations; ext4 enforces actual writes. - -NFS permits only `10.0.0.175` and `10.0.0.112` over the existing LAN. Keep QuietBox -online and its router/network connection intact. No separate switch or direct -cable is required for this pilot. - -## Dataset request limits - -Uploads are bounded by actual dataset free space minus a 64 MiB metadata reserve. -An optional `MIST_MAX_DATASET_GIB` ceiling can lower this (1–1,048,576 GiB). -There is no fixed 64 GiB cap. Nginx streams bodies without buffering; authenticated -API admission enforces capacity. Browser and Go allow 24-hour transfers. ZIP -archives need both staging and expanded space. Cancellation is supported; -interrupted transfers restart from the beginning. - -## Existing store expansion - -`storage/grow-quietbox-store.py --gib 1600` expands only the known, correctly -labelled image, after checking host headroom. It grows sparsely and never formats -physical disks or shrinks existing filesystems. The current image was expanded -online; its initial physical preallocation was released with -`fstrim -v /srv/mist-storage`. Do not unlink or truncate a mounted backing file. -The host had about 1.9 TiB free after release. Logical capacity is not a physical -reservation; account for future image growth alongside existing research files. - -Migration must run without active team jobs and with API submissions paused. -Restart the updated storage agent, wait for both pool capacities to be Ready, -then deploy the updated API. Old policies with no `model_storage_gib` retain the -old allocation independently for each pool. Preserve credentials/auth data. +The October 21 rollout also preserves existing Better Auth password hashes and +deactivated accounts in Go auth, and requires real unexpired sessions. Session +cookies based only on user IDs are rejected. Auth data remains on its existing +local PVC; a consistent pre-cutover database snapshot is retained privately. ## Portal network rules @@ -169,8 +136,8 @@ QuietBox: ```bash systemctl is-active k3s-agent tailscaled nfs-server mist-team-storage mist-portal-firewall -mountpoint /srv/mist-storage -df -h /srv/mist-storage /srv/mist-storage/teams/team-* +findmnt -T /srv/mist-data +df -h /srv/mist-data ``` With root access, inspect `exportfs -v` and the provisioning service journal if a @@ -182,13 +149,12 @@ real login and show two NVIDIA GPUs/four TT boards. Both nodes must be Ready. - **401:** Sign in again. SSH/Tailscale identity is separate from Mist identity. - **403:** Confirm active team membership, scope writing authority and current policy. Administrators also need team membership to use files/jobs. -- **503:** Check auth and API pods, DNS, child filesystem mounts and provisioning +- **503:** Check auth and API pods, DNS, shared directory readiness and provisioning status. Access and missing team mounts fail closed. - **Queued:** Check per-team concurrency/resources, node health and device availability. Aged large requests drain their target node; no preemption. -- **507 / ENOSPC:** Team filesystem is full. An administrator can grow it within - the allocation budget. Delete only unneeded datasets that have no active job - references. No automatic result cleanup was added. +- **507 / ENOSPC:** Team filesystem is full. Check actual QuietBox disk space; advisory allowances do not add capacity. + Delete only unneeded datasets that have no active job references. No automatic result cleanup was added. - **Image rejected:** Use an explicit tag/digest from an allowed registry. No per-image approval is needed. Private pull-credential management is excluded. - **Image pull/runtime failure:** Verify the image is publicly accessible and diff --git a/deploy/private/build-and-deploy.sh b/deploy/private/build-and-deploy.sh index 2c2103c..880c4ba 100755 --- a/deploy/private/build-and-deploy.sh +++ b/deploy/private/build-and-deploy.sh @@ -10,10 +10,10 @@ k3s kubectl get secret mist-auth-bootstrap -n mist-system >/dev/null python3 deploy/private/prepare-internal-secret.py CGO_ENABLED=0 go -C src build -o ../bin/mist-api . CGO_ENABLED=0 go -C cli build -o ../bin/mist-cli . -docker build -f deploy/k3s/Dockerfile.api -t mist-api:departments-20261009 . +docker build -f deploy/k3s/Dockerfile.api -t mist-api:folders-20261021 . archive=$(mktemp /tmp/mist-deploy.XXXXXX.tar) trap 'python3 -c "import os,sys; os.unlink(sys.argv[1])" "$archive"' EXIT -docker save -o "$archive" mist-api:departments-20261009 +docker save -o "$archive" mist-api:folders-20261021 # Existing approved Docker access permits host import, without changing sudo policy. docker run --rm --mount type=bind,src=/,dst=/host alpine:3.16.3 chroot /host /usr/local/bin/k3s ctr images import "$archive" k3s kubectl apply -f deploy/k3s/storage/shared.yaml -f deploy/k3s/team-rbac.yaml -f deploy/k3s/team-workload-policy.yaml -f deploy/k3s/mist-api.yaml diff --git a/docs/agents/simple-storage-execution.md b/docs/agents/simple-storage-execution.md new file mode 100644 index 0000000..7eaa2ca --- /dev/null +++ b/docs/agents/simple-storage-execution.md @@ -0,0 +1,60 @@ +# Shared folders and Ethernet storage — October 21, 2026 + +## Scope + +Implement on `feat/simple-storage-ethernet-20261021` from main `ba2bb86`. +The user authorized implementation and verification on both machines, with fresh +storage, no data migration and no large storage tests. Preserve main's Go auth. + +## Target + +- Dedicated direct Ethernet: NVIDIA `enp6s0` at `192.168.240.1/30`, QuietBox + `enp200s0` at `192.168.240.2/30`. No gateway; retain existing LAN/cluster paths. +- QuietBox ordinary directory `/srv/mist-data`, exported over NFS on the direct + link; no filesystem images or formatting. Keep old storage as rollback material. +- Team layout: `teams//datasets/{common,members,...}` and + `teams//results/{common,members}`. Only trusted API metadata lies at the root. +- Existing dataset/model allowance fields remain API-compatible but become + advisory. Actual disk free space bounds uploads. CPU/GPU/runtime limits remain + enforced. Team grants and scoped workload mounts remain authoritative. +- A small root-owned local provisioning agent creates directories, refuses unsafe + paths/mounts, and publishes logical byte usage. The API validates readiness and + serves the usage record without recursively scanning NFS on every request. +- Preserve `/inputs`, `/outputs`, dataset IDs and wire fields for new jobs. + +## Execution + +1. Establish baseline tests and inspect running Jobs, mounts and network profiles. +2. Persist Ethernet addresses; verify both directions and measure a small transfer. +3. Replace image provisioning with folder provisioning; adapt backend, UI and tests. +4. Install fresh NFS root/provisioner, prepare current team policies there and + switch storage volumes during a controlled API pause. Do not delete old disks. +5. Verify small uploads/downloads, cross-team isolation, and actual NVIDIA/TT jobs. +6. Record sanitized evidence, document operations, commit and push a reviewable PR. + +## Verification boundaries + +No hundreds-of-GB allocations/uploads, host-disk formatting, unrelated auth +redesign or reconfiguration of cluster node addresses. Keep secrets outside Git. +Main's independent-review rule still applies to merging the eventual PR. + +## Completed + +- Both persistent Ethernet profiles configured and bidirectional connectivity + verified. A 16 MiB direct-I/O NFS sample measured 43 MB/s write, 101 MB/s read. +- Fresh ordinary-folder storage deployed; no old-data migration or deletion. + New NFS 4.1 volume bindings avoid reusing the old 4.2 LAN transport. +- Folder provisioner, backend usage/readiness, advisory UI and developer fake + backend updated. Existing grants, scopes and accelerator admission retained. +- Latest main's Go auth required targeted compatibility/session fixes before + deployment: valid database sessions, existing scrypt hashes/schema, and inactive + accounts. Existing credentials were checked privately on a database snapshot. +- Real regular-member upload/download/isolation checks, NVIDIA/TT training, + checkpoint reload/download and browser review passed with small files. +- [Sanitized evidence](../../deploy/k3s/evidence/storage-2026-10-21/README.md) and + [operations](../../deploy/k3s/storage/README.md) contain the current contracts. + +Human-authored October 9 storage operating snapshots under `docs/content` still +describe the previous image-backed release. README/operations now point to the +current guide; maintainers should reconcile those snapshots when updating human +documentation. This task follows the repository's README/agent-doc editing scope. diff --git a/scripts/fake-backend.mjs b/scripts/fake-backend.mjs index d267623..0abce2a 100644 --- a/scripts/fake-backend.mjs +++ b/scripts/fake-backend.mjs @@ -493,6 +493,7 @@ const server = http.createServer(async (req, res) => { sendJSON(res, 200, { teams: state.teams, storage_budget_gib: state.storageBudgetGiB, + storage_limits_enforced: false, }) return } @@ -708,7 +709,22 @@ const server = http.createServer(async (req, res) => { // ---------------------------------------------------- if (pathname === '/storage' && req.method === 'GET') { - sendJSON(res, 200, state.storage) + const team = state.teams.find((t) => t.id === activeTeam) + const datasets = state.datasets.filter((d) => !activeTeam || d.team_id === activeTeam) + const jobs = state.jobs.filter((j) => !activeTeam || j.team_id === activeTeam) + sendJSON(res, 200, { + ...state.storage, + storage_mode: 'shared-folders', + storage_limits_enforced: false, + dataset_used_bytes: datasets.reduce((total, dataset) => total + dataset.size, 0), + model_used_bytes: jobs.reduce((total, job) => + total + (state.jobFiles.get(job.id) ?? []).reduce((sum, file) => sum + file.size, 0), 0), + dataset_allocation_gib: team?.policy.storage_gib ?? state.storage.dataset_allocation_gib, + model_allocation_gib: team?.policy.model_storage_gib ?? state.storage.model_allocation_gib, + usage_observed_at: new Date().toISOString(), + usage_stale: false, + storage_low: state.storage.available_bytes < state.storage.capacity_bytes / 10, + }) return } diff --git a/src/auth.go b/src/auth.go index 507098c..f8d1fe1 100644 --- a/src/auth.go +++ b/src/auth.go @@ -21,6 +21,7 @@ import ( "golang.org/x/crypto/bcrypt" "golang.org/x/crypto/scrypt" + "golang.org/x/text/unicode/norm" _ "modernc.org/sqlite" ) @@ -69,7 +70,8 @@ func newAuthService(dbPath string, log *slog.Logger) (*AuthService, error) { image TEXT, createdAt TEXT NOT NULL, updatedAt TEXT NOT NULL, - role TEXT DEFAULT 'user' + role TEXT DEFAULT 'user', + banned INTEGER NOT NULL DEFAULT 0 ); CREATE TABLE IF NOT EXISTS session ( @@ -106,6 +108,19 @@ func newAuthService(dbPath string, log *slog.Logger) (*AuthService, error) { db.Close() return nil, fmt.Errorf("init auth schema: %w", err) } + // Preserve deactivated Better Auth accounts, and upgrade Go-auth databases + // created before that compatibility column was retained. + var hasBanned int + if err := db.QueryRow("SELECT COUNT(*) FROM pragma_table_info('user') WHERE name='banned'").Scan(&hasBanned); err != nil { + db.Close() + return nil, err + } + if hasBanned == 0 { + if _, err := db.Exec("ALTER TABLE user ADD COLUMN banned INTEGER NOT NULL DEFAULT 0"); err != nil { + db.Close() + return nil, err + } + } s := &AuthService{ db: db, @@ -174,8 +189,8 @@ func (s *AuthService) bootstrapAdmin() error { defer tx.Rollback() _, err = tx.Exec(` - INSERT INTO user (id, name, email, role, createdAt, updatedAt) - VALUES (?, ?, ?, 'admin', ?, ?) + INSERT INTO user (id, name, email, role, emailVerified, createdAt, updatedAt) + VALUES (?, ?, ?, 'admin', 0, ?, ?) `, userID, "Mist Administrator", email, now, now) if err != nil { return err @@ -213,14 +228,14 @@ func verifyPassword(hashed, password string) bool { return bcrypt.CompareHashAndPassword([]byte(hashed), []byte(password)) == nil } parts := strings.Split(hashed, ":") - if len(parts) == 2 { - salt, err1 := hex.DecodeString(parts[0]) - expected, err2 := hex.DecodeString(parts[1]) - if err1 == nil && err2 == nil { - derived, err := scrypt.Key([]byte(password), salt, 16384, 8, 1, len(expected)) - if err == nil && subtle.ConstantTimeCompare(derived, expected) == 1 { - return true - } + if len(parts) == 2 && len(parts[0]) == 32 && len(parts[1]) == 128 { + _, saltErr := hex.DecodeString(parts[0]) + expected, err := hex.DecodeString(parts[1]) + if saltErr == nil && err == nil { + // Better Auth uses the hex salt text (not decoded bytes), NFKC + // normalization and scrypt r=16. Preserve existing account logins. + derived, err := scrypt.Key([]byte(norm.NFKC.String(password)), []byte(parts[0]), 16384, 16, 1, 64) + return err == nil && subtle.ConstantTimeCompare(derived, expected) == 1 } } return false @@ -251,7 +266,7 @@ func (s *AuthService) checkRateLimit(key string, windowSec int64, maxReq int) bo } func extractSessionToken(r *http.Request) string { - for _, cookieName := range []string{"better-auth.session_token", "session_token", "member", "identity"} { + for _, cookieName := range []string{"better-auth.session_token", "session_token"} { if c, err := r.Cookie(cookieName); err == nil && c.Value != "" { val := c.Value if idx := strings.Index(val, "."); idx != -1 { @@ -263,7 +278,7 @@ func extractSessionToken(r *http.Request) string { cookieHeader := r.Header.Get("Cookie") for _, part := range strings.Split(cookieHeader, ";") { part = strings.TrimSpace(part) - for _, name := range []string{"better-auth.session_token=", "session_token=", "member=", "identity="} { + for _, name := range []string{"better-auth.session_token=", "session_token="} { if strings.HasPrefix(part, name) { val := strings.TrimPrefix(part, name) if idx := strings.Index(val, "."); idx != -1 { @@ -287,7 +302,7 @@ func (s *AuthService) session(r *http.Request) (*Member, error) { SELECT u.id, u.name, u.email, u.role FROM session s JOIN user u ON s.userId = u.id - WHERE s.token = ? AND s.expiresAt > ? + WHERE s.token = ? AND s.expiresAt > ? AND COALESCE(u.banned, 0) = 0 `, token, now).Scan(&m.ID, &m.Name, &m.Email, &m.Role) if err == nil { return &m, nil @@ -295,22 +310,11 @@ func (s *AuthService) session(r *http.Request) (*Member, error) { return nil, err } - err = s.db.QueryRowContext(r.Context(), ` - SELECT id, name, email, role - FROM user - WHERE id = ? - `, token).Scan(&m.ID, &m.Name, &m.Email, &m.Role) - if err == nil { - return &m, nil - } else if !errors.Is(err, sql.ErrNoRows) { - return nil, err - } - return nil, nil } func (s *AuthService) activeMembers(ctx context.Context) (map[string]AccountStatus, error) { - rows, err := s.db.QueryContext(ctx, "SELECT id, role FROM user") + rows, err := s.db.QueryContext(ctx, "SELECT id, role FROM user WHERE COALESCE(banned, 0) = 0") if err != nil { return nil, err } @@ -328,7 +332,7 @@ func (s *AuthService) activeMembers(ctx context.Context) (map[string]AccountStat func (s *AuthService) LookupMember(ctx context.Context, id string) (*Member, error) { var m Member - err := s.db.QueryRowContext(ctx, "SELECT id, name, email, role FROM user WHERE id = ?", id). + err := s.db.QueryRowContext(ctx, "SELECT id, name, email, role FROM user WHERE id = ? AND COALESCE(banned, 0) = 0", id). Scan(&m.ID, &m.Name, &m.Email, &m.Role) if err != nil { return nil, errors.New("account unavailable") @@ -429,7 +433,7 @@ func (s *AuthService) handleSignInEmail(w http.ResponseWriter, r *http.Request) err := s.db.QueryRowContext(r.Context(), ` SELECT id, name, email, role, createdAt, updatedAt FROM user - WHERE LOWER(email) = LOWER(?) + WHERE LOWER(email) = LOWER(?) AND COALESCE(banned, 0) = 0 `, req.Email).Scan(&user.ID, &user.Name, &user.Email, &user.Role, &user.CreatedAt, &user.UpdatedAt) if errors.Is(err, sql.ErrNoRows) { writeAuthError(w, 401, "Invalid email or password", "INVALID_EMAIL_OR_PASSWORD") @@ -508,7 +512,7 @@ func (s *AuthService) handleGetSession(w http.ResponseWriter, r *http.Request) { SELECT u.id, u.name, u.email, u.role, s.id, s.userId, s.token, s.expiresAt FROM session s JOIN user u ON s.userId = u.id - WHERE s.token = ? AND s.expiresAt > ? + WHERE s.token = ? AND s.expiresAt > ? AND COALESCE(u.banned, 0) = 0 `, token, now).Scan(&user.ID, &user.Name, &user.Email, &user.Role, &sess.ID, &sess.UserID, &sess.Token, &sess.ExpiresAt) if err == nil { @@ -524,22 +528,6 @@ func (s *AuthService) handleGetSession(w http.ResponseWriter, r *http.Request) { return } - err = s.db.QueryRowContext(r.Context(), ` - SELECT id, name, email, role FROM user WHERE id = ? - `, token).Scan(&user.ID, &user.Name, &user.Email, &user.Role) - if err == nil { - writeJSON(w, 200, map[string]interface{}{ - "user": user, - "session": map[string]interface{}{ - "id": "sess-mock", - "userId": user.ID, - "token": token, - "expiresAt": time.Now().Add(24 * time.Hour).Format(time.RFC3339Nano), - }, - }) - return - } - writeJSON(w, 200, map[string]interface{}{"user": nil, "session": nil}) } @@ -688,8 +676,8 @@ func (s *AuthService) handleCreateUser(w http.ResponseWriter, r *http.Request) { defer tx.Rollback() _, err = tx.ExecContext(r.Context(), ` - INSERT INTO user (id, name, email, role, createdAt, updatedAt) - VALUES (?, ?, ?, ?, ?, ?) + INSERT INTO user (id, name, email, role, emailVerified, createdAt, updatedAt) + VALUES (?, ?, ?, ?, 0, ?, ?) `, userID, req.Name, req.Email, req.Role, now, now) if err != nil { writeAuthError(w, 500, "Failed to create user", "INTERNAL_ERROR") diff --git a/src/auth_test.go b/src/auth_test.go index b30d513..9432a5d 100644 --- a/src/auth_test.go +++ b/src/auth_test.go @@ -2,6 +2,8 @@ package main import ( "bytes" + "context" + "database/sql" "encoding/json" "io" "log/slog" @@ -9,6 +11,7 @@ import ( "path/filepath" "strings" "testing" + "time" ) func newTestAuthService(t *testing.T) *AuthService { @@ -169,13 +172,16 @@ func TestAuthService_AdminUserManagement(t *testing.T) { _, _ = s.db.Exec(`INSERT INTO user (id, name, email, role, createdAt, updatedAt) VALUES ('user1', 'User', 'user@mist.local', 'user', datetime('now'), datetime('now'))`) _, _ = s.db.Exec(`INSERT INTO account (id, userId, accountId, providerId, password, createdAt, updatedAt) VALUES ('a_user', 'user1', 'user@mist.local', 'credential', ?, datetime('now'), datetime('now'))`, hashed) + seedTestSession(t, s, "admin1") + seedTestSession(t, s, "user1") + // Admin creates a new user rec := authRequest(s, "POST", "/auth/admin/create-user", map[string]string{ "name": "Bob", "email": "bob@mist.local", "password": "bobpassword123", "role": "user", - }, "identity=admin1") + }, "better-auth.session_token=test-session-admin1") if rec.Code != 200 { t.Fatalf("create-user failed: %d %s", rec.Code, rec.Body.String()) } @@ -192,13 +198,13 @@ func TestAuthService_AdminUserManagement(t *testing.T) { "name": "Mallory", "email": "mallory@mist.local", "password": "mallorypassword123", - }, "identity=user1") + }, "better-auth.session_token=test-session-user1") if rec.Code != 403 { t.Fatalf("expected 403 for non-admin create-user, got %d", rec.Code) } // List users - rec = authRequest(s, "GET", "/auth/admin/list-users?limit=100", nil, "identity=admin1") + rec = authRequest(s, "GET", "/auth/admin/list-users?limit=100", nil, "better-auth.session_token=test-session-admin1") if rec.Code != 200 { t.Fatalf("list-users failed: %d %s", rec.Code, rec.Body.String()) } @@ -215,7 +221,7 @@ func TestAuthService_AdminUserManagement(t *testing.T) { rec = authRequest(s, "POST", "/auth/admin/set-user-password", map[string]string{ "userId": created.User.ID, "newPassword": "newbobpassword123", - }, "identity=admin1") + }, "better-auth.session_token=test-session-admin1") if rec.Code != 200 { t.Fatalf("set-user-password failed: %d %s", rec.Code, rec.Body.String()) } @@ -264,3 +270,120 @@ func TestAuthService_InternalMembers(t *testing.T) { t.Fatalf("expected 2 members, got %d", len(body.Members)) } } + +// Fixture sessions exercise the same lookup as real sign-in, never user-ID cookies. +func seedTestSession(t *testing.T, service *AuthService, id string) { + t.Helper() + now := time.Now().UTC() + _, err := service.db.Exec(`INSERT INTO session (id,token,userId,expiresAt,createdAt,updatedAt) VALUES (?,?,?,?,?,?)`, "session-"+id, "test-session-"+id, id, now.Add(time.Hour).Format(time.RFC3339Nano), now.Format(time.RFC3339Nano), now.Format(time.RFC3339Nano)) + if err != nil { + t.Fatal(err) + } +} + +func TestAuthServiceRejectsUserIDAndExpiredSessionCookies(t *testing.T) { + service := newTestAuthService(t) + now := time.Now().UTC().Format(time.RFC3339Nano) + _, err := service.db.Exec(`INSERT INTO user (id,name,email,role,createdAt,updatedAt) VALUES (?,?,?,?,?,?)`, "known-user", "Known", "known@mist.local", "admin", now, now) + if err != nil { + t.Fatal(err) + } + seedTestSession(t, service, "known-user") + for _, cookie := range []string{"identity=known-user", "member=known-user", "session_token=known-user", "better-auth.session_token=known-user"} { + request := httptest.NewRequest("GET", "/session", nil) + request.Header.Set("Cookie", cookie) + member, err := service.session(request) + if err != nil || member != nil { + t.Fatalf("forged identity cookie accepted: %v", err) + } + response := authRequest(service, "GET", "/auth/get-session", nil, cookie) + var body struct { + User *Member `json:"user"` + } + if err := json.Unmarshal(response.Body.Bytes(), &body); err != nil || body.User != nil { + t.Fatal("session endpoint accepted a forged identity") + } + } + if _, err := service.db.Exec(`UPDATE session SET expiresAt=?`, time.Now().Add(-time.Hour).UTC().Format(time.RFC3339Nano)); err != nil { + t.Fatal(err) + } + request := httptest.NewRequest("GET", "/session", nil) + request.Header.Set("Cookie", "better-auth.session_token=test-session-known-user") + if member, err := service.session(request); err != nil || member != nil { + t.Fatalf("expired session accepted: %v", err) + } +} + +func TestBetterAuthPasswordCompatibility(t *testing.T) { + // Public test vector generated with Better Auth's Node scrypt parameters. + const hash = "0123456789abcdef0123456789abcdef:8582d1a409fb81869c9f080d49ce2d8839d611a9905642ce9e7b9e5c1cf7e7b71add2af990c5095edf0733f174780d62b2f7ff2fb42cd07a5edf561bef26dc34" + if !verifyPassword(hash, "legacy-password-123") || verifyPassword(hash, "wrong-password") { + t.Fatal("existing Better Auth password compatibility is broken") + } +} + +func TestDeactivatedAccountsCannotUseExistingSessions(t *testing.T) { + service := newTestAuthService(t) + now := time.Now().UTC().Format(time.RFC3339Nano) + _, err := service.db.Exec(`INSERT INTO user (id,name,email,role,banned,createdAt,updatedAt) VALUES (?,?,?,?,?,?,?)`, "inactive", "Inactive", "inactive@mist.local", "admin", 1, now, now) + if err != nil { + t.Fatal(err) + } + seedTestSession(t, service, "inactive") + request := httptest.NewRequest("GET", "/session", nil) + request.Header.Set("Cookie", "better-auth.session_token=test-session-inactive") + if member, err := service.session(request); err != nil || member != nil { + t.Fatal("deactivated account retained access") + } + active, err := service.activeMembers(context.Background()) + if err != nil { + t.Fatal(err) + } + if _, exists := active["inactive"]; exists { + t.Fatal("deactivated account can keep running jobs") + } + if _, err := service.LookupMember(context.Background(), "inactive"); err == nil { + t.Fatal("deactivated account can be enrolled") + } + response := authRequest(service, "GET", "/auth/get-session", nil, "better-auth.session_token=test-session-inactive") + var body struct { + User *Member `json:"user"` + } + if err := json.Unmarshal(response.Body.Bytes(), &body); err != nil || body.User != nil { + t.Fatal("session endpoint exposes deactivated account") + } +} + +func TestExistingBetterAuthSchemaCanCreateAccounts(t *testing.T) { + path := filepath.Join(t.TempDir(), "existing.sqlite") + db, err := sql.Open("sqlite", path) + if err != nil { + t.Fatal(err) + } + _, err = db.Exec(`CREATE TABLE user (id TEXT PRIMARY KEY, name TEXT NOT NULL, email TEXT NOT NULL UNIQUE, emailVerified INTEGER NOT NULL, createdAt TEXT NOT NULL, updatedAt TEXT NOT NULL, role TEXT)`) + db.Close() + if err != nil { + t.Fatal(err) + } + t.Setenv("MIST_ADMIN_EMAIL", "existing-schema@mist.local") + t.Setenv("MIST_ADMIN_PASSWORD", "test-password-123") + service, err := newAuthService(path, nil) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = service.Close() }) + login := authRequest(service, "POST", "/auth/sign-in/email", map[string]string{"email": "existing-schema@mist.local", "password": "test-password-123"}, "") + if login.Code != 200 { + t.Fatal("bootstrap must support old NOT NULL columns") + } + var body struct { + Token string `json:"token"` + } + if err := json.Unmarshal(login.Body.Bytes(), &body); err != nil { + t.Fatal(err) + } + created := authRequest(service, "POST", "/auth/admin/create-user", map[string]string{"name": "New member", "email": "new-schema@mist.local", "password": "test-password-456"}, "better-auth.session_token="+body.Token) + if created.Code != 200 { + t.Fatalf("member creation failed against existing schema: %s", created.Body) + } +} diff --git a/src/go.mod b/src/go.mod index 2653e15..972e1e9 100644 --- a/src/go.mod +++ b/src/go.mod @@ -4,9 +4,12 @@ go 1.26.0 require ( github.com/distribution/reference v0.6.0 + golang.org/x/crypto v0.58.0 + golang.org/x/text v0.43.0 k8s.io/api v0.34.1 k8s.io/apimachinery v0.34.1 k8s.io/client-go v0.34.1 + modernc.org/sqlite v1.60.1 ) require ( @@ -38,12 +41,10 @@ require ( github.com/x448/float16 v0.8.4 // indirect go.yaml.in/yaml/v2 v2.4.2 // indirect go.yaml.in/yaml/v3 v3.0.4 // indirect - golang.org/x/crypto v0.58.0 // indirect golang.org/x/net v0.60.0 // indirect golang.org/x/oauth2 v0.27.0 // indirect golang.org/x/sys v0.49.0 // indirect golang.org/x/term v0.47.0 // indirect - golang.org/x/text v0.43.0 // indirect golang.org/x/time v0.12.0 // indirect google.golang.org/protobuf v1.36.11 // indirect gopkg.in/evanphx/json-patch.v4 v4.12.0 // indirect @@ -55,7 +56,6 @@ require ( modernc.org/libc v1.77.1 // indirect modernc.org/mathutil v1.7.1 // indirect modernc.org/memory v1.12.1 // indirect - modernc.org/sqlite v1.60.1 // indirect sigs.k8s.io/json v0.0.0-20241014173422-cfa47c3a1cc8 // indirect sigs.k8s.io/randfill v1.0.0 // indirect sigs.k8s.io/structured-merge-diff/v6 v6.3.0 // indirect diff --git a/src/go.sum b/src/go.sum index 5e7af4b..1ea1ff4 100644 --- a/src/go.sum +++ b/src/go.sum @@ -29,11 +29,12 @@ github.com/google/gnostic-models v0.7.0/go.mod h1:whL5G0m6dmc5cPxKc5bdKdEN3UjI7O github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8= github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU= github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= -github.com/google/pprof v0.0.0-20241029153458-d1b30febd7db h1:097atOisP2aRj7vFgYQBbFN4U4JNXUNYpxael3UzMyo= -github.com/google/pprof v0.0.0-20241029153458-d1b30febd7db/go.mod h1:vavhavw2zAxS5dIdcRluK6cSGGPlZynqzFM8NdvU144= github.com/google/pprof v0.0.0-20260802141513-ef3492d7dac3 h1:LMLX+LgTNWpfvCBdFebv6EsYotImrt/Ppc5cXIriCSo= +github.com/google/pprof v0.0.0-20260802141513-ef3492d7dac3/go.mod h1:jl5iWTm0/hd5PjEYEOuwAJ57L/CibdZfrqZ5XA5GrCk= github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= +github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k= +github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM= github.com/josharian/intern v1.0.0 h1:vlS4z54oSdjm0bgjRigI+G1HpF+tI+9rE5LLzOg8HmY= github.com/josharian/intern v1.0.0/go.mod h1:5DoeVV0s6jJacbCEi61lwdGj/aVlrQvzHFFd8Hwg//Y= github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM= @@ -103,12 +104,12 @@ golang.org/x/crypto v0.58.0 h1:COkYLr4k7nDI4r6QtMML2AfzeIswWHAJthLS6K/M/54= golang.org/x/crypto v0.58.0/go.mod h1:Xh+kl5A+M0gMiWDytH2Hhr5Z7WbNCpv/eB/EFgfSk+w= golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= +golang.org/x/mod v0.41.0 h1:qJmnOUb4YB+FsEuM3HcWucdZASCPGhsX6uljO6pog0c= +golang.org/x/mod v0.41.0/go.mod h1:Ek9pY8RKWXwsWvd3rQiHYtMqkjSUV+s1Rj7j4H5Ur6o= golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= -golang.org/x/net v0.50.0 h1:ucWh9eiCGyDR3vtzso0WMQinm2Dnt8cFMuQa9K33J60= -golang.org/x/net v0.50.0/go.mod h1:UgoSli3F/pBgdJBHCTc+tp3gmrU4XswgGRgtnwWTfyM= golang.org/x/net v0.60.0 h1:79p50tfZlm0J9YfoDsSi639qSXNGVwEzOPLCxM2FsYU= golang.org/x/net v0.60.0/go.mod h1:2DA/G1UfVbCpQPeWTmMPGY7Cs2PkBkwu743bVX5PIVg= golang.org/x/oauth2 v0.27.0 h1:da9Vo7/tDv5RH/7nZDz1eMGS/q1Vv1N/7FCrBhI9I3M= @@ -116,23 +117,17 @@ golang.org/x/oauth2 v0.27.0/go.mod h1:onh5ek6nERTohokkhCD/y2cV4Do3fxFHFuAejCkRWT golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.24.0 h1:iqiDrGoDgyuvAKrhLQthUEzU/R378N78gXbBjxed7oA= +golang.org/x/sync v0.24.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0= golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.41.0 h1:Ivj+2Cp/ylzLiEU89QhWblYnOE9zerudt9Ftecq2C6k= -golang.org/x/sys v0.41.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks= -golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo= -golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og= golang.org/x/sys v0.49.0 h1:XbzkgYJHdqh/8m2Uu0W/dQv8nktxx4BFHp1M0gROTXA= golang.org/x/sys v0.49.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og= -golang.org/x/term v0.40.0 h1:36e4zGLqU4yhjlmxEaagx2KuYbJq3EwY8K943ZsHcvg= -golang.org/x/term v0.40.0/go.mod h1:w2P8uVp06p2iyKKuvXIm7N/y0UCRt3UfJTfZ7oOpglM= golang.org/x/term v0.47.0 h1:cpyOwQkr2Nzxe2p6KnrTG7Ei2wtzjsuSjhpXXepZ/BM= golang.org/x/term v0.47.0/go.mod h1:efAfAuG6RKABvg1ArGebRkvX6LXul14SM67o2e4mUCY= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= -golang.org/x/text v0.34.0 h1:oL/Qq0Kdaqxa1KbNeMKwQq0reLCCaFtqu2eNuSeNHbk= -golang.org/x/text v0.34.0/go.mod h1:homfLqTYRFyVYemLBFl5GgL/DWEiH5wcsQ5gSh1yziA= golang.org/x/text v0.43.0 h1:1QivrlhwAsnMOcqgOdl9my7s9OyZl8O5/UGUTJVXoWc= golang.org/x/text v0.43.0/go.mod h1:hbSIYA/amXcRXTFSZqh8tWHEoqRvvbKPJ2uz/KU+a+s= golang.org/x/time v0.12.0 h1:ScB/8o8olJvc+CQPWrK3fPZNfh7qgwCrY0zJmoEQLSE= @@ -141,8 +136,8 @@ golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGm golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE= golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= -golang.org/x/tools v0.41.0 h1:a9b8iMweWG+S0OBnlU36rzLp20z1Rp10w+IY2czHTQc= -golang.org/x/tools v0.41.0/go.mod h1:XSY6eDqxVNiYgezAVqqCeihT4j1U2CCsqvH3WhQpnlg= +golang.org/x/tools v0.51.0 h1:k4Xc/1Om9jwkBJBo4NVLMSARBoWtK10mx+W5BnXCeAI= +golang.org/x/tools v0.51.0/go.mod h1:9eEncMayCV6zRMGhR5eZEC2iBx98qWcF1HZ9Z7wJOoA= golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= @@ -171,14 +166,34 @@ k8s.io/kube-openapi v0.0.0-20250710124328-f3f2b991d03b h1:MloQ9/bdJyIu9lb1PzujOP k8s.io/kube-openapi v0.0.0-20250710124328-f3f2b991d03b/go.mod h1:UZ2yyWbFTpuhSbFhv24aGNOdoRdJZgsIObGBUaYVsts= k8s.io/utils v0.0.0-20250604170112-4c0f3b243397 h1:hwvWFiBzdWw1FhfY1FooPn3kzWuJ8tmbZBHi4zVsl1Y= k8s.io/utils v0.0.0-20250604170112-4c0f3b243397/go.mod h1:OLgZIPagt7ERELqWJFomSt595RzquPNLL48iOWgYOg0= +modernc.org/cc/v4 v4.29.7 h1:q+NXGJ0bK3b4TXFYQQVr9pYETGnmwFWkrUzJnMya/Tg= +modernc.org/cc/v4 v4.29.7/go.mod h1:OnovgIhbbMXMu1aISnJ0wvVD1KnW+cAUJkIrAWh+kVI= +modernc.org/ccgo/v4 v4.36.1 h1:ZNIUZAryN0UgnJwtyxrdEzcFc3yD4Cu4AzjfPXsLsIE= +modernc.org/ccgo/v4 v4.36.1/go.mod h1:rrtGc2QkS239nYb/mQNuBMyjq3/y3ZXWbBjPoV3wqzA= +modernc.org/fileutil v1.4.0 h1:j6ZzNTftVS054gi281TyLjHPp6CPHr2KCxEXjEbD6SM= +modernc.org/fileutil v1.4.0/go.mod h1:EqdKFDxiByqxLk8ozOxObDSfcVOv/54xDs/DUHdvCUU= +modernc.org/gc/v2 v2.6.5 h1:nyqdV8q46KvTpZlsw66kWqwXRHdjIlJOhG6kxiV/9xI= +modernc.org/gc/v2 v2.6.5/go.mod h1:YgIahr1ypgfe7chRuJi2gD7DBQiKSLMPgBQe9oIiito= +modernc.org/gc/v3 v3.1.5 h1:21ldfPfRYE31Tb7B3mwAK8gy1AxP4+dKjrOQPfqakoc= +modernc.org/gc/v3 v3.1.5/go.mod h1:HFK/6AGESC7Ex+EZJhJ2Gni6cTaYpSMmU/cT9RmlfYY= +modernc.org/goabi0 v0.2.0 h1:HvEowk7LxcPd0eq6mVOAEMai46V+i7Jrj13t4AzuNks= +modernc.org/goabi0 v0.2.0/go.mod h1:CEFRnnJhKvWT1c1JTI3Avm+tgOWbkOu5oPA8eH8LnMI= modernc.org/libc v1.77.1 h1:Ct8j47QtiZ1Enj2DtFXQtUqrPCAjdCmPjtCuvrYQ0Hs= modernc.org/libc v1.77.1/go.mod h1:87/pZ4L6nD1zqW4nItuS12YO7hN1igAah34xjnQo/W0= modernc.org/mathutil v1.7.1 h1:GCZVGXdaN8gTqB1Mf/usp1Y/hSqgI2vAGGP4jZMCxOU= modernc.org/mathutil v1.7.1/go.mod h1:4p5IwJITfppl0G4sUEDtCr4DthTaT47/N3aT6MhfgJg= modernc.org/memory v1.12.1 h1:nFMiWrpStgZczNl6XI9GnIk/rWhYIyHGUaR04pGbp9g= modernc.org/memory v1.12.1/go.mod h1:/JP4VbVC+K5sU2wZi9bHoq2MAkCnrt2r98UGeSK7Mjw= +modernc.org/opt v0.2.0 h1:tGyef5ApycA7FSEOMraay9SaTk5zmbx7Tu+cJs4QKZg= +modernc.org/opt v0.2.0/go.mod h1:03fq9lsNfvkYSfxrfUhZCWPk1lm4cq4N+Bh//bEtgns= +modernc.org/sortutil v1.2.1 h1:+xyoGf15mM3NMlPDnFqrteY07klSFxLElE2PVuWIJ7w= +modernc.org/sortutil v1.2.1/go.mod h1:7ZI3a3REbai7gzCLcotuw9AC4VZVpYMjDzETGsSMqJE= modernc.org/sqlite v1.60.1 h1:/blz53O951KWFOso4QQvEs/Fq6cDBKLtMVrYNSeJVKw= modernc.org/sqlite v1.60.1/go.mod h1:1dIoEagfDE72QytD5scH1lxARtaUgKgHC/NuApA27r0= +modernc.org/strutil v1.2.1 h1:UneZBkQA+DX2Rp35KcM69cSsNES9ly8mQWD71HKlOA0= +modernc.org/strutil v1.2.1/go.mod h1:EHkiggD70koQxjVdSBM3JKM7k6L0FbGE5eymy9i3B9A= +modernc.org/token v1.1.0 h1:Xl7Ap9dKaEs5kLoOQeQmPWevfnk/DM5qcLcYlA8ys6Y= +modernc.org/token v1.1.0/go.mod h1:UGzOrNV1mAFSEB63lOFHIpNRUVMvYTc6yu1SMY/XTDM= sigs.k8s.io/json v0.0.0-20241014173422-cfa47c3a1cc8 h1:gBQPwqORJ8d8/YNZWEjoZs7npUVDpVXUUOFfW6CgAqE= sigs.k8s.io/json v0.0.0-20241014173422-cfa47c3a1cc8/go.mod h1:mdzfpAEoE6DHQEN0uh9ZbOCuHbLK5wOm7dK4ctXE9Tg= sigs.k8s.io/randfill v1.0.0 h1:JfjMILfT8A6RbawdsK2JXGBR5AQVfd+9TbzrlneTyrU= diff --git a/src/member_auth.go b/src/member_auth.go index 686a8e5..76d6511 100644 --- a/src/member_auth.go +++ b/src/member_auth.go @@ -37,8 +37,8 @@ func (a *App) requestExecutor(r *http.Request) *KubernetesExecutor { if access, _ := r.Context().Value(teamContextKey{}).(*TeamAccess); access != nil { clone := *a.executor clone.namespace, clone.owner = access.Team.namespace(), access.Team.ID - clone.storage, clone.sharedPVC, clone.team = access.Store, "team-models", access - clone.inputPVC = "team-storage" + clone.storage, clone.sharedPVC, clone.team = access.Store, "team-results", access + clone.inputPVC = "team-datasets" return &clone } member, _ := r.Context().Value(memberKey{}).(*Member) diff --git a/src/shared_storage.go b/src/shared_storage.go index 5a9b712..5d7c095 100644 --- a/src/shared_storage.go +++ b/src/shared_storage.go @@ -24,7 +24,7 @@ import ( "time" ) -// Physical per-team dataset capacity is the upload limit; no small fixed ceiling. +// Uploads are bounded by real shared-disk free space and an optional upload cap. const maxDatasetBytes int64 = 1 << 50 var datasetIDPattern = regexp.MustCompile(`^dataset-[a-f0-9]{24}$`) @@ -80,9 +80,6 @@ func (s *SharedStorage) availableUploadLimit(team bool) int64 { return 0 } reserve := int64(1024 * 1024 * 1024) - if team { - reserve = 64 * 1024 * 1024 - } return max(0, min(s.uploadLimit, int64(stats.Bavail)*stats.Bsize-reserve)) } func (s *SharedStorage) dataset(id, owner string) (*Dataset, error) { @@ -141,9 +138,6 @@ func (s *SharedStorage) upload(r *http.Request, owner string) (*Dataset, error) return nil, err } reserve := int64(1024 * 1024 * 1024) - if r.Context().Value(teamContextKey{}) != nil { - reserve = 64 * 1024 * 1024 - } remaining := int64(stats.Bavail)*stats.Bsize - reserve if remaining <= 0 { return nil, fmt.Errorf("dataset storage has insufficient free space: %w", syscall.ENOSPC) @@ -320,12 +314,20 @@ func (a *App) storageInfo(w http.ResponseWriter, r *http.Request) { } info := map[string]interface{}{"enabled": true, "capacity_bytes": stat.Blocks * uint64(stat.Bsize), "available_bytes": stat.Bavail * uint64(stat.Bsize), "upload_limit_bytes": s.availableUploadLimit(e.team != nil), "max_dataset_bytes": s.uploadLimit} if e.team != nil { - var models syscall.Statfs_t - if err := syscall.Statfs(s.resultsRoot(), &models); err != nil { + usage, err := a.teams.storageStatus(e.team.Team.ID) + if err != nil { executorError(w, err) return } - info["model_capacity_bytes"], info["model_available_bytes"] = models.Blocks*uint64(models.Bsize), models.Bavail*uint64(models.Bsize) + info["storage_mode"] = "shared-folders" + info["storage_limits_enforced"] = false + // Retained response fields are aliases for the same shared disk, not + // independent model capacity. New clients use the sampled usage fields. + info["model_capacity_bytes"], info["model_available_bytes"] = info["capacity_bytes"], info["available_bytes"] + info["dataset_used_bytes"], info["model_used_bytes"] = usage.DatasetUsedBytes, usage.ModelUsedBytes + info["usage_observed_at"] = usage.observedAt() + info["usage_stale"] = time.Since(time.Unix(usage.UsageObservedUnix, 0)) > 2*time.Minute + info["storage_low"] = stat.Bavail*uint64(stat.Bsize) < max(uint64(5*1024*1024*1024), stat.Blocks*uint64(stat.Bsize)/10) info["dataset_allocation_gib"], info["model_allocation_gib"] = e.team.Team.Policy.StorageGiB, e.team.Team.Policy.ModelStorageGiB } writeJSON(w, 200, info) diff --git a/src/shared_storage_test.go b/src/shared_storage_test.go index dbf689d..f3e0f78 100644 --- a/src/shared_storage_test.go +++ b/src/shared_storage_test.go @@ -166,6 +166,10 @@ func TestMemberAuthIsolationAndFailureClosed(t *testing.T) { t.Fatal(err) } } + seedTestSession(t, app.auth, "alice") + seedTestSession(t, app.auth, "bob") + t.Cleanup(func() { _ = app.auth.Close() }) + request := func(method, path, cookie, origin, body string) *httptest.ResponseRecorder { req := httptest.NewRequest(method, path, strings.NewReader(body)) req.Header.Set("Cookie", cookie) @@ -178,17 +182,17 @@ func TestMemberAuthIsolationAndFailureClosed(t *testing.T) { if response := request("GET", "/jobs", "", "", ""); response.Code != 401 { t.Fatal("anonymous request accepted") } - if response := request("POST", "/jobs", "member=alice", "http://evil.test", `{"command":["true"]}`); response.Code != 403 { + if response := request("POST", "/jobs", "better-auth.session_token=test-session-alice", "http://evil.test", `{"command":["true"]}`); response.Code != 403 { t.Fatal("cross-origin mutation accepted") } - if response := request("POST", "/jobs", "member=alice", "http://mist.test", `{"command":["true"]}`); response.Code != 201 { + if response := request("POST", "/jobs", "better-auth.session_token=test-session-alice", "http://mist.test", `{"command":["true"]}`); response.Code != 201 { t.Fatalf("submit failed %d %s", response.Code, response.Body.String()) } - if response := request("GET", "/jobs", "member=bob", "", ""); response.Code != 200 || !strings.Contains(response.Body.String(), `"count":0`) { + if response := request("GET", "/jobs", "better-auth.session_token=test-session-bob", "", ""); response.Code != 200 || !strings.Contains(response.Body.String(), `"count":0`) { t.Fatalf("another member sees jobs: %s", response.Body.String()) } app.auth.Close() - if response := request("GET", "/jobs", "member=alice", "", ""); response.Code != 503 { + if response := request("GET", "/jobs", "better-auth.session_token=test-session-alice", "", ""); response.Code != 503 { t.Fatal("auth failure did not close access") } } diff --git a/src/storage_status.go b/src/storage_status.go new file mode 100644 index 0000000..524dd9f --- /dev/null +++ b/src/storage_status.go @@ -0,0 +1,38 @@ +package main + +import ( + "encoding/json" + "errors" + "os" + "path/filepath" + "time" +) + +// Usage is sampled on the owning storage host; HTTP handlers never walk NFS +// trees. Numbers are logical file bytes and may lag active writes by 30 seconds. +type teamStorageStatus struct { + State string `json:"state"` + Layout string `json:"layout"` + DatasetUsedBytes int64 `json:"dataset_used_bytes"` + ModelUsedBytes int64 `json:"model_used_bytes"` + UsageObservedUnix int64 `json:"usage_observed_unix"` +} + +func (s *TeamService) storageStatus(id string) (teamStorageStatus, error) { + var status teamStorageStatus + data, err := os.ReadFile(filepath.Join(s.app.executor.storage.root, "metadata/provision-status", id+".json")) + if err != nil { + return status, err + } + if err = json.Unmarshal(data, &status); err != nil { + return status, err + } + if status.State != "Ready" || status.Layout != sharedFolderLayout || status.DatasetUsedBytes < 0 || status.ModelUsedBytes < 0 || status.UsageObservedUnix <= 0 { + return status, errors.New("team storage usage is unavailable") + } + return status, nil +} + +func (s teamStorageStatus) observedAt() string { + return time.Unix(s.UsageObservedUnix, 0).UTC().Format(time.RFC3339) +} diff --git a/src/team_queue.go b/src/team_queue.go index 6190f48..f9b6389 100644 --- a/src/team_queue.go +++ b/src/team_queue.go @@ -58,7 +58,7 @@ func (s *TeamService) submitTeam(ctx context.Context, e *KubernetesExecutor, req e.inputSubPath = ds.ContentPath if source.ID != t.ID { claim := "input-" + req.DatasetID - if err = s.ensureVolume(ctx, source.ID, e.namespace, claim, "teams/"+source.ID+"/"+ds.ContentPath, source.Policy.StorageGiB, true); err != nil { + if err = s.ensureVolume(ctx, source.ID, e.namespace, claim, "teams/"+source.ID+"/datasets/"+ds.ContentPath, source.Policy.StorageGiB, true); err != nil { return nil, err } e.inputPVC, e.inputSubPath = claim, "" diff --git a/src/team_runtime.go b/src/team_runtime.go index fe5362a..256f21a 100644 --- a/src/team_runtime.go +++ b/src/team_runtime.go @@ -179,10 +179,10 @@ func (s *TeamService) provisionNamespace(ctx context.Context, t *Team) error { return err } } - if err = s.ensureVolume(ctx, t.ID, ns, "team-storage", fmt.Sprintf("teams/%s", t.ID), t.Policy.StorageGiB, false); err != nil { + if err = s.ensureVolume(ctx, t.ID, ns, "team-datasets", fmt.Sprintf("teams/%s/datasets", t.ID), t.Policy.StorageGiB, false); err != nil { return err } - if err = s.ensureVolume(ctx, t.ID, ns, "team-models", fmt.Sprintf("teams/%s/.models", t.ID), t.Policy.ModelStorageGiB, false); err != nil { + if err = s.ensureVolume(ctx, t.ID, ns, "team-results", fmt.Sprintf("teams/%s/results", t.ID), t.Policy.ModelStorageGiB, false); err != nil { return err } s.provisioned[t.ID] = teamProvisioned{signature: string(signature), checked: time.Now()} @@ -209,13 +209,15 @@ func teamNetworkPolicy(ns string) *networkv1.NetworkPolicy { func (s *TeamService) ensureVolume(ctx context.Context, sourceTeam, namespace, claim, path string, gib int64, readonly bool) error { c := s.app.executor.client name := namespace + "-" + claim - pv := &corev1.PersistentVolume{ObjectMeta: metav1.ObjectMeta{Name: name, Labels: map[string]string{"mist.io/team": sourceTeam}}, Spec: corev1.PersistentVolumeSpec{Capacity: corev1.ResourceList{corev1.ResourceStorage: resource.MustParse(fmt.Sprintf("%dGi", gib))}, AccessModes: []corev1.PersistentVolumeAccessMode{corev1.ReadWriteMany}, PersistentVolumeReclaimPolicy: corev1.PersistentVolumeReclaimRetain, StorageClassName: "", ClaimRef: &corev1.ObjectReference{Name: claim, Namespace: namespace}, MountOptions: []string{"nfsvers=4.2", "hard", "timeo=100", "retrans=2"}, PersistentVolumeSource: corev1.PersistentVolumeSource{NFS: &corev1.NFSVolumeSource{Server: envOr("MIST_NFS_SERVER", "10.0.0.112"), Path: "/srv/mist-storage/" + path, ReadOnly: readonly}}}} + pv := &corev1.PersistentVolume{ObjectMeta: metav1.ObjectMeta{Name: name, Labels: map[string]string{"mist.io/team": sourceTeam}}, Spec: corev1.PersistentVolumeSpec{Capacity: corev1.ResourceList{corev1.ResourceStorage: resource.MustParse(fmt.Sprintf("%dGi", gib))}, AccessModes: []corev1.PersistentVolumeAccessMode{corev1.ReadWriteMany}, PersistentVolumeReclaimPolicy: corev1.PersistentVolumeReclaimRetain, StorageClassName: "", ClaimRef: &corev1.ObjectReference{Name: claim, Namespace: namespace}, MountOptions: []string{"nfsvers=4.1", "hard", "timeo=100", "retrans=2"}, PersistentVolumeSource: corev1.PersistentVolumeSource{NFS: &corev1.NFSVolumeSource{Server: envOr("MIST_NFS_SERVER", "192.168.240.2"), Path: envOr("MIST_NFS_ROOT", "/srv/mist-data") + "/" + path, ReadOnly: readonly}}}} if existing, err := c.CoreV1().PersistentVolumes().Get(ctx, name, metav1.GetOptions{}); apierrors.IsNotFound(err) { if _, err = c.CoreV1().PersistentVolumes().Create(ctx, pv, metav1.CreateOptions{}); err != nil { return err } } else if err != nil { return err + } else if existing.Spec.NFS == nil || *existing.Spec.NFS != *pv.Spec.NFS { + return fmt.Errorf("storage volume %s points at a different export; replace its retained binding during storage cutover", name) } else if existing.Spec.Capacity.Storage().Cmp(pv.Spec.Capacity[corev1.ResourceStorage]) < 0 { existing.Spec.Capacity = pv.Spec.Capacity if _, err = c.CoreV1().PersistentVolumes().Update(ctx, existing, metav1.UpdateOptions{}); err != nil { diff --git a/src/team_storage.go b/src/team_storage.go index 5351677..d402fb3 100644 --- a/src/team_storage.go +++ b/src/team_storage.go @@ -376,7 +376,7 @@ func (a *App) datasetFiles(w http.ResponseWriter, r *http.Request) { serveStoredFiles(w, r, path, len(parts) == 3) } -// Keep the public common/member tree stable while files live in two quota pools. +// Keep the public common/member tree stable across dataset and result directories. func serveTeamFolder(w http.ResponseWriter, r *http.Request, store *SharedStorage, scope string) { data := filepath.Join(store.root, scopePath(scope)) models := filepath.Join(store.resultsRoot(), scopePath(scope)) diff --git a/src/teams.go b/src/teams.go index 4232096..9af698c 100644 --- a/src/teams.go +++ b/src/teams.go @@ -1,7 +1,7 @@ package main // Teams are domain policy, independent of the authentication provider. Identity -// still comes exclusively from Better Auth; ordinary clients never supply roles. +// comes from the authentication service; ordinary clients never supply roles. import ( "context" "crypto/rand" @@ -15,7 +15,6 @@ import ( "path/filepath" "regexp" "sort" - "strconv" "strings" "sync" "syscall" @@ -29,6 +28,8 @@ import ( ) const teamRegistry = "mist-teams" +const maxStorageAllowanceGiB int64 = 1 << 20 +const sharedFolderLayout = "shared-folders-v1" var teamIDPattern = regexp.MustCompile(`^team-[a-f0-9]{16}$`) @@ -39,7 +40,7 @@ type TeamPolicy struct { Tenstorrent int `json:"tenstorrent"` Concurrent int `json:"concurrent"` Queued int `json:"queued"` - StorageGiB int64 `json:"storage_gib"` // Dataset pool; retained API field for compatibility. + StorageGiB int64 `json:"storage_gib"` // Advisory dataset allowance; retained wire field. ModelStorageGiB int64 `json:"model_storage_gib"` RuntimeSeconds int64 `json:"runtime_seconds"` Registries []string `json:"registries"` @@ -106,28 +107,18 @@ func (t *Team) namespace() string { return "mist-" + t.ID } func defaultTeamPolicy() TeamPolicy { return TeamPolicy{CPU: "8", Memory: "32Gi", NVIDIA: 2, Tenstorrent: 4, Concurrent: 2, Queued: 50, StorageGiB: 200, ModelStorageGiB: 500, RuntimeSeconds: 86400, Registries: []string{"docker.io", "ghcr.io", "nvcr.io"}} } -func teamStorageBudgetGiB() int64 { - raw := envOr("MIST_TEAM_STORAGE_BUDGET_GIB", "1500") - value, err := strconv.ParseInt(raw, 10, 64) - if err != nil || value < 2 || value > maxDatasetBytes/(1024*1024*1024) { - panic("invalid MIST_TEAM_STORAGE_BUDGET_GIB") - } - return value -} + +// storageBudgetGiB retains the existing response field as informational physical +// shared capacity. Advisory allowances are not added up or reserved on disk. func (s *TeamService) storageBudgetGiB() int64 { - budget := teamStorageBudgetGiB() - if envOr("MIST_REQUIRE_TEAM_FILESYSTEM", "true") == "false" { - return budget - } var fs syscall.Statfs_t if syscall.Statfs(s.app.executor.storage.root, &fs) != nil { return 0 } - return max(0, min(budget, int64(fs.Blocks)*fs.Bsize/(1024*1024*1024)-20)) + return int64(fs.Blocks) * fs.Bsize / (1024 * 1024 * 1024) } func normalizeStoragePolicy(p *TeamPolicy) { - // Old teams had one shared pool. Preserve their allocation for each bucket; - // the host migration moves completed outputs into the new model filesystem. + // Preserve legacy callers which supplied one advisory allowance. if p.ModelStorageGiB == 0 { p.ModelStorageGiB = p.StorageGiB } @@ -146,7 +137,7 @@ func validateTeam(team *Team) error { if err != nil || mem.Cmp(resource.MustParse("64Mi")) < 0 || mem.Cmp(resource.MustParse("128Gi")) > 0 { return errors.New("team memory must be between 64Mi and 128Gi") } - if p.NVIDIA < 0 || p.NVIDIA > 2 || p.Tenstorrent < 0 || p.Tenstorrent > 4 || p.Concurrent < 1 || p.Concurrent > 16 || p.Queued < 1 || p.Queued > 200 || p.StorageGiB < 1 || p.StorageGiB > teamStorageBudgetGiB() || p.ModelStorageGiB < 1 || p.ModelStorageGiB > teamStorageBudgetGiB() || p.RuntimeSeconds < 10 || p.RuntimeSeconds > 86400 { + if p.NVIDIA < 0 || p.NVIDIA > 2 || p.Tenstorrent < 0 || p.Tenstorrent > 4 || p.Concurrent < 1 || p.Concurrent > 16 || p.Queued < 1 || p.Queued > 200 || p.StorageGiB < 1 || p.StorageGiB > maxStorageAllowanceGiB || p.ModelStorageGiB < 1 || p.ModelStorageGiB > maxStorageAllowanceGiB || p.RuntimeSeconds < 10 || p.RuntimeSeconds > 86400 { return errors.New("invalid team device, concurrency, queue, storage or runtime limits") } if len(p.Registries) > 20 { @@ -160,7 +151,6 @@ func validateTeam(team *Team) error { return nil } func newTeamService(a *App) *TeamService { - _ = teamStorageBudgetGiB() return &TeamService{app: a, stores: map[string]*SharedStorage{}, provisioned: map[string]teamProvisioned{}, stop: make(chan struct{}), done: make(chan struct{})} } func (s *TeamService) registry(ctx context.Context) (*corev1.ConfigMap, []Team, error) { @@ -189,13 +179,6 @@ func (s *TeamService) mutate(ctx context.Context, change func(*[]Team) error) er if err = change(&teams); err != nil { return &submissionError{err} } - var budget int64 - for _, t := range teams { - budget += t.Policy.StorageGiB + t.Policy.ModelStorageGiB - } - if budget > s.storageBudgetGiB() { - return &submissionError{fmt.Errorf("total dataset and model allocations cannot exceed the %dGiB storage budget", s.storageBudgetGiB())} - } data, err := json.Marshal(teams) if err != nil { return err @@ -241,59 +224,26 @@ func (s *TeamService) store(t *Team) (*SharedStorage, error) { if err != nil { return nil, errors.New("team storage is being provisioned") } - var ready struct { - State string `json:"state"` - StorageGiB int64 `json:"storage_gib"` - ModelStorageGiB int64 `json:"model_storage_gib"` - } - if json.Unmarshal(status, &ready) != nil || ready.State != "Ready" || ready.StorageGiB < t.Policy.StorageGiB || ready.ModelStorageGiB < t.Policy.ModelStorageGiB { + var ready teamStorageStatus + if json.Unmarshal(status, &ready) != nil || ready.State != "Ready" || ready.Layout != sharedFolderLayout { return nil, errors.New("team storage is not ready") } - if envOr("MIST_REQUIRE_TEAM_FILESYSTEM", "true") != "false" { - var datasetDevice uint64 - for index, pool := range []struct { - path string - gib int64 - }{{filepath.Join(s.app.executor.storage.root, "teams", t.ID), t.Policy.StorageGiB}, {filepath.Join(s.app.executor.storage.root, "teams", t.ID, ".models"), t.Policy.ModelStorageGiB}} { - var fs syscall.Statfs_t - if err := syscall.Statfs(pool.path, &fs); err != nil { - return nil, err - } - // NFS reports a zero statfs fsid on these nodes. The device ID - // identifies distinct mounted superblocks, including NFS child mounts. - entry, err := os.Stat(pool.path) - if err != nil { - return nil, err - } - identity, ok := entry.Sys().(*syscall.Stat_t) - if !ok { - return nil, errors.New("cannot identify team filesystem") - } - if index == 0 { - datasetDevice = uint64(identity.Dev) - } else if uint64(identity.Dev) == datasetDevice { - return nil, errors.New("separate model filesystem is unavailable") - } - capacity := fs.Blocks * uint64(fs.Bsize) - target := uint64(pool.gib) * 1024 * 1024 * 1024 - if capacity > target+64*1024*1024 || capacity < target*9/10 { - return nil, errors.New("bounded team filesystem is unavailable") - } + base := filepath.Join(s.app.executor.storage.root, "teams", t.ID) + for _, path := range []string{base, filepath.Join(base, "datasets"), filepath.Join(base, "results")} { + info, err := os.Lstat(path) + if err != nil || !info.IsDir() || info.Mode()&os.ModeSymlink != 0 { + return nil, errors.New("team storage directories are unavailable") } } if st := s.stores[t.ID]; st != nil { return st, nil } - st, err := NewSharedStorage(filepath.Join(s.app.executor.storage.root, "teams", t.ID)) + st, err := NewSharedStorage(filepath.Join(base, "datasets")) if err != nil { return nil, err } - st.modelRoot = filepath.Join(st.root, ".models") - if envOr("MIST_REQUIRE_TEAM_FILESYSTEM", "true") == "false" { - if err := os.MkdirAll(st.modelRoot, 0700); err != nil { - return nil, err - } - } + st.modelRoot = filepath.Join(base, "results") + s.stores[t.ID] = st return st, nil } @@ -373,7 +323,7 @@ func (a *App) teamEndpoint(w http.ResponseWriter, r *http.Request) { result = append(result, t) } } - writeJSON(w, 200, map[string]any{"teams": result, "storage_budget_gib": s.storageBudgetGiB()}) + writeJSON(w, 200, map[string]any{"teams": result, "storage_budget_gib": s.storageBudgetGiB(), "storage_limits_enforced": false}) return } if !isAdmin(m) { @@ -463,9 +413,6 @@ func (a *App) teamEndpoint(w http.ResponseWriter, r *http.Request) { } if body.Policy != nil { normalizeStoragePolicy(body.Policy) - if body.Policy.StorageGiB < t.Policy.StorageGiB || body.Policy.ModelStorageGiB < t.Policy.ModelStorageGiB { - return errors.New("storage cannot be shrunk online; existing files are preserved") - } t.Policy = *body.Policy } return validateTeam(t) diff --git a/src/teams_test.go b/src/teams_test.go index ee53324..b444812 100644 --- a/src/teams_test.go +++ b/src/teams_test.go @@ -10,7 +10,6 @@ import ( "os" "path/filepath" "strings" - "syscall" "testing" "time" @@ -45,7 +44,6 @@ func departmentTestApp(t *testing.T) (*App, *fake.Clientset, []Team) { t.Fatal(err) } } - t.Setenv("MIST_REQUIRE_TEAM_FILESYSTEM", "false") t.Setenv("MIST_TEAMS_ENABLED", "true") t.Setenv("MIST_AUTH_DATABASE", filepath.Join(t.TempDir(), "auth.sqlite")) e := &KubernetesExecutor{client: c, namespace: "mist", owner: "legacy", storage: store, cpuNode: "main", ttNode: "tt", allowedImages: map[string]bool{cpuImage: true, ttImage: true, nvidiaImage: true}} @@ -63,8 +61,18 @@ func departmentTestApp(t *testing.T) (*App, *fake.Clientset, []Team) { t.Fatal(err) } } + for _, id := range []string{"alice", "bob", "admin"} { + seedTestSession(t, app.auth, id) + } + t.Cleanup(func() { _ = app.auth.Close() }) + for _, team := range teams { - if err = atomicJSON(filepath.Join(root, "metadata/provision-status", team.ID+".json"), map[string]any{"state": "Ready", "storage_gib": team.Policy.StorageGiB, "model_storage_gib": team.Policy.ModelStorageGiB}); err != nil { + for _, pool := range []string{"datasets", "results"} { + if err := os.MkdirAll(filepath.Join(root, "teams", team.ID, pool), 0750); err != nil { + t.Fatal(err) + } + } + if err = atomicJSON(filepath.Join(root, "metadata/provision-status", team.ID+".json"), teamStorageStatus{State: "Ready", Layout: sharedFolderLayout, UsageObservedUnix: time.Now().Unix()}); err != nil { t.Fatal(err) } if _, err = app.teams.store(&team); err != nil { @@ -75,7 +83,7 @@ func departmentTestApp(t *testing.T) (*App, *fake.Clientset, []Team) { } func departmentRequest(a *App, method, path, identity, team, body string) *httptest.ResponseRecorder { req := httptest.NewRequest(method, path, strings.NewReader(body)) - req.Header.Set("Cookie", "identity="+identity) + req.Header.Set("Cookie", "better-auth.session_token=test-session-"+identity) if team != "" { req.Header.Set("X-Mist-Team", team) } @@ -152,7 +160,7 @@ func TestDepartmentTeamPermissionsAndScopedSharing(t *testing.T) { t.Fatal("revoked grant remains effective") } } -func TestDepartmentPolicyAndStorageBudget(t *testing.T) { +func TestDepartmentPolicyAndAdvisoryStorage(t *testing.T) { a, _, teams := departmentTestApp(t) p := defaultTeamPolicy() p.CPU = "1" @@ -172,12 +180,12 @@ func TestDepartmentPolicyAndStorageBudget(t *testing.T) { if _, err := e.normalize(CreateJobRequest{Command: []string{"true"}, TimeoutSeconds: 30}); err != nil { t.Fatalf("valid request rejected: %v", err) } - if err := a.teams.mutate(context.Background(), func(list *[]Team) error { (*list)[0].Policy.StorageGiB = 1500; return nil }); err == nil { - t.Fatal("overcommitted storage budget") + if err := a.teams.mutate(context.Background(), func(list *[]Team) error { (*list)[0].Policy.StorageGiB = 1500; return nil }); err != nil { + t.Fatalf("advisory allowance must not reserve disk space: %v", err) } - _, actual, _ := a.teams.registry(context.Background()) - if actual[0].Policy.StorageGiB != 200 { - t.Fatal("failed transaction changed persisted allocation") + changed := departmentRequest(a, "GET", "/teams", "alice", "", "") + if changed.Code != 200 || !strings.Contains(changed.Body.String(), `"storage_limits_enforced":false`) { + t.Fatalf("advisory semantics missing: %s", changed.Body) } } func TestDepartmentDurableFairAdmission(t *testing.T) { @@ -192,7 +200,7 @@ func TestDepartmentDurableFairAdmission(t *testing.T) { store, _ := a.teams.store(&team) for i := 0; i < 2; i++ { e := *a.executor - e.namespace, e.owner, e.storage, e.sharedPVC = team.namespace(), team.ID, store, "team-storage" + e.namespace, e.owner, e.storage, e.sharedPVC = team.namespace(), team.ID, store, "team-datasets" e.team = &TeamAccess{Team: &team, Member: &Member{ID: team.Members[0].ID}, Service: a.teams} job, err := e.submit(ctx, CreateJobRequest{Command: []string{"true"}, CPU: "1", Memory: "64Mi"}) if err != nil { @@ -402,7 +410,7 @@ func TestDepartmentSplitDatasetAndModelPaths(t *testing.T) { claims[volume.Name] = volume.PersistentVolumeClaim.ClaimName } } - if claims["input"] != "team-storage" || claims["checkpoints"] != "team-models" { + if claims["input"] != "team-datasets" || claims["checkpoints"] != "team-results" { t.Fatalf("storage pools not separated: %+v", claims) } store, err := a.teams.store(&team) @@ -434,21 +442,78 @@ func TestDepartmentSplitDatasetAndModelPaths(t *testing.T) { } } -func TestDepartmentMissingModelMountCannotBorrowDatasetCapacity(t *testing.T) { +func TestDepartmentFoldersFailClosedForMissingOrUnsafePaths(t *testing.T) { a, _, teams := departmentTestApp(t) team := teams[0] - var fs syscall.Statfs_t - if err := syscall.Statfs(a.executor.storage.root, &fs); err != nil { + results := filepath.Join(a.executor.storage.root, "teams", team.ID, "results") + if err := os.RemoveAll(results); err != nil { + t.Fatal(err) + } + if _, err := a.teams.store(&team); err == nil { + t.Fatal("missing results directory accepted") + } + if err := os.Symlink(t.TempDir(), results); err != nil { + t.Fatal(err) + } + if _, err := a.teams.store(&team); err == nil { + t.Fatal("symlink results directory accepted") + } + if err := os.Remove(results); err != nil { + t.Fatal(err) + } + if err := os.Mkdir(results, 0750); err != nil { t.Fatal(err) } - gib := (int64(fs.Blocks)*fs.Bsize + 1024*1024*1024 - 1) / (1024 * 1024 * 1024) - team.Policy.StorageGiB, team.Policy.ModelStorageGiB = gib, gib - if err := atomicJSON(filepath.Join(a.executor.storage.root, "metadata/provision-status", team.ID+".json"), map[string]any{"state": "Ready", "storage_gib": gib, "model_storage_gib": gib}); err != nil { + path := filepath.Join(a.executor.storage.root, "metadata/provision-status", team.ID+".json") + if err := atomicJSON(path, map[string]any{"state": "Ready", "storage_gib": 200, "model_storage_gib": 500}); err != nil { t.Fatal(err) } - t.Setenv("MIST_REQUIRE_TEAM_FILESYSTEM", "true") - if _, err := a.teams.store(&team); err == nil || !strings.Contains(err.Error(), "separate model filesystem") { - t.Fatalf("missing model mount accepted: %v", err) + if _, err := a.teams.store(&team); err == nil { + t.Fatal("old image layout status accepted") + } +} + +func TestDepartmentSharedStorageUsageAndAdvisoryDecrease(t *testing.T) { + a, _, teams := departmentTestApp(t) + team := teams[0] + path := filepath.Join(a.executor.storage.root, "metadata/provision-status", team.ID+".json") + status := teamStorageStatus{State: "Ready", Layout: sharedFolderLayout, DatasetUsedBytes: 123, ModelUsedBytes: 456, UsageObservedUnix: time.Now().Add(-3 * time.Minute).Unix()} + if err := atomicJSON(path, status); err != nil { + t.Fatal(err) + } + response := departmentRequest(a, "GET", "/storage", "alice", team.ID, "") + var info struct { + DatasetUsed int64 `json:"dataset_used_bytes"` + ModelUsed int64 `json:"model_used_bytes"` + Stale bool `json:"usage_stale"` + Enforced bool `json:"storage_limits_enforced"` + Capacity uint64 `json:"capacity_bytes"` + Available uint64 `json:"available_bytes"` + ModelCapacity uint64 `json:"model_capacity_bytes"` + ModelAvailable uint64 `json:"model_available_bytes"` + } + if response.Code != 200 { + t.Fatal(response.Body.String()) + } + if err := json.Unmarshal(response.Body.Bytes(), &info); err != nil { + t.Fatal(err) + } + if info.DatasetUsed != 123 || info.ModelUsed != 456 || !info.Stale || info.Enforced || info.Capacity == 0 || info.Available == 0 { + t.Fatalf("incorrect folder usage: %+v", info) + } + if info.ModelCapacity != info.Capacity || info.ModelAvailable != info.Available { + t.Fatal("compatibility fields must refer to the same shared disk") + } + policy := team.Policy + policy.StorageGiB, policy.ModelStorageGiB = 1, 1 + body, _ := json.Marshal(map[string]any{"policy": policy}) + changed := departmentRequest(a, "PATCH", "/teams/"+team.ID, "admin", "", string(body)) + if changed.Code != 200 { + t.Fatalf("allowance decrease must not require migration: %s", changed.Body) + } + upload := departmentRequest(a, "POST", "/datasets?filename=tiny.csv", "alice", team.ID, "1,2\n") + if upload.Code != 201 { + t.Fatalf("advisory change blocked upload: %s", upload.Body) } } @@ -492,3 +557,31 @@ func TestDepartmentDefaultPoolsAndLegacyPolicy(t *testing.T) { t.Fatal("negative model pool accepted") } } + +func TestTeamVolumesUseEthernetFolderExportsAndRejectStaleBindings(t *testing.T) { + a, client, teams := departmentTestApp(t) + t.Setenv("MIST_NFS_SERVER", "192.168.240.2") + t.Setenv("MIST_NFS_ROOT", "/srv/mist-data") + team := teams[0] + ctx := context.Background() + if err := a.teams.provisionNamespace(ctx, &team); err != nil { + t.Fatal(err) + } + for _, pair := range [][2]string{{"team-datasets", "datasets"}, {"team-results", "results"}} { + pv, err := client.CoreV1().PersistentVolumes().Get(ctx, team.namespace()+"-"+pair[0], metav1.GetOptions{}) + if err != nil { + t.Fatal(err) + } + if pv.Spec.NFS.Server != "192.168.240.2" || pv.Spec.NFS.Path != "/srv/mist-data/teams/"+team.ID+"/"+pair[1] { + t.Fatalf("wrong Ethernet export: %+v", pv.Spec.NFS) + } + pv.Spec.NFS.Path = "/obsolete-export" + if _, err = client.CoreV1().PersistentVolumes().Update(ctx, pv, metav1.UpdateOptions{}); err != nil { + t.Fatal(err) + } + err = a.teams.ensureVolume(ctx, team.ID, team.namespace(), pair[0], "teams/"+team.ID+"/"+pair[1], 1, false) + if err == nil { + t.Fatal("stale export binding accepted") + } + } +} diff --git a/web-interface/package-lock.json b/web-interface/package-lock.json index cf7ce4f..2176980 100644 --- a/web-interface/package-lock.json +++ b/web-interface/package-lock.json @@ -30,7 +30,7 @@ "jsdom": "^28.1.0", "prettier": "^3.8.1", "typescript": "^5.7.2", - "vite": "^7.3.1", + "vite": "^7.3.7", "vite-tsconfig-paths": "^5.1.4", "vitest": "^5.0.3" }, @@ -3043,6 +3043,20 @@ "license": "ISC", "peer": true }, + "node_modules/fsevents": { + "version": "2.3.3", + "resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.3.tgz", + "integrity": "sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw==", + "hasInstallScript": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": "^8.16.0 || ^10.6.0 || >=11.0.0" + } + }, "node_modules/gensync": { "version": "1.0.0-beta.2", "license": "MIT", @@ -3317,12 +3331,14 @@ } }, "node_modules/launch-editor": { - "version": "2.14.1", + "version": "2.14.2", + "resolved": "https://registry.npmjs.org/launch-editor/-/launch-editor-2.14.2.tgz", + "integrity": "sha512-NHJEb30MGBCkmZXLZwz25t37QR4aY03ZzPYUJXOJaOa92uWwExI9SBQUuIznBYc1xkD7cJNGX7oc3Nl3sq2kQQ==", "dev": true, "license": "MIT", "dependencies": { "picocolors": "^1.1.1", - "shell-quote": "^1.8.4" + "shell-quote": "^1.10.0" } }, "node_modules/levn": { @@ -4322,7 +4338,9 @@ "license": "MIT" }, "node_modules/vite": { - "version": "7.3.6", + "version": "7.3.7", + "resolved": "https://registry.npmjs.org/vite/-/vite-7.3.7.tgz", + "integrity": "sha512-AHEzTORMbi7/7aXp7w7KZ5+DCUdSeLsJz0grY8xHpc9DHdYb0vQu6DqJfm2tOw3gDExNgPgnJTXxM6syYFHaCA==", "license": "MIT", "dependencies": { "esbuild": "^0.27.0 || ^0.28.0", diff --git a/web-interface/package.json b/web-interface/package.json index 2475954..a2e274d 100644 --- a/web-interface/package.json +++ b/web-interface/package.json @@ -45,7 +45,7 @@ "jsdom": "^28.1.0", "prettier": "^3.8.1", "typescript": "^5.7.2", - "vite": "^7.3.1", + "vite": "^7.3.7", "vite-tsconfig-paths": "^5.1.4", "vitest": "^5.0.3" }, diff --git a/web-interface/pnpm-lock.yaml b/web-interface/pnpm-lock.yaml index 08effeb..0e4a31e 100644 --- a/web-interface/pnpm-lock.yaml +++ b/web-interface/pnpm-lock.yaml @@ -76,7 +76,7 @@ importers: specifier: ^5.7.2 version: 5.9.3 vite: - specifier: ^7.3.1 + specifier: ^7.3.7 version: 7.3.7(@types/node@22.20.5)(jiti@2.7.0)(lightningcss@1.32.0) vite-tsconfig-paths: specifier: ^5.1.4 diff --git a/web-interface/src/api.ts b/web-interface/src/api.ts index 2ed9c62..ddfeda9 100644 --- a/web-interface/src/api.ts +++ b/web-interface/src/api.ts @@ -180,6 +180,13 @@ export const storageAPI = { available_bytes: number model_capacity_bytes?: number model_available_bytes?: number + storage_mode?: 'shared-folders' + storage_limits_enforced?: boolean + dataset_used_bytes?: number + model_used_bytes?: number + usage_observed_at?: string + usage_stale?: boolean + storage_low?: boolean dataset_allocation_gib?: number model_allocation_gib?: number upload_limit_bytes: number diff --git a/web-interface/src/components/AccountPage.tsx b/web-interface/src/components/AccountPage.tsx index bd1f108..466cdd8 100644 --- a/web-interface/src/components/AccountPage.tsx +++ b/web-interface/src/components/AccountPage.tsx @@ -98,7 +98,6 @@ function Members() { [password, setPassword] = useState(''), [error, setError] = useState(''), [busy, setBusy] = useState(false) - const current = useAccount().user const [creating, setCreating] = useState(false) const [resetUser, setResetUser] = useState(null) const [search, setSearch] = useState('') diff --git a/web-interface/src/components/DatasetsPage.test.tsx b/web-interface/src/components/DatasetsPage.test.tsx new file mode 100644 index 0000000..443dd0d --- /dev/null +++ b/web-interface/src/components/DatasetsPage.test.tsx @@ -0,0 +1,58 @@ +// @vitest-environment jsdom +import { afterEach, beforeEach, expect, test, vi } from 'vitest' +import { cleanup, render, screen } from '@testing-library/react' +import { DatasetsPage } from './DatasetsPage' +import { storageAPI } from '../api' + +vi.mock('../api', () => ({ storageAPI: { list: vi.fn(), info: vi.fn() } })) +vi.mock('../teams', () => ({ + useTeam: () => ({ + selected: 'team-a', + team: { name: 'Team A', members: [] }, + }), +})) +vi.mock('../auth', () => ({ + useAccount: () => ({ user: { id: 'alice', role: 'user' } }), +})) +vi.mock('./TeamStorageBrowser', () => ({ TeamStorageBrowser: () => null })) + +beforeEach(() => { + vi.resetAllMocks() + vi.mocked(storageAPI.list).mockResolvedValue({ + datasets: [], + upload_limit_bytes: 1024, + }) + vi.mocked(storageAPI.info).mockResolvedValue({ + enabled: true, + capacity_bytes: 100 * 1024 ** 3, + available_bytes: 60 * 1024 ** 3, + upload_limit_bytes: 1024, + storage_mode: 'shared-folders', + storage_limits_enforced: false, + dataset_used_bytes: 2 * 1024 ** 3, + model_used_bytes: 3 * 1024 ** 3, + dataset_allocation_gib: 1, + model_allocation_gib: 5, + usage_stale: true, + storage_low: true, + }) +}) +afterEach(cleanup) + +test('shows team file usage independently of shared disk usage and advisory allowances', async () => { + render() + await screen.findByText(/Shared disk:/) + expect(screen.getByText('2 GiB / 1 GiB')).toBeTruthy() + expect(screen.getByText('3 GiB / 5 GiB')).toBeTruthy() + expect(screen.getByText(/Above advisory allowance/)).toBeTruthy() + expect(screen.getByText(/shared disk space is not reserved/)).toBeTruthy() + expect(screen.getByText(/Shared disk:/).textContent).toContain( + '60 GiB free of 100 GiB', + ) +}) + +test('reports stale usage and shared-disk exhaustion risk separately', async () => { + render() + await screen.findByText(/Storage usage reporting is delayed/) + expect(screen.getByText(/Shared disk space is low/)).toBeTruthy() +}) diff --git a/web-interface/src/components/DatasetsPage.tsx b/web-interface/src/components/DatasetsPage.tsx index 323b6bf..2118ef0 100644 --- a/web-interface/src/components/DatasetsPage.tsx +++ b/web-interface/src/components/DatasetsPage.tsx @@ -19,11 +19,13 @@ import { TeamStorageBrowser } from '#/components/TeamStorageBrowser.tsx' function StorageMeter({ label, capacity, - available, + used, + advisory = false, }: { label: string capacity: number - available: number + used: number + advisory?: boolean }) { return (
@@ -31,15 +33,21 @@ function StorageMeter({
- {formatBytes(available)} available + + {advisory + ? used > capacity + ? 'Above advisory allowance — writes are still allowed while disk space remains.' + : 'Advisory allowance — shared disk space is not reserved.' + : `${formatBytes(Math.max(0, capacity - used))} available`} + ) } @@ -155,21 +163,46 @@ export function DatasetsPage() {

)} {storage.data?.enabled && ( -
- - {storage.data.model_capacity_bytes !== undefined && - storage.data.model_available_bytes !== undefined && ( +
+

+ Shared disk: {formatBytes(storage.data.available_bytes)} free of{' '} + {formatBytes(storage.data.capacity_bytes)}. All teams use this disk. +

+ {selected && ( +
+ - )} -
+
+ )} + {selected && ( +

+ File usage updates approximately every 30 seconds. +

+ )} + {storage.data.storage_low && ( +

+ Shared disk space is low. Uploads and jobs may run out of space. +

+ )} + {storage.data.usage_stale && ( +

+ Storage usage reporting is delayed. The displayed usage may be out + of date. +

+ )} + )} {uploading && ( Promise }) { - const { storageBudget } = useTeam() const [policy, setPolicyState] = useState(team.policy) const [editing, setEditing] = useState(false) const signature = JSON.stringify(team.policy) @@ -42,18 +41,8 @@ function PolicyForm({ ['tenstorrent', 'Tenstorrent boards (two chips each)', 0, 4], ['concurrent', 'Simultaneous jobs', 1, 16], ['queued', 'Queued jobs', 1, 200], - [ - 'storage_gib', - 'Dataset storage (GiB)', - team.policy.storage_gib, - storageBudget, - ], - [ - 'model_storage_gib', - 'Model storage (GiB)', - team.policy.model_storage_gib, - storageBudget, - ], + ['storage_gib', 'Dataset allowance (GiB, advisory)', 1, 1048576], + ['model_storage_gib', 'Model allowance (GiB, advisory)', 1, 1048576], ['runtime_seconds', 'Maximum job runtime (seconds)', 10, 86400], ] as const return ( @@ -113,18 +102,20 @@ function PolicyForm({ setPolicy({ ...policy, [key]: Number(e.target.value) }) } /> - - setPolicy({ ...policy, [key]: Number(e.target.value) }) - } - className="mist-range w-full mt-3" - /> + {!key.includes('storage') && ( + + setPolicy({ ...policy, [key]: Number(e.target.value) }) + } + className="mist-range w-full mt-3" + /> + )} ))} @@ -148,9 +139,9 @@ function PolicyForm({ />

- Limits apply to the whole team. Dataset and model pools are separate. - Storage can be expanded; shrinking existing filesystems requires an - offline migration. + Compute limits apply to the whole team. Storage allowances are advisory: + teams share the disk, and changing an allowance does not reserve or + remove space.