Skip to content

notebooks: guard nvidia-smi for CPU-only runtimes (G1.2) - #11

Open
mattdani21 wants to merge 1 commit into
mainfrom
autopilot/nvidia-smi-guard
Open

notebooks: guard nvidia-smi for CPU-only runtimes (G1.2)#11
mattdani21 wants to merge 1 commit into
mainfrom
autopilot/nvidia-smi-guard

Conversation

@mattdani21

Copy link
Copy Markdown
Owner

PR_SUMMARY — nvidia-smi guard for CPU-only runtimes

What

notebooks/colab-phase1-eval.ipynb and notebooks/phase1-pipeline-eval.ipynb called subprocess.run(['nvidia-smi'], ...) unguarded. On a runtime with no NVIDIA driver (CPU-only Colab/Kaggle session), that raises FileNotFoundError and the notebook dies at cell 0 — before any eval work.

Why

Observed in a smoke run (python 3.12.13 | scope: smoke | categories: bugfix,feature): traceback FileNotFoundError: [Errno 2] No such file or directory: 'nvidia-smi'. GPU presence should be reported and handled, not assumed.

Fix

  • New nvidia_smi(*args) helper: returns '' on missing binary / timeout instead of raising.
  • CPU-only runtime now prints nvidia-smi NOT FOUND + GPU count: 0 | total VRAM MiB: 0 + a warning directing to Runtime → Change runtime type → T4 GPU.
  • smoke scope proceeds on CPU (slow but functional — llama.cpp CPU fallback); full scope still fails loudly via the existing >=20GB VRAM assert (vram=0 → assert fires with the clear message).

How tested

  • Both notebooks patched, JSON re-validated, diff limited to cell 0 (30 insertions / 9 deletions).
  • Patched cell-0 logic executed locally on a machine without nvidia-smi (exact repro of the failure): no crash; smoke path proceeds, full path asserts with the intended message.

Numbers

  • Hardware floor (eval path): smoke eval no longer requires an NVIDIA driver to start — CPU-only runtimes can validate the env (slowly) instead of crashing. Runtime hardware-floor targets unchanged; full-scope guard intact.
  • Quality parity / swap latency / cost per task: unaffected (notebook tooling only; no benchmark/task code touched — suite, grader, and baseline runner untouched per governance §2.3).

…eNotFoundError crash (G1.2)

colab+kaggle eval notebooks called subprocess.run(['nvidia-smi']) unguarded;
a runtime with no NVIDIA driver (CPU-only Colab/Kaggle session) dies at cell 0
with FileNotFoundError. Wrap in a helper that returns '' when the binary is
absent; smoke scope now proceeds on CPU with a clear warning, full scope still
fails loudly via the existing >=20GB assert.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant