Repository navigation
Migrate analysis to SmolVM and document the v0.0.1 foundation - #5
Conversation
Add the specialist runtime prerequisites, isolated analysis workspace, and report artifacts. Resolve the specialist model from the CLI, then BACKINTEL_ANALYSIS_MODEL, then ChatGPT-plan Sol with medium reasoning; preserve explicit overrides and subscription limits. Validation: staged snapshot passed 126 tests with 10 optional skips, Ruff checks, and a redacted secret scan.
Separate invalid evaluations from capability coverage, classify subscription outages as infrastructure, reject incomplete acceptance campaigns, and reserve unique local experiments. Decouple ad-hoc questions from evaluator references. Require Docker for model-generated Python, remap local sources outside workspace roots, and isolate assigned endpoints on disposable internal networks. Keep trusted host execution exclusively in repository-owned test fixtures. Validation: 146 tests passed with 11 optional skips; Docker acceptance 8 passed with the large-data test skipped. Ruff, Vulture, and Gitleaks passed.
Use pinned offline microVMs for benchmark Python, specialist analysis and artifact rendering. Preserve read-only sources, bounded resources and assigned-service access; serialize image staging to prevent concurrent cold-start corruption. Validate the scoped snapshot with 155 passing tests and 6 optional skips, including real SmolVM acceptance. Ruff, Vulture and Gitleaks pass. The qualified runtime is macOS ARM64; Docker remains for image preparation and optional source services.
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cc25047eb1
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b6da322781
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| model=env.get("BACKINTEL_MODEL") or DEFAULT_MODEL, | ||
| judge_model=env.get("BACKINTEL_JUDGE_MODEL") or DEFAULT_JUDGE_MODEL, | ||
| sandbox=env.get("BACKINTEL_SANDBOX") or "docker", | ||
| sandbox=env.get("BACKINTEL_SANDBOX") or "smolvm", |
There was a problem hiding this comment.
On a fresh checkout, following docs/benchmark.md:105-116 now builds only the Docker image and immediately runs backintel sandbox check. Because this default selects SmolVM and runtime() rejects an unprepared installation, that documented benchmark setup deterministically stops before its store and benchmark checks; add uv run python -m scripts.prepare_smolvm before the check, as the README already does.
AGENTS.md reference: AGENTS.md:L3-L4
Useful? React with 👍 / 👎.
| if not missing: | ||
| raise RuntimeError(f"SmolVM cleanup failed for {self.name}; retain its identity.") |
There was a problem hiding this comment.
When smolvm machine delete times out or returns a transient non-NOT_FOUND error, cleanup raises here after self.closed was already set at line 561. Every later close() therefore returns immediately, leaving the task-owned VM metadata and overlay behind; repeated runs can accumulate leaked VM resources, so deletion should remain retryable until it succeeds.
Useful? React with 👍 / 👎.
| self.started_at = time.monotonic() | ||
| self._watchdog = threading.Timer(p.run_timeout_s, self.cancel) |
There was a problem hiding this comment.
PythonSandbox(timeout_s=30) and run_analysis(timeout_s=...) both enter start() before any deadline is established, but started_at and the watchdog are created only after the image-lock wait, two independent 120-second CLI calls, and the readiness loop. If image staging or startup stalls, a configured run can therefore exceed its declared limit by several minutes; establish the overall deadline before startup and pass the remaining time to each setup operation.
AGENTS.md reference: AGENTS.md:L28-L30
Useful? React with 👍 / 👎.
| "--workdir", | ||
| "/workspace", | ||
| "--timeout", | ||
| f"{max(1, int(timeout))}s", |
There was a problem hiding this comment.
For PythonSandbox, the run and command limits are equal, so remaining is already fractional when execution starts and int(timeout) schedules SmolVM's deadline almost one second before the host deadline. The official SmolVM timeout report identifies exit 124 as the expected machine exec --timeout result; consequently a command reaching this earlier native limit returns as an ordinary nonzero result with timed_out=false, and the VM remains open for retries instead of being cancelled. Preserve the fractional duration or interpret the native timeout result and cancel the sandbox.
AGENTS.md reference: AGENTS.md:L28-L30
Useful? React with 👍 / 👎.
| ) | ||
| if result.returncode: | ||
| # Never echo arguments: explicitly assigned store credentials may be present. | ||
| raise RuntimeError(f"SmolVM operation failed: {result.stderr[:1000]}") |
There was a problem hiding this comment.
When machine creation, startup, or execution returns nonzero, _checked() raises a plain RuntimeError. Harness.answer() at src/backintel/bench/run.py:229-240 does not classify that type as infrastructure, so it records failure_class="agent"; score() then assigns failed agent scores instead of nulling the invalid run. A broken SmolVM runner can therefore corrupt benchmark comparisons as an apparent model regression, so raise or recognize a dedicated infrastructure exception.
AGENTS.md reference: AGENTS.md:L69-L72
Useful? React with 👍 / 👎.
| os.environ.get("BACKINTEL_SMOLVM") or RUNTIME_HOME / f"runtimes/smolvm-{VERSION}-darwin-arm64/smolvm" | ||
| ) | ||
| binary = engine.parent / "smolvm-bin" | ||
| if not engine.is_file() or not binary.is_file() or digest(binary) != BINARY_SHA256: |
There was a problem hiding this comment.
When BACKINTEL_SMOLVM names a modified launcher, or the installed launcher is corrupted, admission hashes only its sibling smolvm-bin; the unchecked engine file is then returned and executed for every sandbox command. A replacement launcher therefore passes validation whenever a valid smolvm-bin is copied beside it and can run model-generated command arguments or capture store credentials on the host, bypassing the intended SmolVM boundary. Hash the launcher as well, or execute the verified binary directly.
AGENTS.md reference: AGENTS.md:L24-L26
Useful? React with 👍 / 👎.
BackIntel's analysis specialist now executes generated code in SmolVM and requires a structured completion before exporting a report. If a model ends with prose, one bounded recovery prompt runs within the original call budget; a second incomplete response fails without publishing artifacts.
This change also brings the public README, version metadata and roadmap together around an experimental v0.0.1 baseline.
Validation:
Energy experiment
Limits: the prior four-case live run was 3/4; the selected energy rerun does not establish a new 4/4 result. This is an experimental source checkout, not a production release, unattended service or broad reliability claim. SmolVM execution is qualified on macOS ARM64; Linux CI runs the non-live checks. The expanded four-case comparison harness remains separate pending work and is not included here.
Tracking: artifact release and development/release workflow.