Migrate analysis to SmolVM and document the v0.0.1 foundation

AGENTS.md

@@ -1,14 +1,16 @@
-# BackIntel v2 agent instructions
+# BackIntel agent instructions
 
 Use plain, complete English. Lead with the measured result and what it means for the
 analyst's capability. Preserve required behavior and checks.
 
 ## What this is
 
-BackIntel v2 is an ambient data-analyst agent: given connected data stores and a few
-standing business problems, it answers questions on a schedule and tells a consistent story
-over time. This branch (`codex/backintel-v2`) is a fresh build; v1 lives in Git history and on
-the older branches, and nothing from it is imported.
+BackIntel is an experimental data-analyst agent, versioned from **v0.0.1** by owner decision
+on 2026-11-11. It currently answers individual questions and runs historical standing-report
+replays. An opt-in specialist now builds local report/chart/workbook bundles; its live-model
+acceptance remains open. Durable background scheduling and hosted delivery are roadmap work.
+The historical branch name (`codex/backintel-v2`) identifies this rebuild, not product stability;
+the earlier implementation remains in Git history and is not imported.
 
 The build order is deliberate (owner decision, 2026-11-09): benchmark first, agent second,
 infrastructure last. The agent's value is measured, not asserted.
@@ -20,9 +22,17 @@ infrastructure last. The agent's value is measured, not asserted.
   experiment improves the score; keep it only if the score stays better. Number checking is
   code, never another model.
 - Tools (`tools.py`): `list_tables`, `describe_table`, `run_sql` or `run_query` (read-only, 200-row cap),
-  `run_python` (Docker sandbox, no network, read-only data mount). Skills in `agent_files/skills`
+  `run_python` (SmolVM sandbox, no network, read-only data mount). Skills in `agent_files/skills`
   are loaded on demand through the Deep Agents skills middleware; `agent_files/lead.md` is the
   system prompt.
+- `analysis.py`: an opt-in, single specialist graph with a structured business brief,
+  persistent SmolVM shell/Python workspace and bounded artifact output. `analysis_sandbox.py`
+  enforces runtime limits; `artifacts.py` renders tables, charts and sheets. A clean second
+  VM renders the retained specification for delivery. Read `docs/specialist-default.md`
+  and `docs/sandbox-environment.md`.
+  Owner-selected specialist default: `chatgpt-plan/gpt-6.1-sol`, medium reasoning. The `analyze`
+  CLI resolves `--model`, then `BACKINTEL_ANALYSIS_MODEL`, then that built-in default.
+  Scripted integration proves wiring, not live model quality; automatic delegation stays off.
 - Contract (`contracts.py`): the analyst returns an `AnalystAnswer` with `evidence` (the exact
   SQL or Python that produced the number), `method` and `caveats`. Code re-executes the
   evidence; an answer that cannot be reproduced from its evidence is scored as unsupported.

docs/results-analysis-sandbox.md

@@ -0,0 +1,63 @@
+# Analysis sandbox implementation results
+
+## SmolVM migration — 2026-11-12 UTC
+
+Owner-authorized local migration is implemented on macOS ARM64. Both the benchmark's
+`run_python` and the opt-in specialist now execute in SmolVM 1.31.1; final artifact rendering
+uses a separate clean VM. `backintel/dev` in Doppler now selects `BACKINTEL_SANDBOX=smolvm`.
+GPT-6.1 Sol and its ChatGPT subscription route are unchanged. These checks were run before
+committing the migration; PR #4 and its CI results describe the earlier Docker candidate.
+
+The prepared image retains ID
+`sha256:9820b507a21f383ad0af1fac20426a6123ef718df79e35740461ab3ff3087dac`.
+Its current local archive SHA-256 is
+`d12436507cf87f1eb99cc360232e9dc9b2ac0ec8c9addf4167395cdc78ba24a5`.
+The pinned runtime distribution, extracted binary and operating constraints are recorded
+in [sandbox-environment.md](sandbox-environment.md). Docker remains necessary for preparing
+the OCI image and hosting optional database services; prepared file-based execution no
+longer performs Docker image lookups or depends on its Linux VM waking from Resource Saver.
+
+The first concurrent cold-start check exposed a separate SmolVM 1.31.1 image-cache race:
+the hardlink/copy staging fallback can truncate the shared input when machine creations
+overlap. BackIntel now holds a bounded host file lock during image staging only. A real
+regression starts two VMs concurrently from a fresh archive, verifies both return `42`,
+and verifies the archive hash is unchanged. VM execution remains concurrent. The failed
+first four-case attempt remains at `data/output/analysis-smolvm-20261112T155300Z`; its runs
+failed during infrastructure setup and made no model calls. They are not model failures.
+
+Final direct verification:
+
+- `doppler run -p backintel -c dev -- uv run backintel sandbox check`: passed; backend
+  `smolvm`, Python 3.12.16, public-network probe blocked.
+- `BACKINTEL_SMOLVM_TESTS=1 uv run pytest -q --tb=short`: **189 passed, 7 skipped** in
+  71.26 seconds. Real VM tests cover root/source write denial, non-root UID, empty effective
+  capabilities, no-new-privileges, coding recovery, file/output limits, deadlines, assigned
+  service socket isolation, SQLite reads, artifact export, concurrent startup and cleanup.
+  Optional live-source and large-input tests remain outside this run.
+- Ruff lint and format, Vulture and whitespace checks passed.
+- Four-case scripted graph integration: **4/4 passed**, no unresolved cases, **zero provider
+  calls**, 61.169 seconds summed recorded latency. Retail refunds, irregular energy and
+  service availability each produced independently checked report/chart/workbook bundles;
+  missing-cost correctly returned `cannot_answer`. Workbook cells and report values agreed.
+- No task-owned `backintel-*` machines remained after completion.
+
+The final comparison is `data/output/analysis-smolvm-20261112T155700Z/comparison.json`,
+with per-case `run.json` and retained artifacts. Source version is `4db4fdfda73698e3`.
+These four-case results use the separately pending comparison scripts and full local
+working tree; the scoped migration commit does not publish that expanded harness.
+This is scripted runtime/integration evidence, not a new LangSmith live-model comparison.
+Historical Sol 4/4 and Luna 2/4 scores still belong to their original Docker experiments.
+Memory records now report sampled VM used memory, including guest overhead; they are not
+directly comparable to Docker cgroup high-water marks. Linux/Windows SmolVM distributions,
+unseen-domain model quality and large-data acceptance are not claimed. INT-454 remains
+In Progress for its broader release/reliability scope.
+
+### Scoped commit verification
+
+The migration was staged separately from the earlier benchmark expansion and versioning
+changes. An isolated checkout of that staged snapshot passed **155 tests, 6 optional skips**
+with real SmolVM acceptance enabled, in 77.70 seconds. Ruff lint/format, Vulture and Gitleaks
+passed. Its runtime source version is `7e3758f52558caca`; the earlier 189-test count and
+four-case comparison above include the separately pending evaluation files. No runtime
+code changed between the migration checks and commit preparation. The differing source
+fingerprint includes the pending `uv.lock` version metadata excluded from this commit.

docs/roadmap.md

@@ -1,10 +1,130 @@
-# From benchmark evidence to a controlled pilot
-
-This document explains how evidence should guide BackIntel's next product decisions.
-It is a proposed decision sequence, not a second status tracker. Linear owns delivery
-status and priority, including [INT-429](https://linear.app/intelip/issue/INT-429).
-The implementation retains one Deep Agents lead, bounded read-only queries, isolated
-Python, the approved model/provider configuration, and LangSmith evaluation.
+# Capability roadmap and next development increments
+
+BackIntel starts at **v0.0.1, experimental**. Version numbers below describe acceptance
+milestones, not released capability or delivery dates. The internal
+[Linear project](https://linear.app/intelip/project/backintel-0287c22fe5a7) owns priority and
+execution status. This public plan describes the work and its completion criteria without
+requiring tracker access. Planning snapshot: 2026-11-13 UTC.
+
+## Starting point
+
+The agent can discover assigned schemas, execute bounded read-only native queries, run
+Python/shell analysis in SmolVM and return executable evidence. Its opt-in specialist can
+render reports, five chart families, CSV and XLSX through a separate clean VM. The runtime
+uses one Deep Agents graph per request, with no automatic delegation or production scheduler.
+GPT-6.1 Sol is the specialist default; the benchmark analyst remains separately configured.
+
+Its strongest demonstrated capability is **bounded business analysis**. Data engineering
+covers source access, profiling and temporary transformations, not production ETL. Data
+science covers descriptive analysis and calculation checks; forecasting, causal inference
+and general statistical-model validity remain unqualified. Native MongoDB deployments and
+vector databases are not established by the FerretDB benchmark.
+
+The controlled SmolVM specialist baseline passed 3/4 cases. The failed energy case now has
+one successful candidate rerun with verified artifact delivery, while the original failure
+remains retained. That is a targeted fix, not a new four-case candidate score or a reliability
+rate. See [current capabilities and measured outcomes](../README.md#what-has-been-measured).
+
+## Next achievable increments
+
+Each item below is small enough to define a concrete development handoff. A role names the
+work involved, not an assigned person or a new runtime sub-agent. Existing scripts and
+fixtures that are still pending publication must be packaged before another contributor can
+reproduce their experiments from a checkout.
+
+| Order | Deliverable | Completion criteria | Dependency / tracker |
+| --- | --- | --- | --- |
+| 1 | Reproducible foundation and evaluation handoff | Publish the reviewed SmolVM/completion candidate and docs; package the pending four-case runner, case definitions, tests and reference generation as a separately reviewed change; verify a clean checkout can run the offline suite without private files | Engineering + evaluation; INT-451, INT-454 |
+| 2 | Reliable finish, refusal and artifact claims | Clarify virtual skill reads versus shell paths; stop unsupported profit calculations without unnecessary report generation; never claim an unexported file exists; add regressions for these observed behaviors and retain the completion recovery/call-limit tests | Existing failure traces; INT-435, INT-454 |
+| 3 | A measured reliability improvement | Freeze model, inputs, prompt/skills, graders, candidate and finite allowance; execute the 92-response comparison described below; improve mandatory historical failures without regressing controls; assess all development and reserved cases | Readiness and frozen comparison plan; INT-430–438, parent INT-429 |
+| 4 | An unfamiliar-domain data package | Qualify three distinct non-retail sources, including terms, provenance, hashes, units, grain, nulls and joins; reserve one entire domain; define six challenge questions and one bounded discovery report per domain | Data + evaluation; INT-440, INT-441, INT-443 |
+| 5 | First drop-in dataset run | Mount one qualified development folder and its ordinary documentation through configuration; discover its shape with bounded tools; produce a supported report without notebooks, hidden references or handcrafted answer hints | Item 4; runtime + analysis; INT-442 |
+| 6 | Domain-transfer evidence | Record a first-pass baseline on development domains; make a justified general revision; freeze it before seven reserved-domain scenarios; report all 21 outcomes and independently checked discoveries by domain | Items 3–5; INT-444, parent INT-439 |
+| 7 | Real large-data acceptance | Freeze a real corpus of at least 10 GiB, plus deterministic roughly 100 MiB and 1 GiB samples; run equivalent questions at each size with correctness, memory, latency, scan/spill and graceful-limit evidence | Qualified source and compatible host/runtime; INT-446, parent INT-398 |
+| 8 | Reusable decision-report packs | Define metric grain, populations, units and required inputs; compose findings, evidence, charts and sheets for standing business questions; validate temporal consistency and supported conclusions | Reliable analysis and artifact delivery; INT-455–457 |
+
+Items 1–3 are the immediate engineering iteration. Dataset qualification and task design can
+proceed alongside them, but scored expansion waits for frozen criteria and its own authorized
+allowance. This plan does not authorize those runs or provision new compute.
+
+### Reliability comparison: what “done” means
+
+The planned **92 responses** comprise 24 original-regression responses (six cases, two
+trials per baseline/candidate arm), 34 development responses (17 cases per arm), and 34
+reserved responses (17 cases per arm). A response may require many model calls.
+
+- Keep the three historical failures and three healthy controls; require a demonstrated
+  improvement on that comparison and no material control regression.
+- Evaluate all 17 development cases and all 17 reserved variants against the frozen
+  mandatory correctness, refusal, evidence and temporal-consistency criteria.
+- Keep model/provider and input visibility comparable. A model change is a separate
+  experiment, not an unexplained part of the candidate.
+- Preserve failed and invalid attempts, token/cost accounting and the distinction between
+  infrastructure failures, incorrect answers and unfinished delivery.
+- Human calibration remains waived for the current work. Report method scores as
+  uncalibrated; do not turn that waiver into fabricated business-user acceptance.
+
+### Unfamiliar data: what “drop it in the sandbox” means
+
+Transport, building energy and weather are **candidate domains**, not already qualified
+or downloaded corpora. Kaggle pages, competition notebooks and public mirrors do not by
+themselves establish permission or independent ground truth. The data package must record
+its source rights, exact version, transformations, schema and limitations before use.
+
+The evaluated agent receives the business brief, read-only data and ordinary source
+metadata. Solutions, Kaggle notebooks, evaluator code, private references and reserved truth
+remain outside its mounts. An open-ended discovery report may contain up to five supported
+findings; fewer are acceptable when the data warrants them. Novel-sounding claims and long
+narratives do not substitute for independently verified facts.
+
+The two small synthetic energy/service fixtures do not satisfy this domain-transfer goal.
+“Cold-start” means no BackIntel-specific tuning on the held-out domain; public pretraining
+exposure cannot be excluded.
+
+### Large-data prerequisite
+
+The current SmolVM integration is qualified only on macOS ARM64. The earlier proposal to
+use the Windows workstation for heavier workloads is a host candidate, not an implemented
+SmolVM deployment there. Qualify a compatible host/runtime and available RAM, disk and spill
+capacity before the full run. A 4 GiB VM profile, generated TPC-H SF1 or a small replicated
+file does not fulfill the real 10 GiB target. Do not close this milestone from a sample alone.
+
+## Version acceptance map
+
+| Target | Capability delta | Required exit evidence |
+| --- | --- | --- |
+| v0.0.1 | Measured analytical foundation | Reproducible published source; reliable terminal behavior; frozen baseline/candidate reliability results and documented limitations |
+| v0.0.2 | Unfamiliar-domain analysis | Three qualified domains, at least 21 scenarios, separate development/reserved results and independently supported findings |
+| v0.0.3 | Large-data sandbox | Real 10 GiB acceptance with staged scale checks, bounded compute and graceful exhaustion |
+| v0.0.4 | Verified visual artifacts | Reports, charts and workbooks agree with independently checked tables; delivery and refusal semantics hold across acceptance tasks |
+| v0.0.5 | Business-analysis report packs | Reusable metric definitions and standing reports pass factual, evidence and temporal-consistency checks |
+| v0.1.0 | Controlled business-analysis pilot | One named workflow, approved read-only data contract, representative development/reserved cases and an explicit pilot-entry decision |
+| v0.2.0 | Durable background execution | Persisted jobs, schedule/data-ready triggers, cancellation, deadlines and bounded retries survive whole-job failure scenarios |
+| v0.3.0 | Operational monitoring | Qualified connectors, freshness/missing-data handling, measured alert quality, deduplication and authorized delivery |
+| v0.4.0 | Contextual/statistical analysis | Document retrieval and statistical/forecasting workflows improve held-out tasks with honest uncertainty; vector retrieval is optional |
+| v0.5.0 | Managed background-intelligence beta | User controls, run history, artifact access and spending limits; sustained-operation evidence; bounded improvement proposals |
+| v0.9.0 | Release candidate | Measured beta gaps resolved, operational/support boundaries defined, reproducible candidate and rollback plan |
+| v1.0.0 | Supported background-analysis service | A declared workload and connector matrix, supported artifact delivery, operational objectives and verified recovery/rollback |
+
+Renderer and artifact-contract components already exist ahead of their numbered milestone.
+Implementation progress does not imply that the corresponding release criteria have passed.
+Fine-tuning is an optional later experiment only if approved data and held-out results show
+value beyond prompt/tool improvements. LangSmith provides measurement and comparison; it
+does not automatically train or promote a model.
+
+## Version development loop
+
+1. Select the capability and its Linear acceptance issue; state the baseline, boundaries,
+   mandatory/regression/reserved cases, candidate owner and finite experiment allowance.
+2. Implement the smallest justified change; review the actual diff and run relevant code,
+   runtime and artifact checks.
+3. Compare frozen baseline and candidate in LangSmith. Diagnose the failed layer; retain
+   the attempt and revise only the responsible behavior. Never rerun until a lucky pass.
+4. Review the exact candidate, hosted CI and evaluation evidence. Record what improved,
+   what regressed, usage and remaining uncertainty.
+5. Publish a version only through its authorized release process, retain rollback identity,
+   and feed observed failures into the next development cycle. Green CI alone is not
+   analytical acceptance; `accept_for_review` is not deployment authority.
 
 ## What the current evidence establishes
 
@@ -88,9 +208,9 @@ from the current controlled cases.
 
 An improvement-loop result of `accept_for_review` remains a review recommendation. It does
 not authorize a commit, merge, deployment, purchase, new provider or expanded data access.
-Commit/push authority for the current implementation comes separately from the recorded
-owner approval; the approved experiment allowance is $5 total. Deployment and purchases
-remain outside that approval.
+Each execution campaign needs its own recorded authorization and allowance. Historical
+allowances do not fund new domain/scale experiments. Deployment and purchases remain
+separately authorized decisions.
 Development subagents coordinate engineering; they do not imply runtime delegation inside
 the analyst. Scheduling, customer integrations and production infrastructure remain separate
 proposed work until the evidence and owner authorization support them.