diff --git a/AGENTS.md b/AGENTS.md index 7bfde7a..27dbe01 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,14 +1,16 @@ -# BackIntel v2 agent instructions +# BackIntel agent instructions Use plain, complete English. Lead with the measured result and what it means for the analyst's capability. Preserve required behavior and checks. ## What this is -BackIntel v2 is an ambient data-analyst agent: given connected data stores and a few -standing business problems, it answers questions on a schedule and tells a consistent story -over time. This branch (`codex/backintel-v2`) is a fresh build; v1 lives in Git history and on -the older branches, and nothing from it is imported. +BackIntel is an experimental data-analyst agent, versioned from **v0.0.1** by owner decision +on 2026-11-11. It currently answers individual questions and runs historical standing-report +replays. An opt-in specialist now builds local report/chart/workbook bundles; its live-model +acceptance remains open. Durable background scheduling and hosted delivery are roadmap work. +The historical branch name (`codex/backintel-v2`) identifies this rebuild, not product stability; +the earlier implementation remains in Git history and is not imported. The build order is deliberate (owner decision, 2026-11-09): benchmark first, agent second, infrastructure last. The agent's value is measured, not asserted. @@ -20,9 +22,17 @@ infrastructure last. The agent's value is measured, not asserted. experiment improves the score; keep it only if the score stays better. Number checking is code, never another model. - Tools (`tools.py`): `list_tables`, `describe_table`, `run_sql` or `run_query` (read-only, 200-row cap), - `run_python` (Docker sandbox, no network, read-only data mount). Skills in `agent_files/skills` + `run_python` (SmolVM sandbox, no network, read-only data mount). Skills in `agent_files/skills` are loaded on demand through the Deep Agents skills middleware; `agent_files/lead.md` is the system prompt. +- `analysis.py`: an opt-in, single specialist graph with a structured business brief, + persistent SmolVM shell/Python workspace and bounded artifact output. `analysis_sandbox.py` + enforces runtime limits; `artifacts.py` renders tables, charts and sheets. A clean second + VM renders the retained specification for delivery. Read `docs/specialist-default.md` + and `docs/sandbox-environment.md`. + Owner-selected specialist default: `chatgpt-plan/gpt-6.1-sol`, medium reasoning. The `analyze` + CLI resolves `--model`, then `BACKINTEL_ANALYSIS_MODEL`, then that built-in default. + Scripted integration proves wiring, not live model quality; automatic delegation stays off. - Contract (`contracts.py`): the analyst returns an `AnalystAnswer` with `evidence` (the exact SQL or Python that produced the number), `method` and `caveats`. Code re-executes the evidence; an answer that cannot be reproduced from its evidence is scored as unsupported. diff --git a/README.md b/README.md index 7975e55..1d5675f 100644 --- a/README.md +++ b/README.md @@ -1,92 +1,312 @@ -# BackIntel v2 +# BackIntel -An ambient data-analyst agent built on the Deep Agents SDK, benchmarked before it is -productised. One lead agent, no sub-agents, native read-only queries, schema discovery, sandboxed -Python and storage skills, and an answer contract whose numbers are re-checked by code. +**Business analysis from assigned data, with executable evidence and useful deliverables.** -## Quick start +BackIntel is an experimental data-analyst agent built with the LangChain Deep Agents SDK. +It discovers schemas, queries read-only sources, runs Python in SmolVM, and produces +structured answers or reports with charts, CSV tables and Excel workbooks. It is designed +for developers and analysts building reproducible business and operational analysis. + +**Status: v0.0.1, experimental.** Use a source checkout. This is not a supported production +service or an unattended 24/7 analyst. The historical `codex/backintel-v2` branch name is +not a product version. Code is available under the [MIT license](LICENSE). + +[Get started](#get-started) · [Capabilities](#capabilities-today) · +[Architecture](#how-it-works) · [Evaluation](#what-has-been-measured) · +[Roadmap](docs/roadmap.md) · [Contributing](#contributing) + +## Capabilities today + +| Area | Implemented behavior | Current boundary | +| --- | --- | --- | +| Business analysis | Filters, aggregates, joins, rankings, weighted metrics, time-window calculations and explicit missing-data refusals | Evaluated on bounded historical and synthetic tasks; arbitrary-domain accuracy is not established | +| Data engineering | Schema discovery, bounded profiling, read-only source queries and transformations inside a run | No production ETL orchestration, source writes or customer connector service | +| Data science | Descriptive analysis and executable calculation checks using Python/SQL | Forecasting, causal inference and trained predictive workflows are not validated capabilities | +| Reporting | HTML reports; bar, line, scatter, histogram and heatmap charts; PNG/SVG; offline interactive HTML; CSV and XLSX | Local file bundles with bounded tables; no hosted dashboard or automated delivery service | +| Repeated analysis | Standing questions and historical snapshot replays that hide future observations | Replayed invocations, not a durable scheduler | +| Improvement | Versioned cases, reproducible evidence, LangSmith traces/scores and supervised candidate comparisons | No automatic weight training, deployment or self-promotion | + +Two entry points share the same source boundaries: + +- **`ask`** runs the benchmark analyst and returns a structured answer with evidence, method + and caveats. Its default is `openrouter/openai/gpt-6-luna`. +- **`analyze`** runs an opt-in coding specialist with a persistent per-run workspace and + artifact export. Its default is `chatgpt-plan/gpt-6.1-sol` with medium reasoning. + +Each entry point runs one Deep Agents graph. The analyst does not automatically delegate +to the specialist, and runtime sub-agents are disabled. + +### Data sources + +| Store | Native access | Evidence and limits | +| --- | --- | --- | +| SQLite | SQL over a local database | Historical benchmark runs and the controlled specialist tasks | +| PostgreSQL | Native read-only SQL | Historical single-store and PostgreSQL/SQLite cross-store runs | +| FerretDB | MongoDB-compatible aggregation through PyMongo | Tested against FerretDB; this does not qualify arbitrary MongoDB deployments. Its benchmark backend is frozen to enforce read-only access | +| DuckDB / Parquet | SQL over local Parquet | Historical benchmark and generated TPC-H SF1 workloads | +| Iceberg / object storage | DuckDB over Iceberg metadata and Parquet on Garage S3 | Historical native-store benchmark; not every S3 provider or Iceberg deployment | +| ClickHouse | Native read-only SQL | Historical native-store benchmark | +| CSV and mixed CSV/JSON | Bounded file discovery and Python | CSV benchmark plus a small mixed-format probe; universal folder ingestion is planned | +| Multiple stores | Explicitly assigned store combinations | Six historical PostgreSQL/SQLite cases | +| Vector databases | Not implemented | Optional document retrieval is roadmap work | + +A connector's existence is distinct from a successful model evaluation. These data sources +have been used for testing; BackIntel has not trained or fine-tuned its models on them. + +## Get started + +### Requirements + +- Python **3.12** and [uv](https://docs.astral.sh/uv/). Package metadata also permits Python + 3.13; 3.12 is the documented, CI-tested path. +- **macOS on Apple Silicon** for the qualified SmolVM execution path. Linux and Windows + runtime support has not been qualified. Python-only CI runs on Linux. +- Docker for the initial sandbox image build/export and optional database services. + Prepared SQLite/file analysis runs in SmolVM without Docker execution. +- For live analysis, an authorized ChatGPT subscription connection or a configured + supported OpenRouter model. No model account is needed for the offline checks below. + +### 1. Check out and verify the project + +Until a versioned release is published, use the development branch: + +```bash +git clone --branch codex/backintel-v2 https://github.com/IntelIP/BackIntel.git +cd BackIntel +uv sync --frozen --python 3.12 +uv run backintel --help +uv run pytest -q +``` + +The standard tests make no model-provider calls. Optional rendering, VM and live-service +checks are skipped when their dependencies or opt-in flags are absent. + +Prepare the controlled development fixtures without inference or source downloads: + +```bash +uv run backintel acceptance build --split development +``` + +Expected result: a manifest for **17 development cases** with data and reference hashes. +This prepares an evaluation dataset; it does not demonstrate agent performance. + +### 2. Prepare SmolVM + +```bash +docker build -f Dockerfile.sandbox -t backintel-sandbox:py312 . +uv run python -m scripts.prepare_smolvm +uv run backintel sandbox check +``` + +Preparation pins the SmolVM distribution and exported image. The check runs Python in a +VM and should report a successful exit with public networking blocked. See +[sandbox setup, packages and resource profiles](docs/sandbox-environment.md). + +### 3. Generate a report + +Download and build the historical Online Retail II test source, then connect your own +ChatGPT account and run the supplied example brief: ```bash -uv sync -doppler run -p backintel -c dev -- uv run backintel data build all # download + canonical + SQLite -doppler run -p backintel -c dev -- uv run backintel oracle build all # reference answers -docker build -f Dockerfile.sandbox -t backintel-sandbox:py312 . # the Python sandbox -doppler run -p backintel -c dev -- uv run backintel sandbox check -doppler run -p backintel -c dev -- uv run backintel ask --store sqlite:olist "How many orders were delivered?" -doppler run -p backintel -c dev -- uv run backintel bench run --dataset olist --levels 1,2,3 -doppler run -p backintel -c dev -- uv run backintel bench scorecard +uv run backintel data build online_retail_ii +uv run backintel chatgpt login +uv run backintel chatgpt models +uv run backintel analyze \ + --store sqlite:online_retail_ii \ + --brief examples/retail-brief.json \ + --output data/output/retail-report \ + --max-model-calls 24 --max-tool-calls 50 --timeout 900 ``` -Environment (from Doppler): `OPENROUTER_API_KEY`, `LANGSMITH_API_KEY`, `LANGSMITH_PROJECT`, -`LANGSMITH_TRACING`, optional `BACKINTEL_MODEL`, `BACKINTEL_JUDGE_MODEL`, -`BACKINTEL_SANDBOX` (`docker` only), `BACKINTEL_DATA_DIR`. Kaggle downloads read -`~/.kaggle/access_token` or `KAGGLE_API_TOKEN`. - -## Datasets - -| Dataset | Source | Licence | Used for | -|---|---|---|---| -| Online Retail II | UCI Machine Learning Repository (id 502) | CC BY 4.0 | single-table and messy-data levels | -| Brazilian E-Commerce (Olist) | Kaggle `olistbr/brazilian-ecommerce` | CC BY-NC-SA 4.0 | joins and messy-data levels | - -Raw files never enter Git. `backintel data build` downloads them, verifies the pinned SHA-256, -writes typed Parquet copies under `data/canonical/` and loads SQLite stores under `data/stores/`. - -## Benchmark - -Seven levels; all have scenarios, stores and scoring: - -1. **Single table**: filters, aggregates, rankings with stated conventions. -2. **Joins / multi-step**: three-table joins, ratios, identity resolution. -3. **Messy data**: duplicated exports, cancellations as negative quantities, null ids, - non-product codes, untranslated categories, grain mismatches, and questions the data - cannot answer (the right answer is to decline). -4. **Cross-store**: the same dataset split across PostgreSQL and SQLite; answers need both. -5. **Standing problems**: one problem, several facets, answered as a report that tells a story. -6. **Ambient replay**: monthly delivery, quarterly repeat-purchasing, and weekly category/seller questions against - snapshots that hide future observations, with a - planted step change (declared in `bench/replay.py`) among the data's natural variation. -7. **Scale**: TPC-H generated at a chosen scale factor and queried as a Parquet lake. - -Stores: `sqlite`, `postgres`, `ferretdb`, `lake-iceberg` (Iceberg on Garage S3 through DuckDB), -`clickhouse`, `files` (CSV), `files-mixed` (CSV/JSON), `lake-parquet` (local-file baseline), and -`multi` (several at once). Build them with `backintel data build all`, -`backintel split build`, `backintel replay build`, `backintel tpch generate --sf 1`. - -Scores, all recorded as LangSmith feedback and in `results/`: - -- `correct`: answer matches the polars-computed reference within the scenario's tolerance. -- `evidence_reproduces`: the cited SQL/Python is re-executed read-only and yields the answer. -- `abstention_correct`: answered when answerable, declined when not. -- `judge_method`: a cross-family LLM rubric on conventions, method fit, overreach, caveats. -- `fully_supported`: correct + evidence + abstention, the headline number. - -See [the capability report](docs/results.md) for the measured outcome and limitations, -[the scorecard](results/scorecard.md) for all experiments, and [the benchmark specification](docs/benchmark.md) -for exact scoring and reproduction commands. `doppler run -p backintel -c dev -- -scripts/prepare_testbed.sh` prepares every local store; `uv run python scripts/verify_testbed.py` -checks both datasets through each native path without model calls. `scripts/run_matrix.sh` -runs the full matrix with the configured paid analyst and judge. The measured matrix is -listed in the report; the three-question mixed-file probe is explicitly a probe. - -## Reliability and supervised improvement - -The [improvement goal](docs/improvement-goal.md) defines the additional acceptance work. -The [data-quality report](docs/data-quality.md) explains the source limitations and independent -reference checks. The [acceptance specification](docs/acceptance-spec.md) adds 17 development -cases and 17 reserved variants for data sufficiency, report consistency and changes over time. -These controlled cases supplement the original benchmark; their existence is not a measured -improvement in the analyst. - -Use the [improvement workflow](docs/improvement-loop.md) for bounded model calls, frozen -baseline/candidate comparisons, human calibration and the coding worker's restricted inputs. -`uv run backintel acceptance build --split development` prepares cases without model calls. -Paid trials require an explicit allowance. A passing comparison produces a candidate for -review; it does not publish or deploy it. The [roadmap](docs/roadmap.md) explains how this -evidence informs a controlled pilot and the later scheduling and integration work. -Completed LangSmith trials, costs, rejection decisions and invalid provider attempts are -recorded in the [improvement pilot report](results/improvement-pilots.md). Human calibration -was waived by the owner; method-judge scores remain uncalibrated. - -Python-only checks run in GitHub Actions. Publication waits for owner approval under -AGENTS.md; no production service is deployed by this build. +Confirm that the account's model catalog includes `gpt-6.1-sol`. `analyze` also accepts +`--model` and `--reasoning` overrides. Source acquisition may use the network; the analytical +VM cannot browse the internet or install packages. Reuse `chatgpt status` for an existing +connection rather than starting another login. + +A successful run writes: + +```text +data/output/retail-report/ +├── run.json # completion, transcript, usage and runtime identity +└── artifacts/report/ + ├── report.html # findings, tables, evidence and caveats + ├── report.json # retained report specification + ├── supporting.xlsx + ├── tables/*.csv + ├── charts/*.{html,png,svg} + └── manifest.json # delivered-file inventory and hashes +``` + +Use a **new output directory** for every run. A valid refusal explains the missing inputs; +it does not export a substitute report. A failed or unfinished analysis is not a successful +delivery. The example's figures are computed from the downloaded source, not hardcoded. + +### Configuration and credentials + +BackIntel reads environment variables; it does not load `.env` files. Supply credentials +through your secret manager or shell environment. IntelIP development uses Doppler, but +access to IntelIP's Doppler project is **not required** by the application. + +| Setting | Purpose | +| --- | --- | +| `BACKINTEL_ANALYSIS_MODEL` | Override the specialist default; explicit `--model` takes precedence | +| `BACKINTEL_MODEL` | Benchmark analyst model | +| `BACKINTEL_JUDGE_MODEL` | Method judge; default `openrouter/anthropic/claude-haiku-5.5` | +| `OPENROUTER_API_KEY` | Required when using the OpenRouter route | +| `LANGSMITH_API_KEY`, `LANGSMITH_PROJECT`, `LANGSMITH_TRACING` | Optional tracing and experiment configuration | +| `BACKINTEL_DATA_DIR` | Data directory; defaults to the checkout's `data/` | +| `BACKINTEL_SANDBOX` | `smolvm`; other execution backends are rejected | + +ChatGPT credentials stay in BackIntel's private local account cache, outside the source tree +and VM mounts. Subscription usage reports calls and tokens; it does not provide a +per-request API dollar charge. There is no paid-API fallback. An OpenRouter specialist run +requires both `--max-cost-usd` and `--reservation-usd`; these are application admission +limits, not a provider-enforced billing cap. See [model selection and usage](docs/specialist-default.md). + +Olist acquisition additionally needs your own Kaggle access. Dataset permissions are +separate from this repository's code license; see the [benchmark specification](docs/benchmark.md). + +## How it works + +```mermaid +flowchart LR + Brief[Business brief] --> Agent[One Deep Agents graph] + Agent --> Query[Read-only store tools] + Query --> Data[Assigned sources] + Agent --> VM[SmolVM shell and Python] + VM --> Data + Agent --> Check[Structured completion and reproduced tables] + Check --> Render[Clean renderer VM] + Render --> Files[Report, charts, CSV and XLSX] +``` + +The host supplies business context, source access, model choice and limits. Store tools +execute native bounded reads. Generated shell/Python code runs inside a per-run VM with +read-only sources, bounded writable directories and no public network. Assigned database +services use restricted endpoints. Model inference and optional LangSmith uploads occur +on the host, outside the VM; the model receives the brief and tool results. + +For reports, the host retains the specification and re-executes cited calculations to +reconcile its tables. Valid structured completion is required before a separate clean VM +renders the delivered files. One prose-only stop can receive a recovery prompt within the +original limits; it is never silently converted into success. + +The standard specialist profile uses **2 CPUs and 2 GiB RAM**, with separate workspace, +artifact and temporary-file quotas. A larger profile exists, but is not evidence of real +large-data reliability. Aggregate capacity reservation across concurrent runs is not implemented. + +### Prompts, skills and tools + +- System prompts: [benchmark analyst](src/backintel/agent_files/lead.md) and + [analysis specialist](src/backintel/agent_files/analysis.md). +- Runtime skills: [analytical methods and source-specific guidance](src/backintel/agent_files/skills). + The coding assistant's installed skills are not automatically available inside BackIntel. +- Store tools: `list_tables`, `describe_table`, `run_sql` / `run_query` and `run_python`. +- Specialist tools additionally include `run_shell`, `write_analysis_file`, + `read_analysis_file`, `render_report` and `inspect_chart`. +- Virtual documentation tools: `ls`, `read_file`, `glob` and `grep`. Virtual `/skills/` + paths are distinct from the VM filesystem; use `read_file` to read them. +- Output contract: [artifact specification](src/backintel/agent_files/docs/artifact-contract.md). + +The agent may revise its programs and reports inside a run. It cannot change its governing +prompt, model weights, permissions or release; write to source databases; browse for extra +facts; or deploy itself. Installed scientific libraries do not establish validated forecasting +or causal-analysis capabilities. + +## What has been measured + +These are versioned development observations, not a production accuracy guarantee. + +| Experiment | Observed result | Scope | +| --- | --- | --- | +| Historical Olist storage matrix | 143/144 supported responses | GPT-6 Luna; 24 questions repeated across six stores on the earlier runtime | +| Historical UCI storage runs | 80/80 | 20 questions on four stores; repetitions are not independent domains | +| Historical standing reports and replay | 15/17 | Three standing reports plus 14 historical replay cases; no live scheduler | +| Historical TPC-H SF1 | 6/6 | Generated analytical workload; not real 10 GiB data or an official performance result | +| Controlled Sol specialist on SmolVM | 3/4 | Retail refunds, missing-cost refusal, energy coverage and service availability; one trial each | +| Energy after completion recovery fix | 1/1 | Targeted rerun delivered the correct report, heatmap, CSVs and workbook; other cases were not rerun on that candidate | + +The specialist cases use **three small synthetic SQLite fixtures**, with sales reused for +the missing-cost question. The energy failure had correct intermediate numbers but no +structured completion or exported bundle. The fix's live rerun completed directly without +needing recovery; a deterministic SmolVM regression separately exercised recovery and export. +The 3/4 baseline remains unchanged: a targeted success is not a new 4/4 experiment. + +Historical results and failures are in the [capability report](docs/results.md) and +[scorecard](results/scorecard.md). The [SmolVM baseline](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/datasets/d2e934c2-77e8-4926-93b3-0bcd644b200d/compare?selectedSessions=f667f6fc-4e8d-4bba-bcbf-fcde7d808d40) +and [energy rerun](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/datasets/d2e934c2-77e8-4926-93b3-0bcd644b200d/compare?selectedSessions=ff29645c-8e96-4cd8-875c-302ae006acf8) +are separate LangSmith experiments; workspace access may be required. Their expanded +four-case runner remains pending publication, so it is not part of the checkout quickstart. + +### Evaluation and data quality + +The original suite covers seven levels: single-table analysis, joins, messy/unanswerable +data, cross-store queries, standing reports, historical replay and generated scale. +Independent code computes numerical references. Evidence is re-executed; a separate model +judge evaluates method and caveats where configured. LangSmith stores experiments, traces +and feedback—it does not automatically fine-tune the model. + +The 34 additional controlled acceptance cases comprise **17 development and 17 reserved +variants**. Fixture readiness is not a passing model result. References, solutions and +reserved truth stay outside the analyst's accessible data. Source hashes and frozen +baseline/candidate comparisons preserve provenance; failures are retained. + +Data coverage includes historical [Online Retail II](https://archive.ics.uci.edu/dataset/502/online+retail+ii), +[Olist](https://www.kaggle.com/datasets/olistbr/brazilian-ecommerce), generated TPC-H and the +controlled fixtures. Public data may have appeared in model pretraining; the benchmark does +not establish training-data novelty. Read the [data-quality assessment](docs/data-quality.md), +[acceptance specification](docs/acceptance-spec.md) and [improvement workflow](docs/improvement-loop.md). +Method-judge scores have not been calibrated against human labels. + +## Roadmap + +These are acceptance milestones, not shipped versions or promised dates. Some later-stage +components already exist; a milestone closes only when its evaluation and delivery criteria pass. + +| Target | Capability to establish | +| --- | --- | +| **v0.0.1** | Reproducible foundation, reliable completion/refusal, executable evidence and a frozen baseline/candidate comparison | +| **v0.0.2** | Drop-in analysis across three unfamiliar domains, with one entire domain reserved from tuning | +| **v0.0.3** | Real large-data analysis at staged sizes up to at least 10 GiB on a qualified host | +| **v0.0.4** | Consistent, verified report/chart/workbook delivery across the acceptance suite | +| **v0.0.5** | Reusable business metrics and decision-report packs | +| **v0.1.0** | One controlled customer pilot with an explicit data and workload contract | +| **v0.2.0** | Durable scheduled jobs, cancellation, bounded retries and recovery | +| **v0.3.0** | Qualified operational connectors, change detection and deduplicated alerts | +| **v0.4.0** | Evaluated document context and statistical/forecasting workflows | +| **v0.5.0** | Managed background-analysis beta with user controls, history and spending limits | +| **v0.9.0 → v1.0.0** | Operational hardening, reversible releases and a supported, bounded service | + +The [detailed roadmap](docs/roadmap.md) gives the next implementable tasks, dependencies and +completion criteria. Linear remains the internal delivery tracker; the public plan is readable +without access to it. No production hosting or continuous background operation is available today. + +## Contributing + +Start with the [roadmap](docs/roadmap.md), [benchmark specification](docs/benchmark.md) and +[repository instructions](AGENTS.md). Propose a bounded capability or reproduce a concrete +failure before changing models, infrastructure or evaluation criteria. + +```bash +uv run ruff check src tests scripts +uv run ruff format --check src tests scripts +uv run vulture src --min-confidence 100 +uv run pytest -q +# Optional: prepared macOS ARM64 SmolVM runtime required; no model calls +BACKINTEL_SMOLVM_TESTS=1 uv run pytest tests/test_analysis_completion.py -q +``` + +Keep changes focused, add a regression for non-trivial behavior, and explain what changed, +what was measured and what remains unsupported. Preserve frozen benchmark truth and earlier +failed attempts. Do not commit credentials, account caches, downloaded datasets or raw traces. +Hosted CI checks Python, formatting, dead code and secrets; it does not run live model experiments. + +Report reproducible bugs and proposals through [GitHub Issues](https://github.com/IntelIP/BackIntel/issues). +Include the source revision, platform, command and sanitized failure details. Keep private +data and credentials out of public reports. Model runs, infrastructure changes and publication +require the applicable project authorization. + +## License + +BackIntel's code is [MIT licensed](LICENSE), copyright IntelIP. Dataset and model/provider +terms are separate; the repository's license does not relicense third-party data or services. diff --git a/docs/results-analysis-sandbox.md b/docs/results-analysis-sandbox.md new file mode 100644 index 0000000..721ece1 --- /dev/null +++ b/docs/results-analysis-sandbox.md @@ -0,0 +1,63 @@ +# Analysis sandbox implementation results + +## SmolVM migration — 2026-11-12 UTC + +Owner-authorized local migration is implemented on macOS ARM64. Both the benchmark's +`run_python` and the opt-in specialist now execute in SmolVM 1.31.1; final artifact rendering +uses a separate clean VM. `backintel/dev` in Doppler now selects `BACKINTEL_SANDBOX=smolvm`. +GPT-6.1 Sol and its ChatGPT subscription route are unchanged. These checks were run before +committing the migration; PR #4 and its CI results describe the earlier Docker candidate. + +The prepared image retains ID +`sha256:9820b507a21f383ad0af1fac20426a6123ef718df79e35740461ab3ff3087dac`. +Its current local archive SHA-256 is +`d12436507cf87f1eb99cc360232e9dc9b2ac0ec8c9addf4167395cdc78ba24a5`. +The pinned runtime distribution, extracted binary and operating constraints are recorded +in [sandbox-environment.md](sandbox-environment.md). Docker remains necessary for preparing +the OCI image and hosting optional database services; prepared file-based execution no +longer performs Docker image lookups or depends on its Linux VM waking from Resource Saver. + +The first concurrent cold-start check exposed a separate SmolVM 1.31.1 image-cache race: +the hardlink/copy staging fallback can truncate the shared input when machine creations +overlap. BackIntel now holds a bounded host file lock during image staging only. A real +regression starts two VMs concurrently from a fresh archive, verifies both return `42`, +and verifies the archive hash is unchanged. VM execution remains concurrent. The failed +first four-case attempt remains at `data/output/analysis-smolvm-20261112T155300Z`; its runs +failed during infrastructure setup and made no model calls. They are not model failures. + +Final direct verification: + +- `doppler run -p backintel -c dev -- uv run backintel sandbox check`: passed; backend + `smolvm`, Python 3.12.16, public-network probe blocked. +- `BACKINTEL_SMOLVM_TESTS=1 uv run pytest -q --tb=short`: **189 passed, 7 skipped** in + 71.26 seconds. Real VM tests cover root/source write denial, non-root UID, empty effective + capabilities, no-new-privileges, coding recovery, file/output limits, deadlines, assigned + service socket isolation, SQLite reads, artifact export, concurrent startup and cleanup. + Optional live-source and large-input tests remain outside this run. +- Ruff lint and format, Vulture and whitespace checks passed. +- Four-case scripted graph integration: **4/4 passed**, no unresolved cases, **zero provider + calls**, 61.169 seconds summed recorded latency. Retail refunds, irregular energy and + service availability each produced independently checked report/chart/workbook bundles; + missing-cost correctly returned `cannot_answer`. Workbook cells and report values agreed. +- No task-owned `backintel-*` machines remained after completion. + +The final comparison is `data/output/analysis-smolvm-20261112T155700Z/comparison.json`, +with per-case `run.json` and retained artifacts. Source version is `4db4fdfda73698e3`. +These four-case results use the separately pending comparison scripts and full local +working tree; the scoped migration commit does not publish that expanded harness. +This is scripted runtime/integration evidence, not a new LangSmith live-model comparison. +Historical Sol 4/4 and Luna 2/4 scores still belong to their original Docker experiments. +Memory records now report sampled VM used memory, including guest overhead; they are not +directly comparable to Docker cgroup high-water marks. Linux/Windows SmolVM distributions, +unseen-domain model quality and large-data acceptance are not claimed. INT-454 remains +In Progress for its broader release/reliability scope. + +### Scoped commit verification + +The migration was staged separately from the earlier benchmark expansion and versioning +changes. An isolated checkout of that staged snapshot passed **155 tests, 6 optional skips** +with real SmolVM acceptance enabled, in 77.70 seconds. Ruff lint/format, Vulture and Gitleaks +passed. Its runtime source version is `7e3758f52558caca`; the earlier 189-test count and +four-case comparison above include the separately pending evaluation files. No runtime +code changed between the migration checks and commit preparation. The differing source +fingerprint includes the pending `uv.lock` version metadata excluded from this commit. diff --git a/docs/roadmap.md b/docs/roadmap.md index 964af4d..3a77f1b 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -1,10 +1,130 @@ -# From benchmark evidence to a controlled pilot - -This document explains how evidence should guide BackIntel's next product decisions. -It is a proposed decision sequence, not a second status tracker. Linear owns delivery -status and priority, including [INT-429](https://linear.app/intelip/issue/INT-429). -The implementation retains one Deep Agents lead, bounded read-only queries, isolated -Python, the approved model/provider configuration, and LangSmith evaluation. +# Capability roadmap and next development increments + +BackIntel starts at **v0.0.1, experimental**. Version numbers below describe acceptance +milestones, not released capability or delivery dates. The internal +[Linear project](https://linear.app/intelip/project/backintel-0287c22fe5a7) owns priority and +execution status. This public plan describes the work and its completion criteria without +requiring tracker access. Planning snapshot: 2026-11-13 UTC. + +## Starting point + +The agent can discover assigned schemas, execute bounded read-only native queries, run +Python/shell analysis in SmolVM and return executable evidence. Its opt-in specialist can +render reports, five chart families, CSV and XLSX through a separate clean VM. The runtime +uses one Deep Agents graph per request, with no automatic delegation or production scheduler. +GPT-6.1 Sol is the specialist default; the benchmark analyst remains separately configured. + +Its strongest demonstrated capability is **bounded business analysis**. Data engineering +covers source access, profiling and temporary transformations, not production ETL. Data +science covers descriptive analysis and calculation checks; forecasting, causal inference +and general statistical-model validity remain unqualified. Native MongoDB deployments and +vector databases are not established by the FerretDB benchmark. + +The controlled SmolVM specialist baseline passed 3/4 cases. The failed energy case now has +one successful candidate rerun with verified artifact delivery, while the original failure +remains retained. That is a targeted fix, not a new four-case candidate score or a reliability +rate. See [current capabilities and measured outcomes](../README.md#what-has-been-measured). + +## Next achievable increments + +Each item below is small enough to define a concrete development handoff. A role names the +work involved, not an assigned person or a new runtime sub-agent. Existing scripts and +fixtures that are still pending publication must be packaged before another contributor can +reproduce their experiments from a checkout. + +| Order | Deliverable | Completion criteria | Dependency / tracker | +| --- | --- | --- | --- | +| 1 | Reproducible foundation and evaluation handoff | Publish the reviewed SmolVM/completion candidate and docs; package the pending four-case runner, case definitions, tests and reference generation as a separately reviewed change; verify a clean checkout can run the offline suite without private files | Engineering + evaluation; INT-451, INT-454 | +| 2 | Reliable finish, refusal and artifact claims | Clarify virtual skill reads versus shell paths; stop unsupported profit calculations without unnecessary report generation; never claim an unexported file exists; add regressions for these observed behaviors and retain the completion recovery/call-limit tests | Existing failure traces; INT-435, INT-454 | +| 3 | A measured reliability improvement | Freeze model, inputs, prompt/skills, graders, candidate and finite allowance; execute the 92-response comparison described below; improve mandatory historical failures without regressing controls; assess all development and reserved cases | Readiness and frozen comparison plan; INT-430–438, parent INT-429 | +| 4 | An unfamiliar-domain data package | Qualify three distinct non-retail sources, including terms, provenance, hashes, units, grain, nulls and joins; reserve one entire domain; define six challenge questions and one bounded discovery report per domain | Data + evaluation; INT-440, INT-441, INT-443 | +| 5 | First drop-in dataset run | Mount one qualified development folder and its ordinary documentation through configuration; discover its shape with bounded tools; produce a supported report without notebooks, hidden references or handcrafted answer hints | Item 4; runtime + analysis; INT-442 | +| 6 | Domain-transfer evidence | Record a first-pass baseline on development domains; make a justified general revision; freeze it before seven reserved-domain scenarios; report all 21 outcomes and independently checked discoveries by domain | Items 3–5; INT-444, parent INT-439 | +| 7 | Real large-data acceptance | Freeze a real corpus of at least 10 GiB, plus deterministic roughly 100 MiB and 1 GiB samples; run equivalent questions at each size with correctness, memory, latency, scan/spill and graceful-limit evidence | Qualified source and compatible host/runtime; INT-446, parent INT-398 | +| 8 | Reusable decision-report packs | Define metric grain, populations, units and required inputs; compose findings, evidence, charts and sheets for standing business questions; validate temporal consistency and supported conclusions | Reliable analysis and artifact delivery; INT-455–457 | + +Items 1–3 are the immediate engineering iteration. Dataset qualification and task design can +proceed alongside them, but scored expansion waits for frozen criteria and its own authorized +allowance. This plan does not authorize those runs or provision new compute. + +### Reliability comparison: what “done” means + +The planned **92 responses** comprise 24 original-regression responses (six cases, two +trials per baseline/candidate arm), 34 development responses (17 cases per arm), and 34 +reserved responses (17 cases per arm). A response may require many model calls. + +- Keep the three historical failures and three healthy controls; require a demonstrated + improvement on that comparison and no material control regression. +- Evaluate all 17 development cases and all 17 reserved variants against the frozen + mandatory correctness, refusal, evidence and temporal-consistency criteria. +- Keep model/provider and input visibility comparable. A model change is a separate + experiment, not an unexplained part of the candidate. +- Preserve failed and invalid attempts, token/cost accounting and the distinction between + infrastructure failures, incorrect answers and unfinished delivery. +- Human calibration remains waived for the current work. Report method scores as + uncalibrated; do not turn that waiver into fabricated business-user acceptance. + +### Unfamiliar data: what “drop it in the sandbox” means + +Transport, building energy and weather are **candidate domains**, not already qualified +or downloaded corpora. Kaggle pages, competition notebooks and public mirrors do not by +themselves establish permission or independent ground truth. The data package must record +its source rights, exact version, transformations, schema and limitations before use. + +The evaluated agent receives the business brief, read-only data and ordinary source +metadata. Solutions, Kaggle notebooks, evaluator code, private references and reserved truth +remain outside its mounts. An open-ended discovery report may contain up to five supported +findings; fewer are acceptable when the data warrants them. Novel-sounding claims and long +narratives do not substitute for independently verified facts. + +The two small synthetic energy/service fixtures do not satisfy this domain-transfer goal. +“Cold-start” means no BackIntel-specific tuning on the held-out domain; public pretraining +exposure cannot be excluded. + +### Large-data prerequisite + +The current SmolVM integration is qualified only on macOS ARM64. The earlier proposal to +use the Windows workstation for heavier workloads is a host candidate, not an implemented +SmolVM deployment there. Qualify a compatible host/runtime and available RAM, disk and spill +capacity before the full run. A 4 GiB VM profile, generated TPC-H SF1 or a small replicated +file does not fulfill the real 10 GiB target. Do not close this milestone from a sample alone. + +## Version acceptance map + +| Target | Capability delta | Required exit evidence | +| --- | --- | --- | +| v0.0.1 | Measured analytical foundation | Reproducible published source; reliable terminal behavior; frozen baseline/candidate reliability results and documented limitations | +| v0.0.2 | Unfamiliar-domain analysis | Three qualified domains, at least 21 scenarios, separate development/reserved results and independently supported findings | +| v0.0.3 | Large-data sandbox | Real 10 GiB acceptance with staged scale checks, bounded compute and graceful exhaustion | +| v0.0.4 | Verified visual artifacts | Reports, charts and workbooks agree with independently checked tables; delivery and refusal semantics hold across acceptance tasks | +| v0.0.5 | Business-analysis report packs | Reusable metric definitions and standing reports pass factual, evidence and temporal-consistency checks | +| v0.1.0 | Controlled business-analysis pilot | One named workflow, approved read-only data contract, representative development/reserved cases and an explicit pilot-entry decision | +| v0.2.0 | Durable background execution | Persisted jobs, schedule/data-ready triggers, cancellation, deadlines and bounded retries survive whole-job failure scenarios | +| v0.3.0 | Operational monitoring | Qualified connectors, freshness/missing-data handling, measured alert quality, deduplication and authorized delivery | +| v0.4.0 | Contextual/statistical analysis | Document retrieval and statistical/forecasting workflows improve held-out tasks with honest uncertainty; vector retrieval is optional | +| v0.5.0 | Managed background-intelligence beta | User controls, run history, artifact access and spending limits; sustained-operation evidence; bounded improvement proposals | +| v0.9.0 | Release candidate | Measured beta gaps resolved, operational/support boundaries defined, reproducible candidate and rollback plan | +| v1.0.0 | Supported background-analysis service | A declared workload and connector matrix, supported artifact delivery, operational objectives and verified recovery/rollback | + +Renderer and artifact-contract components already exist ahead of their numbered milestone. +Implementation progress does not imply that the corresponding release criteria have passed. +Fine-tuning is an optional later experiment only if approved data and held-out results show +value beyond prompt/tool improvements. LangSmith provides measurement and comparison; it +does not automatically train or promote a model. + +## Version development loop + +1. Select the capability and its Linear acceptance issue; state the baseline, boundaries, + mandatory/regression/reserved cases, candidate owner and finite experiment allowance. +2. Implement the smallest justified change; review the actual diff and run relevant code, + runtime and artifact checks. +3. Compare frozen baseline and candidate in LangSmith. Diagnose the failed layer; retain + the attempt and revise only the responsible behavior. Never rerun until a lucky pass. +4. Review the exact candidate, hosted CI and evaluation evidence. Record what improved, + what regressed, usage and remaining uncertainty. +5. Publish a version only through its authorized release process, retain rollback identity, + and feed observed failures into the next development cycle. Green CI alone is not + analytical acceptance; `accept_for_review` is not deployment authority. ## What the current evidence establishes @@ -88,9 +208,9 @@ from the current controlled cases. An improvement-loop result of `accept_for_review` remains a review recommendation. It does not authorize a commit, merge, deployment, purchase, new provider or expanded data access. -Commit/push authority for the current implementation comes separately from the recorded -owner approval; the approved experiment allowance is $5 total. Deployment and purchases -remain outside that approval. +Each execution campaign needs its own recorded authorization and allowance. Historical +allowances do not fund new domain/scale experiments. Deployment and purchases remain +separately authorized decisions. Development subagents coordinate engineering; they do not imply runtime delegation inside the analyst. Scheduling, customer integrations and production infrastructure remain separate proposed work until the evidence and owner authorization support them. diff --git a/docs/sandbox-environment.md b/docs/sandbox-environment.md new file mode 100644 index 0000000..3c117a0 --- /dev/null +++ b/docs/sandbox-environment.md @@ -0,0 +1,122 @@ +# Analytical sandbox environment + +Both `AnalysisSandbox` and the benchmark `PythonSandbox` execute in **SmolVM 1.31.1**. +The qualified local target is macOS ARM64. Each run owns a Linux microVM; the final trusted +artifact renderer uses a separate clean VM. There is no Docker or host-Python fallback. +Docker remains an image preparation tool and the host of optional database services. +File/SQLite/Parquet analysis does not invoke Docker after preparation. + +## Prepare once + +```bash +uv sync --frozen +docker build -f Dockerfile.sandbox -t backintel-sandbox:py312 . +uv run python -m scripts.prepare_smolvm +uv run backintel sandbox check +``` + +Preparation downloads a checksum-pinned official SmolVM distribution and exports the +already-built local OCI image. `--runtime-archive /path/to/release.tar.gz` reuses a downloaded +release. Nothing is installed or downloaded by a running analysis. Set +`BACKINTEL_SANDBOX=smolvm` if the environment previously selected Docker. + +The default installation is `~/.local/share/backintel/runtimes/smolvm-1.31.1-darwin-arm64/`. +The image is `~/.local/share/backintel/images/analysis-py312.tar`, with an adjacent JSON +preparation receipt. `BACKINTEL_SMOLVM` and `BACKINTEL_SMOLVM_IMAGE` can select their locations; +they do not bypass runtime or image checksums. Missing/corrupt archives fail closed. +BackIntel serializes local image staging with a host file lock (120-second wait limit). +This avoids a SmolVM 1.31.1 cold-cache hardlink/copy race that can truncate an archive +when two machines are created concurrently. VM execution remains concurrent after staging. + +The release archive SHA-256 is +`69196e9880931344411eaf70b667c6633bb3015dc9f741cb85600f85a85d480c`. +The binary extracted from that archive hashes to +`ae78eda57cb3a4d88960c74077055972fc058441ed49fded26a5b2f146faf086`. +The same archive was already present for Condere. A historical IntelIPGTM binary lock has +a different hash; this integration records the bytes actually extracted, rather than +copying that old binary identity. + +## Packages and source provenance + +The existing Python 3.12.16 Debian Bookworm image and hash-locked binary wheels are reused. +Packages include Polars, pandas, NumPy, SciPy, DuckDB and database clients; Matplotlib +3.11.2, Plotly 7.1.0 and openpyxl 3.1.5 provide PNG/SVG, offline interactive HTML and Excel. +Bash, ripgrep, grep, findutils, coreutils, file and DejaVu fonts are preinstalled. +The tested workflow needs no runtime compiler or package downloads. Package records are +in `/opt/backintel/package-versions.txt` and `/opt/backintel/system-packages.txt`; offline +guidance is under `/docs`. The trusted guest setup is mounted read-only from the application. + +SmolVM is the upstream [smol-machines project](https://github.com/smol-machines/smolvm/tree/v1.31.1), +not the unrelated SmolLM model family. [Y Combinator's company record](https://www.ycombinator.com/companies/smol-machines) +identifies a San Francisco company led by its founder/CEO; the runtime uses +[libkrun](https://github.com/libkrun/libkrun) and Apple's Hypervisor.framework on this host. +The public records establish the project and its operating organization; no private cap-table +verification is claimed. No new model, hosted provider, cloud account or inference dependency +is introduced. The existing Debian, GNU, Python and analytical package provenance remains +applicable to the unchanged image. + +## Limits and workspace + +| Profile | RAM / CPUs | Workspace / artifacts / temp | Command / run deadline | Commands | +| --- | --- | --- | --- | --- | +| analysis | 2 GiB / 2 | 512 / 128 / 256 MiB | 120 / 900 seconds | 40 | +| large | 4 GiB / 3 | 1024 / 128 / 512 MiB | 300 / 1800 seconds | 40 | +| benchmark Python | 768 MiB / 1 | 128 / 64 / 128 MiB | caller timeout, normally 120 seconds | 1 | + +CPU counts must be whole numbers. Admission requires a profile within the host's CPU count +and at most 85% of its physical RAM; it does not reserve capacity across concurrent runs. SmolVM caps guest +RAM and CPUs. Each run has a 3 GiB image/storage disk and a 1 GiB overlay; analyst writes +are restricted to separately bounded tmpfs mounts. `/dev/shm` is bounded by the temp limit. + +Trusted bootstrap sets up the filesystems. Every analyst command gets a private read-only +root mount, UID/GID 10001, no supplemental groups, an empty capability bounding set, +`no_new_privs`, 128 processes per UID and 1024 open files. Applying the read-only root per +command avoids blocking SmolVM's own resolver-metadata refresh. Work files persist between +commands within that run. Source mounts are read-only; individual files are staged as +private copies so their parent directories do not expose other host files. + +The host watchdog and command deadline cancel the entire VM. The native workload exits at +the run deadline with `--stop-on-exit`, so it also stops if the host orchestrator disappears. +Explicit close stops and deletes only the task-owned machine. Cancellation is not durable +job resumption. No host home, Docker socket, model credential or writable host directory is +mounted. Host subprocesses run the trusted SmolVM CLI only, never analyst-generated code. + +Memory telemetry is sampled **VM used memory**, including guest runtime overhead. Its +sampled maximum is retained in `peak_memory_bytes`; it is not the former Docker cgroup +high-water mark and must not be directly compared without that qualification. Records +include `backend=smolvm`, runtime version, OCI image ID, archive hash and limits. + +## Assigned database services + +The VM has no outbound network. For each trusted store endpoint, the host validates its +unique identity on the existing internal Docker network and its localhost-only published +TCP port. A task-owned Unix socket forwards only to that specific endpoint. SmolVM mounts +that socket into the guest; a trusted loopback listener preserves the existing database +hostname and port. PostgreSQL, FerretDB, ClickHouse and Garage use the same connection +fields and native database clients. No general localhost/LAN/public-network access is given. + +The endpoint is checked before commands and connections. An endpoint that stops or changes +cancels access. The host bridge admits at most 32 concurrent connections. Cleanup removes +all sockets and closes forwards, preserving the source containers and their networks. +Database roles must independently remain read-only; FerretDB's frozen backend remains +necessary. The Docker host is trusted not to replace a binding during a connection. + +Same-path local sources are relocated under `/sources`, updating `BACKINTEL_SQLITE_PATH`, +`BACKINTEL_FILES_DIR` and `BACKINTEL_PARQUET_DIR`. The original source remains unchanged. +Artifact reads and exports reuse the bounded tar-stream validation: regular files only, +no links/traversal/special files, supported suffixes, at most 256 files and the byte budget. + +## Verification + +```bash +uv run pytest -q +BACKINTEL_SMOLVM_TESTS=1 uv run pytest tests/test_analysis_sandbox.py -q +BACKINTEL_SMOLVM_TESTS=1 BACKINTEL_LARGE_TEST=1 uv run pytest tests/test_analysis_sandbox.py -q +``` + +The real VM checks cover coding and error recovery, source/root write denial, UID and +capabilities, networking, endpoint isolation, storage/output limits, cancellation, binary +artifact transfer and cleanup. The separately pending four-case comparison harness defaults +to a scripted model and makes no provider calls. Historical live Sol/Luna scores were measured on Docker and remain +labelled with that runtime. See [measured results](results-analysis-sandbox.md) for current +verification receipts. Linux and Windows distributions are not qualified by this migration. diff --git a/docs/specialist-default.md b/docs/specialist-default.md index 6b49bac..a44fb70 100644 --- a/docs/specialist-default.md +++ b/docs/specialist-default.md @@ -1,7 +1,8 @@ # Default analysis specialist The owner selected **GPT-6.1 Sol through the ChatGPT subscription**, with **medium reasoning**, -after a controlled four-case comparison. Sol passed four cases; Luna passed two. This is a +after a controlled four-case comparison on the previous Docker runtime. Sol passed four cases; +Luna passed two. This is a specialist configuration decision, not a production release or a general model ranking. `backintel analyze` resolves its model in this order: @@ -10,12 +11,13 @@ specialist configuration decision, not a production release or a general model r 2. `BACKINTEL_ANALYSIS_MODEL` environment setting. 3. Built-in `chatgpt-plan/gpt-6.1-sol` default. -Use `--reasoning` to override medium. The specialist keeps its Docker isolation, read-only +Use `--reasoning` to override medium. The specialist uses SmolVM isolation, read-only source access and explicit model/tool/time limits. Subscription errors stop work; there is no paid API fallback. Token usage is reported without inventing a per-request dollar charge. ```bash docker build -f Dockerfile.sandbox -t backintel-sandbox:py312 . +uv run python -m scripts.prepare_smolvm uv run backintel chatgpt login uv run backintel chatgpt models uv run backintel analyze --store sqlite:olist --brief brief.json --output data/output/my-report @@ -25,7 +27,8 @@ The brief is a JSON object with a required `objective`; it can include `business `audience`, `metric_definitions`, `time_window`, `known_limitations` and `required_outputs`. Use a new output directory for each run. The result contains a run record and, on success, verified HTML reports, charts, CSV tables and an XLSX workbook. Model inference uses the -subscription; Python, SQL and rendering use local Docker resources. +subscription; Python and rendering use local SmolVM resources. Native source SQL queries +use the assigned store. The qualified host is macOS ARM64; see [sandbox setup](sandbox-environment.md). An explicit OpenRouter override requires both `--max-cost-usd` and `--reservation-usd`. These dollar flags are rejected for subscription runs. Comparison experiments require diff --git a/examples/retail-brief.json b/examples/retail-brief.json new file mode 100644 index 0000000..1d0f082 --- /dev/null +++ b/examples/retail-brief.json @@ -0,0 +1,15 @@ +{ + "objective": "Report net invoiced sales by calendar month, with a monthly trend chart, supporting CSV tables and an Excel workbook. Describe coverage and data-quality limitations.", + "business_context": "Exploratory review of the assigned historical Online Retail II source. Inspect the schema and observed date range before calculating.", + "audience": "Business analyst", + "metric_definitions": { + "net_invoiced_sales": "Sum Quantity multiplied by Price for rows with a valid invoice date and non-null quantity and price. Preserve negative quantities as returns or cancellations. Report currency as GBP. This is a source-row invoiced-sales measure, not cash collected or profit.", + "population": "Include source rows regardless of whether a customer identifier is present. Explain missingness and any exact duplicate rows; do not silently remove duplicates or non-product codes. Report any exclusions explicitly." + }, + "time_window": "All observed invoice dates in the assigned source; identify partial boundary months.", + "known_limitations": [ + "Historical test data, not a current operating business.", + "No costs, expenses or causal conclusions are supplied by this brief." + ], + "required_outputs": ["report", "monthly line chart", "CSV", "XLSX"] +} diff --git a/pyproject.toml b/pyproject.toml index 52cfdac..031934e 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,7 +1,7 @@ [project] name = "backintel" -version = "2.0.0a0" -description = "BackIntel v2: an ambient data-analyst agent on Deep Agents, benchmarked across storage types" +version = "0.0.1" +description = "BackIntel: an experimental data-analyst agent on Deep Agents, benchmarked across storage types" readme = "README.md" license = "MIT" requires-python = ">=3.12,<3.14" diff --git a/scripts/prepare_smolvm.py b/scripts/prepare_smolvm.py new file mode 100644 index 0000000..89cb361 --- /dev/null +++ b/scripts/prepare_smolvm.py @@ -0,0 +1,84 @@ +"""Prepare the pinned local SmolVM runtime and export an already-built analysis image. + +Docker is needed only for image preparation and optional database services, never for +file-based sandbox execution. No model calls or cloud resources are created. +""" + +import argparse +import hashlib +import json +import platform +import subprocess +import tarfile +import tempfile +from pathlib import Path + +import httpx + +from backintel.smol_runtime import ARCHIVE_SHA256, RELEASE_URL, RUNTIME_HOME, VERSION, digest + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--runtime-archive", type=Path) + parser.add_argument("--image", default="backintel-sandbox:py312") + parser.add_argument("--output", type=Path, default=RUNTIME_HOME / "images/analysis-py312.tar") + args = parser.parse_args() + if (platform.system(), platform.machine()) != ("Darwin", "arm64"): + parser.error("This qualified runtime distribution requires macOS ARM64.") + with tempfile.TemporaryDirectory(prefix="backintel-prepare-") as temp: + archive = args.runtime_archive or Path(temp) / "smolvm.tar.gz" + if args.runtime_archive is None: + with httpx.stream("GET", RELEASE_URL, follow_redirects=True, timeout=120) as response: + response.raise_for_status() + with archive.open("wb") as output: + for chunk in response.iter_bytes(): + output.write(chunk) + if digest(archive) != ARCHIVE_SHA256: + raise ValueError("SmolVM release archive checksum mismatch.") + root = RUNTIME_HOME / "runtimes" + root.mkdir(parents=True, exist_ok=True) + with tarfile.open(archive) as source: + for member in source: + destination = root / member.name + if ( + Path(member.name).is_absolute() + or ".." in Path(member.name).parts + or not destination.parent.resolve().is_relative_to(root.resolve()) + ): + raise ValueError("Unsafe runtime archive path.") + # Verified official guest rootfs includes absolute guest symlinks (/run). + # Parent confinement above prevents following those links on the host. + source.extract(member, root, filter="fully_trusted") + output = args.output.resolve() + output.parent.mkdir(parents=True, exist_ok=True) + temporary = output.with_suffix(".partial") + try: + with temporary.open("wb") as target: + subprocess.run( + ["docker", "image", "save", args.image], stdout=target, check=True, timeout=300 + ) + with tarfile.open(temporary) as tar: + manifest = json.load(tar.extractfile("manifest.json")) + if len(manifest) != 1: + raise ValueError("Expected one prepared sandbox image.") + config = tar.extractfile(manifest[0]["Config"]).read() + image_id = "sha256:" + hashlib.sha256(config).hexdigest() + if json.loads(config)["architecture"] != "arm64": + raise ValueError("The prepared image must be ARM64.") + temporary.replace(output) + receipt = { + "image_id": image_id, + "archive_sha256": digest(output), + "runtime_version": VERSION, + "runtime_archive_sha256": ARCHIVE_SHA256, + "source_image": args.image, + } + output.with_suffix(".json").write_text(json.dumps(receipt, indent=2) + "\n") + print(json.dumps({"archive": str(output), **receipt}, indent=2)) + finally: + temporary.unlink(missing_ok=True) + + +if __name__ == "__main__": + main() diff --git a/scripts/prepare_testbed.sh b/scripts/prepare_testbed.sh index 1fdf429..89ba1f0 100755 --- a/scripts/prepare_testbed.sh +++ b/scripts/prepare_testbed.sh @@ -26,4 +26,5 @@ done uv run backintel tpch generate --sf 1 uv run backintel oracle build tpch_sf1 docker build -f Dockerfile.sandbox -t backintel-sandbox:py312 . +uv run python -m scripts.prepare_smolvm uv run backintel sandbox check diff --git a/scripts/verify_runtime_evidence.py b/scripts/verify_runtime_evidence.py index 452f8b7..0cacea1 100644 --- a/scripts/verify_runtime_evidence.py +++ b/scripts/verify_runtime_evidence.py @@ -2,7 +2,7 @@ Run with ``uv run python scripts/verify_runtime_evidence.py``. Only existing service containers are started; containers already running are preserved. Raw-file evidence runs -in the normal read-only Docker Python sandbox. Results contain no connection credentials. +in the normal read-only SmolVM Python sandbox. Results contain no connection credentials. """ from __future__ import annotations @@ -102,7 +102,7 @@ def verify_case(store, sandbox: PythonSandbox) -> dict: network=store.sandbox_network(), ) if result.exit_code != 0: - raise RuntimeError("Docker Python count failed") + raise RuntimeError("Sandboxed Python count failed") count = int(result.stdout.strip()) else: kind, good = ("query", COUNT_QUERY) if store.kind == "ferretdb" else ("sql", COUNT_SQL) @@ -143,7 +143,7 @@ def verify_case(store, sandbox: PythonSandbox) -> dict: def main() -> int: settings = Settings.from_env() - sandbox = PythonSandbox("docker", timeout_s=30) + sandbox = PythonSandbox(settings.sandbox, timeout_s=30) report = { "created_at": datetime.now(UTC).isoformat(), "scope": "Runtime evidence execution smoke; not a model quality or full data parity evaluation", diff --git a/src/backintel/agent_files/analysis.md b/src/backintel/agent_files/analysis.md index 7bc90d4..10dcfdb 100644 --- a/src/backintel/agent_files/analysis.md +++ b/src/backintel/agent_files/analysis.md @@ -45,9 +45,12 @@ tools; they do not grant additional permissions. check titles, axes, units, labels, order, overlap, contrast and whether the visual answers the question. Fix a demonstrated presentation or correctness problem, then verify that fix. If visual inspection is unavailable, state that limit rather than claim a visual review. -8. Finish with the result, important assumptions/caveats and the returned artifact paths or - manifest. Report a partial result or specific failure honestly when a resource limit or - data gap prevents the complete request. Never fabricate an artifact or successful check. +8. Finish by calling the `AnalysisCompletion` tool; an ordinary text response cannot complete + the run. Set `status="completed"` only after `render_report` succeeds and the requested + checks pass. Include the result and important assumptions/caveats in the tool arguments. + If required inputs are absent or a constraint prevents completion, use `status="cannot_answer"` + with a specific `reason`. The host exports a completed report after this call; temporary + VM paths are not delivered files. Never fabricate an artifact or successful check. ## Boundaries and stopping diff --git a/src/backintel/analysis.py b/src/backintel/analysis.py index 6037a51..cfe7abc 100644 --- a/src/backintel/analysis.py +++ b/src/backintel/analysis.py @@ -1,7 +1,7 @@ """Opt-in analytical coding specialist; the benchmark analyst remains independent. The orchestrator owns the brief, sources, model and limits. The specialist can revise -programs and artifacts inside one Docker workspace, never its own runtime or release. +programs and artifacts inside one SmolVM workspace, never its own runtime or release. """ from __future__ import annotations @@ -15,15 +15,21 @@ import uuid from datetime import UTC, datetime from pathlib import Path -from typing import Literal +from typing import Literal, NotRequired from deepagents import FilesystemPermission, HarnessProfile, create_deep_agent, register_harness_profile from deepagents._models import get_model_identifier, get_model_provider from deepagents.backends import FilesystemBackend from deepagents.middleware.filesystem import FilesystemMiddleware -from langchain.agents.middleware import ModelCallLimitMiddleware, ToolCallLimitMiddleware +from langchain.agents.middleware import ( + AgentState, + ModelCallLimitMiddleware, + ToolCallLimitMiddleware, + after_model, +) from langchain.agents.structured_output import ToolStrategy from langchain_core.language_models import BaseChatModel +from langchain_core.messages import AIMessage, HumanMessage from langchain_core.tools import ToolException, tool from pydantic import BaseModel, ConfigDict, Field, model_validator @@ -62,8 +68,43 @@ def explain_refusal(self): return self +class AnalysisState(AgentState): + completion_repair_attempted: NotRequired[bool] + + +@after_model(state_schema=AnalysisState, can_jump_to=["model"]) +def require_analysis_completion(state: AnalysisState, runtime) -> dict | None: + """Recover one text-only stop, retaining the original graph's budgets and transcript.""" + if state.get("structured_response") is not None: + return None + last = state["messages"][-1] + if not isinstance(last, AIMessage) or last.tool_calls: + return None + if state.get("completion_repair_attempted"): + raise ValueError("Missing AnalysisCompletion after one recovery prompt; no artifacts delivered.") + # ChatGPT plan transport uses automatic tool selection. ToolStrategy alone cannot + # prevent a prose-only stop. Never infer completion or a refusal from that prose. + return { + "completion_repair_attempted": True, + "messages": [ + HumanMessage( + content=( + "The host did not receive the required AnalysisCompletion tool call. " + "Your previous text has been retained, but it cannot finish this run. " + "Call AnalysisCompletion with your supported status, summary and caveats; " + "cannot_answer also requires a specific reason. Use completed only after " + "a successful render_report and the requested checks. Do not repeat finished " + "analysis or claim that temporary VM paths have already been delivered. " + "The original tool, model-call and time limits still apply." + ) + ) + ], + "jump_to": "model", + } + + def _safe_text(text: str, env: dict[str, str]) -> str: - # Source access stays inside Docker; do not return connection secrets in tool traces. + # Source access stays inside SmolVM; do not return connection secrets in tool traces. for key, value in env.items(): if len(value) >= 8 and any( part in key.upper() for part in ("PASSWORD", "TOKEN", "SECRET", "DSN", "URI") @@ -164,7 +205,7 @@ def run_shell(command: str) -> str: @tool def run_python(code: str) -> str: - """Run Python in the persistent Docker workspace with assigned read-only sources. + """Run Python in the persistent SmolVM workspace with assigned read-only sources. Use installed analytical/plotting packages. Save programs in /workspace; save final outputs through render_report. Print bounded results, not entire raw datasets. @@ -240,7 +281,7 @@ def render_report(spec_path: str) -> str: return execute(result) # Reuse native read-only schema/query tools; replace the old ephemeral Python tool. - domain = [t for t in build_tools(store, PythonSandbox("docker")) if t.name != "run_python"] + domain = [t for t in build_tools(store, PythonSandbox("smolvm")) if t.name != "run_python"] selected = [ *domain, run_shell, @@ -292,6 +333,8 @@ def build_analysis_agent( permissions=[FilesystemPermission(operations=["write"], paths=["/**"], mode="deny")], middleware=[ FilesystemMiddleware(backend=backend, tools=["ls", "read_file", "glob", "grep"]), + # after_model hooks run in reverse order: count the call before any recovery jump. + require_analysis_completion, ModelCallLimitMiddleware(run_limit=max_model_calls, exit_behavior="error"), ToolCallLimitMiddleware(run_limit=max_tool_calls, exit_behavior="error"), ], @@ -439,6 +482,7 @@ async def run_analysis( record["usage"] = usage.report() record["budget"] = budget.report() if budget else None record["latency_s"] = round(time.monotonic() - started, 3) + record["completion_recovery_attempts"] = int(state.get("completion_repair_attempted", False)) record["messages"] = [m.model_dump(mode="json") for m in state.get("messages", [])] (output / "run.json").write_text(json.dumps(record, indent=2, default=str)) return record diff --git a/src/backintel/analysis_sandbox.py b/src/backintel/analysis_sandbox.py index fd59245..022083f 100644 --- a/src/backintel/analysis_sandbox.py +++ b/src/backintel/analysis_sandbox.py @@ -1,6 +1,6 @@ -"""A Docker-only, per-run coding workspace with bounded persistent outputs. +"""A SmolVM-only, per-run coding workspace with bounded persistent outputs. -The workspace persists between commands, not across container failures. Only explicitly +The workspace persists between commands, not across VM failures. Only explicitly exported regular artifact files survive close/cancellation. No writable host mount is used. """ @@ -12,6 +12,7 @@ import shutil import subprocess import tarfile +import tempfile import threading import time import uuid @@ -19,7 +20,9 @@ from pathlib import Path, PurePosixPath from urllib.parse import urlsplit -from backintel.sandbox import DOCKER_IMAGE, MAX_OUTPUT_CHARS, SandboxResult +from backintel.sandbox import MAX_OUTPUT_CHARS, SandboxResult +from backintel.smol_runtime import capacity, host_environment, image_archive, image_staging_lock, runtime +from backintel.smol_services import ServiceBridges @dataclass(frozen=True) @@ -56,27 +59,12 @@ def __post_init__(self): } -def _checked(command: list[str], *, timeout: float = 20) -> str: - result = subprocess.run(command, capture_output=True, text=True, timeout=timeout) - if result.returncode: - # Do not echo command arguments: store credentials may be among them. - raise RuntimeError(f"Docker operation failed: {result.stderr[:1000]}") - return result.stdout - - -def docker_capacity() -> dict: - """Only capacity fields, never Docker configuration or credentials.""" - raw = _checked(["docker", "info", "--format", "{{json .MemTotal}} {{json .NCPU}}"]) - memory, cpus = raw.split() - return {"memory_bytes": int(memory), "cpus": int(cpus)} - - class AnalysisSandbox: """One isolated analytical run. Use as a context manager and export before close. `env` and `mounts` come from a trusted Store, never directly from a model. Optional - `network` identifies internal store infrastructure. Only assigned endpoints are attached - to a fresh internal run network; their backend peers remain on the infrastructure network. + `network` identifies internal store infrastructure. Assigned endpoints are bridged through + individual Unix sockets. The VM has no outbound network or general host access. """ def __init__( @@ -87,7 +75,7 @@ def __init__( network: str | None = None, allowed_hosts: set[str] | None = None, profile: str | ResourceProfile = "analysis", - image: str = DOCKER_IMAGE, + image: str | Path | None = None, timeout_s: int | None = None, max_commands: int | None = None, ): @@ -112,8 +100,10 @@ def __init__( if self.env.get(key) == inside: self.env[key] = target self.network = network - self._owned_network: str | None = None - self._source_peers: list[str] = [] + self._services: ServiceBridges | None = None + self._staging: tempfile.TemporaryDirectory | None = None + self.engine: str | None = None + self._created = False self.allowed_hosts = set(allowed_hosts or ()) for key in ("BACKINTEL_POSTGRES_DSN", "BACKINTEL_MONGO_URI", "BACKINTEL_CH_URL"): if key in self.env and (host := urlsplit(self.env[key]).hostname): @@ -134,7 +124,7 @@ def __init__( @property def label(self) -> str: - return "docker" + return "smolvm" def __enter__(self): return self.start() @@ -142,97 +132,170 @@ def __enter__(self): def __exit__(self, *_): self.close() + def _checked(self, arguments, *, timeout=30): + result = subprocess.run( + [self.engine, *arguments], capture_output=True, text=True, timeout=timeout, env=host_environment() + ) + if result.returncode: + # Never echo arguments: explicitly assigned store credentials may be present. + raise RuntimeError(f"SmolVM operation failed: {result.stderr[:1000]}") + return result.stdout + + def _exec_command(self, arguments, *, input_data=False, timeout=30): + return [ + self.engine, + "machine", + "exec", + "--name", + self.name, + "--user", + "0", + "--workdir", + "/workspace", + "--timeout", + f"{max(1, int(timeout))}s", + *(["-i"] if input_data else ["--stream"]), + "--", + "python", + "-I", + "/opt/backintel/runtime/execute.py", + json.dumps(asdict(self.profile)), + *arguments, + ] + def start(self): if self.closed: raise RuntimeError("A closed sandbox cannot be restarted; create a new run.") if self.started_at is not None: return self - if shutil.which("docker") is None: - raise RuntimeError("Analysis requires Docker; there is no host subprocess fallback.") - capacity = docker_capacity() - if self.profile.memory_mib * 1024**2 > capacity["memory_bytes"] * 0.85: - raise RuntimeError("Requested profile leaves insufficient Docker VM memory headroom.") - if self.profile.cpus > capacity["cpus"]: - raise RuntimeError("Requested profile exceeds available Docker CPUs.") - self.image_id = _checked( - ["docker", "inspect", "--type=image", self.image, "--format", "{{.Id}}"] - ).strip() + self.engine = runtime() + available = capacity() + if self.profile.memory_mib * 1024**2 > available["memory_bytes"] * 0.85: + raise RuntimeError("Requested profile leaves insufficient host memory headroom.") + if self.profile.cpus > available["cpus"] or int(self.profile.cpus) != self.profile.cpus: + raise ValueError("SmolVM requires whole CPUs within available host capacity.") + archive, self._image_receipt = image_archive(self.image) + self.image_id = self._image_receipt["image_id"] p = self.profile command = [ - "docker", - "run", - "--detach", - "--rm", + "machine", + "create", "--name", self.name, "--label", - "backintel.analysis=true", - "--network", - self.network or "none", - "--dns", - "127.0.0.1", # Keep Docker's peer-name resolution; disable external DNS forwarding. - "--read-only", - "--user", - "10001:10001", - "--memory", - f"{p.memory_mib}m", - "--memory-swap", - f"{p.memory_mib}m", + "owner=backintel", + "--image", + str(archive), "--cpus", - str(p.cpus), - "--pids-limit", - str(p.pids), - "--cap-drop", - "ALL", - "--security-opt", - "no-new-privileges", - "--init", - "--ulimit", - "nofile=1024:1024", - "--workdir", - "/workspace", + str(int(p.cpus)), + "--mem", + str(p.memory_mib), + "--storage", + "3", + "--overlay", + "1", + "--shm-size", + f"{p.temporary_mib}m", + "--user", + "0", + "--stop-on-exit", + "--stop-timeout", + "1", + "--volume", + f"{Path(__file__).parent / 'smol_guest'}:/opt/backintel/runtime:ro", ] - for path, mib in [ - ("/workspace", p.workspace_mib), - ("/artifacts", p.artifacts_mib), - ("/tmp", p.temporary_mib), - ]: - command += ["--tmpfs", f"{path}:rw,nosuid,nodev,size={mib}m,uid=10001,gid=10001,mode=0700"] - protected = ( - "/workspace", - "/artifacts", - "/tmp", - "/opt", - "/usr", - "/bin", - "/etc", - "/proc", - "/sys", - "/dev", - ) - for host, inside in self.mounts.items(): - source = Path(host).resolve(strict=True) - target = PurePosixPath(inside) - if not target.is_absolute() or ".." in target.parts or str(target) == "/": - raise ValueError("Source mounts require an absolute, non-root container path.") - if any(str(target) == part or str(target).startswith(part + "/") for part in protected): - raise ValueError("Source mounts cannot replace runtime or output directories.") - if source in {Path("/"), Path.home(), Path.home().parent} or "," in str(source): - raise ValueError("Refusing an overly broad or malformed source mount.") - if "," in str(target): - raise ValueError("Malformed source mount target.") - command += ["--mount", f"type=bind,src={source},dst={target},readonly"] - for key, value in self.env.items(): - if not key.startswith("BACKINTEL_") or not key.replace("_", "").isalnum(): - raise ValueError("Only explicit BACKINTEL_ store context can enter the analysis sandbox.") - command += ["--env", f"{key}={value}"] - # The container expires even if the host process is killed before its watchdog runs. - command += [self.image_id, "sleep", str(p.run_timeout_s)] + self._staging = tempfile.TemporaryDirectory(prefix="backintel-sources-") + file_directories = {} try: + for host, inside in self.mounts.items(): + source, target = Path(host).resolve(strict=True), PurePosixPath(inside) + protected = ( + "/workspace", + "/artifacts", + "/tmp", + "/opt", + "/usr", + "/bin", + "/etc", + "/proc", + "/sys", + "/dev", + "/run", + "/backintel-sockets", + ) + if ( + not target.is_absolute() + or ".." in target.parts + or str(target) == "/" + or any(target == PurePosixPath(part) or target.is_relative_to(part) for part in protected) + ): + raise ValueError("Source mounts cannot replace runtime or output directories.") + if source in {Path("/"), Path.home(), Path.home().parent} or ":" in str(source) + str(target): + raise ValueError("Refusing an overly broad or malformed source mount.") + if source.is_dir(): + command += ["--volume", f"{source}:{target}:ro"] + elif source.is_file() and str(target.parent) != "/": + directory = file_directories.setdefault( + str(target.parent), Path(self._staging.name) / str(len(file_directories)) + ) + directory.mkdir(exist_ok=True) + shutil.copyfile(source, directory / target.name) + else: + raise ValueError("Sources must be regular files or directories.") + for target, source in file_directories.items(): + command += ["--volume", f"{source}:{target}:ro"] + for key, value in self.env.items(): + if not key.startswith("BACKINTEL_") or not key.replace("_", "").isalnum(): + raise ValueError("Only explicit BACKINTEL_ store context can enter the analysis sandbox.") + command += ["--env", f"{key}={value}"] + services = [] if self.network: - self._isolate_network() - command[command.index("--network") + 1] = self.network - _checked(command, timeout=45) + self._services = ServiceBridges(self.network, self.env, self.allowed_hosts) + services = self._services.services + for service in services: + command += ["--mount-socket", f"{service['host_socket']}:{service['socket']}"] + command += [ + "--", + "python", + "-I", + "/opt/backintel/runtime/boot.py", + json.dumps(asdict(p)), + json.dumps( + [{k: v for k, v in item.items() if k in {"host", "port", "socket"}} for item in services] + ), + ] + self._created = True + # SmolVM 1.31.1's concurrent cold-cache staging can copy onto a hardlink + # of its input, truncating both. Serialize staging, not VM execution. + with image_staging_lock(): + self._checked(command, timeout=120) + self._checked(["machine", "start", "--name", self.name], timeout=120) + deadline = time.monotonic() + 15 + while True: + ready = subprocess.run( + [ + self.engine, + "machine", + "exec", + "--name", + self.name, + "--timeout", + "2s", + "--", + "test", + "-f", + "/run/backintel-ready", + ], + capture_output=True, + timeout=5, + env=host_environment(), + ) + if ready.returncode == 0: + break + if time.monotonic() >= deadline: + logs = self._checked(["machine", "logs", "--name", self.name]) + raise RuntimeError(f"SmolVM workspace setup failed: {logs[-1500:]}") + time.sleep(0.1) self.started_at = time.monotonic() self._watchdog = threading.Timer(p.run_timeout_s, self.cancel) self._watchdog.daemon = True @@ -242,91 +305,31 @@ def start(self): raise return self - def _isolate_network(self): - """Attach only selected source endpoints; never modify their existing connections.""" - details = json.loads( - _checked(["docker", "network", "inspect", self.network, "--format", "{{json .}}"]) - ) - if not details.get("Internal") or not self.allowed_hosts: - raise ValueError("Networked analysis requires an internal network and assigned source hosts.") - selected, found = {}, set() - for container_id, peer in details.get("Containers", {}).items(): - networks = json.loads( - _checked( - ["docker", "inspect", container_id, "--format", "{{json .NetworkSettings.Networks}}"] - ) - ) - names = {peer.get("Name"), *(networks.get(self.network, {}).get("Aliases") or [])} - authorized = names & self.allowed_hosts - if found & authorized: - raise ValueError("Assigned source hostname resolves to more than one container.") - if authorized: - selected[container_id] = authorized - found.update(authorized) - if found != self.allowed_hosts: - raise ValueError("Not all assigned source hosts exist on the selected internal network.") - private = f"{self.name}-net" - _checked(["docker", "network", "create", "--internal", "--label", "backintel.analysis=true", private]) - self._owned_network = self.network = private - for container_id, names in selected.items(): - # Track before connecting so even an interrupted Docker response gets cleaned up. - self._source_peers.append(container_id) - aliases = [arg for name in sorted(names) for arg in ("--alias", name)] - _checked(["docker", "network", "connect", *aliases, private, container_id]) - self._validate_network() - - def _validate_network(self): - details = json.loads( - _checked(["docker", "network", "inspect", self.network, "--format", "{{json .}}"]) - ) - if not details.get("Internal"): - raise ValueError("Analysis may join only an explicitly internal Docker network.") - if not self.allowed_hosts: - raise ValueError("Networked analysis requires explicit assigned source hostnames.") - found = set() - for container_id, peer in details.get("Containers", {}).items(): - if peer.get("Name") == self.name: - continue - networks = json.loads( - _checked( - ["docker", "inspect", container_id, "--format", "{{json .NetworkSettings.Networks}}"] - ) - ) - endpoint = networks.get(self.network, {}) - names = {peer.get("Name"), *(endpoint.get("Aliases") or [])} - authorized = names & self.allowed_hosts - if not authorized: - raise ValueError( - "Shared network includes an unassigned peer; use a source-only internal network." - ) - found.update(authorized) - if found != self.allowed_hosts: - raise ValueError("Not all assigned source hosts exist on the selected internal network.") - def _execute(self, arguments: list[str], *, input_data: bytes | None = None) -> SandboxResult: with self._command_lock: if self.started_at is None: self.start() if self.closed: - return SandboxResult("", "Sandbox is closed or cancelled.", -1, 0, "docker") + return SandboxResult("", "Sandbox is closed or cancelled.", -1, 0, "smolvm") if self.network: try: - self._validate_network() + self._services.validate() except (ValueError, RuntimeError): self.cancel() raise remaining = self.profile.run_timeout_s - (time.monotonic() - self.started_at) if self.commands >= self.profile.max_commands or remaining <= 0: self.cancel() - return SandboxResult("", "Whole-run command/time budget exhausted.", -1, 0, "docker", True) + return SandboxResult("", "Whole-run command/time budget exhausted.", -1, 0, "smolvm", True) self.commands += 1 timeout = min(self.profile.command_timeout_s, remaining) started = time.monotonic() proc = subprocess.Popen( - ["docker", "exec", *(["-i"] if input_data is not None else []), self.name, *arguments], + self._exec_command(arguments, input_data=input_data is not None, timeout=timeout), stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE if input_data is not None else None, + env=host_environment(), ) writer = None if input_data is not None: @@ -387,7 +390,7 @@ def feed_input(): outputs["stderr"], -1 if timed_out else proc.returncode, int((time.monotonic() - started) * 1000), - "docker", + "smolvm", timed_out, ) @@ -434,18 +437,10 @@ def read_bytes(self, path: str, max_bytes: int = 2 * 1024 * 1024) -> bytes: raise RuntimeError("Read from an active sandbox before closing it.") source_path = PurePosixPath(path) proc = subprocess.Popen( - [ - "docker", - "exec", - self.name, - "tar", - "-C", - str(source_path.parent), - "-cf", - "-", - "--", - source_path.name, - ], + self._exec_command( + ["tar", "-C", str(source_path.parent), "-cf", "-", "--", source_path.name] + ), + env=host_environment(), stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, ) @@ -473,23 +468,23 @@ def read_bytes(self, path: str, max_bytes: int = 2 * 1024 * 1024) -> bytes: def resource_usage(self) -> dict: if self.started_at is not None and not self.closed: try: - result = subprocess.run( - ["docker", "exec", self.name, "cat", "/sys/fs/cgroup/memory.peak"], - capture_output=True, - text=True, - timeout=10, + stats = json.loads( + self._checked(["machine", "stats", "--name", self.name, "--json"], timeout=10) ) - if result.returncode == 0 and result.stdout.strip().isdigit(): - self._peak_memory_bytes = max(self._peak_memory_bytes, int(result.stdout.strip())) - except subprocess.TimeoutExpired: - pass # Observability must not turn a completed command into a failed analysis. + self._peak_memory_bytes = max(self._peak_memory_bytes, stats["memory"]["used_bytes"]) + except (subprocess.TimeoutExpired, RuntimeError, KeyError, ValueError): + pass return { + "backend": "smolvm", + "runtime_version": "1.31.1", "peak_memory_bytes": self._peak_memory_bytes, + "memory_measurement": "sampled VM used memory; includes guest runtime", "commands": self.commands, "elapsed_ms": int(((self.finished_at or time.monotonic()) - self.started_at) * 1000) if self.started_at else 0, "image_id": self.image_id, + "image_archive_sha256": getattr(self, "_image_receipt", {}).get("archive_sha256"), "profile": asdict(self.profile), "cancelled": self.cancelled, } @@ -509,7 +504,8 @@ def export_artifacts(self, destination: str | Path) -> list[str]: output.mkdir(parents=True, exist_ok=True) output = output.resolve() proc = subprocess.Popen( - ["docker", "exec", self.name, "tar", "-C", "/artifacts", "-cf", "-", "."], + self._exec_command(["tar", "-C", "/artifacts", "-cf", "-", "."], timeout=60), + env=host_environment(), stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, ) @@ -566,38 +562,44 @@ def close(self): self.finished_at = time.monotonic() if self._watchdog: self._watchdog.cancel() - if shutil.which("docker"): - try: - subprocess.run(["docker", "rm", "--force", self.name], capture_output=True, timeout=15) - finally: - if self._owned_network: - # Even an unassigned peer that triggered cancellation must be detached - # from our disposable network. Its other networks/container stay intact. - peers = set(self._source_peers) - inspected = subprocess.run( + try: + if self.engine and self._created: + try: + subprocess.run( + [self.engine, "machine", "stop", "--name", self.name], + capture_output=True, + timeout=20, + env=host_environment(), + ) + finally: + deleted = subprocess.run( [ - "docker", - "network", - "inspect", - self._owned_network, - "--format", - "{{json .Containers}}", + self.engine, + "machine", + "delete", + "--name", + self.name, + "--force", + "--error-format", + "json", ], capture_output=True, text=True, - timeout=15, - ) - if inspected.returncode == 0: - peers.update(json.loads(inspected.stdout) or {}) - for peer in peers: - subprocess.run( - ["docker", "network", "disconnect", "--force", self._owned_network, peer], - capture_output=True, - timeout=15, - ) - subprocess.run( - ["docker", "network", "rm", self._owned_network], capture_output=True, timeout=15 + timeout=30, + env=host_environment(), ) + if deleted.returncode: + try: + missing = json.loads(deleted.stderr)["error"]["code"] == "NOT_FOUND" + except (ValueError, KeyError): + missing = False + if not missing: + raise RuntimeError(f"SmolVM cleanup failed for {self.name}; retain its identity.") + finally: + if self._services: + self._services.close() + if self._staging: + self._staging.cleanup() def cancel(self): self.cancelled = True diff --git a/src/backintel/cli.py b/src/backintel/cli.py index d4e6b9b..adde230 100644 --- a/src/backintel/cli.py +++ b/src/backintel/cli.py @@ -205,8 +205,8 @@ def cmd_analyze(args, settings): from backintel.models import build_model from backintel.stores import open_store - if settings.sandbox != "docker": - raise ValueError("The analysis specialist requires Docker; host subprocess mode is not allowed.") + if settings.sandbox != "smolvm": + raise ValueError("The analysis specialist requires SmolVM; host subprocess mode is not allowed.") model_id = args.model or settings.analysis_model plan = model_id.startswith("chatgpt-plan/") if plan: @@ -322,7 +322,7 @@ def main(argv: list[str] | None = None) -> None: sandbox.add_argument("action", choices=("check",)) sandbox.set_defaults(func=cmd_sandbox) - analyze = sub.add_parser("analyze", help="run the opt-in Docker analysis/report specialist") + analyze = sub.add_parser("analyze", help="run the opt-in SmolVM analysis/report specialist") analyze.add_argument("--brief", required=True, help="business brief JSON; see docs/specialist-default.md") analyze.add_argument("--store", required=True) analyze.add_argument( diff --git a/src/backintel/sandbox.py b/src/backintel/sandbox.py index d0c65d9..b318853 100644 --- a/src/backintel/sandbox.py +++ b/src/backintel/sandbox.py @@ -1,25 +1,15 @@ """Sandboxed Python for the analyst. -Docker is the required isolation boundary for model-generated code: -no network, read-only root filesystem, memory and CPU caps, the data mounted read-only, and a -hard timeout. There is no host-interpreter fallback. +SmolVM is the required isolation boundary for model-generated code. The shared runtime +enforces offline networking, read-only source mounts, resource caps and a hard timeout. """ from __future__ import annotations -import json -import os -import shutil -import subprocess -import tempfile -import uuid from dataclasses import dataclass -from pathlib import Path MAX_OUTPUT_CHARS = 12_000 DOCKER_IMAGE = "backintel-sandbox:py312" -DOCKER_MEMORY = "768m" -DOCKER_CPUS = "1.0" @dataclass @@ -42,127 +32,41 @@ def _clip(text: str) -> str: class PythonSandbox: - def __init__(self, backend: str = "docker", *, timeout_s: int = 120): - if backend != "docker": - raise ValueError("Docker is required for model-generated Python; host execution is disabled.") + def __init__(self, backend: str = "smolvm", *, timeout_s: int = 120): + if backend != "smolvm": + raise ValueError("SmolVM is required for model-generated Python; host execution is disabled.") self.backend = backend self.timeout_s = timeout_s @property def label(self) -> str: - return "docker" + return "smolvm" def run( self, code: str, *, env: dict[str, str], mounts: dict[str, str], network: str | None = None ) -> SandboxResult: - """Run `code` with `env` set; `mounts` maps host paths to the same path inside (read-only). - - `network` names an internal Docker network that reaches a store container; it never - reaches the internet (the network is created with `--internal`). - """ - return self._run_docker(code, env=env, mounts=mounts, network=network) - - def _run_docker( - self, code: str, *, env: dict[str, str], mounts: dict[str, str], network: str | None - ) -> SandboxResult: - if shutil.which("docker") is None: - raise RuntimeError("Docker is required for model-generated Python; no host fallback is allowed.") - with tempfile.TemporaryDirectory(prefix="backintel-sbx-") as tmp: - container_name = f"backintel-sbx-{uuid.uuid4().hex}" - script = Path(tmp) / "main.py" - script.write_text(code) - cmd = [ - "docker", - "run", - "--rm", - "--name", - container_name, - "--network", - network or "none", - "--read-only", - "--memory", - DOCKER_MEMORY, - "--cpus", - DOCKER_CPUS, - "--pids-limit", - "128", - "--cap-drop", - "ALL", - "--security-opt", - "no-new-privileges", - "--tmpfs", - "/tmp:rw,size=256m", - "-v", - f"{tmp}:/work:ro", - "-w", - "/work", - ] - for host, inside in mounts.items(): - cmd += ["-v", f"{host}:{inside}:ro"] - for key, value in env.items(): - cmd += ["-e", f"{key}={value}"] - cmd += [DOCKER_IMAGE, "python", "-I", "/work/main.py"] - try: - return self._execute(cmd, label="docker") - finally: - # Killing a timed-out Docker CLI does not kill its container. Remove only - # this invocation's UUID-named container, including on a Python exception. - subprocess.run(["docker", "rm", "--force", container_name], capture_output=True, timeout=15) - - def _execute(self, cmd: list[str], *, label: str, env=None, cwd=None) -> SandboxResult: - import time - - started = time.monotonic() - try: - proc = subprocess.run( - cmd, capture_output=True, text=True, timeout=self.timeout_s, env=env, cwd=cwd - ) - return SandboxResult( - stdout=_clip(proc.stdout), - stderr=_clip(proc.stderr), - exit_code=proc.returncode, - elapsed_ms=int((time.monotonic() - started) * 1000), - backend=label, - ) - except subprocess.TimeoutExpired as error: - return SandboxResult( - stdout=_clip(error.stdout or "" if isinstance(error.stdout, str) else ""), - stderr=f"Timed out after {self.timeout_s}s.", - exit_code=-1, - elapsed_ms=int((time.monotonic() - started) * 1000), - backend=label, - timed_out=True, - ) - - -def docker_image_available() -> bool: - if shutil.which("docker") is None: - return False - proc = subprocess.run(["docker", "image", "inspect", DOCKER_IMAGE], capture_output=True) - return proc.returncode == 0 + # Import lazily: both sandbox entry points share the result contract in this module. + from backintel.analysis_sandbox import AnalysisSandbox, ResourceProfile + + profile = ResourceProfile( + memory_mib=768, + cpus=1, + workspace_mib=128, + artifacts_mib=64, + temporary_mib=128, + command_timeout_s=self.timeout_s, + run_timeout_s=self.timeout_s, + max_commands=1, + ) + with AnalysisSandbox(env=env, mounts=mounts, network=network, profile=profile) as sandbox: + return sandbox.run_python(code) def describe_backend(sandbox: PythonSandbox) -> dict: return { "backend": sandbox.label, "timeout_s": sandbox.timeout_s, - **( - {"image": DOCKER_IMAGE, "memory": DOCKER_MEMORY, "cpus": DOCKER_CPUS} - if sandbox.backend == "docker" - else {} - ), + "memory": "768m", + "cpus": "1", + "runtime_version": "1.31.1", } - - -if __name__ == "__main__": # Manual check: python -m backintel.sandbox - sb = PythonSandbox(os.environ.get("BACKINTEL_SANDBOX", "docker")) - print( - json.dumps( - sb.run( - "import sys, polars; print(sys.version); print(polars.__version__)", - env={}, - mounts={}, - ).to_dict(), - indent=2, - ) - ) diff --git a/src/backintel/settings.py b/src/backintel/settings.py index 7fd3f04..3b42112 100644 --- a/src/backintel/settings.py +++ b/src/backintel/settings.py @@ -20,7 +20,7 @@ class Settings: data_dir: Path model: str judge_model: str - sandbox: str # Docker only; the runtime rejects other backends. + sandbox: str # SmolVM only; the runtime rejects other backends. langsmith_project: str extra: dict[str, str] = field(default_factory=dict) analysis_model: str = DEFAULT_ANALYSIS_MODEL @@ -32,7 +32,7 @@ def from_env(cls, env: dict[str, str] | None = None) -> Settings: data_dir=Path(env.get("BACKINTEL_DATA_DIR") or repo_root() / "data").resolve(), model=env.get("BACKINTEL_MODEL") or DEFAULT_MODEL, judge_model=env.get("BACKINTEL_JUDGE_MODEL") or DEFAULT_JUDGE_MODEL, - sandbox=env.get("BACKINTEL_SANDBOX") or "docker", + sandbox=env.get("BACKINTEL_SANDBOX") or "smolvm", langsmith_project=env.get("LANGSMITH_PROJECT") or "backintel-v2-dev", analysis_model=env.get("BACKINTEL_ANALYSIS_MODEL") or DEFAULT_ANALYSIS_MODEL, ) diff --git a/src/backintel/smol_guest/boot.py b/src/backintel/smol_guest/boot.py new file mode 100644 index 0000000..c8bc0ee --- /dev/null +++ b/src/backintel/smol_guest/boot.py @@ -0,0 +1,70 @@ +"""Trusted per-run setup, before any analyst command is admitted.""" + +import json +import os +import selectors +import socket +import subprocess +import sys +import threading +import time +from pathlib import Path + + +def relay(client, destination): + with client, socket.socket(socket.AF_UNIX) as upstream: + upstream.connect(destination) + with selectors.DefaultSelector() as selector: + selector.register(client, selectors.EVENT_READ, upstream) + selector.register(upstream, selectors.EVENT_READ, client) + while selector.get_map(): + for key, _ in selector.select(timeout=1): + data = key.fileobj.recv(65536) + if not data: + return + key.data.sendall(data) + + +def serve(listener, destination): + while True: + client, _ = listener.accept() + threading.Thread(target=relay, args=(client, destination), daemon=True).start() + + +def main(): + profile, services = json.loads(sys.argv[1]), json.loads(sys.argv[2]) + for path, size in [ + ("/workspace", profile["workspace_mib"]), + ("/artifacts", profile["artifacts_mib"]), + ("/tmp", profile["temporary_mib"]), + ("/dev/shm", profile["temporary_mib"]), + ]: + Path(path).mkdir(parents=True, exist_ok=True) + subprocess.run( + [ + "mount", + "-t", + "tmpfs", + "-o", + f"size={size}m,nosuid,nodev,uid=10001,gid=10001,mode=0700", + "tmpfs", + path, + ], + check=True, + ) + for service in services: + with open("/etc/hosts", "a") as hosts: + hosts.write(f"\n127.0.0.1 {service['host']}\n") + listener = socket.socket() + listener.bind(("127.0.0.1", service["port"])) + listener.listen(32) + threading.Thread(target=serve, args=(listener, service["socket"]), daemon=True).start() + # execute.py gives each analyst command its own read-only root view. + Path("/run/backintel-ready").write_text("ready") + time.sleep(profile["run_timeout_s"]) + # --stop-on-exit tears down the VM even if the host orchestrator disappears. + os._exit(0) + + +if __name__ == "__main__": + main() diff --git a/src/backintel/smol_guest/execute.py b/src/backintel/smol_guest/execute.py new file mode 100644 index 0000000..7c20eba --- /dev/null +++ b/src/backintel/smol_guest/execute.py @@ -0,0 +1,41 @@ +"""Trusted guest entry point: apply limits before executing analyst-controlled code.""" + +import ctypes +import json +import os +import resource +import subprocess +import sys + + +def main(): + limits = json.loads(sys.argv[1]) + libc = ctypes.CDLL(None, use_errno=True) + # crun exec reopens the image root for each command. Give this process its + # own read-only root view without freezing the agent's image metadata path. + if libc.unshare(0x00020000) != 0: # CLONE_NEWNS + raise OSError(ctypes.get_errno(), "Cannot isolate workload mounts") + subprocess.run(["mount", "--make-rprivate", "/"], check=True) + subprocess.run(["mount", "--rbind", "/", "/"], check=True) + subprocess.run(["mount", "-o", "remount,bind,ro", "/"], check=True) + os.chroot("/") + if not os.statvfs("/").f_flag & os.ST_RDONLY: + raise RuntimeError("Workload root is not read-only") + # Drop the bounding set while still privileged, then permanently become the analyst. + for capability in range(41): + if libc.prctl(24, capability, 0, 0, 0) != 0: # PR_CAPBSET_DROP + raise OSError(ctypes.get_errno(), "Cannot drop guest capabilities") + if libc.prctl(38, 1, 0, 0, 0) != 0: # PR_SET_NO_NEW_PRIVS + raise OSError(ctypes.get_errno(), "Cannot disable privilege escalation") + resource.setrlimit(resource.RLIMIT_NPROC, (limits["pids"], limits["pids"])) + resource.setrlimit(resource.RLIMIT_NOFILE, (1024, 1024)) + resource.setrlimit(resource.RLIMIT_CORE, (0, 0)) + os.setgroups([]) + os.setgid(10001) + os.setuid(10001) + os.chdir("/workspace") + os.execvp(sys.argv[2], sys.argv[2:]) + + +if __name__ == "__main__": + main() diff --git a/src/backintel/smol_runtime.py b/src/backintel/smol_runtime.py new file mode 100644 index 0000000..9e95b7c --- /dev/null +++ b/src/backintel/smol_runtime.py @@ -0,0 +1,86 @@ +"""Pinned local runtime and image admission. Preparation is separate from execution.""" + +import fcntl +import hashlib +import json +import os +import platform +import subprocess +import time +from contextlib import contextmanager +from functools import lru_cache +from pathlib import Path + +VERSION = "1.31.1" +ARCHIVE_SHA256 = "69196e9880931344411eaf70b667c6633bb3015dc9f741cb85600f85a85d480c" +BINARY_SHA256 = "ae78eda57cb3a4d88960c74077055972fc058441ed49fded26a5b2f146faf086" +RELEASE_URL = f"https://github.com/smol-machines/smolvm/releases/download/v{VERSION}/smolvm-{VERSION}-darwin-arm64.tar.gz" +RUNTIME_HOME = Path.home() / ".local/share/backintel" + + +@lru_cache(maxsize=8) +def _digest(path: str, _stat_identity: tuple[int, int, int]) -> str: + with open(path, "rb") as handle: + return hashlib.file_digest(handle, "sha256").hexdigest() + + +def digest(path: Path) -> str: + info = path.stat() + return _digest(str(path.resolve()), (info.st_size, info.st_mtime_ns, info.st_ctime_ns)) + + +def runtime(): + if (platform.system(), platform.machine()) != ("Darwin", "arm64"): + raise RuntimeError("This SmolVM release is qualified on macOS ARM64; no runtime fallback is allowed.") + engine = Path( + os.environ.get("BACKINTEL_SMOLVM") or RUNTIME_HOME / f"runtimes/smolvm-{VERSION}-darwin-arm64/smolvm" + ) + binary = engine.parent / "smolvm-bin" + if not engine.is_file() or not binary.is_file() or digest(binary) != BINARY_SHA256: + raise RuntimeError("Prepare the pinned SmolVM runtime with uv run python -m scripts.prepare_smolvm.") + return str(engine) + + +def image_archive(image: str | Path | None): + path = Path( + image or os.environ.get("BACKINTEL_SMOLVM_IMAGE") or RUNTIME_HOME / "images/analysis-py312.tar" + ).resolve() + receipt = path.with_suffix(".json") + if not path.is_file() or not receipt.is_file(): + raise RuntimeError( + "Prepare a local SmolVM image archive with scripts.prepare_smolvm; runtime pulls are disabled." + ) + identity = json.loads(receipt.read_text()) + if digest(path) != identity["archive_sha256"]: + raise ValueError("SmolVM image archive checksum does not match its preparation receipt.") + return path, identity + + +@contextmanager +def image_staging_lock(): + """Serialize CLI image staging, whose hardlink/copy fallback is not atomic.""" + RUNTIME_HOME.mkdir(parents=True, exist_ok=True) + with (RUNTIME_HOME / "image-staging.lock").open("a") as lock: + deadline = time.monotonic() + 120 + while True: + try: + fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB) + break + except BlockingIOError: + if time.monotonic() >= deadline: + raise TimeoutError("Another sandbox is still staging its SmolVM image.") from None + time.sleep(0.1) + try: + yield + finally: + fcntl.flock(lock, fcntl.LOCK_UN) + + +def host_environment(): + # Never inherit model credentials, proxy settings, or SMOLVM_* policy overrides. + return {"PATH": "/usr/bin:/bin:/usr/sbin:/sbin", "HOME": str(Path.home()), "TMPDIR": "/tmp"} + + +def capacity(): + memory = subprocess.check_output(["sysctl", "-n", "hw.memsize"], text=True, timeout=5) + return {"memory_bytes": int(memory), "cpus": os.cpu_count() or 1} diff --git a/src/backintel/smol_services.py b/src/backintel/smol_services.py new file mode 100644 index 0000000..34c086b --- /dev/null +++ b/src/backintel/smol_services.py @@ -0,0 +1,142 @@ +"""Explicit service sockets: offline VMs can reach only their assigned store endpoints.""" + +import json +import selectors +import socket +import subprocess +import tempfile +import threading +from pathlib import Path +from urllib.parse import urlsplit + + +def docker_json(*args): + result = subprocess.run(["docker", *args], capture_output=True, text=True, timeout=15) + if result.returncode: + raise RuntimeError("Cannot inspect the assigned store infrastructure.") + return json.loads(result.stdout) + + +class ServiceBridges: + def __init__(self, network, env, allowed_hosts): + self.network = network + self.directory = tempfile.TemporaryDirectory(prefix="bi-svc-", dir="/tmp") + self.services = [] + self.listeners = [] + self.connections = set() + self.closed = False + self._lock = threading.Lock() + self._slots = threading.BoundedSemaphore(32) + endpoints = set() + for key, default_port in [ + ("BACKINTEL_POSTGRES_DSN", 5432), + ("BACKINTEL_MONGO_URI", 27017), + ("BACKINTEL_CH_URL", 8123), + ]: + if key in env: + url = urlsplit(env[key]) + endpoints.add((url.hostname, url.port or default_port)) + if "BACKINTEL_ICEBERG_TABLES" in env: + endpoints.add(("backintel-garage", 3900)) + if not network or not endpoints or {host for host, _ in endpoints} != allowed_hosts: + self.close() + raise ValueError("Networked analysis requires explicit assigned store endpoints.") + try: + details = docker_json("network", "inspect", network)[0] + if not details.get("Internal"): + raise ValueError("Assigned store infrastructure must use an internal network.") + for index, (host, port) in enumerate(sorted(endpoints)): + matches = [] + for identity, peer in details.get("Containers", {}).items(): + info = docker_json("inspect", identity)[0] + aliases = info["NetworkSettings"]["Networks"].get(network, {}).get("Aliases") or [] + if host in {peer.get("Name"), *aliases}: + matches.append((identity, info)) + if len(matches) != 1: + raise ValueError("An assigned hostname must identify exactly one store container.") + identity, info = matches[0] + bindings = info["NetworkSettings"]["Ports"].get(f"{port}/tcp") or [] + bindings = [item for item in bindings if item["HostIp"] == "127.0.0.1"] + if len(bindings) != 1: + raise ValueError("Assigned stores need a single localhost-only published TCP port.") + host_socket = str(Path(self.directory.name) / f"{index}.sock") + service = dict( + host=host, + port=port, + socket=f"/backintel-sockets/{index}.sock", + identity=identity, + host_port=int(bindings[0]["HostPort"]), + host_socket=host_socket, + ) + self.services.append(service) + listener = socket.socket(socket.AF_UNIX) + listener.bind(host_socket) + listener.listen(32) + listener.settimeout(0.2) + self.listeners.append(listener) + threading.Thread(target=self._serve, args=(listener, service), daemon=True).start() + except BaseException: + self.close() + raise + + def validate(self): + for service in self.services: + info = docker_json("inspect", service["identity"])[0] + bindings = info["NetworkSettings"]["Ports"].get(f"{service['port']}/tcp") or [] + if ( + not info["State"]["Running"] + or self.network not in info["NetworkSettings"]["Networks"] + or {"HostIp": "127.0.0.1", "HostPort": str(service["host_port"])} not in bindings + ): + raise ValueError("An assigned store endpoint changed; cancelling the sandbox.") + + def _serve(self, listener, service): + while not self.closed: + try: + client, _ = listener.accept() + except TimeoutError: + continue + except OSError: + return + if not self._slots.acquire(blocking=False): + client.close() + continue + threading.Thread(target=self._relay, args=(client, service), daemon=True).start() + + def _relay(self, client, service): + upstream = None + try: + self.validate() + upstream = socket.create_connection(("127.0.0.1", service["host_port"]), timeout=5) + with self._lock: + if self.closed: + return + self.connections.update((client, upstream)) + with selectors.DefaultSelector() as selector: + selector.register(client, selectors.EVENT_READ, upstream) + selector.register(upstream, selectors.EVENT_READ, client) + while not self.closed: + for key, _ in selector.select(timeout=0.2): + data = key.fileobj.recv(65536) + if not data: + return + key.data.sendall(data) + except (OSError, ValueError, RuntimeError, subprocess.TimeoutExpired): + return + finally: + self._slots.release() + with self._lock: + for connection in (client, upstream): + if connection: + self.connections.discard(connection) + connection.close() + + def close(self): + self.closed = True + for listener in self.listeners: + listener.close() + with self._lock: + for connection in self.connections: + connection.close() + self.connections.clear() + self.directory.cleanup() diff --git a/src/backintel/stores/base.py b/src/backintel/stores/base.py index 688b685..ffa1df1 100644 --- a/src/backintel/stores/base.py +++ b/src/backintel/stores/base.py @@ -77,7 +77,7 @@ def python_mounts(self) -> dict[str, str]: ... def sandbox_network(self) -> str | None: - """Docker network the sandbox must join to reach the store, or None for no network.""" + """Internal source network used to validate assigned service sockets, or None for local files.""" ... def close(self) -> None: ... diff --git a/src/backintel/stores/postgres.py b/src/backintel/stores/postgres.py index 395f905..7571a64 100644 --- a/src/backintel/stores/postgres.py +++ b/src/backintel/stores/postgres.py @@ -1,7 +1,8 @@ """PostgreSQL store: a local container on an internal Docker network, read through a read-only role. -The runner connects over localhost; the sandbox joins the same internal network (no internet) -and connects by container name. The `analyst` role has SELECT only and read-only transactions. +The runner connects over localhost; the SmolVM sandbox receives only a socket bridge to +the assigned service, using the same container hostname inside its offline guest. +The `analyst` role has SELECT only and read-only transactions. """ from __future__ import annotations diff --git a/tests/test_analysis_completion.py b/tests/test_analysis_completion.py new file mode 100644 index 0000000..32c217e --- /dev/null +++ b/tests/test_analysis_completion.py @@ -0,0 +1,135 @@ +"""Structured completion must survive text-only model stops without inventing success.""" + +import asyncio +import json +import os +import sqlite3 +from types import SimpleNamespace + +import pytest +from langchain_core.language_models.fake_chat_models import FakeMessagesListChatModel +from langchain_core.messages import AIMessage + +from backintel.analysis import AnalysisBrief, build_analysis_agent, run_analysis +from backintel.artifacts import verify_bundle +from backintel.stores.sqlite import SQLiteStore + + +class CompletionTestModel(FakeMessagesListChatModel): + def bind_tools(self, tools, **kwargs): + return self + + +def call(name, args, index): + return AIMessage(content="", tool_calls=[{"name": name, "args": args, "id": f"test-{index}"}]) + + +def completion(): + return call( + "AnalysisCompletion", + {"status": "cannot_answer", "summary": "Missing required inputs.", "reason": "Costs are absent."}, + 99, + ) + + +@pytest.fixture +def store(tmp_path): + source = tmp_path / "source.sqlite" + with sqlite3.connect(source) as connection: + connection.execute("CREATE TABLE items(value INTEGER)") + connection.executemany("INSERT INTO items VALUES (?)", [(1,), (2,)]) + store = SQLiteStore(source, dataset="completion-test") + try: + yield store + finally: + store.close() + + +def graph(store, responses, *, max_model_calls=4): + sandbox = SimpleNamespace(env={}, resource_usage=lambda: {"profile": {}}) + return build_analysis_agent( + CompletionTestModel(responses=responses), + store, + sandbox, + {}, + {}, + max_model_calls=max_model_calls, + ) + + +@pytest.mark.parametrize("recover", [False, True]) +def test_valid_completion_and_text_only_recovery(store, recover): + responses = ([AIMessage(content="I cannot calculate this.")] if recover else []) + [completion()] + result = asyncio.run(graph(store, responses).ainvoke({"messages": [("user", "Analyze the data.")]})) + assert result["structured_response"].status == "cannot_answer" + assert bool(result.get("completion_repair_attempted")) is recover + if recover: + assert any(m.type == "human" and "AnalysisCompletion" in m.content for m in result["messages"]) + assert any(m.type == "ai" and m.content == "I cannot calculate this." for m in result["messages"]) + + +def test_repeated_text_stop_fails_without_resetting_call_limit(store): + agent = graph(store, [AIMessage(content="Done."), AIMessage(content="Still done.")]) + with pytest.raises(ValueError, match="Missing AnalysisCompletion after one recovery prompt"): + asyncio.run(agent.ainvoke({"messages": [("user", "Analyze the data.")]})) + + +def test_completion_recovery_respects_original_call_limit(store): + from langchain.agents.middleware.model_call_limit import ModelCallLimitExceededError + + agent = graph(store, [AIMessage(content="Done."), completion()], max_model_calls=1) + with pytest.raises(ModelCallLimitExceededError): + asyncio.run(agent.ainvoke({"messages": [("user", "Analyze the data.")]})) + + +@pytest.mark.skipif(os.environ.get("BACKINTEL_SMOLVM_TESTS") != "1", reason="Requires prepared SmolVM") +def test_recovered_completion_exports_verified_bundle(tmp_path, store): + spec = { + "title": "Completion recovery fixture", + "summary": "The two assigned values sum to three.", + "evidence": [ + { + "id": "sum", + "kind": "sql", + "code": "SELECT SUM(value) AS Total FROM items", + "source_ids": ["assigned"], + } + ], + "tables": [ + { + "id": "total", + "title": "Sum", + "columns": ["Total"], + "rows": [[3]], + "units": {"Total": "items"}, + "evidence_ids": ["sum"], + } + ], + "metrics": [], + "charts": [], + } + model = CompletionTestModel( + responses=[ + call("write_analysis_file", {"path": "/workspace/report.json", "content": json.dumps(spec)}, 0), + call("render_report", {"spec_path": "/workspace/report.json"}, 1), + AIMessage(content="The report is ready."), + call("AnalysisCompletion", {"status": "completed", "summary": "The report is ready."}, 2), + ] + ) + output = tmp_path / "run" + record = asyncio.run( + run_analysis( + AnalysisBrief(objective="Report the sum of the assigned values."), + model=model, + model_id="scripted-reference", + store=store, + output=output, + scripted=True, + ) + ) + assert record["status"] == "completed", record.get("error") + assert record["completion_recovery_attempts"] == 1 + assert record["resources"]["backend"] == "smolvm" + bundle = output / "artifacts" / "report" + assert verify_bundle(bundle) == record["artifacts"] + assert json.loads((bundle / "report.json").read_text())["tables"][0]["rows"] == [[3]] diff --git a/tests/test_analysis_sandbox.py b/tests/test_analysis_sandbox.py index 82e876b..90eb721 100644 --- a/tests/test_analysis_sandbox.py +++ b/tests/test_analysis_sandbox.py @@ -1,4 +1,4 @@ -"""Real container acceptance is opt-in; the default suite never needs Docker or a model.""" +"""Real SmolVM acceptance is opt-in; the default suite never needs Docker or a model.""" import json import os @@ -10,10 +10,11 @@ import pytest from backintel.analysis_sandbox import AnalysisSandbox, ResourceProfile -from backintel.sandbox import DOCKER_IMAGE +from backintel.sandbox import DOCKER_IMAGE, PythonSandbox +from backintel.smol_runtime import runtime -docker = pytest.mark.skipif( - os.environ.get("BACKINTEL_DOCKER_TESTS") != "1", reason="Docker acceptance opt-in" +smolvm = pytest.mark.skipif( + os.environ.get("BACKINTEL_SMOLVM_TESTS") != "1", reason="SmolVM acceptance opt-in" ) @@ -29,7 +30,7 @@ def test_reject_invalid_resource_and_write_contracts(): sandbox.write_file("/workspace/program.py", "x" * (256 * 1024 + 1)) -@docker +@smolvm def test_real_persistent_workspace_boundary_and_export(tmp_path): source = tmp_path / "source.csv" source.write_text("category,revenue\nA,12\nB,18\n") @@ -42,6 +43,10 @@ def test_real_persistent_workspace_boundary_and_export(tmp_path): assert sandbox.read_file("/workspace/program.py") == "print('recovered')" assert sandbox.run_shell("printf forbidden >> /assigned/source.csv").exit_code != 0 assert sandbox.run_shell("touch /etc/forbidden").exit_code != 0 + assert sandbox.run_shell("touch /home/analyst/forbidden").exit_code != 0 + result = sandbox.run_python("import os; print(os.getuid()); print(open('/proc/self/status').read())") + assert "10001" in result.stdout and "CapEff:\t0000000000000000" in result.stdout + assert "NoNewPrivs:\t1" in result.stdout assert sandbox.run_shell("test ! -e /var/run/docker.sock && test ! -d /Users/hudson").exit_code == 0 denied = sandbox.run_python("import socket; socket.create_connection(('1.1.1.1',443),timeout=1)") assert denied.exit_code != 0 @@ -75,10 +80,10 @@ def test_real_persistent_workspace_boundary_and_export(tmp_path): assert usage["image_id"].startswith("sha256:") assert source.read_text() == "category,revenue\nA,12\nB,18\n" assert (tmp_path / "delivered/chart.png").is_file() - assert subprocess.run(["docker", "inspect", sandbox.name], capture_output=True).returncode != 0 + assert sandbox.name not in subprocess.check_output([runtime(), "machine", "ls", "--quiet"], text=True) -@docker +@smolvm def test_real_command_deadline_output_and_storage_limits(tmp_path): profile = ResourceProfile(workspace_mib=1, artifacts_mib=1, command_timeout_s=2, run_timeout_s=30) with AnalysisSandbox(profile=profile) as sandbox: @@ -92,7 +97,7 @@ def test_real_command_deadline_output_and_storage_limits(tmp_path): assert sandbox.run_shell("echo cannot-run").exit_code != 0 -@docker +@smolvm def test_real_whole_run_deadline_and_command_budget(): with AnalysisSandbox(timeout_s=2) as sandbox: time.sleep(3) @@ -103,10 +108,10 @@ def test_real_whole_run_deadline_and_command_budget(): assert sandbox.closed -@docker -def test_real_internal_network_allows_only_assigned_peers(): +@smolvm +def test_real_offline_vm_reaches_only_assigned_store_socket(): suffix = uuid.uuid4().hex[:12] - network, source, extra = (f"backintel-test-{kind}-{suffix}" for kind in ("net", "source", "extra")) + network, source = (f"backintel-test-{kind}-{suffix}" for kind in ("net", "source")) subprocess.run(["docker", "network", "create", "--internal", network], check=True, capture_output=True) try: subprocess.run( @@ -116,11 +121,11 @@ def test_real_internal_network_allows_only_assigned_peers(): "-d", "--name", source, - "--network", - network, "--read-only", "--memory", "128m", + "-p", + "127.0.0.1::8080", DOCKER_IMAGE, "python", "-m", @@ -132,65 +137,40 @@ def test_real_internal_network_allows_only_assigned_peers(): check=True, capture_output=True, ) - subprocess.run( - [ - "docker", - "run", - "-d", - "--name", - extra, - "--network", - network, - "--read-only", - "--memory", - "32m", - DOCKER_IMAGE, - "sleep", - "infinity", - ], - check=True, - capture_output=True, - ) + subprocess.run(["docker", "network", "connect", network, source], check=True, capture_output=True) env = {"BACKINTEL_CH_URL": f"http://{source}:8080"} with AnalysisSandbox(env=env, network=network) as sandbox: - private_network = sandbox.network - assert private_network != network peers = json.loads( subprocess.check_output( - ["docker", "network", "inspect", private_network, "--format", "{{json .Containers}}"] + ["docker", "network", "inspect", network, "--format", "{{json .Containers}}"], text=True ) ) - assert {peer["Name"] for peer in peers.values()} == {source, sandbox.name} + assert {peer["Name"] for peer in peers.values()} == {source} result = sandbox.run_python( f"import urllib.request; print(urllib.request.urlopen('http://{source}:8080/README.md').read(40))" ) assert result.exit_code == 0, result.stderr assert "Offline analytical workspace" in result.stdout - assert ( - sandbox.run_python( - "import socket; socket.create_connection(('1.1.1.1',443),timeout=1)" - ).exit_code - != 0 - ) - # A new unrelated testbed peer is harmless, but one on the private run - # network must still stop execution. + for host, port in [("1.1.1.1", 443), ("127.0.0.1", 55432), ("host.docker.internal", 80)]: + assert ( + sandbox.run_python( + f"import socket; socket.create_connection(({host!r},{port}),timeout=1)" + ).exit_code + != 0 + ) subprocess.run( - ["docker", "network", "connect", sandbox.network, extra], check=True, capture_output=True + ["docker", "network", "disconnect", network, source], check=True, capture_output=True ) - with pytest.raises(ValueError, match="unassigned peer"): + with pytest.raises(ValueError, match="endpoint changed"): sandbox.run_shell("echo must-not-execute") - assert ( - subprocess.run(["docker", "network", "inspect", private_network], capture_output=True).returncode - != 0 - ) + assert sandbox.closed assert subprocess.run(["docker", "inspect", source], capture_output=True).returncode == 0 finally: - for name in (source, extra): - subprocess.run(["docker", "rm", "--force", name], capture_output=True) + subprocess.run(["docker", "rm", "--force", source], capture_output=True) subprocess.run(["docker", "network", "rm", network], capture_output=True) -@docker +@smolvm def test_real_large_scan_with_small_memory_budget(tmp_path): """A real on-disk input larger than the process memory, created without model calls.""" # Opt-in large acceptance generates a 320 MiB text corpus, not a >10 GiB roadmap claim. @@ -244,7 +224,7 @@ def test_store_mounts_are_relocated_outside_protected_workspace(): assert env[key].startswith("/workspace/") # Does not mutate the host-side Store context. -@docker +@smolvm def test_real_store_path_remap_reads_sqlite_and_rejects_writes(tmp_path): import sqlite3 @@ -270,40 +250,89 @@ def test_real_store_path_remap_reads_sqlite_and_rejects_writes(tmp_path): store.close() -def test_private_network_is_cleaned_if_connect_fails(monkeypatch): +def test_service_socket_cleanup_when_endpoint_admission_fails(monkeypatch, tmp_path): + import backintel.smol_services as module + + created = [] + real_temp = module.tempfile.TemporaryDirectory + + def temporary(**kwargs): + result = real_temp(dir=tmp_path) + created.append(Path(result.name)) + return result + + monkeypatch.setattr(module.tempfile, "TemporaryDirectory", temporary) + monkeypatch.setattr(module, "docker_json", lambda *args: [{"Internal": True, "Containers": {}}]) + with pytest.raises(ValueError, match="exactly one"): + module.ServiceBridges("test", {"BACKINTEL_CH_URL": "http://selected:8123"}, {"selected"}) + assert created and all(not path.exists() for path in created) + + +@smolvm +def test_real_benchmark_python_uses_smolvm(): + result = PythonSandbox().run("import os; print(os.getuid())", env={}, mounts={}) + assert result.backend == "smolvm" + assert result.exit_code == 0 and result.stdout.strip() == "10001", result.stderr + + +@smolvm +def test_real_concurrent_cold_startup_preserves_archive(tmp_path): + import shutil + from concurrent.futures import ThreadPoolExecutor + from threading import Barrier + + from backintel.smol_runtime import digest, image_archive + + original, receipt = image_archive(None) + archive = tmp_path / "concurrent.tar" + shutil.copyfile(original, archive) + # Trailing padding gives this valid tar a never-staged identity, exposing the + # upstream cold-cache hardlink race rather than testing an existing seed. + with archive.open("ab") as stream: + stream.write(uuid.uuid4().bytes + bytes(10240)) + expected = digest(archive) + archive.with_suffix(".json").write_text(json.dumps({**receipt, "archive_sha256": expected})) + barrier = Barrier(2) + profile = ResourceProfile(memory_mib=768, cpus=1, workspace_mib=128, temporary_mib=128) + + def execute(): + barrier.wait(timeout=10) + with AnalysisSandbox(image=archive, profile=profile) as sandbox: + result = sandbox.run_python("print(6 * 7)") + assert result.exit_code == 0 and result.stdout.strip() == "42", result.stderr + + with ThreadPoolExecutor(max_workers=2) as pool: + list(pool.map(lambda _: execute(), range(2))) + assert digest(archive) == expected + + +def test_missing_image_archive_fails_without_remote_pull(tmp_path): + from backintel.smol_runtime import image_archive + + with pytest.raises(RuntimeError, match="runtime pulls are disabled"): + image_archive(tmp_path / "absent.tar") + + +def test_partial_machine_creation_failure_is_cleaned(monkeypatch, tmp_path): import backintel.analysis_sandbox as module actions = [] - sandbox = AnalysisSandbox(env={"BACKINTEL_CH_URL": "http://selected:8123"}, network="shared") - monkeypatch.setattr(module, "docker_capacity", lambda: {"memory_bytes": 8 * 1024**3, "cpus": 4}) - monkeypatch.setattr(module.shutil, "which", lambda _: "/docker") - - def checked(args, **kwargs): - actions.append(args) - if args[1:3] == ["inspect", "--type=image"]: - return "sha256:fixture" - if args[1:3] == ["network", "inspect"]: - return json.dumps( - { - "Internal": True, - "Containers": {"selected-id": {"Name": "selected"}, "other-id": {"Name": "other"}}, - } - ) - if args[1] == "inspect": - return json.dumps({"shared": {"Aliases": []}}) - if args[1:3] == ["network", "connect"]: - raise RuntimeError("controlled connect failure") - return "" - - def cleanup(args, **kwargs): - actions.append(args) - return subprocess.CompletedProcess(args, 0, stdout="{}") - - monkeypatch.setattr(module, "_checked", checked) - monkeypatch.setattr(module.subprocess, "run", cleanup) - with pytest.raises(RuntimeError, match="controlled connect failure"): + monkeypatch.setattr(module, "runtime", lambda: "/smolvm") + monkeypatch.setattr(module, "capacity", lambda: {"memory_bytes": 8 * 1024**3, "cpus": 4}) + monkeypatch.setattr( + module, "image_archive", lambda _: (tmp_path / "image.tar", {"image_id": "sha256:test"}) + ) + + def run(arguments, **kwargs): + actions.append(arguments) + if "create" in arguments: + raise subprocess.TimeoutExpired(arguments, 120) + return subprocess.CompletedProcess(arguments, 0, stdout="", stderr="") + + monkeypatch.setattr(module.subprocess, "run", run) + sandbox = AnalysisSandbox() + with pytest.raises(subprocess.TimeoutExpired): sandbox.start() - assert ["docker", "network", "rm", sandbox.network] in actions - assert ["docker", "network", "disconnect", "--force", sandbox.network, "selected-id"] in actions - assert not any(a[:3] == ["docker", "network", "connect"] and a[-1] == "other-id" for a in actions) assert sandbox.closed + assert any(args[1:3] == ["machine", "delete"] and sandbox.name in args for args in actions) + assert not Path(sandbox._staging.name).exists() diff --git a/tests/test_chatgpt_plan.py b/tests/test_chatgpt_plan.py index 9892f49..b277b68 100644 --- a/tests/test_chatgpt_plan.py +++ b/tests/test_chatgpt_plan.py @@ -96,7 +96,7 @@ async def run(brief, **kwargs): monkeypatch.setattr(backintel.models, "build_model", model) monkeypatch.setattr(backintel.analysis, "run_analysis", run) monkeypatch.setattr(backintel.stores, "open_store", lambda *args: SimpleNamespace(close=lambda: None)) - monkeypatch.setenv("BACKINTEL_SANDBOX", "docker") + monkeypatch.setenv("BACKINTEL_SANDBOX", "smolvm") monkeypatch.setenv("BACKINTEL_MODEL", "openrouter/openai/gpt-6-luna") monkeypatch.delenv("BACKINTEL_ANALYSIS_MODEL", raising=False) args = ["analyze", "--brief", str(brief_path), "--store", "test", "--output", str(tmp_path / "out")] diff --git a/tests/test_pr4_regressions.py b/tests/test_pr4_regressions.py index 511e97e..061ca8e 100644 --- a/tests/test_pr4_regressions.py +++ b/tests/test_pr4_regressions.py @@ -142,6 +142,6 @@ def test_local_experiments_are_unique_and_explicit_names_cannot_append(tmp_path) def test_host_python_backend_is_rejected_before_code_execution(tmp_path): marker = tmp_path / "must-not-exist" - with pytest.raises(ValueError, match="Docker"): + with pytest.raises(ValueError, match="SmolVM"): PythonSandbox("subprocess").run(f"open({str(marker)!r}, 'w').write('escaped')", env={}, mounts={}) assert not marker.exists() diff --git a/uv.lock b/uv.lock index bc1de92..f7f604d 100644 --- a/uv.lock +++ b/uv.lock @@ -45,7 +45,7 @@ wheels = [ [[package]] name = "backintel" -version = "2.0.0a0" +version = "0.0.1" source = { editable = "." } dependencies = [ { name = "boto3" },