Publish the reproducible specialist evaluation suite

.agents/skills/backintel-benchmark-world/SKILL.md

@@ -3,7 +3,7 @@ name: backintel-benchmark-world
 description: Reuse BackIntel's local benchmark setup and independent grading conventions.
 ---
 
-Status: unreviewed agent-authored knowledge, 2026-11-09.
+Status: unreviewed agent-authored knowledge, updated 2026-11-12.
 
 BackIntel uses LangSmith Evals over a single Deep Agents analyst. See docs/benchmark.md
 for the task specification and README.md for commands. Add storage adapters behind the
@@ -21,3 +21,33 @@ Fix a harness defect before treating its score as agent capability. Keep the ori
 and label the rerun. Read a complete relevant trace, not only aggregate percentages. A
 numerical score does not establish narrative quality; use the independently versioned
 method judge and its written reasons. Cost reports must say whether judge cost is included.
+
+The opt-in analysis graph uses persistent SmolVM execution and a bounded report specification.
+After the analyst stops, a clean renderer VM produces the delivered files from the
+host-retained specification. The artifact manifest establishes file integrity; table citations
+establish reproducibility. Neither establishes analytical truth. New artifact tasks need an
+independent reference and reconciliation across report tables, charts, CSV and workbook cells.
+See `docs/analysis-sandbox.md` and `evals/analysis/Task.md` for the implemented pattern.
+Scripted graph runs exercise real tools and export but cannot establish model judgment or
+unseen-domain competence. Keep their results separate from live LangSmith experiments.
+
+Cross-format reconciliation must use declared numerical tolerances: XLSX serialization
+can change a floating-point value's last digit. Preserve null versus zero and text types;
+do not weaken population, unit or metric-binding checks. The controlled energy/availability
+cases in `scripts/analysis_cases.py` provide independent-oracle patterns for irregular
+observations and overlapping intervals. Their exact requests and truth stay in the task
+specification and evaluator-owned references. A comparison with unclassified failures or
+unknown charges cannot qualify an arm by excluding those cases from its denominator.
+
+A successful render inside an analyst VM is not a delivered bundle. A completed analysis
+requires structured completion and the host's clean export. Classify a text-only terminal
+response separately from infrastructure failure; retain the provisional score and trace
+when adding reviewed failure feedback. A correct refusal must not imply that temporary
+artifacts were delivered. See the live SmolVM rerun in `docs/results-analysis-sandbox.md`.
+
+The specialist now explicitly requires its completion tool and permits one recovery prompt
+for a text-only stop. Keep recovery inside the original graph's call/time limits and count
+the failed completion attempt before jumping back to the model. Test both recovered export
+and exhaustion; a live success with zero recovery prompts does not exercise the fallback.
+Selected-case reruns retain full frozen dataset validation and have their own experiment;
+never combine their passes with an earlier baseline to claim a complete candidate suite.

README.md

@@ -68,10 +68,10 @@ have been used for testing; BackIntel has not trained or fine-tuned its models o
 
 ### 1. Check out and verify the project
 
-Until a versioned release is published, use the development branch:
+Until a versioned release is published, use a source checkout of `main`:
 
 ```bash
-git clone --branch codex/backintel-v2 https://github.com/IntelIP/BackIntel.git
+git clone https://github.com/IntelIP/BackIntel.git
 cd BackIntel
 uv sync --frozen --python 3.12
 uv run backintel --help
@@ -235,8 +235,26 @@ The 3/4 baseline remains unchanged: a targeted success is not a new 4/4 experime
 Historical results and failures are in the [capability report](docs/results.md) and
 [scorecard](results/scorecard.md). The [SmolVM baseline](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/datasets/d2e934c2-77e8-4926-93b3-0bcd644b200d/compare?selectedSessions=f667f6fc-4e8d-4bba-bcbf-fcde7d808d40)
 and [energy rerun](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/datasets/d2e934c2-77e8-4926-93b3-0bcd644b200d/compare?selectedSessions=ff29645c-8e96-4cd8-875c-302ae006acf8)
-are separate LangSmith experiments; workspace access may be required. Their expanded
-four-case runner remains pending publication, so it is not part of the checkout quickstart.
+are separate LangSmith experiments; workspace access may be required. The
+[four-case suite](evals/analysis/cases.md) includes fixture builders, independent references,
+wrong-answer controls and a runnable comparison harness.
+
+After preparing SmolVM, verify all four cases with scripted decisions and no model calls:
+
+```bash
+uv run python -m scripts.run_analysis_comparison
+```
+
+This exercises the actual graph, sandbox and artifact delivery; it is integration evidence,
+not a live quality score. To prepare only the fixtures without starting a VM:
+
+```bash
+uv run python -m scripts.analysis_cases data/output/my-new-analysis-fixtures
+```
+
+Use a new output directory for every run. The [evaluation guide](docs/analysis-sandbox.md)
+documents the subscription and explicitly budgeted API modes. Live comparisons require
+explicit model IDs, retain failed attempts and never promote a model automatically.
 
 ### Evaluation and data quality
 

docs/analysis-artifacts.md

@@ -0,0 +1,77 @@
+# Analysis artifact contract
+
+`backintel.artifacts` turns a JSON analysis specification into a self-contained report bundle.
+It uses Matplotlib for PNG/SVG, Plotly for interactive HTML, and openpyxl for workbooks.
+No browser, remote font, CDN, map service or rendering service is required.
+
+The module is installed in the sandbox image. Call it from an analysis program:
+
+```python
+import json
+from backintel.artifacts import render_report
+
+spec = json.load(open('/workspace/report-spec.json'))
+manifest = render_report(spec, '/artifacts/report')
+print(json.dumps(manifest))
+```
+
+The output directory must not already exist. For a revision, use a fresh directory or let
+the orchestrator replace the exact previous sandbox output. Existing user files are not
+overwritten. The default bundle limit is 64 MiB, within the sandbox's separate storage quota.
+
+See `examples/analysis-report.json` for a complete **synthetic** example. It is authoring
+guidance, not a benchmark answer or a measurement of customer outcomes.
+
+## Specification
+
+The canonical schema and analytical instructions are in
+[`artifact-contract.md`](../src/backintel/agent_files/docs/artifact-contract.md), packaged
+in the sandbox and available through the analyst's instruction filesystem. It documents
+source/evidence bindings, table-backed metrics, the five supported chart families and
+bounds. Python table evidence must print one JSON object containing `columns` and `rows`;
+SQL/native-query evidence must reproduce the same table. The host owns run/source identity.
+
+## Output and trust
+
+The bundle contains `report.html`, `report.json`, `supporting.xlsx`, table CSVs, chart
+PNG/SVG/interactive HTML files, and `manifest.json`. The manifest records run identity,
+sources, metric definitions/bindings, evidence IDs, MIME types, byte counts and SHA-256 hashes.
+
+The report is responsive, printable and uses local system fonts. Interactive files embed
+the installed Plotly JavaScript and declare `connect-src 'none'`; no external resource is
+needed. Supporting exact values remain readable below each interactive chart.
+
+Excel cells containing text are forced to text, never formulas. Formula-like CSV text is
+prefixed with an apostrophe; the unchanged value remains in JSON and Excel text cells.
+Numeric values remain numeric. HTML and script payloads in data are escaped.
+
+`validate_report` checks schema, references, grain, units, finite values and bounds.
+`verify_bundle` checks exact file inventory, path/symlink restrictions, hashes, canonical
+report HTML and supporting CSV consistency. These checks do **not** prove the supplied
+calculation or narrative is true and are not a signature on untrusted files.
+
+The delivery boundary must regenerate the normalized specification using the trusted
+renderer in a fresh sandbox after the analyst stops. Do not serve arbitrary HTML or an
+agent-edited manifest directly from its writable working directory. Independent source
+calculations and benchmark references remain the evaluation harness's responsibility.
+
+## Package sources
+
+New renderer versions were checked on 2026-11-11: Matplotlib 3.11.2, Plotly 7.1.0 and
+the existing openpyxl 3.1.5. The sandbox dependency lock supplies exact transitive versions.
+
+- [Matplotlib governance](https://github.com/matplotlib/governance/blob/main/governance.md)
+  describes community governance and NumFOCUS affiliation; the
+  [license](https://matplotlib.org/stable/project/license.html) records shared development-team copyright.
+- [Plotly's company page](https://plotly.com/about-us/) identifies the company maintaining
+  its open-source libraries. Its [registry-validated LEI](https://lei.bloomberg.com/leis/view/549300IQLNOF8XICEZ30)
+  identifies Technologies Plotly Inc., Canadian legal/headquarters addresses, and natural-person
+  direct/ultimate parents. [Real Ventures](https://realventures.com/portfolio/) lists its investment.
+  These are documented ownership/control signals; a complete private beneficial-ownership register
+  is not published by those sources.
+- [openpyxl](https://openpyxl.readthedocs.io/en/stable/) identifies its independent authors,
+  volunteer maintenance, license and professional support organizations.
+
+Relevant APIs: [Plotly offline HTML](https://plotly.com/python/interactive-html-export/),
+[Matplotlib export](https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.savefig.html),
+and [openpyxl formatting](https://openpyxl.readthedocs.io/en/stable/styles.html).