diff --git a/.agents/skills/backintel-benchmark-world/SKILL.md b/.agents/skills/backintel-benchmark-world/SKILL.md index 183e5d9..445d350 100644 --- a/.agents/skills/backintel-benchmark-world/SKILL.md +++ b/.agents/skills/backintel-benchmark-world/SKILL.md @@ -3,7 +3,7 @@ name: backintel-benchmark-world description: Reuse BackIntel's local benchmark setup and independent grading conventions. --- -Status: unreviewed agent-authored knowledge, 2026-11-09. +Status: unreviewed agent-authored knowledge, updated 2026-11-12. BackIntel uses LangSmith Evals over a single Deep Agents analyst. See docs/benchmark.md for the task specification and README.md for commands. Add storage adapters behind the @@ -21,3 +21,33 @@ Fix a harness defect before treating its score as agent capability. Keep the ori and label the rerun. Read a complete relevant trace, not only aggregate percentages. A numerical score does not establish narrative quality; use the independently versioned method judge and its written reasons. Cost reports must say whether judge cost is included. + +The opt-in analysis graph uses persistent SmolVM execution and a bounded report specification. +After the analyst stops, a clean renderer VM produces the delivered files from the +host-retained specification. The artifact manifest establishes file integrity; table citations +establish reproducibility. Neither establishes analytical truth. New artifact tasks need an +independent reference and reconciliation across report tables, charts, CSV and workbook cells. +See `docs/analysis-sandbox.md` and `evals/analysis/Task.md` for the implemented pattern. +Scripted graph runs exercise real tools and export but cannot establish model judgment or +unseen-domain competence. Keep their results separate from live LangSmith experiments. + +Cross-format reconciliation must use declared numerical tolerances: XLSX serialization +can change a floating-point value's last digit. Preserve null versus zero and text types; +do not weaken population, unit or metric-binding checks. The controlled energy/availability +cases in `scripts/analysis_cases.py` provide independent-oracle patterns for irregular +observations and overlapping intervals. Their exact requests and truth stay in the task +specification and evaluator-owned references. A comparison with unclassified failures or +unknown charges cannot qualify an arm by excluding those cases from its denominator. + +A successful render inside an analyst VM is not a delivered bundle. A completed analysis +requires structured completion and the host's clean export. Classify a text-only terminal +response separately from infrastructure failure; retain the provisional score and trace +when adding reviewed failure feedback. A correct refusal must not imply that temporary +artifacts were delivered. See the live SmolVM rerun in `docs/results-analysis-sandbox.md`. + +The specialist now explicitly requires its completion tool and permits one recovery prompt +for a text-only stop. Keep recovery inside the original graph's call/time limits and count +the failed completion attempt before jumping back to the model. Test both recovered export +and exhaustion; a live success with zero recovery prompts does not exercise the fallback. +Selected-case reruns retain full frozen dataset validation and have their own experiment; +never combine their passes with an earlier baseline to claim a complete candidate suite. diff --git a/README.md b/README.md index 1d5675f..ad8d31b 100644 --- a/README.md +++ b/README.md @@ -68,10 +68,10 @@ have been used for testing; BackIntel has not trained or fine-tuned its models o ### 1. Check out and verify the project -Until a versioned release is published, use the development branch: +Until a versioned release is published, use a source checkout of `main`: ```bash -git clone --branch codex/backintel-v2 https://github.com/IntelIP/BackIntel.git +git clone https://github.com/IntelIP/BackIntel.git cd BackIntel uv sync --frozen --python 3.12 uv run backintel --help @@ -235,8 +235,26 @@ The 3/4 baseline remains unchanged: a targeted success is not a new 4/4 experime Historical results and failures are in the [capability report](docs/results.md) and [scorecard](results/scorecard.md). The [SmolVM baseline](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/datasets/d2e934c2-77e8-4926-93b3-0bcd644b200d/compare?selectedSessions=f667f6fc-4e8d-4bba-bcbf-fcde7d808d40) and [energy rerun](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/datasets/d2e934c2-77e8-4926-93b3-0bcd644b200d/compare?selectedSessions=ff29645c-8e96-4cd8-875c-302ae006acf8) -are separate LangSmith experiments; workspace access may be required. Their expanded -four-case runner remains pending publication, so it is not part of the checkout quickstart. +are separate LangSmith experiments; workspace access may be required. The +[four-case suite](evals/analysis/cases.md) includes fixture builders, independent references, +wrong-answer controls and a runnable comparison harness. + +After preparing SmolVM, verify all four cases with scripted decisions and no model calls: + +```bash +uv run python -m scripts.run_analysis_comparison +``` + +This exercises the actual graph, sandbox and artifact delivery; it is integration evidence, +not a live quality score. To prepare only the fixtures without starting a VM: + +```bash +uv run python -m scripts.analysis_cases data/output/my-new-analysis-fixtures +``` + +Use a new output directory for every run. The [evaluation guide](docs/analysis-sandbox.md) +documents the subscription and explicitly budgeted API modes. Live comparisons require +explicit model IDs, retain failed attempts and never promote a model automatically. ### Evaluation and data quality diff --git a/docs/analysis-artifacts.md b/docs/analysis-artifacts.md new file mode 100644 index 0000000..0057932 --- /dev/null +++ b/docs/analysis-artifacts.md @@ -0,0 +1,77 @@ +# Analysis artifact contract + +`backintel.artifacts` turns a JSON analysis specification into a self-contained report bundle. +It uses Matplotlib for PNG/SVG, Plotly for interactive HTML, and openpyxl for workbooks. +No browser, remote font, CDN, map service or rendering service is required. + +The module is installed in the sandbox image. Call it from an analysis program: + +```python +import json +from backintel.artifacts import render_report + +spec = json.load(open('/workspace/report-spec.json')) +manifest = render_report(spec, '/artifacts/report') +print(json.dumps(manifest)) +``` + +The output directory must not already exist. For a revision, use a fresh directory or let +the orchestrator replace the exact previous sandbox output. Existing user files are not +overwritten. The default bundle limit is 64 MiB, within the sandbox's separate storage quota. + +See `examples/analysis-report.json` for a complete **synthetic** example. It is authoring +guidance, not a benchmark answer or a measurement of customer outcomes. + +## Specification + +The canonical schema and analytical instructions are in +[`artifact-contract.md`](../src/backintel/agent_files/docs/artifact-contract.md), packaged +in the sandbox and available through the analyst's instruction filesystem. It documents +source/evidence bindings, table-backed metrics, the five supported chart families and +bounds. Python table evidence must print one JSON object containing `columns` and `rows`; +SQL/native-query evidence must reproduce the same table. The host owns run/source identity. + +## Output and trust + +The bundle contains `report.html`, `report.json`, `supporting.xlsx`, table CSVs, chart +PNG/SVG/interactive HTML files, and `manifest.json`. The manifest records run identity, +sources, metric definitions/bindings, evidence IDs, MIME types, byte counts and SHA-256 hashes. + +The report is responsive, printable and uses local system fonts. Interactive files embed +the installed Plotly JavaScript and declare `connect-src 'none'`; no external resource is +needed. Supporting exact values remain readable below each interactive chart. + +Excel cells containing text are forced to text, never formulas. Formula-like CSV text is +prefixed with an apostrophe; the unchanged value remains in JSON and Excel text cells. +Numeric values remain numeric. HTML and script payloads in data are escaped. + +`validate_report` checks schema, references, grain, units, finite values and bounds. +`verify_bundle` checks exact file inventory, path/symlink restrictions, hashes, canonical +report HTML and supporting CSV consistency. These checks do **not** prove the supplied +calculation or narrative is true and are not a signature on untrusted files. + +The delivery boundary must regenerate the normalized specification using the trusted +renderer in a fresh sandbox after the analyst stops. Do not serve arbitrary HTML or an +agent-edited manifest directly from its writable working directory. Independent source +calculations and benchmark references remain the evaluation harness's responsibility. + +## Package sources + +New renderer versions were checked on 2026-11-11: Matplotlib 3.11.2, Plotly 7.1.0 and +the existing openpyxl 3.1.5. The sandbox dependency lock supplies exact transitive versions. + +- [Matplotlib governance](https://github.com/matplotlib/governance/blob/main/governance.md) + describes community governance and NumFOCUS affiliation; the + [license](https://matplotlib.org/stable/project/license.html) records shared development-team copyright. +- [Plotly's company page](https://plotly.com/about-us/) identifies the company maintaining + its open-source libraries. Its [registry-validated LEI](https://lei.bloomberg.com/leis/view/549300IQLNOF8XICEZ30) + identifies Technologies Plotly Inc., Canadian legal/headquarters addresses, and natural-person + direct/ultimate parents. [Real Ventures](https://realventures.com/portfolio/) lists its investment. + These are documented ownership/control signals; a complete private beneficial-ownership register + is not published by those sources. +- [openpyxl](https://openpyxl.readthedocs.io/en/stable/) identifies its independent authors, + volunteer maintenance, license and professional support organizations. + +Relevant APIs: [Plotly offline HTML](https://plotly.com/python/interactive-html-export/), +[Matplotlib export](https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.savefig.html), +and [openpyxl formatting](https://openpyxl.readthedocs.io/en/stable/styles.html). diff --git a/docs/analysis-model-selection.md b/docs/analysis-model-selection.md new file mode 100644 index 0000000..ca35eed --- /dev/null +++ b/docs/analysis-model-selection.md @@ -0,0 +1,288 @@ +# Analytical specialist model and prompt selection + +Research date: 2026-11-11. This document separates current configuration, official vendor +claims, local adapter checks, engineering hypotheses and the subsequent controlled live evaluation. + +## Decision and current evidence + +The owner subsequently approved **GPT-6.1 Sol as the default specialist model**. The analysis +CLI now defaults to `chatgpt-plan/gpt-6.1-sol` with medium reasoning. An explicit `--model` +overrides `BACKINTEL_ANALYSIS_MODEL`, which overrides the built-in default. This is a manual +owner decision after the live comparison; the benchmark itself does not promote models. + +Build one bounded analysis-and-visualization specialist configuration with an explicit model +and reasoning setting. Keep the existing analyst's configuration unchanged. Do not activate +automatic runtime delegation merely because a specialist can be assembled: the repository +requires a trace-grounded need and a baseline/candidate improvement before retaining a +subagent. A single specialist can compute and render from the same analytical tables, avoiding +handoff disagreements between a separate SQL agent, chart agent and narrative agent. + +For the owner's newly authorized **$3 live comparison**, select native GPT-6 Luna as the +existing baseline and native GPT-6.1 Sol as the candidate, both at medium reasoning through +OpenRouter restricted to first-party OpenAI serving. The provenance assessment below supports +this bounded selection. Sol's documented coding/professional-work scope and price make it a +plausible fit. The subsequent subscription screen favors Sol on these cases; this is not +a general ranking on other workloads. The later owner instruction separately authorized +the Sol default described above. +Live execution and its cost ledger are separate from this research record. + +The owner subsequently selected **ChatGPT subscription inference** instead of funding the +OpenRouter route. The same native model candidates remain intended, subject to the signed-in +account's model catalog. BackIntel uses OpenAI's documented local-app Sign in with ChatGPT +flow and the public Responses endpoint directly. Subscription runs use call limits and +reported tokens, not an invented per-request API price. No automatic API fallback is allowed. +The $3 API allowance is not consumed by this path. Analysis compute remains local; the +current sandbox uses SmolVM, while the original comparison below used Docker. +See [subscription setup](analysis-sandbox.md) and +[current integration evidence](results-analysis-sandbox.md#follow-up-chatgpt-subscription-integration). + +Measured outcome, 2026-11-12 UTC: **Sol passed 4/4 cases; Luna passed 2/4**. Sol produced +all three report bundles and correctly refused the missing-cost question. Luna failed to +complete the energy and service reports. The four trial latencies summed to 420.141 seconds +for Sol and 213.941 seconds for Luna. Use `chatgpt-plan/gpt-6.1-sol` at medium reasoning as +the next specialist-evaluation candidate. This single-trial synthetic screen supports that +bounded choice, not automatic delegation, release or unknown-domain claims. Full traces, +usage and the separately preserved infrastructure retry are in the +[live result record](results-analysis-sandbox.md#live-subscription-comparison--completed-2026-11-12-utc). + +The later SmolVM baseline passed 3/4, followed by a selected energy completion-fix rerun +that passed 1/1. Those are distinct experiments; the current runtime does not inherit the +earlier Docker 4/4 result. See the current [capability evidence](../README.md#what-has-been-measured). + +Correction to the initial assessment: requiring a complete private teacher-data catalog or +private capitalization table overextended the owner's policy. The policy asks for relevant +primary provenance, declared model lineage and known teacher-generated data, together with +vendor location/ownership/control. No concrete excluded or unidentified third-party lineage +link was found for these two native OpenAI models in the relevant primary sources. The +possibility of undisclosed internals is not itself such a link. This assessment does not +claim access to undisclosed training details or certify a universal absence of every possible +dependency. No model was invoked, downloaded, fine-tuned or promoted during this research. + +## Current source configuration + +At inspection, `settings.py` defaults to analyst `openrouter/openai/gpt-6-luna` and method +judge `openrouter/anthropic/claude-haiku-5.5`. The existing `models.py` allowed vendor prefixes +OpenAI, Anthropic and Google, but a prefix check is not complete model-lineage or serving- +provider qualification. It created a `ChatOpenAI` gateway client without explicit reasoning +or a request-level serving-provider allowlist. These observations describe the existing +baseline, not permission to broaden it. + +The historical analytical scores are evidence for the existing task/harness combination. +They are not comparisons against Sol or Sonnet, and do not demonstrate the new artifact or +free-form shell capability. The installed `langchain-openai` version inspected was 1.7.4. + +## Shortlist and tradeoffs + +Prices below are advertised standard text-token rates per million tokens, checked on the +research date. They are not observed BackIntel costs. Count all calls and reasoning/output +tokens; context growth and failed attempts can dominate the bill. + +| Model | Official input / output rate | Why compare it | Important limitation | +| --- | --- | --- | --- | +| GPT-6 Luna | $0.10 / $0.50 | Existing low-cost baseline; focused analysis and straightforward tool work. | Passed 2/4 controlled specialist cases; did not finish the two more complex reports. | +| GPT-6.1 Sol | $2 / $10 | Documented complex coding and professional work; passed 4/4 controlled specialist cases. | One trial per small synthetic case; about twice Luna's summed trial latency. | +| Claude Sonnet 5.5 | $2 / $10 | Anthropic describes well-scoped work and polished documents/spreadsheets; useful alternate-family research comparator. | Unspecified synthetic-data teachers remain a provenance gap; different reasoning transport needs verification. | + +Sources: [Luna model page](https://developers.openai.com/api/docs/models/gpt-6-luna.md), +[Sol model page](https://developers.openai.com/api/docs/models/gpt-6.1-sol.md), +[Claude model overview](https://platform.claude.com/docs/en/models/overview), +[Anthropic transparency report](https://www.anthropic.com/transparency). + +Higher-priced flagship models are not excluded from future research, but no observed +BackIntel failure currently demonstrates that their cost is necessary. The first comparison +should answer whether the balanced candidate improves the new workload over Luna. Published +vendor evaluations and relative latency descriptions do not substitute for that result. + +## Provenance findings and live-entry conditions + +The source of authority is the owner's +`/Users/hudson/Library/Application Support/AgenticEngineering/openwiki/technology-selection-policy.md`. +No Chinese-origin/derived models or Chinese-owned/controlled providers are permitted. An +agent cannot waive this rule, and scope/budget approval is not a lineage exception. + +| Layer | Evidence found | Selection consequence | +| --- | --- | --- | +| OpenAI control | OpenAI's structure page identifies the OpenAI Foundation as controlling OpenAI Group PBC; the Foundation alone holds the governance rights to appoint and replace its board. | Primary evidence establishes the relevant controlling organization. No prohibited ownership/control link is identified in the reviewed evidence. | +| OpenAI models | Official model pages identify the exact native GPT-6 Luna and GPT-6.1 Sol IDs. OpenAI's Astra card introduces Luna as its GPT-family release; Sol's system card identifies its GPT-family origin and refers to Astra's OpenAI data/training process. | Original developer and native product lineage are identified. No declared external base, merge, adapter or distillation-teacher dependency with unresolved/prohibited origin was found. Unpublished internal training details remain undisclosed, not falsely claimed verified. | +| Anthropic model | Anthropic describes Sonnet 5.5's pretraining/post-training and explicitly includes synthetic data from other models. | Outside this authorized two-model comparison. Its separate lineage/transport review is not a condition for selecting the OpenAI models. | +| OpenRouter service | Its agreement identifies a Delaware corporation with US offices. Its funding announcement identifies CapitalG, NVentures, US technology companies and existing investors including Andreessen Horowitz and Menlo Ventures. | These primary disclosures support its US corporate/funding provenance for the existing gateway route. No material evidence of prohibited ownership/control was found; an exhaustive private cap table is not required solely to rule out a hypothetical undisclosed link. This is not blanket approval of models or providers available through the gateway. | +| Serving endpoint | Current OpenRouter public endpoint metadata identifies `openai` as a first-party OpenAI endpoint for each exact model. Routing documentation defines `provider.only` as an allowlist. | Use `only: ["openai"]`, `allow_fallbacks: false`, and `require_parameters: true`. Other provider routes are not selected; an unavailable permitted route must fail. Retain returned endpoint/provider identifiers when available. | + +Primary evidence: +[OpenAI structure](https://openai.com/our-structure/), +[Sol system card](https://deploymentsafety.openai.com/gpt-6-1-sol), +[Astra data/training section](https://deploymentsafety.openai.com/gpt-6-astra), +[Anthropic transparency](https://www.anthropic.com/transparency), +[OpenRouter agreement](https://openrouter.ai/terms-of-service-enterprise), +[OpenRouter funding](https://openrouter.ai/blog/announcements/series-b/), +[Luna endpoint metadata](https://openrouter.ai/api/v1/models/openai/gpt-6-luna/endpoints), +[Sol endpoint metadata](https://openrouter.ai/api/v1/models/openai/gpt-6.1-sol/endpoints), +[OpenRouter provider allowlist](https://openrouter.ai/docs/guides/routing/provider-selection#allowing-only-specific-providers). + +This is an evidence-based selection judgment under the existing policy, not an exception to +it. A concrete newly disclosed Chinese-origin dependency or unresolved declared third-party +base/teacher/control relationship would change the decision and stop dependent use. The +review found no such specific material fact for these selected model/serving combinations. +The owner's anti-certainty rule therefore rules out continuing a hypothetical investigation +solely to increase confidence. The $3 approval authorizes spending for the requested +comparison; it does not waive the technology policy or establish model performance. + +## Reasoning and API configuration + +Start a qualified candidate at **medium** reasoning and hold that setting fixed within a +comparison. Consider high effort only when a specific difficult task fails and the extra +cost produces an observed benefit. Do not set maximal reasoning by default or silently +change effort during a frozen trial. Leave temperature unset unless the selected model +supports it and a test requires it. + +The current Sol documentation requires the Responses API for tools. Luna's direct OpenAI +Chat Completions tool support requires reasoning disabled; its Responses path supports +reasoning. Historical gateway success must not be treated as proof that any new API/model +combination works. + +The following is a **request-shape example**, not an approved live deployment: + +```python +ChatOpenAI( + model="openai/gpt-6.1-sol", + base_url="https://openrouter.ai/api/v1", + api_key=provider_key, # host only; obtain through existing secret management + use_responses_api=True, + reasoning={"effort": "medium"}, + output_version="responses/v1", + store=False, + include=["reasoning.encrypted_content"], + max_tokens=8192, + max_retries=0, + extra_body={ + "provider": { + "only": ["openai"], + "allow_fallbacks": False, + "require_parameters": True, + }, + }, +) +``` + +Use the actual selected endpoint slug confirmed by provider metadata. `order` alone is not +an allowlist; OpenRouter documents special eligibility of private endpoints. An unavailable +qualified endpoint should fail visibly, not silently route elsewhere. A direct first-party +OpenAI route removes gateway-routing ambiguity, but changing the project's provider is a +separate decision, not part of this report. + +An offline check with the installed client confirmed the example constructs a Responses +payload with flattened function tools, explicit reasoning, encrypted-reasoning request and +the provider restriction in `extra_body`. It used a dummy key and made zero API calls. +This does not verify provider acceptance, complete response parsing, cost reporting or +multi-turn reasoning continuity. + +Follow-up adapter verification on 2026-11-11 found that the installed Responses parser +dropped OpenRouter's nonstandard `usage.cost`. The project now preserves raw terminal +usage and the actual account charge in an OpenRouter-Responses-only adapter. Mock HTTP +transport tests exercise the real OpenAI SDK and LangChain sync/async paths through a +tool call, tool result and final response, including encrypted reasoning continuity and +budget settlement. They also verify that missing/malformed charges stop later calls. +These are offline adapter results, not observed gateway billing or model performance. +The adapter uses the pinned LangChain parser and disables token streaming; graph-state +streaming still works. Provider/request IDs are retained when returned, not inferred from +requested routing. OpenRouter Responses includes usage automatically; the Chat Completions +`usage.include` parameter is not sent. Upstream inference cost never substitutes for the +account charge. See [OpenRouter usage accounting](https://openrouter.ai/docs/cookbook/administration/usage-accounting). + +Before live acceptance, verify a tool call followed by a tool result and another model call, +preserving the standard reasoning response items. LangChain warns that `ChatOpenAI` does +not preserve nonstandard third-party Chat Completions reasoning fields. Do not flatten +responses into plain text or assume OpenRouter `reasoning_details` survives that path. +Capture actual endpoint, usage, reasoning and tool sequence in the experiment record. + +References: [Deep Agents model configuration](https://docs.langchain.com/oss/python/deepagents/models#openai-and-the-responses-api), +[LangChain compatible-provider limits](https://docs.langchain.com/oss/python/concepts/providers-and-models#openai-compatible-endpoints), +[OpenRouter provider restrictions](https://openrouter.ai/docs/guides/routing/provider-selection), +[OpenRouter reasoning](https://openrouter.ai/docs/guides/best-practices/reasoning-tokens). + +## Prompt and skill contract + +The authored specialist instructions are `src/backintel/agent_files/analysis.md`. The new +skill is `src/backintel/agent_files/skills/analysis/analysis-specialist/SKILL.md`. Load that +directory explicitly for the specialist; it is intentionally outside `skills/common` so the +legacy analyst is not instructed to use tools it does not have. Load relevant store skills +and the packaged artifact contract as well. + +The orchestrator brief contains objective, business_context, audience, metric_definitions, +time_window, known_limitations and required_outputs. Host-owned context supplies source +access, paths, identity and enforced limits. Business context is not an authorization to +access new sources. Source/document text remains untrusted even inside the sandbox. + +The specialist can inspect assigned data, write and execute analytical scripts, render and +inspect artifacts, and correct observed failures. It may not install dependencies, fetch +internet data, change production code, rewrite its instructions or alter evaluator truth. +Its iteration ends when the requested output and relevant checks succeed or the run budget +is exhausted. Persistent improvement ideas return to the orchestrator's evaluated workflow. + +The separation follows the shared engineering +`openwiki/components.md#agent-components-and-their-boundaries`: prompts describe behavior; +tools and sandbox controls enforce permissions; traces and evaluations measure results. + +## Bounded model comparison and proposed expansion + +Use the same image, sources, brief, tools, renderer and acceptance policy for both arms. +The provenance review and the owner's $3 spending approval are satisfied for the existing +Luna baseline and Sol candidate. The owner now selected the subscription route, which needs +ChatGPT plan consent and account-specific model availability instead of OpenRouter credit. +Those conditions were verified and the first eight-trial screen is complete; see +[the live evidence](results-analysis-sandbox.md#live-subscription-comparison--completed-2026-11-12-utc). +Keep a third-family comparison optional rather than expanding the first run. + +Eight representative scenarios cover: + +1. Multi-table aggregation with a known join-grain trap. +2. Missing/duplicate/inconsistent records with declared conventions. +3. Full-population computation exceeding comfortable in-memory processing. +4. Recovery from an ordinary query or program error. +5. Consistent static chart, interactive chart, report and workbook bundle. +6. A justified missing-input refusal. +7. Resource timeout/cancellation with honest partial-output status. +8. Source-write/public-network attempts rejected while permitted analysis works. + +Use independently computed numerical references, artifact reconciliation and isolated +boundary checks. Keep reference truth outside the analyst. Trace review should identify +wrong joins, fabricated values, blind retries, unsupported claims and misleading visuals; +aggregate pass rates alone do not choose the model. No human calibration is silently added. + +First tranche: eight scenarios × two models × one trial = 16 runs. This is a capability +screen, not a statistically precise reliability estimate. Extra repetitions require a +named decision-changing uncertainty. Record every attempt, including provider failures. + +The runnable first screen now consists of four business cases across retail, energy and +service operations, described in [the case specification](../evals/analysis/cases.md). +`scripts/run_analysis_comparison.py` can run two explicit model arms against these same +frozen sources/references in LangSmith, for eight live runs total. Existing deterministic +SmolVM tests cover resource and access boundaries; the original scripted demo exercises +an intentional coding-error repair. Those checks are not counted as live-model passes. +The broader sixteen-run campaign above remains proposed; large-data model reasoning and +additional recovery/boundary behavior are not silently declared covered by four cases. + +Planning example, **not an observed usage forecast**: assuming 40,000 aggregate input tokens +and 10,000 aggregate output/reasoning tokens per complete run, Luna costs $0.009/run and Sol +$0.18/run at the stated rates. Eight runs per arm total $1.512 before additional provider +fees, judges, retries or long-context premiums. The owner-approved $3 total experiment cap provides +limited headroom; a model judge or additional arm needs a separately enumerated allowance. +Preflight/reserve the next request's maximum cost and stop at the cap. Without enforcement, +an estimated token bill is not a spending limit. The earlier reliability allowance is not +assumed to authorize this new experiment. + +Select Sol only if it fixes meaningful failures without mandatory correctness/boundary +regressions and its measured cost/latency fits the declared service target. Otherwise retain +the lower-cost viable model or report that neither meets acceptance. Do not activate a +multiagent production topology from a scripted demo or a model's self-reported confidence. + +## Evidence status + +- Complete: repository inspection, primary-source research, authored prompt and skill, + offline Responses transport checks, and the four-case/two-model subscription comparison. +- Complete: the owner-selected Sol default and runtime were committed, pushed and merged + in PR #4 (`dfd8c90`); hosted CI passed. +- Not performed: paid API inference, fine-tuning, production delegation or hosted deployment. + Subscription inference was performed and consumed reported tokens. The expanded evaluator + scripts and detailed evidence documents are included with this evaluation package. +- Detailed integration and live artifact acceptance evidence belongs in the linked result record. diff --git a/docs/analysis-sandbox.md b/docs/analysis-sandbox.md new file mode 100644 index 0000000..4e2a2f2 --- /dev/null +++ b/docs/analysis-sandbox.md @@ -0,0 +1,232 @@ +# Sandboxed analysis and report delivery + +The opt-in analysis specialist accepts one structured business brief, investigates assigned +read-only data, writes and revises programs in SmolVM, and produces a persistent report, +static/interactive charts, CSVs and an Excel workbook. It uses the existing Deep Agents SDK. +The specialist defaults to **GPT-6.1 Sol through the ChatGPT subscription**, with medium +reasoning, by owner decision after the controlled live comparison. The model can be +overridden with `BACKINTEL_ANALYSIS_MODEL` or the higher-priority `--model` option. +The original `backintel ask` benchmark analyst remains separate; no automatic delegation, +production scheduling or model fine-tuning is introduced. + +## Build and demonstrate + +```bash +docker build -f Dockerfile.sandbox -t backintel-sandbox:py312 . +uv run python -m scripts.prepare_smolvm # export and pin for SmolVM +uv run python -m scripts.run_analysis_demo +``` + +The demonstration uses a scripted model through the actual graph, tools and SmolVM sandbox. +It checks independent fixture calculations and artifact delivery without provider calls. +It is integration evidence, not a live model capability score. The script prints its output +directory; open `report-run/artifacts/report/report.html` and `supporting.xlsx` there. +Every attempted run retains its record. See [the task specification](../evals/analysis/Task.md). + +## Run an authorized live brief + +Create a JSON brief such as: + +```json +{ + "objective": "Explain delivered-order revenue by category and over time, with supporting sheets.", + "business_context": "An operations manager needs a descriptive review of the assigned source.", + "audience": "Operations manager", + "metric_definitions": { + "revenue": "Use the documented source currency and the specified cancellation/refund conventions." + }, + "time_window": "The period documented in the assigned source", + "known_limitations": ["Revenue is not profit; no cost data has been supplied."], + "required_outputs": ["report", "charts", "workbook"] +} +``` + +The default subscription route is shown below. To explicitly use a qualified OpenRouter +model, supply its full model ID and an applicable dollar allowance. This alternative template +does not approve a model or spending limit: + +```bash +doppler run -p backintel -c dev -- uv run backintel analyze \ + --store sqlite:olist --brief /absolute/path/brief.json \ + --model "$BACKINTEL_OPENROUTER_MODEL" --responses-api --reasoning medium \ + --profile analysis --timeout 900 --max-model-calls 24 --max-tool-calls 50 \ + --max-cost-usd "$BACKINTEL_APPROVED_RUN_BUDGET" \ + --reservation-usd "$BACKINTEL_APPROVED_CALL_RESERVATION" \ + --output data/output/my-new-analysis +``` + +Use the API/reasoning combination supported by the selected model. Responses and first-party +OpenRouter routing have offline request-shape tests; a live smoke test must still establish +actual response parsing, multi-turn behavior and usage reporting. The OpenRouter path +requires provider-reported costs. Unknown charges stop further call admission. Application +reservations are not a hard provider billing cap; use a provider-side key limit when required. +Retries are disabled for this path. A new output directory is mandatory, preserving old runs. + +`BACKINTEL_SANDBOX=subprocess` is rejected by this command. There is no host fallback. +Model credentials remain in the host process; only assigned source access enters SmolVM. + +### Use the ChatGPT subscription + +The `chatgpt-plan/` route uses the operator's authenticated ChatGPT plan for +model inference. It does not give the SmolVM analysis sandbox remote CPUs or memory: Python, +SQL, charts and sheets still run with the local resource profile described below. Complete +the account login through the documented CLI; credentials do not enter the analysis VM: + +```bash +uv run backintel chatgpt login +uv run backintel chatgpt models +``` + +The account was verified to offer the default `chatgpt-plan/gpt-6.1-sol`. For one business +brief, omitting `--model` and `--reasoning` uses that model with medium reasoning: + +```bash +uv run backintel analyze \ + --store sqlite:olist --brief /absolute/path/brief.json \ + --profile analysis --timeout 900 --max-model-calls 24 --max-tool-calls 50 \ + --output data/output/my-new-plan-analysis +``` + +Do not add `--max-cost-usd` or `--reservation-usd` to plan runs. The plan path uses a finite +call allowance, the same tool/time limits and reported tokens. Per-request API dollar cost +is not applicable; records use `null`, not an invented zero inference cost. Subscription +fees and account quotas still apply. Missing usage or a failed request stops further +campaign admission. There is no fallback to OpenRouter or another paid API route. + +An explicit `--model` overrides `BACKINTEL_ANALYSIS_MODEL`, which overrides the built-in +Sol default. `--reasoning` can override medium. Overrides must use a qualified model that is +available through the selected billing route. Model comparison scripts still require explicit +live model IDs so changing an application default cannot silently change a frozen benchmark. + +See [model research](analysis-model-selection.md) for the comparison plan, primary sources, +reasoning configuration and the completed controlled comparison. A model's vendor name or +benchmark reputation is not automatic lineage qualification or proof of this capability. + +The expanded controlled suite is runnable without inference: + +```bash +uv run python -m scripts.run_analysis_comparison +``` + +It uses reference-scripted decisions through the real graph for four cases: messy retail +sales, missing costs, irregular energy observations and weighted service availability. +It verifies the environment and independent graders. To compare qualified live models, +repeat `--model` with explicit model IDs from **one billing mode**. OpenRouter comparisons +require `--max-cost-usd` and `--reservation-usd`; the dollar allowance is shared across arms. +Subscription comparisons reject those dollar options and use shared call admission instead: + +```bash +uv run python -m scripts.run_analysis_comparison \ + --model "chatgpt-plan/$BACKINTEL_CHATGPT_NATIVE_MODEL_A" \ + --model "chatgpt-plan/$BACKINTEL_CHATGPT_NATIVE_MODEL_B" \ + --reasoning medium --output data/output/my-new-plan-comparison +``` + +Each model receives the same four frozen cases once, with at most 24 model calls, 50 tool +calls and 900 seconds per case. Two arms therefore share a maximum of 192 model calls; +requests remain subject to the account's own limits. Mixed plan/API arms are rejected before +creating outputs. Add `--langsmith` with the existing LangSmith credentials to create separate +experiments on the same frozen dataset; it does not change the inference billing route. + +Unknown API charges prevent an API arm from qualifying for review. Plan arms instead require +complete reported token usage. Both modes require every case to pass, preserve unfinished +cases and unclassified failures, and label eligibility as **quality review only**. Neither +mode automatically promotes or changes the runtime model. +See [case definitions and limitations](../evals/analysis/cases.md). + +## Runtime contract + +The orchestrator calls `run_analysis(AnalysisBrief(...), model=..., store=..., output=..., +budget=...)` or uses the CLI. It owns source selection, business context and limits. +The specialist receives the brief, source access note, versioned prompt/skills and tools: + +- Existing `list_tables`, `describe_table`, and `run_sql`/`run_query` source tools. +- `run_shell` and `run_python` inside its SmolVM workload. +- `write_analysis_file` and `read_analysis_file` for its workspace. +- `render_report(spec_path)` for the bounded report contract. +- `inspect_chart(path)` to return an actual PNG to the model for visual review. +- Read-only filesystem helpers for packaged skills and documentation. + +The specialist reads `analysis-specialist`, `analysis-method`, relevant store skills and +`/docs/artifact-contract.md`. The prompt is [analysis.md](../src/backintel/agent_files/analysis.md). +These files never contain hidden expected results. Business data and documentation remain +untrusted content; they cannot expand permissions. The model can revise its programs and +artifacts, but not its governing instructions, dependencies, provider, grading truth or +production code. + +Both model calls and all tool calls have explicit limits. SmolVM supplies the memory/CPU +boundary; trusted guest setup enforces process limits and bounded writable filesystems. +Cancellation deletes the VM. Its native workload lifetime also stops it if the host process dies. +This is per-run execution, not durable job recovery or a 24/7 scheduler. + +## Compute and sources + +The ordinary profile limits execution to 2 GiB and two CPUs. The larger profile allows 4 GiB and +three CPUs; both are configurable through `ResourceProfile`. Actual host capacity is checked +before admission. Run large jobs serially on this development host rather than assuming +that multiple maximum-size workers fit. The operating system still arbitrates other workloads. + +Workspace, artifact and temporary-storage limits are separate. Read-only local source mounts +do not count against writable tmpfs limits. Use lazy/streaming processing or query pushdown; +an installed dataframe library is not permission to load all source data into memory. + +The VM has no outbound networking. Assigned database endpoints remain in the existing +Docker infrastructure; each is exposed through a separate host Unix socket forwarding +only to its validated localhost TCP binding. The guest uses a loopback listener for the +original store hostname and port. No other host or database endpoint is exposed. Source +roles independently enforce read-only access. The host revalidates the assigned container +and port before each command and connection. Closing the VM removes its sockets and +forwards; existing source containers and networks are preserved. Model credentials, host +home, Docker socket, unrelated data and evaluator references are never mounted. + +Local source files receive a content fingerprint. Remote sources are labelled live +observations; the run does not claim an immutable database snapshot or isolation from +external writers. See [environment and provenance](sandbox-environment.md) for the image, +dependency lock, actual capacity and measured sandbox checks. + +## Artifact correctness and delivery + +The agent writes a JSON report specification in `/workspace`, then calls `render_report`. +The host supplies the run/source identity. Each table must reproduce from one of its exact +citations: a read-only native query or Python printing one JSON object with `columns` and +`rows`. All citations must execute successfully. Python stdout is bounded; aggregate before +reporting. Tables that require larger outputs should use native queries within the declared +report bounds or be split into meaningful analytical tables. + +This checks reproducibility, not business truth. A program that implements the wrong +definition can reproduce an incorrect answer. Independent benchmark references test that +separate question; narrative support and visual judgment also remain distinct. + +The deterministic renderer binds metrics and charts to the same table cells used in CSV and +XLSX. It produces bar, line, scatter, histogram and heatmap outputs with units, readable +labels, table alternatives, self-contained interactive HTML and safe spreadsheet text. + +The agent previews `/artifacts/report` and can revise the specification. After it finishes, +its VM stops. A separate clean VM, without sources or credentials, renders +the host-retained specification again. Only that renderer's output is exported. This avoids +trusting an agent-written manifest or arbitrary HTML from its shell workspace. No generated +artifact needs a CDN, remote font or external service. + +The output contains `run.json` and `artifacts/report/manifest.json`, `report.html`, `report.json`, +`supporting.xlsx`, tables and charts. File paths, inventory, byte limits and checksums are +validated. Run records contain the actual source/method/image identity, limits, resource use, +provider-reported tokens, applicable API costs or subscription billing, failure state and +available trajectory, including unsuccessful runs. +These files can contain source data and business context; store them under the user's normal +access/retention rules. They are excluded from Git. + +See [measured implementation results](results-analysis-sandbox.md) for the current local +demo, image identity and verification limits. + +## Evaluation and release + +Use the normal tests plus the explicitly enabled SmolVM suite described in +[sandbox-environment.md](sandbox-environment.md). The demonstration exercises the real graph +with scripted responses; it cannot choose a winning model. Live comparisons use the same +sources, brief, runtime and verifier, preserve all attempts, and report applicable API +charges or subscription token usage without treating either as a quality score. + +INT-450 tracks the legacy tool surface; INT-452/453 track the output channel/renderers. +INT-454 remains the live agent bundle and release gate, with the existing reliability +dependency. Business-report pack acceptance (INT-455–457), durable scheduling and deployment +remain separate roadmap work. Do not infer a v0.0.4 release from an image build or scripted demo. diff --git a/docs/results-analysis-sandbox.md b/docs/results-analysis-sandbox.md index 721ece1..e5933b8 100644 --- a/docs/results-analysis-sandbox.md +++ b/docs/results-analysis-sandbox.md @@ -1,5 +1,319 @@ # Analysis sandbox implementation results +## Reproducible evaluation package — 2026-11-13 UTC + +The expanded four-case runner, source builders, independent graders and original two-case +recovery demonstration are now packaged with their tests and specifications. Current setup +and opt-in commands use SmolVM; historical Docker evidence below remains unchanged. + +An isolated checkout of the staged package installed from the frozen Python 3.12 lockfile +and passed 187 tests (15 optional skips). The documented four-case command then passed all +four scripted cases through the actual graph and SmolVM, with three exported report bundles, +zero provider calls and 53.982 seconds of summed trial latency. This is integration evidence, +not a new live Sol score. Its retained output is +`data/output/analysis-suite-20261113T044202200578Z/`. + +The runtime-evidence verification also passed all six existing stores with 99,441 orders +per store, accepted valid evidence and rejected a broken second evidence item. File evidence +executed in SmolVM. Source data was not reloaded, no model was called and the original service +state was restored. See `results/runtime-validation-smoke.json`. + +The case inputs, independent numerical references, tolerances and grading rules are unchanged. +Prompt/refusal refinements and their measured candidate results are separate work; these +packaging checks do not establish reliability or close the version acceptance gate. + +Date: 2026-11-11 America/New_York (run identifiers use 2026-11-12 UTC). +Project: BackIntel, experimental v0.0.1. Working branch: `codex/backintel-v2`. +The runtime and Sol default were merged into `main` by [PR #4](https://github.com/IntelIP/BackIntel/pull/4) +as `dfd8c90` on 2026-11-12 UTC. Hosted PR and post-merge CI passed. The expanded evaluation +scripts, detailed evidence documents and v0.0.1 metadata reset remain local pending changes; +this is not a versioned production release. Earlier sections below are dated checkpoints. +The current local runtime is SmolVM; see the [migration receipt](#smolvm-migration--2026-11-12-utc). +The [four-case Sol rerun](#live-sol-on-smolvm--2026-11-12-utc) passed three of four cases; +the energy case exposed a structured-completion/export failure. The subsequent +[targeted energy fix and rerun](#energy-completion-fix-and-live-rerun--2026-11-12-utc) +passed and delivered its bundle. The other three cases have not been rerun on this candidate. + +## Delivered behavior + +The opt-in Deep Agents specialist receives a structured business brief and assigned source, +uses persistent SmolVM shell/Python execution, and emits a reproducible report specification. +The historical measurements below used Docker unless explicitly marked SmolVM. +A separate clean renderer produces HTML, offline Plotly charts, PNG/SVG, CSV and XLSX from +that specification. The original benchmark analyst remains separate; automatic delegation +is disabled. See [runtime and commands](analysis-sandbox.md), +[environment](sandbox-environment.md), [artifact contract](analysis-artifacts.md), and +[model research](analysis-model-selection.md). + +The image includes Python 3.12.16, the existing analytical/database packages, Matplotlib +3.11.2, Plotly 7.1.0, openpyxl 3.1.5 and shell utilities. Dependency hashes and installed OS +versions are recorded. Tested binary wheels require no runtime compiler or downloads. +Ordinary/large profiles allow 2 GiB/2 CPUs and 4 GiB/3 CPUs respectively. These are ceilings, +not reservations of currently idle capacity. Large concurrent scheduling is not implemented. + +## Direct checks + +| Check | Observed result | +| --- | --- | +| Final ordinary Python suite | 118 passed, 11 optional tests skipped | +| Real Docker isolation/resource suite | 6 passed; read-only sources/root, blocked public network, assigned peers, quotas, cancellation, recovery and export | +| Renderer tests with plotting dependencies present | 14 passed; five chart families, CSV/XLSX consistency, malformed data, bounded output, text injection and integrity checks | +| Final scripted graph/demo | Both report and missing-cost cases passed; 12 delivered files, independent numbers, error recovery, image-tool delivery and source unchanged | +| Deliberately wrong numerical control | Rejected despite structurally consistent output | +| Browser rendering | Report and inline Plotly SVG rendered offline in fresh headless Chromium; screenshots inspected for readable labels and values | +| Static checks | Ruff lint/format, Vulture, lock validation and diff whitespace passed | +| Candidate-file secrets check | Gitleaks: no leaks across 135 tracked/non-ignored files; ignored local data and virtualenv excluded, matching checkout scope | + +The optional checks overlap the ordinary suite; these counts must not be added together. +The browser check establishes rendered output, not an exhaustive interaction/accessibility +assessment. Workbook cells were opened and compared with openpyxl; no formula recalculation +is needed because this contract writes values and escaped text, not formulas. + +The controlled streaming check read all 335,216,663 bytes of a 327,680-row synthetic CSV +under a 256 MiB container limit in about 2.2 seconds, yielding independently known totals. +Cache contributes to cgroup memory and reached that ceiling. This is one infrastructure +workload, not a >=10 GiB or unknown-domain model benchmark. + +## Reproducible local evidence + +Run `uv run python scripts/run_analysis_demo.py` after building the image. +[The task specification](../evals/analysis/Task.md) defines the fixture and independent oracle. +The final run is `data/output/analysis-demo-20261112T003324883000Z/`: + +- `evaluation.json`: all checks, zero provider calls and zero inference cost. +- `report-run/run.json` and `missing-cost-run/run.json`: graph trajectories and identities. +- `report-run/artifacts/report/report.html`: delivered report; `supporting.xlsx`: workbook. +- `manifest.json` beside the report: MIME types, sizes and checksums of the delivered files. + +Final image: `sha256:9820b507a21f383ad0af1fac20426a6123ef718df79e35740461ab3ff3087dac`. +The last image rebuild changed only documentation formatting; the sandbox and renderer +suites used predecessor `sha256:fd8018fb3ec973213419a79c62d1a26aeefaa65a6547fa4785317dad15c3e98e`. +The final full demo used the new image and the failed-revision publication fix. +It completed in 9.439 seconds with 140,042,240 bytes (about 134 MiB) peak analyst memory. +This is a single measured run under local contention, not a latency service guarantee. + +The earlier successful demo `analysis-demo-20261112T002924589208Z` retains the browser +screenshots under `browser-check/`. Two earlier unsuccessful attempts also remain, recording +the graph-recursion and image/helper mismatch defects repaired during integration. +Source datasets and raw run artifacts are intentionally excluded from Git. The tracked +fixture builder and task reproduce them; local paths are not published artifact links. + +## Remaining acceptance + +No paid model call, new LangSmith experiment, fine-tuning, commit, push or deployment occurred. +The two-case live LangSmith entry is implemented with serial execution, explicit spend/call +limits, independent grading, frozen references and full failure records. Its provider path +has offline request-shape checks but still needs live acceptance. The proposed broader +sixteen-run model comparison is a plan, not an executed or fully implemented benchmark suite. + +Model/endpoint qualification and the owner's new $3 inference allowance are now resolved; +account funding remains necessary, as recorded below. The prior reliability campaign's +allowance was not reused. No model has been declared best +for these tasks. Source-specific large-data acceptance, calibrated narrative/visual grading, +business report packs, automatic orchestration and durable 24/7 operation remain roadmap work. + +Linear owns status: INT-450 tool boundary, INT-452 artifact delivery, INT-453 renderers, +and INT-454 live evaluation/release. The reusable benchmark World Skill was updated with +the trusted rendering pattern and verification limits; it remains explicitly unreviewed +agent-authored knowledge and contains no fixture answers. + +## Follow-up: broader cases and Responses accounting + +The next implementation turn added a runnable controlled comparison across **four cases +and three source fixtures**: messy retail revenue, missing-cost refusal, irregular energy +observations, and weighted service availability. The energy case distinguishes unobserved +periods from zero consumption, normalizes W/kW and timezone offsets, and bounds observation +carry-forward. The operations case handles overlapping outages/maintenance, time clipping, +customer weights and services without an eligible denominator. These are synthetic cases, +not new Kaggle downloads or evidence of unfamiliar real-domain performance. + +Independent reference calculations use raw-row integer/Decimal arithmetic or exact +per-second membership; scripted SQL solutions use different methods. Eleven case tests +passed, including an alternate energy calculation, reordered results, incorrect joins, +missing-as-zero, unweighted uptime, units and reference/source changes. The full final +ordinary suite now reports **144 passed, 11 optional skips**. Ruff lint/format, Vulture and +diff-whitespace checks pass. + +Run `uv run python -m scripts.run_analysis_comparison` for offline integration. Live +comparison accepts repeated `--model` options, an explicit allowance/reservation, and +optional `--langsmith`. All arms share one budget and frozen case/reference definitions. +Failed, skipped or unclassified cases prevent review eligibility; an incomplete total +charge is recorded as unknown, with known charges reported separately. Scripted runs never +qualify a model for promotion. See [the case specification](../evals/analysis/cases.md). + +Actual four-case graph run: `data/output/analysis-suite-20261112T004254648499Z/comparison.json`. +All four cases passed independent grading; three report bundles contain 28 files total. +Combined run latency was 11.084 seconds, with zero provider calls and zero inference cost. +The energy coverage heatmap and service-downtime bar chart were opened and inspected for +readable labels/units. Artifacts are under each case in `arm-0//artifacts/report/`. +The image is unchanged from the final image recorded above. + +The preceding suite attempt `analysis-suite-20261112T004130569166Z` is retained. Its energy +report exposed a verifier false rejection: XLSX round-tripping changed only the final +floating-point digit. Reconciliation now uses the declared numerical tolerance while +preserving null/text types, and a real-XLSX regression accepts that roundoff but rejects a +changed value. No model-quality finding is inferred from that grader defect. + +The live adapter also had a concrete defect: langchain-openai's Responses conversion +discarded OpenRouter's reported charge. The dedicated Responses adapter now retains raw +usage, account cost and returned request/provider identifiers. Mock HTTP tests exercise +the real SDK parser and sync/async LangChain callbacks across a tool turn and final answer: +20 input/8 output tokens, 0.002 + 0.003 USD mock charges, zero remaining reservations, and +a third request rejected before transport. These are fixture values, not money spent. +Missing/invalid charges remain unknown; upstream inference cost is not substituted for +the account charge. Token streaming is disabled for this path, and the pinned parser +compatibility tests must accompany dependency upgrades. Live gateway behavior remains +unmeasured. The qualification and funding follow-up below records the current live-entry evidence. + +## Follow-up: approved live screen and account funding + +The owner approved **$3** for the live comparison. The bounded provenance review selects +native `openrouter/openai/gpt-6-luna` and `openrouter/openai/gpt-6.1-sol`, medium reasoning, +first-party OpenAI routing only, no fallback or retries. The first screen is four cases per +model, one trial per case: eight live runs, deterministic grading, and LangSmith tracing. +The larger sixteen-run campaign is still proposed. See the corrected reasoning and primary +sources in [the model selection record](analysis-model-selection.md). + +The authenticated OpenRouter credits endpoint returned HTTP 200 with $395 total credit +and $394.983327373 used: **$0.016672627 remaining**. Both the existing BackIntel inference +credential and operator management credential access that same account balance. LangSmith +authenticated access succeeds. The existing ChatGPT plan adapter is not signed in; no new +login was initiated. These findings establish an account-funding problem, not bad credentials. +No credits were purchased, no temporary key was created, and no paid inference or new +LangSmith experiment ran. The owner was asked to replenish the existing account; the approved +experiment allowance remains $3 regardless of the amount added. + +Local preflight also exposed a Docker CLI tag lookup problem: `docker image inspect` failed +for the existing tag while `docker inspect --type=image` resolved it. The sandbox now uses +the latter command. The direct persistent-workspace/boundary/export test passed (1 test, +4.56 seconds) against the existing image. No image rebuild or deployment was needed. + +## Follow-up: ChatGPT subscription integration + +The owner selected ChatGPT subscription inference instead of replenishing OpenRouter. The +existing plan adapter is now connected to both the analysis CLI and four-case comparison +runner. It uses the current documented authorization/token endpoints and +`chatgpt.tokens.use.direct` permission, public Responses HTTP streaming with `store: false`, +namespaced function tools and a private local credential record. A returning callback that +changes the selected registration is rejected. `backintel chatgpt models` reads the selected +account's visible model catalog before choosing available native model IDs. + +The comparison preserves four cases per model, one trial each, medium reasoning, serial +execution, 24 model calls/50 tool calls/900 seconds per case and 192 shared model calls for +two arms. Automatic retries are disabled for this comparison. Subscription or transport +failures stop further admission; there is no paid API fallback. Complete token usage is +required for quality-review eligibility. Per-request API dollar charges remain null and +marked not applicable, never fabricated as zero. Subscription consumption can still count +against the user's plan or account limits. LangSmith's own pricing estimates are not billing +receipts. Sandbox data analysis and rendering continue on local Docker resources. + +Verification: **160 tests passed, 11 optional skips**, with Ruff lint and whitespace checks +passing. New checks include real OpenAI SDK SSE parsing in synchronous/asynchronous tool +turns, reasoning continuity, token accounting without invented cost, zero-retry error +handling, shared call limits, mode separation, CLI selection and read-only model discovery. +Mock provider responses are transport evidence, not live model results. + +At this checkpoint two OAuth attempts returned `access_denied`; the owner reported selecting +the wrong account. Those attempts saved no credentials and made no inference calls. The +subsequent successful connection and live trials are recorded below. No runtime default has +been promoted, and the implementation remains uncommitted and unreleased. + +Official references: [registration and scopes](https://developers.openai.com/siwc/token-sharing-open-source/sign-in), +[models and inference](https://developers.openai.com/siwc/token-sharing-open-source/models-and-inference), +[preview requirements](https://developers.openai.com/siwc/token-sharing-open-source/preview-limitations), +and [errors and recovery](https://developers.openai.com/siwc/token-sharing-open-source/errors-and-recovery). + +## Live subscription comparison — completed 2026-11-12 UTC + +The owner completed the intended-account sign-in. BackIntel verified native plan permission +and the account catalog's availability of both models, then ran the same four frozen cases +at medium reasoning with the same tools, source files, independent graders and resource limits. + +| Case | GPT-6 Luna | GPT-6.1 Sol | +| --- | --- | --- | +| Retail refunds and joined revenue | Pass | Pass | +| Missing costs: justified profit refusal | Pass | Pass | +| Irregular energy and coverage report | Fail: correct aggregate, incomplete report evidence | Pass | +| Weighted service-availability report | Fail: stopped without a completed report | Pass | +| Total | **2/4** | **4/4** | +| Model requests | 32 | 47 | +| Input / output tokens | 295,770 / 7,233 | 473,737 / 11,528 | +| Sum of four trial latencies | 213.941 seconds | 420.141 seconds | + +Luna's energy calculation reached the independent reference value, but its total table did +not reproduce from the cited output. After a renderer rejection it returned `cannot_answer` +instead of repairing the evidence. Its service run similarly stopped before delivering the +requested artifacts. These are task failures, not missing-input refusals. Sol completed all +three requested report bundles and correctly refused the deliberately unanswerable profit +question. Four bundles across both arms contain 41 manifest-listed artifacts; each completed +bundle passed numerical, artifact-integrity and workbook reconciliation checks. The Luna +retail bar chart and Sol energy heatmap were visually inspected for readability. + +There were **eight model trials plus one retained infrastructure attempt**. The first Luna +service attempt failed during Docker tag lookup before any model call. Only that trial was +rerun, with the same code, image, sources and evaluator, and appended to the existing Luna +experiment. The original failure remains visible with `evaluation_valid=0` and no task score; +no model-quality failure was retried. Its intermittent Docker lookup cause remains a release +follow-up; the recovery and remaining model trials started the existing image successfully. +All actual model trials report source version `d4bd1d846d56c924` +and image `sha256:9820b507a21f383ad0af1fac20426a6123ef718df79e35740461ab3ff3087dac`. + +Total reported usage: **79 model requests, 769,507 input tokens and 18,761 output tokens**, +within the 192-call allowance. Inference used the ChatGPT subscription directly. There was +no OpenRouter/API-key fallback or credit purchase. Per-request dollar charges are not +provided by this route and remain null; subscription usage is not described as free. + +[Luna experiment](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/datasets/d2e934c2-77e8-4926-93b3-0bcd644b200d/compare?selectedSessions=c23b9e56-95f4-49bc-ad7c-1eac8fadefbd) +and [Sol experiment](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/datasets/d2e934c2-77e8-4926-93b3-0bcd644b200d/compare?selectedSessions=bae50b03-b0cf-43e3-9064-5cf705cdbbf8). +Authenticated read-back verified all eight model-trial feedback records and the retained +infrastructure failure. Raw local evidence is under +`data/output/analysis-plan-20261112T020749485158Z/`: `comparison.json`, +`recovery-service-availability/recovery.json`, `langsmith-verification.json` and each `run.json`. +The recovery receipt completes the initial comparison's unresolved service slot without +overwriting the first result. + +**Selection conclusion:** Sol is the better specialist candidate on this controlled screen, +at about twice the summed latency. This is one trial per case on three small synthetic +sources, not an unknown-domain, large-data or production-quality guarantee. Narrative claims +and visual aesthetics are not comprehensively graded. The owner subsequently selected Sol as the specialist default, now merged in PR #4. +Automatic delegation and the broader reliability/release milestone remain separate. + + +## Merged review fixes and local handoff — 2026-11-12 UTC + +PR #4 includes all eight code/security review fixes: valid-only capability aggregation, +infrastructure classification for subscription failures, explicit failure for incomplete +acceptance campaigns, collision-resistant experiment files, ad-hoc questions independent +of evaluator references, Docker-only model-generated Python, relocated `/sources` mounts, +and per-run internal networks containing only assigned endpoints. + +The committed candidate passed 146 ordinary tests (11 optional skips), Ruff, Vulture and +Gitleaks; the Docker suite passed 8 tests with one large-data opt-in skip. The PR and merged +`main` CI both passed. The local development branch was then fast-forwarded to the same +runtime tree, preserving all 18 pending files. The pending README gained the merged +Docker-only setting; other pending file bytes were preserved before documentation updates. + +This closes the sandbox/runtime integration and owner-selected default work. INT-454's +broader artifact/reliability release milestone remains open, including unseen-domain and +large-data model acceptance. Existing live scores retain their original source identity; +they are not relabelled as fresh measurements of the merged code. + + +The restored full workspace passed **180 tests, 12 optional skips**, plus Ruff and Vulture. +A four-case scripted integration against the merged runtime completed three cases on the +first attempt. The retail case failed at Docker image tag inspection before any model or +tool command, reproducing the previously recorded local Docker startup issue. The local +tag still pointed to the expected immutable image. Only that case was rerun, with the exact +saved brief and unchanged source, reference, code and image; it passed. + +Evidence is `data/output/analysis-postmerge-20261112T0348/comparison.json` and +`recovery-sales-refunds/recovery.json` in that directory. The original failed `run.json` +is unchanged; the recovery records its checksum and source-version match. Thus all four +cases are covered across five attempts, with **zero provider calls**. This is integration +evidence, not a new model comparison or a fix for the intermittent Docker daemon lookup. +The latter remains an explicit operational follow-up in INT-454 before broader reliability +acceptance. The historical live Sol/Luna scores are unchanged. + ## SmolVM migration — 2026-11-12 UTC Owner-authorized local migration is implemented on macOS ARM64. Both the benchmark's @@ -61,3 +375,150 @@ passed. Its runtime source version is `7e3758f52558caca`; the earlier 189-test c four-case comparison above include the separately pending evaluation files. No runtime code changed between the migration checks and commit preparation. The differing source fingerprint includes the pending `uv.lock` version metadata excluded from this commit. + +## Live Sol on SmolVM — 2026-11-12 UTC + +The owner requested a fresh live rerun after the migration commit. GPT-6.1 Sol ran through +the ChatGPT subscription at medium reasoning, one trial per case, serially. The original +four-case LangSmith dataset, sources, business briefs, references and graders were reused. +No model, prompt or runtime changes were made during the experiment; no failed trial was +retried. The shared allowance was 96 model calls, with 24 model calls, 50 tool calls and +900 seconds per case. No paid-API fallback, API credit purchase or model judge was used. + +[LangSmith experiment](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/datasets/d2e934c2-77e8-4926-93b3-0bcd644b200d/compare?selectedSessions=f667f6fc-4e8d-4bba-bcbf-fcde7d808d40) +(`backintel-specialist-gpt-6.1-sol-e2bce7f1`). + +| Case | Reviewed outcome | Calls | Trial seconds | +| --- | --- | ---: | ---: | +| Retail sales/refunds | Pass: USD 135; category/day tables and delivered bundle agree with references | 11 | 104.521 | +| Missing costs | Pass under the existing refusal criterion: costs/expenses are absent; no profit fabricated | 12 | 86.928 | +| Irregular energy | Fail: correct intermediate calculation/render, but missing structured completion prevented final export | 12 | 110.779 | +| Service availability | Pass: weighted availability 59.43396226415094%; delivered bundle and reconciliations agree | 12 | 117.030 | + +**Result: 3/4 passed, one completion-contract failure, zero unresolved cases after trace +classification.** All four trials booted and executed in SmolVM 1.31.1 without startup, +provider, quota or timeout failure. Two verified report/chart/workbook bundles were delivered. +The service downtime chart was opened and checked for readable labels, units and values; +this is a representative visual check, not a comprehensive aesthetic grade. + +Energy's final provider response had status `completed` and contained ordinary prose with +no `AnalysisCompletion` call. `run_analysis` consequently received no structured response, +failed validation and never entered its clean renderer/export stage. The trace contains +the expected 2.1791666666666667 kWh and 127/540 meter-minutes, successful reproduction/render +and PNG inspection, but the temporary VM's report is not a delivered artifact. This is a +valid observed end-to-end failure under the existing completion requirement, not a numerical +failure or an invalid infrastructure trial. The unsuccessful attempt remains unchanged. + +The automatic grader initially left this failed run unclassified (`evaluation_valid=0`, +`task_pass=null`). After reading its full trajectory, agent review classified it as +`agent_completion_contract` and updated those two LangSmith feedback entries to 1 and 0, +with explanatory comments. Original feedback was captured before the update. Read-back +verified all four stored run states/backends and reviewed scores. This was agent review, +not human calibration. `comparison.json` retains the original provisional results; +`review.json` supplies the reviewed summary and the original record hashes. + +Usage was **47/96 model calls**, **468,059 input tokens** and **11,830 output tokens**, +with complete token reporting. Summed trial latency was **419.258 seconds**. Subscription +billing supplies no per-request dollar charge; this is not labelled free inference. +The previous Docker Sol trial passed 4/4 with 47 calls and 420.141 seconds. One trial per +case, with different successful-delivery counts, cannot establish a runtime speed gain or +prove that SmolVM caused the completion failure. + +Full trajectories also exposed two narrower improvement opportunities. Every case first +tried to read virtual skill paths through the shell, then recovered with `read_file`. +The missing-cost case correctly refused but unnecessarily rendered a population/schema +report and mentioned temporary artifact paths; refusal outputs are not exported. Its pass +establishes the frozen cost-sufficiency criterion only, not efficient stopping or reliable +artifact delivery claims. Keep that limitation visible rather than silently strengthening +the grader after observing this trial. + +Next development target: require a valid terminal completion before accepting a successful +analysis, preserve honest delivery state, and prevent refusal summaries from promising +unexported files. Clarify the distinction between virtual skill files and sandbox files. +Develop and test that candidate separately, then compare it against these retained failures; +do not rerun this baseline until it happens to pass. Broader INT-429/INT-454 acceptance remains +open; these controlled synthetic cases do not establish unfamiliar-domain or large-data reliability. + +Evidence directory: `data/output/analysis-sol-smolvm-20261112T162410382489Z/`, containing +`comparison.json`, each `arm-0//run.json`, two delivered bundles, `review.json`, +`langsmith-feedback-before-review.json` and `langsmith-verification.json`. Runtime commit: +`d62995a`; full working source: `4db4fdfda73698e3`; method: `60464eed3476`. The comparison harness +remains separately pending local work. Its evaluator hash is +`81e82ca777660c5920a582f7b1b18d01dded5eb3e82f7658e4fb6de798918412`; the sole change from the +Docker harness is its documentation's runtime label. Dataset hash +`7294f8c0ca8ea4ef54e1667535e845a4d1cbd8591b82b5d235583cf9ff006615` and case-module hash +`7e2aacfd0d507dac657a8b0f8c7ad126fc682bbfbdfe8a27f22a5c0a68b43cd7` match the original comparison. +Image ID and archive hash match the migration receipt above. No task-owned BackIntel VMs +remained after the run. No code was committed or pushed as part of this rerun. + +## Energy completion fix and live rerun — 2026-11-12 UTC + +The owner requested a fix for the missing structured completion and a fresh energy trial. +The specialist prompt previously asked for a final result without explicitly requiring the +`AnalysisCompletion` tool. LangChain's ToolStrategy requests a tool call, but the ChatGPT +plan adapter sends automatic tool selection. A normal prose-only model stop could therefore +end the graph with no structured response; validation then failed before export. + +The candidate explicitly requires the terminal tool call and adds one recovery prompt when +the model stops without it. Recovery stays in the same graph, retaining the transcript, +model/tool counters, inference allowance and whole-run deadline. A second text-only stop +fails explicitly. It never synthesizes a completion from prose or publishes a report based +only on an intermediate render. Existing completion validation, successful-render checks, +clean rendering and bundle verification remain required. The completion middleware runs +after call accounting (middleware hooks execute in reverse order); a regression caught and +prevented skipping that accounting on a recovery jump. + +`scripts/run_analysis_comparison.py --case irregular-energy` selects the requested case while +checking the full existing LangSmith dataset identity. The subscription allowance scales to +24 calls for this single-case run. All case definitions, reference answers and grading logic +are unchanged. No energy-specific answers or calculation hints were added to runtime files. + +[LangSmith experiment](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/datasets/d2e934c2-77e8-4926-93b3-0bcd644b200d/compare?selectedSessions=ff29645c-8e96-4cd8-875c-302ae006acf8) +(`backintel-specialist-gpt-6.1-sol-b26bb9b1`), run `01a1d132-09e0-7070-a491-d46ad33015cc`. + +**Result: 1/1 passed.** GPT-6.1 Sol, medium reasoning, ChatGPT subscription, one energy trial +in SmolVM 1.31.1. Sol directly submitted a valid `AnalysisCompletion`; the live run needed +zero recovery prompts. Its clean exported report contains 2.1791666666666667 kWh, 127 observed +of 540 eligible meter-minutes and 23.51851851851852% coverage. All nine meter/window rows, +null-versus-zero semantics, units, chart bindings and workbook reconciliation passed the +unchanged grader. Eight manifest-listed artifacts were delivered, plus the manifest: report +HTML/JSON, heatmap HTML/PNG/SVG, two CSV tables and an XLSX workbook. The PNG was opened and +its labels, units and values were readable. This is not a general visual-aesthetic score. + +Usage: **17/24 model calls, 191,455 input and 3,806 output tokens, 141.795 seconds**. +Token reporting is complete; subscription billing does not provide a per-request dollar +charge. There were no trial retries or paid-API fallbacks. The model independently used SQL +interval clipping and Python datetime reconciliation, rendered and inspected the heatmap, +and completed through the terminal tool. The earlier virtual-skill shell lookup mistake +still occurred and recovered through `read_file`; that separate improvement remains open. + +Direct verification: the original text-only failure was reproduced in the actual graph. +After the fix, the focused suite passed **23 tests**, including a real SmolVM energy run +that deliberately stopped with prose, recovered once, and exported a reference-matching +bundle. Other checks cover direct completion, repeated prose rejection and the unchanged +model-call cap. The standard suite passed **187 tests with 15 optional skips**; Ruff lint, +format, CI's Vulture check (`--min-confidence 100`) and diff whitespace checks passed. +These overlapping test counts must not be added together. + +Evidence: `data/output/analysis-sol-smolvm-energy-fix-20261112T191758439825Z/`, with +`comparison.json`, `arm-0/irregular-energy/run.json`, the exported bundle and +`langsmith-verification.json`. The full trajectory was reviewed; packaged skill results +matched the previously reviewed trace. LangSmith read-back confirmed `completed`, SmolVM, +zero recovery prompts, `evaluation_valid=1` and `task_pass=1`; no score correction was needed. +Working source: `e903bc5341eb19cd`; method: `a544130f2127`; base commit: `d62995a`. +Dataset/case-module hashes match the original four-case run. Harness hash: +`841b8d1236d9050592ce823a4cbc5ae5fe201ca28983e5d849920fea9123e5d7`. +The runtime and image identities match the migration receipt. + +Reproduce the selected live run under an authorized ChatGPT subscription allowance: + +```bash +doppler run -p backintel -c dev -- uv run python -m scripts.run_analysis_comparison \ + --model chatgpt-plan/gpt-6.1-sol --reasoning medium --langsmith --case irregular-energy +``` + +This demonstrates one successful candidate trial, not a measured long-run completion rate. +The fallback was exercised by the deterministic SmolVM regression, not by this live run. +The original failed energy attempt and 3/4 baseline remain unchanged; do not combine them +with this selected rerun into a claimed 4/4 candidate experiment. Broader release acceptance +remains open. Changes are local and uncommitted; nothing was pushed or deployed. diff --git a/docs/roadmap.md b/docs/roadmap.md index 3a77f1b..6e3e7af 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -28,13 +28,13 @@ rate. See [current capabilities and measured outcomes](../README.md#what-has-bee ## Next achievable increments Each item below is small enough to define a concrete development handoff. A role names the -work involved, not an assigned person or a new runtime sub-agent. Existing scripts and -fixtures that are still pending publication must be packaged before another contributor can -reproduce their experiments from a checkout. +work involved, not an assigned person or a new runtime sub-agent. The +[four-case evaluation package](../evals/analysis/cases.md) supplies its scripts and fixtures +in the checkout; publication and release evidence belongs in the linked Linear issues. | Order | Deliverable | Completion criteria | Dependency / tracker | | --- | --- | --- | --- | -| 1 | Reproducible foundation and evaluation handoff | Publish the reviewed SmolVM/completion candidate and docs; package the pending four-case runner, case definitions, tests and reference generation as a separately reviewed change; verify a clean checkout can run the offline suite without private files | Engineering + evaluation; INT-451, INT-454 | +| 1 | Reproducible foundation and evaluation handoff | Publish the reviewed SmolVM/completion candidate and docs; publish the four-case runner, case definitions, tests and reference generation as a separately reviewed change; verify a clean checkout can run the offline suite without private files | Engineering + evaluation; INT-451, INT-454 | | 2 | Reliable finish, refusal and artifact claims | Clarify virtual skill reads versus shell paths; stop unsupported profit calculations without unnecessary report generation; never claim an unexported file exists; add regressions for these observed behaviors and retain the completion recovery/call-limit tests | Existing failure traces; INT-435, INT-454 | | 3 | A measured reliability improvement | Freeze model, inputs, prompt/skills, graders, candidate and finite allowance; execute the 92-response comparison described below; improve mandatory historical failures without regressing controls; assess all development and reserved cases | Readiness and frozen comparison plan; INT-430–438, parent INT-429 | | 4 | An unfamiliar-domain data package | Qualify three distinct non-retail sources, including terms, provenance, hashes, units, grain, nulls and joins; reserve one entire domain; define six challenge questions and one bounded discovery report per domain | Data + evaluation; INT-440, INT-441, INT-443 | diff --git a/evals/analysis/Task.md b/evals/analysis/Task.md new file mode 100644 index 0000000..9999ea6 --- /dev/null +++ b/evals/analysis/Task.md @@ -0,0 +1,188 @@ +# Sandboxed analytical report and missing-cost refusal + +Status: Draft, agent-reviewed implementation under the owner's instruction to execute +without a new approval pause. This is an offline integration task, not a human-calibrated +or live-model benchmark result. Date: 2026-11-11. + +The broader, separately documented [four-case comparison](cases.md) now adds energy and +service-availability tasks. This original demo remains the explicit coding-error-recovery +integration; its two cases and historical results are preserved. + +## Purpose and source + +Exercise BackIntel's Deep Agents analysis graph, persistent SmolVM workspace, ordinary +coding-error recovery, read-only source access, report renderer, PNG inspection tool and +exported artifact verification. Prior benchmark outputs were structured answers; this task +adds an actual report/chart/workbook bundle and independent numerical reconciliation. + +Governing implementation: `src/backintel/analysis.py`, `analysis_sandbox.py`, `artifacts.py`. +Runnable task: `scripts/run_analysis_demo.py`. Direct checks: `tests/test_analysis_workflow.py`. +The project's benchmark World Skill supplies the reusable rule that verifier references, +grading logic and scenario-specific expected values stay outside analyst mounts. It also +requires distinguishing harness failures from model behavior. No new World Skill truth or +human approval is implied here. + +## Exact business input + +Primary objective: “Produce a business report of delivered-order net revenue by category +and day, including a total metric, static and interactive charts, and supporting sheets.” + +Business context: a synthetic retailer's operational sales review; all money is USD. +Audience: operations manager. Period: 2026-11-01 through 2026-11-03 inclusive. +Definition: deduplicate exact repeated line rows, include delivered orders only, sum refunded +quantity per line before subtracting it, then multiply by integer unit cents. Revenue is +not profit. The source has no product-cost or expense fields. + +Second run objective: “Calculate net profit for the delivered orders in the assigned period.” +Other definitions and limitations are unchanged. A correct response identifies the missing +cost/expense inputs and does not substitute revenue for profit. + +## Environment and visibility + +The evaluator creates a new SQLite database with four tables: orders, lines, catalog and +refunds. It contains a cancelled order, an exact repeated line and two refund records for +one line. Only this source file is mounted, read-only. The normal source schema/query tools, +preinstalled packages and local documentation are available. Writable per-run paths and +resource limits are supplied by the sandbox, with no public network or host shell fallback. + +The fixture builder, this Task, scripted client and verifier remain on the host. Expected +numbers are not mounted into the VM. Each run uses a fresh VM/workspace and +new output directory. The report run and refusal run share the unchanged immutable source. + +A scripted `FakeMessagesListChatModel` supplies predetermined actions through the actual +Deep Agents graph. It discovers tables, reads relevant schemas/docs, writes a program with +an ordinary missing-column error, executes it, writes a corrected SQL-based program, +renders the report, receives a PNG through `inspect_chart`, and submits structured completion. +The refusal script inspects source availability and submits `cannot_answer`. + +This verifies tool wiring and artifact behavior, not autonomous discovery, LLM reasoning, +vision judgment, unseen-domain transfer or production reliability. It uses zero provider +inference calls; the graph still executes scripted model steps. +The live entry uses the same business input with an explicit output-layout requirement: +table IDs `categories`, `daily`, `total`, with columns respectively `Category, Revenue`, +`Date, Revenue`, and `Revenue`. Values must be USD; row/table presentation order is free. +This exposes a formatting contract, not answers, SQL, scripted actions or verifier logic. + +## Independent verification + +The verifier reads raw SQLite records and independently calculates values with integer +arithmetic, without reusing the analyst's aggregation SQL. Expected category revenues: +Home 75 USD, Outdoor 40 USD, Essentials 20 USD. Total: 135 USD. Daily totals: 35, 50 and +50 USD. The analytical SQL and independent calculation must agree. + +The complete primary pass condition requires: + +- Graph status completed; bundle exists and passes inventory/checksum validation. +- Canonical tables match the independent reference, including category/period labels; + row and table presentation order may differ. +- The requested total metric and category/day charts exist. +- CSV and XLSX cells reconcile with those canonical tables. +- A real missing-column execution error appears in the trace before the successful output. +- Actual PNG data returns through the chart-inspection tool. +- A deliberately wrong but structurally valid control fails independent numerical grading. +- The second graph run returns an explanatory missing-cost refusal. +- The source file's hash is unchanged after both runs. + +PNG tool delivery is not a visual-aesthetic grade. The implementation owner separately +opens representative output to inspect rendering. The deterministic renderer supplies the +static/interactive figures from the same canonical tables; numerical reference matching +does not establish the truth of every narrative sentence or an arbitrary live question. + +Correct equivalent calculations are acceptable for later live-model runs. This scripted +integration checks a chosen path, not a required tool sequence for general agents. +The wrong-result control changes table/metric totals consistently; structural consistency +alone cannot pass it. A malformed/corrupt bundle or missing evidence must fail the existing +bundle/runtime verification. Source modifications are prohibited collateral changes. + +SmolVM/image startup errors, unavailable dependencies, broken graph assembly or renderer +errors are infrastructure/harness failures. Keep failed run records and repair those faults +before interpreting success as a functioning integration. They are not model failures. + +## Running and evidence + +Build and prepare the project's sandbox image as described in +[sandbox setup](../../docs/analysis-sandbox.md#build-and-demonstrate), then run: + +```bash +uv run python -m scripts.run_analysis_demo +``` + +Outputs live under a fresh `data/output/analysis-demo-...` directory. `evaluation.json` +records independent checks, measured resources and explicit limits. Each run retains +`run.json` with the full graph trajectory. Delivered files are under +`report-run/artifacts/report/`, including report HTML/JSON, chart PNG/SVG/HTML, CSV tables, +supporting XLSX and manifest. Source datasets and run artifacts stay out of Git. + +Ordinary unit test: + +```bash +uv run pytest -q tests/test_analysis_workflow.py +``` + +Real SmolVM integration (opt-in, no provider calls): + +```bash +BACKINTEL_SMOLVM_TESTS=1 uv run pytest -q tests/test_analysis_workflow.py +``` + +This task does not claim a large-data result, a live model comparison, user-facing +deployment, LangSmith score improvement, or a completed release milestone. + +## Optional live LangSmith entry + +This historical OpenRouter-only entry also prepares two actual model runs through +`run_analysis`, one repetition each, +serially. It is deliberately smaller than the proposed eight-scenario, sixteen-run model +comparison. That broader campaign still needs additional tasks and approved execution. +Choose an independently qualified model/endpoint and a separately authorized allowance +before using this template; placeholders are not model selection or spend approval: + +```bash +doppler run -p backintel -c dev -- uv run python scripts/run_analysis_demo.py \ + --live --model "$BACKINTEL_ANALYSIS_MODEL" --reasoning medium \ + --max-cost-usd "$BACKINTEL_APPROVED_RUN_BUDGET" \ + --reservation-usd "$BACKINTEL_APPROVED_CALL_RESERVATION" \ + --langsmith --dataset backintel-analysis-artifacts-v1 +``` + +The current transport is Responses through OpenRouter, first-party provider only, without +fallbacks or retries. The reasoning/API combination must be supported by the qualified model. +Limits are 24 model calls, 50 tool calls and 900 seconds per case; the shared admission ledger +also caps both runs at 48 model calls and the explicit dollar allowance. Unknown charges or +a per-call reservation breach stop later calls. This is application admission control, not a +hard provider billing cap. A provider-side key limit is necessary for a strict dollar ceiling. + +`--langsmith` uploads the synthetic case inputs/references, actual run outputs and traces, +and deterministic feedback into LangSmith. Existing dataset contents must match the frozen +case/reference definitions; changes require a new dataset version. Without `--langsmith`, +live results stay local and tracing is disabled for those runs. Neither mode uses an LLM judge. +Both retain `run.json`, artifact bundles, usage and `evaluation.json`, including source, +reference and evaluator hashes. Reference outputs never enter the target's model input. + +Live scoring reuses the artifact/numerical verifier, but does not require an intentional +coding error or particular action sequence. `evaluation_valid=0` and an unset `task_pass` +mark failed/cancelled runs for fault classification. Completed wrong results receive +`task_pass=0`; a missing-cost refusal receives a deterministic status/reason check. That +check does not establish narrative nuance, visual aesthetics or general domain competence. +`cost_complete` is separate feedback; absent provider charges are never reported as free. + +For the current subscription-capable entry, use the [four-case comparison](cases.md). +Its completed live results are recorded separately from this original scripted task. +SDK behavior follows the installed LangSmith client and official +[evaluation guide](https://docs.langchain.com/langsmith/evaluation-quickstart) and +[local evaluation guidance](https://docs.langchain.com/langsmith/local). + +## Observed offline integration + +The real graph and rebuilt Docker image completed both scripted cases on 2026-11-11 +(UTC output directory `data/output/analysis-demo-20261112T002924589208Z`). All independent +checks passed: 12 report artifacts, intended coding-error recovery, PNG delivery, wrong-result +rejection, missing-cost refusal and unchanged source. Combined measured run latency was +5.19 seconds. The analyst container's peak memory was 129,146,880 bytes (about 123 MiB). +Model inference cost was zero because no provider was called. Both actual PNGs were opened +and inspected: labels, USD units and plotted category/day values were readable. + +The two preceding failed run records remain available. The first exposed an undersized +graph-recursion fuse; the second exposed an image/helper version mismatch. Those harness +faults were fixed before interpreting the successful result. This small integration does +not measure model judgment or establish large-dataset compute capacity. diff --git a/evals/analysis/cases.md b/evals/analysis/cases.md new file mode 100644 index 0000000..0051e0b --- /dev/null +++ b/evals/analysis/cases.md @@ -0,0 +1,150 @@ +# Controlled analysis cases, version 1 + +Status: controlled development suite, implemented and agent-reviewed. No human calibration. +The four-case live comparison completed on 2026-11-12 UTC; see +[the measured results and limitations](../../docs/results-analysis-sandbox.md#live-subscription-comparison--completed-2026-11-12-utc). +This specification does not establish broader reliability or release acceptance. + +The 2026-11-12 [Sol rerun on SmolVM](../../docs/results-analysis-sandbox.md#live-sol-on-smolvm--2026-11-12-utc) +retained these same inputs, reference answers and acceptance rules. It passed 3/4 after +trace classification: the energy trial returned prose without structured completion, so +its correct intermediate report was never exported. Existing refusal grading remains +narrow; it does not assess unnecessary work or temporary artifact-path claims. + +The subsequent [targeted completion fix and live energy rerun](../../docs/results-analysis-sandbox.md#energy-completion-fix-and-live-rerun--2026-11-12-utc) +passed with the same case data, references and graders. Only energy was rerun on that candidate; +the original four-case result remains 3/4. Use `--case irregular-energy` on the comparison +runner to select it while retaining full-dataset identity validation. + +This suite adds concrete analytical tasks to the isolated analysis specialist. It contains **four cases over three small synthetic SQLite fixtures**: retail sales, facilities energy and service operations. The profit-refusal case deliberately reuses the sales fixture. These are development cases with evaluator-written solutions, not four unseen datasets, production data, large-data performance evidence or a statistical model ranking. + +The implementation is [`scripts/analysis_cases.py`](../../scripts/analysis_cases.py); the serial graph runner is [`scripts/run_analysis_comparison.py`](../../scripts/run_analysis_comparison.py). The renderer contract is [`artifact-contract.md`](../../src/backintel/agent_files/docs/artifact-contract.md). The original two-case demonstration remains available separately. + +## Data and evaluator separation + +`build_cases(new_directory)` writes read-only source files under `sources/`, private JSON references under `private-references/`, and a manifest. Each case exposes an `AnalysisBrief` and a `SQLiteStore` mounting **only its assigned source file**. The brief includes business definitions, requested tables/units/chart bindings and limitations; it includes no expected numerical values or solution queries. Fixture builders, this document, references, graders and SQL reference solutions stay evaluator-side. A synthetic fixture is not independently sourced ground truth; its declared definitions make its expected answer inspectable. + +The manifest records the source file SHA-256, canonical source-row content SHA-256, private-reference SHA-256, brief SHA-256, and case-module SHA-256. The runner records its own evaluator fingerprint. Source/reference hashes are checked during grading, including refusal cases. Each reference stores its table values, requirements and explicit numeric tolerance. The original sales builder/oracle is imported from `run_analysis_demo.py`; preserve that file and the repository revision when reproducing a run. A manifest is run provenance, not proof of analytical correctness. + +SQLite engine versions can encode identical rows into different file bytes. A changed +source-byte fingerprint cannot reuse an existing frozen LangSmith dataset; retain the +original fixture/environment or use a separately named dataset and report the difference. +Selecting only one case still validates the entire four-case dataset identity. + +Each answerable case has two distinct calculation paths: + +| Path | Purpose | Access | +| --- | --- | --- | +| Raw-row stdlib oracle | Generates frozen expected values independently of the reference SQL | Evaluator only | +| Executable SQL solution, `reference_queries(case_id)` | Demonstrates solvability through the existing read-only Store | Evaluator only | +| `scripted_spec(case)` | Executes that SQL and creates an artifact specification from returned rows | Scripted integration only | +| Live specialist | Discovers schema and writes its own calculations from source and brief | Source, brief, sandbox tools and packaged documentation | + +The scripted solution is privileged: it is useful for testing the graph, evidence reproduction and renderer, and cannot measure model reasoning. A live run must never receive that solution or the hidden reference. + +## Task 1: sales-refunds + +**Business request:** Report delivered-order net revenue by category and day, with a total metric, charts and supporting sheets for 1–3 November 2026. All money is USD. Exact duplicate line rows must count once, cancelled orders are excluded, and multiple refunds must be summed per line before subtracting refunded quantity. Prices are integer cents. Revenue does not establish profit. + +**Source:** `sales.sqlite`, tables `orders`, `lines`, `catalog`, `refunds`. The fixture contains an exact repeated line, a cancelled order and split refunds. It has no costs or expenses. + +| Required table | Columns and units | Frozen expected rows | +| --- | --- | --- | +| `categories` | `Category`, `Revenue` (USD) | Home 75; Outdoor 40; Essentials 20 | +| `daily` | `Date`, `Revenue` (USD) | 2026-11-01: 35; 2026-11-02: 50; 2026-11-03: 50 | +| `total` | `Revenue` (USD) | 135 | + +Required metric: `total`, row 0, `Revenue`. Required charts: category bar chart (`Category` → `Revenue`) and daily line chart (`Date` → `Revenue`). Table/column/row presentation order is flexible. + +The independent oracle iterates deduplicated raw lines, aggregates raw refunds and calculates integer cents. The SQL solution deduplicates and preaggregates before joining. A realistic wrong control directly joins raw lines to refunds and reports 190 USD; the grader rejects it even though it can form a structurally valid report. + +## Task 2: missing-cost + +**Business request:** Calculate net profit for the same delivered orders. **Expected outcome:** `cannot_answer`, with a reason identifying missing or required product costs/expenses. It must not relabel revenue as profit or deliver a substitute profit report. + +The same `sales.sqlite` source and its limitations are used. There is no numerical reference solution. The deterministic refusal check requires an explicit statement that cost/expense/COGS inputs are absent. “Product costs are missing” passes; generic refusals, a requirement alone, “Costs are not missing” and “There is no cost issue” fail. This conservative language check is not a semantic judge of every possible refusal; ambiguous wording requires review. + +The 2026-11-13 evaluator correction also requires the declared metric ID alongside its table/row/column binding. Inputs and numerical references are unchanged. Preserve the original feedback and identify any regrade with the new case-module and runner hashes; apply the same evaluator to both comparison arms. Live CLI success additionally requires complete applicable accounting and review eligibility. Scripted checks remain integration evidence only. + +## Task 3: irregular-energy + +**Business request:** Report observed energy and time coverage for every assigned meter and hourly window, including empty observations, for 2026-03-15 00:00–03:00 UTC. Produce a coverage heatmap, observed-energy metric and supporting sheets. + +**Source:** `energy.sqlite`, tables `meters`, `readings`, `windows`. Three meters and three hourly windows define nine required rows. Readings include irregular timing, W/kW units, a timestamp with a UTC offset, valid zero power, bad/null samples, freshness gaps, a pre-window reading, an exclusive-end reading and a meter with no readings. + +Definitions supplied to the specialist: + +1. Normalize instants to UTC and power to watts. A good, non-null sample is constant until the next observation for its meter, including a bad/null observation, but for at most 15 minutes. +2. Clip those intervals to half-open reporting windows. Integrate `watts × seconds / 3,600,000` to kWh. Do not interpolate or fill uncovered time. +3. A valid zero-power interval counts as covered time. No valid covered interval means `Energy=null`, not zero. +4. Include every inventory-meter/window pair. Coverage is covered time divided by eligible time. The total denominator includes all assigned meters over the full reporting duration. + +`energy_by_window` requires `Meter`, `WindowStart`, `Energy` (kWh), `CoveredMinutes` (minutes), `CoveragePercent` (%). Instants below abbreviate the required full UTC strings `2026-03-15T00:00:00Z`, `...01:00:00Z`, `...02:00:00Z`. + +| Meter | Window UTC | Energy, kWh | Covered minutes | Coverage, % | +| --- | --- | ---: | ---: | ---: | +| alpha | 00:00 | 0.6166666667 | 37 | 61.6666666667 | +| alpha | 01:00 | 0.375 | 15 | 25 | +| alpha | 02:00 | 0.125 | 15 | 25 | +| beta | 00:00 | 0.3125 | 30 | 50 | +| beta | 01:00 | 0.75 | 30 | 50 | +| beta | 02:00 | null | 0 | 0 | +| gamma | 00:00 | null | 0 | 0 | +| gamma | 01:00 | null | 0 | 0 | +| gamma | 02:00 | null | 0 | 0 | + +`energy_total` requires one row: `Energy` = 2.1791666666666667 kWh; `ObservedMeterMinutes` = 127 meter-minutes; `EligibleMeterMinutes` = 540 meter-minutes; `CoveragePercent` = 23.51851851851852%. The metric binds to its `Energy` cell. The heatmap binds `energy_by_window`, x=`WindowStart`, y=`Meter`, z=`CoveragePercent`. + +The independent oracle parses timestamps with stdlib `datetime` and integrates interval intersections with `Decimal`. The reference SQL uses window functions and interval clipping. Tests also execute a different minute-grid solution; it is exact for these whole-minute fixture endpoints and accepted by the same grader. Missing/bad readings prevent full-window consumption estimates. Tariffs, costs, forecasts and causal conclusions are outside the supplied data. + +## Task 4: service-availability + +**Business request:** Reconcile per-service and customer-weighted fleet availability outside planned maintenance during 2026-10-01 00:00–01:00 UTC. Include every service and provide a downtime chart and supporting sheets. + +**Source:** `availability.sqlite`, tables `services`, `reporting_window`, `incidents`, `maintenance`. It contains overlapping and adjacent outages, overlapping maintenance, incidents crossing reporting boundaries, a fully maintained service, and a service with no incidents. Incident/calendar completeness is an explicit synthetic fixture assumption. In this fixture an eligible service with no incident was available. + +Definitions supplied to the specialist: + +1. Use half-open UTC intervals, clipped to the reporting window. Union each service’s maintenance intervals; eligible time excludes their union. +2. Union each service’s outages and intersect with eligible time. Neither outage overlaps nor maintenance overlaps count twice. +3. Per-service availability is `100 × (eligible − downtime) / eligible`; return null when eligible time is zero. +4. Fleet availability is `100 × (1 − Σ(weight × downtime) / Σ(weight × eligible))`. Do not average service percentages. Weights express exposure, not affected headcount or revenue loss. + +`service_availability` requires: + +| Service | EligibleMinutes (minutes) | DowntimeMinutes (minutes) | Availability (%) | Weight (customer-weight) | +| --- | ---: | ---: | ---: | ---: | +| api | 50 | 37 | 26 | 5 | +| batch | 50 | 15 | 70 | 2 | +| chat | 0 | 0 | null | 1 | +| search | 60 | 0 | 100 | 3 | + +`fleet_availability` requires one row: `WeightedEligibleMinutes` = 530 weighted-minutes, `WeightedDowntimeMinutes` = 215 weighted-minutes, `Availability` = 59.43396226415094%. The metric binds to this `Availability` cell. The bar chart binds `service_availability`, x=`Service`, y=`DowntimeMinutes`. + +The independent oracle counts raw outage/maintenance membership for each second; this is exact for the fixture’s whole-second endpoints. The SQL solution segments interval endpoints and sums eligible pieces. The naive average of eligible service percentages gives 65.3333333333%, which the grader rejects. These fixed synthetic intervals do not establish production monitoring completeness, root causes or revenue impact. + +## Acceptance and limits + +`verify_case_report` compares required columns by name and rows as an unordered multiset, accepts equivalent calculations, requires declared units and metric/chart bindings, and applies numeric tolerance `rel=1e-9, abs=1e-8`. It distinguishes null, zero and booleans; missing population rows and duplicate rows fail. Additional supporting tables are allowed. It does not demand the reference SQL, a fixed tool sequence, chart titles or prose wording. + +Numerical task grading is one layer. Artifact schema validation, executed evidence reproduction, clean-VM regeneration, bundle hashes and CSV/XLSX reconciliation are separate runtime/runner checks. A structurally consistent report can still be numerically wrong. These deterministic graders do not score narrative quality, visual aesthetics, unsupported causal prose, production reliability or unseen-domain generalization. The scripted reports explicitly identify their evaluator-written calculations. + +Prepare fixtures only, with no provider calls: + +```bash +uv run python -m scripts.analysis_cases data/output/analysis-cases-v1 +``` + +Run the focused independent-oracle and wrong-control checks: + +```bash +uv run pytest tests/test_analysis_cases.py -q +``` + +Run the four cases through the actual graph using privileged scripted reference actions; the output directory must be new and the analysis sandbox image must already be built: + +```bash +uv run python -m scripts.run_analysis_comparison --output data/output/analysis-comparison-scripted +``` + +The latter exercises infrastructure at zero model-provider cost. A live comparison is a separate mode requiring explicitly qualified model endpoints, reasoning settings and finite budget options. It receives only sources and business briefs. Identical frozen cases allow a useful next comparison, but four controlled cases with one trial each cannot support a broad competence claim; future held-out tasks should vary source domains, ambiguity, scale and real data quality without leaking evaluator solutions. diff --git a/results/runtime-validation-smoke.json b/results/runtime-validation-smoke.json index a7de421..8fe2d2d 100644 --- a/results/runtime-validation-smoke.json +++ b/results/runtime-validation-smoke.json @@ -1,5 +1,9 @@ { - "created_at": "2026-11-11T02:14:34.401914+00:00", + "created_at": "2026-11-13T04:57:47.664482+00:00", + "git_sha": "3d61a72", + "source_version": "e903bc5341eb19cd", + "working_tree": "clean", + "runner_sha256": "d5a41cebff0132758892f487a633f729994f257569dc8f4164fc492a0d1afd81", "scope": "Runtime evidence execution smoke; not a model quality or full data parity evaluation", "dataset": "olist", "model_calls": 0, @@ -90,7 +94,7 @@ "evidence_kind": "python", "valid_answer_accepted": true, "broken_second_evidence_rejected": true, - "sandbox": "docker" + "sandbox": "smolvm" } ], "final_running_containers": [], diff --git a/scripts/analysis_cases.py b/scripts/analysis_cases.py new file mode 100644 index 0000000..83b04d1 --- /dev/null +++ b/scripts/analysis_cases.py @@ -0,0 +1,872 @@ +"""Four controlled analysis tasks; fixtures/solutions/references stay evaluator-side. + +Only ``AnalysisCase.brief`` and its SQLite source enter the runtime. These cases cover +three source fixtures, not four independent datasets or unseen-domain competence. +""" + +from __future__ import annotations + +import copy +import hashlib +import json +import math +import re +import sqlite3 +from dataclasses import dataclass +from datetime import UTC, datetime +from decimal import Decimal +from pathlib import Path + +from backintel.analysis import AnalysisBrief +from backintel.stores.sqlite import SQLiteStore +from scripts.run_analysis_demo import ANALYSIS_QUERIES, brief, create_source, independent_reference + +CASE_IDS = ("sales-refunds", "missing-cost", "irregular-energy", "service-availability") + + +def _canonical(value: object) -> bytes: + return json.dumps(value, sort_keys=True, separators=(",", ":"), allow_nan=False).encode() + + +def _sha(path: Path) -> str: + with path.open("rb") as handle: + return hashlib.file_digest(handle, "sha256").hexdigest() + + +def _read_source(path: Path) -> dict[str, list]: + with sqlite3.connect(f"file:{path}?mode=ro", uri=True) as connection: + names = [ + r[0] + for r in connection.execute("SELECT name FROM sqlite_master WHERE type='table' ORDER BY name") + ] + return {name: connection.execute(f'SELECT * FROM "{name}"').fetchall() for name in names} + + +def _source_content_sha(path: Path) -> str: + rows = _read_source(path) + return hashlib.sha256(_canonical({k: sorted(v, key=_canonical) for k, v in rows.items()})).hexdigest() + + +@dataclass(frozen=True) +class AnalysisCase: + case_id: str + brief: AnalysisBrief + source: Path + reference: Path + source_sha256: str + source_content_sha256: str + reference_sha256: str + evaluator_sha256: str + expected_status: str + requirements: dict + + def open_store(self) -> SQLiteStore: + return SQLiteStore(self.source, dataset=self.case_id) + + +def _energy_source(path: Path) -> None: + with sqlite3.connect(path) as connection: + connection.executescript(""" + CREATE TABLE meters(meter_id TEXT PRIMARY KEY, site TEXT); + CREATE TABLE readings(meter_id TEXT, observed_at TEXT, power REAL, unit TEXT, quality TEXT); + CREATE TABLE windows(window_start_utc TEXT PRIMARY KEY, window_end_utc TEXT); + """) + connection.executemany( + "INSERT INTO meters VALUES (?,?)", [("alpha", "East"), ("beta", "West"), ("gamma", "Annex")] + ) + connection.executemany( + "INSERT INTO windows VALUES (?,?)", + [ + ("2026-03-15T00:00:00Z", "2026-03-15T01:00:00Z"), + ("2026-03-15T01:00:00Z", "2026-03-15T02:00:00Z"), + ("2026-03-15T02:00:00Z", "2026-03-15T03:00:00Z"), + ], + ) + connection.executemany( + "INSERT INTO readings VALUES (?,?,?,?,?)", + [ + ("alpha", "2026-03-14T23:55:00Z", 4000, "W", "good"), + ("alpha", "2026-03-15T00:00:00Z", 1000, "W", "good"), + ("alpha", "2026-03-15T00:07:00Z", 2, "kW", "good"), + ("alpha", "2026-03-15T00:25:00Z", None, "kW", "missing"), + ("alpha", "2026-03-15T00:40:00Z", 0, "W", "good"), + ("alpha", "2026-03-15T01:00:00Z", 1.5, "kW", "good"), + ("alpha", "2026-03-15T02:00:00Z", 500, "W", "good"), + ("alpha", "2026-03-15T03:00:00Z", 999, "kW", "good"), + ("beta", "2026-03-15T01:05:00+01:00", 0.5, "kW", "good"), + ("beta", "2026-03-15T00:20:00Z", 750, "W", "good"), + ("beta", "2026-03-15T00:50:00Z", 100, "kW", "bad"), + ("beta", "2026-03-15T01:10:00Z", 1000, "W", "good"), + ("beta", "2026-03-15T01:45:00Z", 2, "kW", "good"), + ("beta", "2026-03-15T02:15:00Z", None, "W", "missing"), + ], + ) + + +def _availability_source(path: Path) -> None: + with sqlite3.connect(path) as connection: + connection.executescript(""" + CREATE TABLE services(service_id TEXT PRIMARY KEY, customer_weight INTEGER); + CREATE TABLE reporting_window(window_start_utc TEXT, window_end_utc TEXT); + CREATE TABLE incidents(incident_id TEXT, service_id TEXT, started_at TEXT, ended_at TEXT); + CREATE TABLE maintenance(maintenance_id TEXT, service_id TEXT, started_at TEXT, ended_at TEXT); + """) + connection.executemany( + "INSERT INTO services VALUES (?,?)", [("api", 5), ("batch", 2), ("chat", 1), ("search", 3)] + ) + connection.execute( + "INSERT INTO reporting_window VALUES (?,?)", ("2026-10-01T00:00:00Z", "2026-10-01T01:00:00Z") + ) + connection.executemany( + "INSERT INTO incidents VALUES (?,?,?,?)", + [ + ("i1", "api", "2026-09-30T23:55:00Z", "2026-10-01T00:10:00Z"), + ("i2", "api", "2026-10-01T00:05:00Z", "2026-10-01T00:20:00Z"), + ("i3", "api", "2026-10-01T00:30:00Z", "2026-10-01T00:40:00Z"), + ("i4", "api", "2026-10-01T00:35:00Z", "2026-10-01T00:45:00Z"), + ("i5", "api", "2026-10-01T00:50:00Z", "2026-10-01T01:10:00Z"), + ("i6", "batch", "2026-10-01T00:10:00Z", "2026-10-01T00:25:00Z"), + ("i7", "batch", "2026-10-01T00:25:00Z", "2026-10-01T00:35:00Z"), + ], + ) + connection.executemany( + "INSERT INTO maintenance VALUES (?,?,?,?)", + [ + ("m1", "api", "2026-10-01T00:12:00Z", "2026-10-01T00:18:00Z"), + ("m2", "api", "2026-10-01T00:16:00Z", "2026-10-01T00:22:00Z"), + ("m3", "batch", "2026-10-01T00:20:00Z", "2026-10-01T00:30:00Z"), + ("m4", "chat", "2026-10-01T00:00:00Z", "2026-10-01T01:00:00Z"), + ], + ) + + +def _seconds(value: str) -> int: + return int(datetime.fromisoformat(value.replace("Z", "+00:00")).astimezone(UTC).timestamp()) + + +def _energy_oracle(path: Path) -> dict: + """Decimal interval integration over raw rows; independent of SQL windows/joins.""" + source = _read_source(path) + rows = [] + total_energy, total_covered, total_eligible = Decimal(0), 0, 0 + for meter_id, _ in sorted(source["meters"]): + observations = sorted( + (r for r in source["readings"] if r[0] == meter_id), key=lambda r: _seconds(r[1]) + ) + for begin, end in sorted(source["windows"]): + lower, upper = _seconds(begin), _seconds(end) + energy, covered = Decimal(0), 0 + for index, (_, instant, power, unit, quality) in enumerate(observations): + if quality != "good" or power is None: + continue + start = _seconds(instant) + next_start = _seconds(observations[index + 1][1]) if index + 1 < len(observations) else upper + stop = min(next_start, start + 900) + duration = max(0, min(stop, upper) - max(start, lower)) + watts = Decimal(str(power)) * (1000 if unit == "kW" else 1) + energy += watts * duration / Decimal(3600000) + covered += duration + total_energy += energy + total_covered += covered + total_eligible += upper - lower + rows.append( + [ + meter_id, + begin, + float(energy) if covered else None, + covered / 60, + covered * 100 / (upper - lower), + ] + ) + return { + "energy_by_window": { + "columns": ["Meter", "WindowStart", "Energy", "CoveredMinutes", "CoveragePercent"], + "rows": rows, + }, + "energy_total": { + "columns": ["Energy", "ObservedMeterMinutes", "EligibleMeterMinutes", "CoveragePercent"], + "rows": [ + [ + float(total_energy), + total_covered / 60, + total_eligible / 60, + total_covered * 100 / total_eligible, + ] + ], + }, + } + + +def _availability_oracle(path: Path) -> dict: + """Independent exact per-second membership; fixture endpoints have whole-second precision.""" + source = _read_source(path) + lower, upper = map(_seconds, source["reporting_window"][0]) + service_rows = [] + weighted_eligible = weighted_down = 0 + for service, weight in sorted(source["services"]): + outages = [(_seconds(s), _seconds(e)) for _, sid, s, e in source["incidents"] if sid == service] + maintenance = [(_seconds(s), _seconds(e)) for _, sid, s, e in source["maintenance"] if sid == service] + eligible = down = 0 + for second in range(lower, upper): + if any(start <= second < stop for start, stop in maintenance): + continue + eligible += 1 + down += any(start <= second < stop for start, stop in outages) + weighted_eligible += eligible * weight + weighted_down += down * weight + service_rows.append( + [ + service, + eligible / 60, + down / 60, + (eligible - down) * 100 / eligible if eligible else None, + weight, + ] + ) + return { + "service_availability": { + "columns": ["Service", "EligibleMinutes", "DowntimeMinutes", "Availability", "Weight"], + "rows": service_rows, + }, + "fleet_availability": { + "columns": ["WeightedEligibleMinutes", "WeightedDowntimeMinutes", "Availability"], + "rows": [ + [ + weighted_eligible / 60, + weighted_down / 60, + (weighted_eligible - weighted_down) * 100 / weighted_eligible, + ] + ], + }, + } + + +_ENERGY_SQL = """ +WITH observations AS ( + SELECT meter_id, CAST(strftime('%s',observed_at) AS INTEGER) AS instant, + CASE WHEN quality='good' AND power IS NOT NULL + THEN power * CASE unit WHEN 'kW' THEN 1000 ELSE 1 END END AS watts + FROM readings +), segments AS ( + SELECT meter_id, instant, watts, LEAD(instant) OVER (PARTITION BY meter_id ORDER BY instant) AS next_instant + FROM observations +), bounds AS ( + SELECT meter_id, window_start_utc, + CAST(strftime('%s',window_start_utc) AS INTEGER) AS lower, + CAST(strftime('%s',window_end_utc) AS INTEGER) AS upper + FROM meters CROSS JOIN windows +), pieces AS ( + SELECT b.*, s.watts, + MAX(0, MIN(COALESCE(s.next_instant,s.instant+900),s.instant+900,b.upper) + -MAX(s.instant,b.lower)) AS seconds + FROM bounds b LEFT JOIN segments s ON s.meter_id=b.meter_id + AND s.instant < b.upper AND s.instant+900 > b.lower +), hourly AS ( + SELECT meter_id AS Meter, window_start_utc AS WindowStart, + CASE WHEN SUM(CASE WHEN watts IS NOT NULL THEN seconds ELSE 0 END)>0 + THEN SUM(CASE WHEN watts IS NOT NULL THEN watts*seconds/3600000.0 ELSE 0 END) END AS Energy, + SUM(CASE WHEN watts IS NOT NULL THEN seconds ELSE 0 END)/60.0 AS CoveredMinutes, + (upper-lower)/60.0 AS EligibleMinutes + FROM pieces GROUP BY meter_id,window_start_utc +) +""".strip() + +_AVAILABILITY_SQL = """ +WITH bounds AS ( + SELECT CAST(strftime('%s',window_start_utc) AS INTEGER) AS lower, + CAST(strftime('%s',window_end_utc) AS INTEGER) AS upper FROM reporting_window +), outages AS ( + SELECT service_id, CAST(strftime('%s',started_at) AS INTEGER) AS start, + CAST(strftime('%s',ended_at) AS INTEGER) AS stop FROM incidents +), maintenance_intervals AS ( + SELECT service_id, CAST(strftime('%s',started_at) AS INTEGER) AS start, + CAST(strftime('%s',ended_at) AS INTEGER) AS stop FROM maintenance +), endpoints AS ( + SELECT service_id,lower AS point FROM services CROSS JOIN bounds + UNION SELECT service_id,upper FROM services CROSS JOIN bounds + UNION SELECT service_id,MAX(lower,MIN(upper,start)) FROM outages CROSS JOIN bounds + UNION SELECT service_id,MAX(lower,MIN(upper,stop)) FROM outages CROSS JOIN bounds + UNION SELECT service_id,MAX(lower,MIN(upper,start)) FROM maintenance_intervals CROSS JOIN bounds + UNION SELECT service_id,MAX(lower,MIN(upper,stop)) FROM maintenance_intervals CROSS JOIN bounds +), segments AS ( + SELECT service_id,point,LEAD(point) OVER (PARTITION BY service_id ORDER BY point) AS next_point + FROM endpoints +), eligible AS ( + SELECT s.*, next_point-point AS duration, + EXISTS(SELECT 1 FROM outages o WHERE o.service_id=s.service_id + AND o.start<=s.point AND o.stop>s.point) AS is_down + FROM segments s WHERE next_point IS NOT NULL AND NOT EXISTS( + SELECT 1 FROM maintenance_intervals m WHERE m.service_id=s.service_id + AND m.start<=s.point AND m.stop>s.point + ) +), totals AS ( + SELECT services.service_id AS Service,customer_weight AS Weight, + COALESCE(SUM(e.duration),0)/60.0 AS EligibleMinutes, + COALESCE(SUM(e.duration*e.is_down),0)/60.0 AS DowntimeMinutes + FROM services LEFT JOIN eligible e ON e.service_id=services.service_id GROUP BY services.service_id +) +""".strip() + + +def reference_queries(case_id: str) -> dict[str, str]: + """Host-only reference solution. Never include these queries in a live brief or mount.""" + if case_id == "sales-refunds": + return dict(ANALYSIS_QUERIES) + if case_id == "missing-cost": + return {} + if case_id == "irregular-energy": + return { + "energy_by_window": _ENERGY_SQL + + ( + "\nSELECT " + "Meter,WindowStart,Energy,CoveredMinutes,100.0*CoveredMinutes/EligibleMinutes AS " + "CoveragePercent FROM hourly ORDER BY Meter,WindowStart" + ), + "energy_total": _ENERGY_SQL + + ( + "\nSELECT SUM(Energy) AS Energy,SUM(CoveredMinutes) AS " + "ObservedMeterMinutes,SUM(EligibleMinutes) AS " + "EligibleMeterMinutes,100.0*SUM(CoveredMinutes)/SUM(EligibleMinutes) AS " + "CoveragePercent FROM hourly" + ), + } + if case_id == "service-availability": + return { + "service_availability": _AVAILABILITY_SQL + + ( + "\nSELECT Service,EligibleMinutes,DowntimeMinutes,CASE WHEN EligibleMinutes>0 THEN" + " 100.0*(EligibleMinutes-DowntimeMinutes)/EligibleMinutes END AS " + "Availability,Weight FROM totals ORDER BY Service" + ), + "fleet_availability": _AVAILABILITY_SQL + + ( + "\nSELECT SUM(Weight*EligibleMinutes) AS " + "WeightedEligibleMinutes,SUM(Weight*DowntimeMinutes) AS WeightedDowntimeMinutes," + "100.0*(SUM(Weight*EligibleMinutes)-" + "SUM(Weight*DowntimeMinutes))/SUM(Weight*EligibleMinutes) AS Availability FROM " + "totals" + ), + } + raise ValueError(f"Unknown analysis case: {case_id}") + + +def _requirements(case_id: str) -> dict: + if case_id == "missing-cost": + return { + "tables": {}, + "metrics": [], + "charts": [], + "refusal": "Explain the absent cost inputs; do not substitute revenue for profit.", + } + if case_id == "sales-refunds": + return { + "tables": { + "categories": {"columns": ["Category", "Revenue"], "units": {"Revenue": "USD"}}, + "daily": {"columns": ["Date", "Revenue"], "units": {"Revenue": "USD"}}, + "total": {"columns": ["Revenue"], "units": {"Revenue": "USD"}}, + }, + "metrics": [ + { + "id": "net-revenue", + "label": "Net revenue", + "table": "total", + "row": 0, + "column": "Revenue", + "definition": ( + "Delivered-order revenue after exact-row deduplication and refunded units; USD." + ), + } + ], + "charts": [ + { + "id": "category-revenue", + "title": "Delivered revenue by category", + "kind": "bar", + "table": "categories", + "x": "Category", + "y": "Revenue", + }, + { + "id": "daily-revenue", + "title": "Delivered revenue by day", + "kind": "line", + "table": "daily", + "x": "Date", + "y": "Revenue", + }, + ], + } + if case_id == "irregular-energy": + return { + "tables": { + "energy_by_window": { + "columns": ["Meter", "WindowStart", "Energy", "CoveredMinutes", "CoveragePercent"], + "units": {"Energy": "kWh", "CoveredMinutes": "minutes", "CoveragePercent": "%"}, + }, + "energy_total": { + "columns": ["Energy", "ObservedMeterMinutes", "EligibleMeterMinutes", "CoveragePercent"], + "units": { + "Energy": "kWh", + "ObservedMeterMinutes": "meter-minutes", + "EligibleMeterMinutes": "meter-minutes", + "CoveragePercent": "%", + }, + }, + }, + "metrics": [ + { + "id": "observed-energy", + "label": "Energy during covered intervals", + "table": "energy_total", + "row": 0, + "column": "Energy", + "definition": ( + "Integrated power only over valid, freshness-bounded intervals; not an " + "estimate for missing time." + ), + } + ], + "charts": [ + { + "id": "coverage-grid", + "title": "Observation coverage by meter and hour", + "kind": "heatmap", + "table": "energy_by_window", + "x": "WindowStart", + "y": "Meter", + "z": "CoveragePercent", + } + ], + } + return { + "tables": { + "service_availability": { + "columns": ["Service", "EligibleMinutes", "DowntimeMinutes", "Availability", "Weight"], + "units": { + "EligibleMinutes": "minutes", + "DowntimeMinutes": "minutes", + "Availability": "%", + "Weight": "customer-weight", + }, + }, + "fleet_availability": { + "columns": ["WeightedEligibleMinutes", "WeightedDowntimeMinutes", "Availability"], + "units": { + "WeightedEligibleMinutes": "weighted-minutes", + "WeightedDowntimeMinutes": "weighted-minutes", + "Availability": "%", + }, + }, + }, + "metrics": [ + { + "id": "weighted-availability", + "label": "Weighted fleet availability", + "table": "fleet_availability", + "row": 0, + "column": "Availability", + "definition": ( + "One minus weighted eligible downtime divided by weighted eligible service " + "time, expressed as a percentage." + ), + } + ], + "charts": [ + { + "id": "service-downtime", + "title": "Outage time outside planned maintenance", + "kind": "bar", + "table": "service_availability", + "x": "Service", + "y": "DowntimeMinutes", + } + ], + } + + +def _case_brief(case_id: str, requirements: dict) -> AnalysisBrief: + if case_id in {"sales-refunds", "missing-cost"}: + result = brief(profit=case_id == "missing-cost") + elif case_id == "irregular-energy": + result = AnalysisBrief( + objective=( + "Report observed energy and time coverage for every assigned meter in every " + "reporting window, including meters/windows with no usable observations. Produce " + "a coverage heatmap, total observed-energy metric and supporting sheets." + ), + business_context=( + "Synthetic facilities operations. Readings are instantaneous power samples, not " + "cumulative energy. The meter inventory and windows define the eligible " + "population." + ), + audience="Facilities operations manager", + metric_definitions={ + "energy": ( + "Normalize all timestamps to UTC and W/kW to watts. Hold a good, non-null " + "sample constant from its timestamp until the next observation for that meter" + " (including a bad/null observation), but for at most 15 minutes. Clip " + "intervals to each half-open reporting window. Integrate " + "watts*seconds/3600000 to kWh. Do not interpolate or fill missing coverage. A" + " valid zero sample counts as covered time; a window with no valid interval " + "has Energy=null, not zero." + ), + "coverage": ( + "CoveredMinutes counts usable interval time. " + "CoveragePercent=100*covered/window duration. Include every meter/window " + "pair. Totals sum observed energy and covered meter-minutes; denominator is " + "every assigned meter times the full reporting duration. Never average per-" + "window percentages if durations differ." + ), + }, + time_window=( + "2026-03-15T00:00:00Z inclusive to 2026-03-15T03:00:00Z exclusive; use the hourly" + " windows table." + ), + known_limitations=[ + "Controlled synthetic source, not real building operations.", + ( + "Missing/bad samples and freshness gaps prevent full-window consumption " + "estimates. No tariff, cost or causal explanation is supported." + ), + ], + required_outputs=["report", "coverage heatmap", "CSV", "XLSX"], + ) + else: + result = AnalysisBrief( + objective=( + "Report service and customer-weighted fleet availability outside planned " + "maintenance. Reconcile eligible time and downtime, include every service, and " + "produce a service-downtime chart and supporting sheets." + ), + business_context=( + "Synthetic service-operations incident review. Incident and maintenance records " + "are complete for the assigned window; an eligible service with no incident was " + "available. Customer weights are fixed exposure weights, not counts of affected " + "people." + ), + audience="Service operations lead", + metric_definitions={ + "eligible_time": ( + "Use half-open UTC intervals and clip everything to the reporting window. For" + " each service, union overlapping or adjacent maintenance intervals and " + "remove them from the reporting duration. Do not subtract overlaps twice." + ), + "downtime": ( + "Union incidents per service. Count only their intersection with eligible " + "time, excluding maintenance even when an incident overlaps it. Overlapping " + "incidents count once." + ), + "availability": ( + "Per service: 100*(eligible-downtime)/eligible; null if eligible=0. Fleet: " + "100*(1-SUM(weight*downtime)/SUM(weight*eligible)). Do not take an unweighted" + " average of service percentages. Fully maintained services have no eligible " + "exposure and must not become zero-percent availability." + ), + }, + time_window="2026-10-01T00:00:00Z inclusive to 2026-10-01T01:00:00Z exclusive.", + known_limitations=[ + ( + "Controlled synthetic, whole-second incident intervals. Completeness is a " + "declared fixture assumption, not a production monitoring guarantee." + ), + "Availability is exposure weighted; it does not estimate revenue loss or incident causes.", + ], + required_outputs=["report", "service downtime chart", "CSV", "XLSX"], + ) + if requirements["tables"]: + result.objective += " Required output contract (table/column order may vary): " + json.dumps( + requirements + ) + return result + + +def build_cases(root: Path) -> list[AnalysisCase]: + """Create three source fixtures and four tasks in a new host-owned directory.""" + root = Path(root).resolve() + root.mkdir(parents=True, exist_ok=False, mode=0o700) + sources = root / "sources" + private = root / "private-references" + sources.mkdir() + private.mkdir(mode=0o700) + sales, energy, availability = (sources / f"{name}.sqlite" for name in ("sales", "energy", "availability")) + create_source(sales) + _energy_source(energy) + _availability_source(availability) + for source in (sales, energy, availability): + source.chmod(0o444) + source_by_case = dict(zip(CASE_IDS, (sales, sales, energy, availability), strict=True)) + evaluator_sha = _sha(Path(__file__)) + cases = [] + for case_id in CASE_IDS: + source = source_by_case[case_id] + requirements = _requirements(case_id) + status = "cannot_answer" if case_id == "missing-cost" else "completed" + tables = ( + independent_reference(source) + if case_id == "sales-refunds" + else _energy_oracle(source) + if case_id == "irregular-energy" + else _availability_oracle(source) + if case_id == "service-availability" + else {} + ) + reference = { + "schema_version": 1, + "case_id": case_id, + "expected_status": status, + "source_sha256": _sha(source), + "source_content_sha256": _source_content_sha(source), + "tables": tables, + "numeric_tolerance": {"rel": 1e-9, "abs": 1e-8}, + "requirements": requirements, + "refusal_rule": ( + "An absent/missing/required cost input must be named; no report substituting " + "revenue for profit." + ), + } + path = private / f"{case_id}.json" + path.write_bytes(_canonical(reference) + b"\n") + path.chmod(0o600) + cases.append( + AnalysisCase( + case_id=case_id, + brief=_case_brief(case_id, requirements), + source=source, + reference=path, + source_sha256=reference["source_sha256"], + source_content_sha256=reference["source_content_sha256"], + reference_sha256=_sha(path), + evaluator_sha256=evaluator_sha, + expected_status=status, + requirements=requirements, + ) + ) + (root / "manifest.json").write_bytes( + _canonical( + { + "suite": "backintel-analysis-controlled-v1", + "cases": [ + { + "case_id": c.case_id, + "source": str(c.source.relative_to(root)), + "source_sha256": c.source_sha256, + "source_content_sha256": c.source_content_sha256, + "reference": str(c.reference.relative_to(root)), + "reference_sha256": c.reference_sha256, + "brief_sha256": hashlib.sha256(_canonical(c.brief.model_dump())).hexdigest(), + "evaluator_sha256": c.evaluator_sha256, + "expected_status": c.expected_status, + } + for c in cases + ], + "unique_source_fixtures": 3, + "model_calls": 0, + } + ) + + b"\n" + ) + return cases + + +def scripted_spec(case: AnalysisCase) -> dict: + """Execute host-only SQL reference solution; no oracle values are copied into the report.""" + if case.expected_status != "completed": + raise ValueError("A refusal case does not have a report solution") + queries = reference_queries(case.case_id) + store = case.open_store() + tables = [] + try: + for table_id, query in queries.items(): + result = store.query(query, max_rows=10001) + if result.truncated: + raise ValueError("Reference solution returned an unexpectedly truncated table") + tables.append( + { + "id": table_id, + "title": table_id.replace("_", " ").title(), + "columns": result.columns, + "rows": result.rows, + "units": case.requirements["tables"][table_id]["units"], + "evidence_ids": [table_id + "-query"], + } + ) + finally: + store.close() + return { + "run_id": "case-" + case.case_id, + "schema_version": 1, + "title": { + "sales-refunds": "Delivered revenue after refunds", + "irregular-energy": "Observed energy and missing coverage", + "service-availability": "Service availability outside planned maintenance", + }[case.case_id], + "summary": ( + "Controlled reference integration: calculated tables answer the declared business " + "brief. This execution uses evaluator-written code and does not measure model " + "reasoning." + ), + "findings": [], + "caveats": list(case.brief.known_limitations), + "sources": [ + { + "id": "assigned", + "snapshot": "sha256:" + case.source_sha256, + "description": "Controlled SQLite source: " + case.case_id, + } + ], + "evidence": [ + {"id": name + "-query", "kind": "sql", "code": query, "source_ids": ["assigned"]} + for name, query in queries.items() + ], + "tables": tables, + "metrics": copy.deepcopy(case.requirements["metrics"]), + "charts": copy.deepcopy(case.requirements["charts"]), + } + + +def _same_cell(actual, expected, tolerance: dict) -> bool: + if type(expected) in (int, float): + return ( + type(actual) in (int, float) + and math.isfinite(actual) + and math.isclose(actual, expected, rel_tol=tolerance["rel"], abs_tol=tolerance["abs"]) + ) + return type(actual) is type(expected) and actual == expected + + +def _matching_rows(actual: list, expected: list, tolerance: dict) -> bool: + remaining = list(expected) + for row in actual: + for index, target in enumerate(remaining): + if len(row) == len(target) and all( + _same_cell(a, b, tolerance) for a, b in zip(row, target, strict=True) + ): + remaining.pop(index) + break + else: + return False + return not remaining + + +def verify_case_report(case: AnalysisCase, report: dict) -> None: + """Check independent numbers/units/bindings, accepting equivalent computations/order. + + Bundle integrity, executable evidence and narrative/visual grading are separate checks. + This function never requires a particular SQL program or tool sequence. + """ + if case.expected_status != "completed": + raise ValueError("This case requires refusal, not an analytical report") + if _sha(case.source) != case.source_sha256 or _sha(case.reference) != case.reference_sha256: + raise ValueError("Source or hidden reference changed after case preparation") + reference = json.loads(case.reference.read_text()) + tables = report.get("tables", []) + if not isinstance(tables, list) or any(not isinstance(t, dict) for t in tables): + raise ValueError("Report tables must be objects") + actual = {t.get("id"): t for t in tables} + if len(actual) != len(tables): + raise ValueError("Duplicate table identifiers") + for name, expected in reference["tables"].items(): + table = actual.get(name) + if table is None: + raise ValueError(f"Missing required table {name}") + columns = table.get("columns", []) + if len(columns) != len(set(columns)) or set(columns) != set(expected["columns"]): + raise ValueError(f"Table {name} has incorrect columns") + order = [columns.index(c) for c in expected["columns"]] + rows = table.get("rows", []) + if any(not isinstance(r, list) or len(r) != len(columns) for r in rows): + raise ValueError(f"Table {name} has malformed rows") + arranged = [[r[i] for i in order] for r in rows] + if not _matching_rows(arranged, expected["rows"], reference["numeric_tolerance"]): + raise ValueError(f"Table {name} differs from the independent raw-source reference") + if any( + table.get("units", {}).get(c) != unit + for c, unit in case.requirements["tables"][name]["units"].items() + ): + raise ValueError(f"Table {name} has incorrect units") + for required in case.requirements["metrics"]: + if not any( + all(m.get(k) == required[k] for k in ("id", "table", "row", "column")) + for m in report.get("metrics", []) + ): + raise ValueError("Required summary metric is missing or incorrectly bound") + for required in case.requirements["charts"]: + keys = [k for k in ("kind", "table", "x", "y", "z") if k in required] + if not any(all(c.get(k) == required[k] for k in keys) for c in report.get("charts", [])): + raise ValueError("Required analytical chart is missing or incorrectly bound") + + +def _explains_missing_cost(reason: str) -> bool: + """Recognize explicit missing-input statements, not disconnected keywords. + + This is a conservative language check, not a semantic judge. Ambiguous wording + remains a failed automatic check for review rather than an inferred pass. + """ + cost = r"\b(?:costs?|expenses?|cogs)\b" + absence = r"(?:missing|absent|unavailable|not\s+(?:provided|supplied|available|present|included))" + modifiers = r"(?:(?:required|product|unit|operating|operational|purchase|actual|the|any)\s+)*" + has_gap = False + for clause in re.split(r"[.!?;\n]", reason.lower()): + # Negating an absence (or asserting presence) cannot establish a data gap. + if re.search(r"\b(?:not|never|isn't|aren't|isn’t|aren’t)\s+(?:missing|absent|unavailable)\b", clause): + return False + negative_subject = re.search(r"\b(?:no|without|lacks?|lacking)\s+" + modifiers + cost, clause) + if not negative_subject and re.search( + cost + r".{0,60}\b(?:are|is)\s+(?:present|available|provided|supplied)\b", clause + ): + return False + if re.search(r"\bnot\s+(?:true|the case)\b", clause): + continue + # Costs may be qualified by units or grouped with other absent inputs. + if re.search(cost + r".{0,100}\b(?:are|is|were|was)\s+" + absence + r"\b", clause): + has_gap = True + if re.search(r"\b" + absence + r"[:\s]+" + modifiers + cost, clause): + has_gap = True + if re.search( + r"\b(?:no|without|lacks?|lacking)\s+" + + modifiers + + cost + + r"(?:\s+(?:data|inputs?|fields?|records?|information))?" + + r"(?=\s*(?:[,/:]|$|\b(?:is|are|for|to|in|and|or)\b))", + clause, + ): + has_gap = True + return has_gap + + +def verify_case_completion(case: AnalysisCase, completion: dict, report: dict | None = None) -> None: + if _sha(case.source) != case.source_sha256 or _sha(case.reference) != case.reference_sha256: + raise ValueError("Source or hidden reference changed after case preparation") + if completion.get("status") != case.expected_status: + raise ValueError(f"Expected {case.expected_status}, received {completion.get('status')}") + if case.expected_status == "completed": + if report is None: + raise ValueError("Completed analysis has no report") + verify_case_report(case, report) + return + reason = completion.get("reason") or "" + if not isinstance(reason, str) or not _explains_missing_cost(reason): + raise ValueError("Refusal must explain the absent required cost inputs") + if report is not None: + raise ValueError("Missing-cost refusal must not deliver a substitute profit report") + + +if __name__ == "__main__": + import argparse + + parser = argparse.ArgumentParser( + description="Prepare controlled analytical cases without calling a model" + ) + parser.add_argument("output", type=Path, help="A new evaluator-owned directory") + args = parser.parse_args() + cases = build_cases(args.output) + print( + json.dumps( + { + "case_ids": [case.case_id for case in cases], + "manifest": str(args.output / "manifest.json"), + "model_calls": 0, + }, + indent=2, + ) + ) diff --git a/scripts/run_analysis_comparison.py b/scripts/run_analysis_comparison.py new file mode 100644 index 0000000..7363422 --- /dev/null +++ b/scripts/run_analysis_comparison.py @@ -0,0 +1,456 @@ +"""Run the four-case specialist suite offline, or compare explicitly qualified models. + +The default exercises reference scripts through the real graph and SmolVM. It measures +integration only. Live arms use either ChatGPT subscription call limits or an explicitly +budgeted OpenRouter API route, with no fallback between billing modes. +""" + +from __future__ import annotations + +import argparse +import asyncio +import hashlib +import json +from datetime import UTC, datetime +from pathlib import Path + +from langsmith import Client, tracing_context + +from backintel.analysis import _same_cells, run_analysis +from backintel.artifacts import verify_bundle +from backintel.bench.usage import RunBudget +from backintel.models import build_model +from scripts.analysis_cases import ( + CASE_IDS, + AnalysisCase, + build_cases, + scripted_spec, + verify_case_completion, +) +from scripts.run_analysis_demo import ScriptedAnalysisModel, call + + +def fingerprint(path: Path) -> str: + with path.open("rb") as stream: + return hashlib.file_digest(stream, "sha256").hexdigest() + + +def examples_for(cases: list[AnalysisCase]) -> list[dict]: + """The target receives inputs only; reference values are evaluator-owned outputs.""" + return [ + { + "inputs": {"case_id": case.case_id, "brief": case.brief.model_dump()}, + "outputs": { + "source_sha256": case.source_sha256, + "reference_sha256": case.reference_sha256, + "expected_status": case.expected_status, + "reference": json.loads(case.reference.read_text()), + }, + } + for case in cases + ] + + +def reference_model(case: AnalysisCase) -> ScriptedAnalysisModel: + """Privileged reference solution: never use these scripted actions for live trials.""" + steps = [("list_tables", {})] + if case.expected_status == "cannot_answer": + steps.append( + ( + "AnalysisCompletion", + { + "status": "cannot_answer", + "summary": "Profit needs cost and expense inputs absent from the source.", + "reason": "Product costs and operating expenses are unavailable.", + "caveats": ["Scripted integration; not a live model decision."], + }, + ) + ) + else: + spec = scripted_spec(case) + steps.extend( + [ + ("run_shell", {"command": "cat /docs/artifact-contract.md"}), + ( + "write_analysis_file", + {"path": "/workspace/report.json", "content": json.dumps(spec)}, + ), + ("render_report", {"spec_path": "/workspace/report.json"}), + ( + "inspect_chart", + {"path": f"/artifacts/report/charts/{spec['charts'][0]['id']}.png"}, + ), + ( + "AnalysisCompletion", + { + "status": "completed", + "summary": "The reference-scripted graph rendered the requested bundle.", + "caveats": ["Integration evidence, not autonomous model capability."], + }, + ), + ] + ) + return ScriptedAnalysisModel(responses=[call(name, args, i) for i, (name, args) in enumerate(steps)]) + + +def grade(case: AnalysisCase, record: dict, directory: Path) -> list[dict]: + """Numerical/artifact checks; unresolved failures cannot qualify an arm for review.""" + if fingerprint(case.source) != case.source_sha256 or fingerprint(case.reference) != case.reference_sha256: + return [ + {"key": "evaluation_valid", "score": 0, "comment": "Source/reference identity changed."}, + {"key": "task_pass", "score": None, "comment": "Rebuild a new frozen trial."}, + ] + if record.get("status") in {"failed", "cancelled", "not_run"}: + return [ + {"key": "evaluation_valid", "score": 0, "comment": "Failure needs layer classification."}, + {"key": "task_pass", "score": None, "comment": record.get("error", "Run unfinished.")}, + ] + try: + report = None + if record.get("status") == "completed": + bundle = directory / "artifacts" / "report" + verify_bundle(bundle) + report = json.loads((bundle / "report.json").read_text()) + from openpyxl import load_workbook + + workbook = load_workbook(bundle / "supporting.xlsx", read_only=True, data_only=True) + try: + for index, table in enumerate(report["tables"], 1): + sheet = workbook[f"T{index}-{table['id']}"[:31]] + # XLSX serializes finite floats to decimal text; use the same declared + # numeric tolerance as evidence reproduction, preserving null/text types. + if not _same_cells( + [list(row) for row in sheet.values], [table["columns"], *table["rows"]] + ): + raise ValueError("Workbook differs from the report's analytical tables.") + finally: + workbook.close() + completion = record.get("completion") or {} + if completion.get("status") != record.get("status"): + raise ValueError("Completion and run status disagree.") + verify_case_completion(case, completion, report) + passed, comment = True, "Independent case reference and delivered output agree." + except (ValueError, KeyError, OSError) as error: + passed, comment = False, str(error) + usage = record.get("usage", {}) + plan_billing = usage.get("billing_mode") == "chatgpt_plan" + return [ + {"key": "evaluation_valid", "score": 1, "comment": "Completed result available for grading."}, + {"key": "task_pass", "score": int(passed), "comment": comment}, + { + "key": "cost_complete", + "score": None if plan_billing else int(usage.get("cost_complete", False)), + "comment": "Subscription billing; per-request API dollar cost is not applicable." + if plan_billing + else "Provider-reported cost completeness, separate from task quality.", + }, + { + "key": "usage_complete", + "score": int(usage.get("usage_complete", False)), + "comment": "Reported token completeness, separate from task quality.", + }, + ] + + +def summarize_arm( + case_ids: list[str], records: dict, scores: dict, *, scripted: bool, billing_mode: str = "api" +) -> dict: + billing_mode = "scripted" if scripted else billing_mode + plan_billing = billing_mode == "chatgpt_plan" + feedback = {case: {item["key"]: item["score"] for item in items} for case, items in scores.items()} + passed = [case for case in case_ids if feedback.get(case, {}).get("task_pass") == 1] + wrong = [case for case in case_ids if feedback.get(case, {}).get("task_pass") == 0] + unresolved = [case for case in case_ids if feedback.get(case, {}).get("task_pass") is None] + costs = [record.get("usage", {}).get("cost_usd") for record in records.values()] + complete_cost = len(records) == len(case_ids) and all( + record.get("usage", {}).get("cost_complete", False) for record in records.values() + ) + complete_usage = len(records) == len(case_ids) and all( + record.get("usage", {}).get("usage_complete", False) for record in records.values() + ) + complete_accounting = complete_usage if plan_billing else complete_cost + if scripted: + provider_cost = known_cost = 0 + review_basis = "Scripted integration is not eligible for live quality review" + elif plan_billing: + provider_cost = known_cost = None + review_basis = "Task quality and complete reported tokens under subscription billing" + else: + provider_cost = sum(costs) if complete_cost else None + known_cost = sum(cost for cost in costs if cost is not None) + review_basis = "Task quality and complete provider-reported API charges" + return { + "billing_mode": billing_mode, + "case_count": len(case_ids), + "passed": passed, + "wrong": wrong, + "unresolved": unresolved, + "all_tasks_passed": len(passed) == len(case_ids), + "eligible_for_live_review": not scripted and len(passed) == len(case_ids) and complete_accounting, + "review_eligibility_basis": review_basis, + "provider_cost_usd": provider_cost, + "known_reported_cost_usd": known_cost, + "cost_applicable": not scripted and not plan_billing, + "cost_complete": True if scripted else False if plan_billing else complete_cost, + "usage_complete": complete_usage, + "input_tokens": sum(record.get("usage", {}).get("input_tokens", 0) for record in records.values()), + "output_tokens": sum(record.get("usage", {}).get("output_tokens", 0) for record in records.values()), + "total_latency_s": sum(record.get("latency_s", 0) for record in records.values()), + "provider_calls": 0 + if scripted + else sum(record.get("usage", {}).get("llm_calls", 0) for record in records.values()), + "scope": "Scripted integration only" + if scripted + else "One trial per case; quality-review eligibility only, no automatic model promotion", + } + + +def run_suite( + output: Path, + *, + model_ids: list[str] | None = None, + reasoning: str = "medium", + max_cost: float | None = None, + reservation: float | None = None, + langsmith: bool = False, + dataset_name: str = "backintel-analysis-four-cases-v1", + case_ids: list[str] | None = None, +) -> dict: + selected = list(CASE_IDS) if case_ids is None else case_ids + if not selected or len(set(selected)) != len(selected) or set(selected) - set(CASE_IDS): + raise ValueError("Select distinct known case IDs; at least one case is required.") + scripted = not model_ids + if scripted and (langsmith or max_cost is not None or reservation is not None): + raise ValueError("LangSmith and spend options require explicit live model IDs.") + billing_mode, budget = "scripted", None + if model_ids: + providers = {model_id.split("/", 1)[0] for model_id in model_ids} + if ( + len(set(model_ids)) != len(model_ids) + or len(providers) != 1 + or not providers.issubset({"openrouter", "chatgpt-plan"}) + or any("/" not in model_id or not model_id.split("/", 1)[1] for model_id in model_ids) + ): + raise ValueError("Use distinct qualified models in one billing mode: OpenRouter or chatgpt-plan.") + if providers == {"chatgpt-plan"}: + if max_cost is not None or reservation is not None: + raise ValueError("ChatGPT plan comparisons use call limits, not API dollar budget options.") + from backintel.bench.usage import PlanRunBudget + + billing_mode = "chatgpt_plan" + budget = PlanRunBudget(max_calls=24 * len(selected) * len(model_ids)) + else: + if max_cost is None or reservation is None: + raise ValueError("OpenRouter comparisons require an allowance and per-call reservation.") + billing_mode = "api" + budget = RunBudget(max_cost, 24 * len(selected) * len(model_ids), reservation) + if budget and not budget.can_admit(): + raise ValueError("The allowance cannot admit one reserved request.") + output = output.resolve() + output.mkdir(parents=True, exist_ok=False) + cases = build_cases(output / "fixtures") + examples = examples_for(cases) + by_id = {case.case_id: case for case in cases if case.case_id in selected} + expected_inputs = {example["inputs"]["case_id"]: example["inputs"] for example in examples} + summary = { + "execution": "scripted integration; not a model comparison" if scripted else "live model comparison", + "billing_mode": billing_mode, + "reasoning": None if scripted else reasoning, + "repetitions": 1, + "max_model_calls_per_case": 24, + "max_tool_calls_per_case": 50, + "timeout_s_per_case": 900, + "provider_fallbacks": False, + "case_ids": list(by_id), + "evaluator_sha256": fingerprint(Path(__file__)), + "cases_sha256": fingerprint(Path(__file__).with_name("analysis_cases.py")), + "dataset_sha256": hashlib.sha256(json.dumps(examples, sort_keys=True).encode()).hexdigest(), + "arms": [], + "limitations": [ + "Controlled small fixtures; not unseen real-domain or large-data acceptance.", + "Deterministic task checks do not grade every narrative claim or visual aesthetic.", + "Unresolved failures block acceptance and need classification; they are not silently excluded.", + ], + } + + def handlers(arm_dir, model, model_id, records, scores, arm): + def target(inputs: dict) -> dict: + case_id = inputs.get("case_id") + if case_id not in by_id or inputs != expected_inputs[case_id]: + raise ValueError("Inputs differ from the frozen case.") + case = by_id[case_id] + destination = arm_dir / case_id + if budget and not budget.can_admit(): + destination.mkdir() + record = { + "status": "not_run", + "error": "Shared budget cannot admit another call.", + "usage": { + "billing_mode": billing_mode, + "cost_usd": None, + "cost_applicable": billing_mode == "api", + "cost_complete": False, + "usage_complete": False, + "llm_calls": 0, + }, + } + (destination / "run.json").write_text(json.dumps(record, indent=2)) + else: + store = case.open_store() + try: + record = asyncio.run( + run_analysis( + case.brief, + model=reference_model(case) if scripted else model, + model_id=model_id, + store=store, + output=destination, + budget=budget, + max_model_calls=24, + max_tool_calls=50, + timeout_s=900, + scripted=scripted, + ) + ) + finally: + store.close() + records[case_id] = record + arm["records"][case_id] = str(destination / "run.json") + report = destination / "artifacts" / "report" / "report.json" + return { + **{key: value for key, value in record.items() if key != "messages"}, + "report": json.loads(report.read_text()) if report.exists() else None, + } + + def evaluator(inputs: dict, outputs: dict, reference_outputs: dict) -> list[dict]: + case_id = inputs["case_id"] + expected = next( + example["outputs"] for example in examples if example["inputs"]["case_id"] == case_id + ) + if reference_outputs != expected: + raise ValueError("Reference outputs differ from the frozen case.") + scores[case_id] = grade(by_id[case_id], outputs, arm_dir / case_id) + return scores[case_id] + + return target, evaluator + + client, stored = None, None + try: + if langsmith: + client = Client() + if client.has_dataset(dataset_name=dataset_name): + dataset = client.read_dataset(dataset_name=dataset_name) + else: + dataset = client.create_dataset( + dataset_name=dataset_name, + description=( + "BackIntel: retail, energy and service availability; frozen independent references." + ), + ) + client.create_examples(dataset_id=dataset.id, examples=examples) + stored = list(client.list_examples(dataset_id=dataset.id)) + actual = [{"inputs": item.inputs, "outputs": item.outputs} for item in stored] + if sorted(actual, key=lambda x: x["inputs"]["case_id"]) != sorted( + examples, key=lambda x: x["inputs"]["case_id"] + ): + raise ValueError("Existing LangSmith dataset differs. Use a new version; never overwrite it.") + stored = [item for item in stored if item.inputs["case_id"] in by_id] + for index, model_id in enumerate(model_ids or ["scripted-reference"]): + arm_dir = output / f"arm-{index}" + arm_dir.mkdir() + records, scores = {}, {} + arm = {"model": model_id, "billing_mode": billing_mode, "records": {}, "scores": scores} + summary["arms"].append(arm) + model = ( + None + if scripted + else build_model( + model_id, + reasoning_effort=reasoning, + responses_api=True, + first_party_only=billing_mode == "api", + max_retries=0, + timeout_s=300, + ) + ) + + target, evaluator = handlers(arm_dir, model, model_id, records, scores, arm) + + try: + if client: + results = client.evaluate( + target, + data=stored, + evaluators=[evaluator], + experiment_prefix="backintel-specialist-" + model_id.rsplit("/", 1)[-1], + metadata={key: value for key, value in summary.items() if key != "arms"}, + max_concurrency=1, + num_repetitions=1, + blocking=False, + ) + arm["runs"] = [ + {"case_id": row["example"].inputs["case_id"], "run_id": str(row["run"].id)} + for row in results + ] + arm["experiment"] = results.experiment_name + arm["url"] = client.read_project(project_name=results.experiment_name).url + else: + with tracing_context(enabled=False): + for example in examples: + if example["inputs"]["case_id"] not in by_id: + continue + result = target(example["inputs"]) + evaluator(example["inputs"], result, example["outputs"]) + finally: + arm["summary"] = summarize_arm( + list(by_id), records, scores, scripted=scripted, billing_mode=billing_mode + ) + finally: + summary["budget"] = budget.report() if budget else None + (output / "comparison.json").write_text(json.dumps(summary, indent=2, default=str)) + return summary + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--output", + type=Path, + default=Path("data/output") / ("analysis-suite-" + datetime.now(UTC).strftime("%Y%m%dT%H%M%S%fZ")), + ) + parser.add_argument( + "--model", + action="append", + help="Qualified openrouter/... or chatgpt-plan/... model; repeat within one billing mode.", + ) + parser.add_argument( + "--reasoning", choices=["none", "minimal", "low", "medium", "high", "xhigh"], default="medium" + ) + parser.add_argument( + "--max-cost-usd", type=float, help="OpenRouter only: shared approved dollar allowance." + ) + parser.add_argument("--reservation-usd", type=float, help="OpenRouter only: per-call dollar reservation.") + parser.add_argument("--langsmith", action="store_true") + parser.add_argument("--dataset", default="backintel-analysis-four-cases-v1") + parser.add_argument( + "--case", action="append", choices=CASE_IDS, help="Run only selected cases; repeatable." + ) + args = parser.parse_args() + result = run_suite( + args.output, + model_ids=args.model, + reasoning=args.reasoning, + max_cost=args.max_cost_usd, + reservation=args.reservation_usd, + langsmith=args.langsmith, + dataset_name=args.dataset, + case_ids=args.case, + ) + print(json.dumps(result, indent=2)) + passed_key = "eligible_for_live_review" if args.model else "all_tasks_passed" + if any(not arm["summary"][passed_key] for arm in result["arms"]): + raise SystemExit(1) + + +if __name__ == "__main__": + main() diff --git a/scripts/run_analysis_demo.py b/scripts/run_analysis_demo.py new file mode 100644 index 0000000..6a2a5dd --- /dev/null +++ b/scripts/run_analysis_demo.py @@ -0,0 +1,660 @@ +"""Exercise the specialist graph with an offline script or an explicit live model. + +By default this is an offline integration demonstration, not a live model evaluation. Its fixture +builder, scripted actions and independent verifier are evaluator-side and never mounted +into the analyst VM. Only the SQLite source file is mounted read-only. The optional +live mode uses the same two cases and requires an explicit model and finite budget. +""" + +from __future__ import annotations + +import argparse +import asyncio +import copy +import csv +import hashlib +import json +import sqlite3 +from collections import defaultdict +from datetime import UTC, datetime +from pathlib import Path + +from langchain_core.language_models.fake_chat_models import FakeMessagesListChatModel +from langchain_core.messages import AIMessage +from pydantic import Field + +from backintel.analysis import AnalysisBrief, run_analysis +from backintel.artifacts import validate_report, verify_bundle +from backintel.stores.sqlite import SQLiteStore + + +class ScriptedAnalysisModel(FakeMessagesListChatModel): + """A fixed sequence of tool calls; no inference, credentials or provider requests.""" + + model_name: str = "backintel-scripted-analysis-demo" + bindings: list[list[str]] = Field(default_factory=list) + + def bind_tools(self, tools, **kwargs): + self.bindings.append([t.name if hasattr(t, "name") else t["name"] for t in tools]) + return self + + +def call(name: str, args: dict, index: int) -> AIMessage: + return AIMessage(content="", tool_calls=[{"name": name, "args": args, "id": f"demo-{index}"}]) + + +def create_source(path: Path) -> None: + """Small synthetic business dataset, deliberately unlike a clean SUM fixture.""" + path.parent.mkdir(parents=True, exist_ok=True) + if path.exists(): + raise ValueError("Create a new fixture; do not overwrite an existing database.") + with sqlite3.connect(path) as connection: + connection.executescript(""" + CREATE TABLE orders(order_id INTEGER PRIMARY KEY, ordered_on TEXT, status TEXT); + CREATE TABLE lines(line_id INTEGER, order_id INTEGER, sku TEXT, + quantity INTEGER, unit_cents INTEGER); + CREATE TABLE catalog(sku TEXT PRIMARY KEY, category TEXT); + CREATE TABLE refunds(refund_id INTEGER PRIMARY KEY, line_id INTEGER, quantity INTEGER); + """) + connection.executemany( + "INSERT INTO orders VALUES (?,?,?)", + [ + (1, "2026-11-01", "delivered"), + (2, "2026-11-02", "delivered"), + (3, "2026-11-02", "cancelled"), + (4, "2026-11-03", "delivered"), + ], + ) + connection.executemany( + "INSERT INTO catalog VALUES (?,?)", + [ + ("a", "Essentials"), + ("b", "Home"), + ("c", "Outdoor"), + ], + ) + connection.executemany( + "INSERT INTO lines VALUES (?,?,?,?,?)", + [ + (1, 1, "a", 2, 1000), + (2, 1, "b", 1, 2500), + (2, 1, "b", 1, 2500), + (3, 2, "a", 3, 1000), + (4, 2, "c", 1, 4000), + (5, 3, "c", 8, 4000), + (6, 4, "b", 2, 2500), + ], + ) + connection.executemany( + "INSERT INTO refunds VALUES (?,?,?)", + [ + (1, 1, 1), + (2, 3, 1), + (3, 3, 1), + ], + ) + + +def brief(*, profit: bool = False) -> AnalysisBrief: + return AnalysisBrief( + objective=( + "Calculate net profit for the delivered orders in the assigned period." + if profit + else "Produce a business report of delivered-order net revenue by category and day, " + "including a total metric, static and interactive charts, and supporting sheets." + ), + business_context="A synthetic retailer wants an operational sales review. All money is USD.", + audience="Operations manager", + metric_definitions={ + "net_revenue": "Deduplicate exact repeated line rows; include delivered orders only; " + "subtract the summed refunded quantity for each line before multiplying by unit price. " + "Prices are integer cents. Revenue is not profit.", + }, + time_window="2026-11-01 through 2026-11-03 inclusive", + known_limitations=["Synthetic fixture, not customer data. No product-cost or expense fields."], + required_outputs=["report", "static charts", "interactive charts", "CSV", "XLSX"], + ) + + +_NET = """ +WITH unique_lines AS (SELECT DISTINCT line_id, order_id, sku, quantity, unit_cents FROM lines), +refund_totals AS (SELECT line_id, SUM(quantity) AS refunded FROM refunds GROUP BY line_id), +net AS ( + SELECT o.ordered_on, c.category, + (l.quantity - COALESCE(r.refunded, 0)) * l.unit_cents AS net_cents + FROM unique_lines l JOIN orders o ON o.order_id = l.order_id + JOIN catalog c ON c.sku = l.sku + LEFT JOIN refund_totals r ON r.line_id = l.line_id + WHERE o.status = 'delivered' AND o.ordered_on BETWEEN '2026-11-01' AND '2026-11-03' +) +""".strip() + +ANALYSIS_QUERIES = { + "categories": _NET + "\nSELECT category AS Category, SUM(net_cents)/100.0 AS Revenue " + "FROM net GROUP BY category ORDER BY Revenue DESC, Category", + "daily": _NET + "\nSELECT ordered_on AS Date, SUM(net_cents)/100.0 AS Revenue " + "FROM net GROUP BY ordered_on ORDER BY Date", + "total": _NET + "\nSELECT SUM(net_cents)/100.0 AS Revenue FROM net", +} + + +def analysis_program() -> str: + """The model's scripted calculation uses SQL, with no expected answers embedded.""" + return f"""import json, os, sqlite3 +from pathlib import Path +connection = sqlite3.connect("file:" + os.environ["BACKINTEL_SQLITE_PATH"] + "?mode=ro", uri=True) +queries = {ANALYSIS_QUERIES!r} +tables = [] +for table_id, query in queries.items(): + result = connection.execute(query) + tables.append({{"id": table_id, "title": table_id.title() + " net revenue", + "columns": [c[0] for c in result.description], + "rows": [list(row) for row in result.fetchall()], + "units": {{"Revenue": "USD"}}, "evidence_ids": [table_id + "-sql"]}}) +connection.close() +by_id = {{t["id"]: t for t in tables}} +assert abs(sum(row[1] for row in by_id["categories"]["rows"]) - by_id["total"]["rows"][0][0]) < 1e-9 +assert abs(sum(row[1] for row in by_id["daily"]["rows"]) - by_id["total"]["rows"][0][0]) < 1e-9 +report = {{ + "title": "Delivered sales after refunds", + "summary": "A consistent view of delivered-order revenue across categories and days. " + "Cancelled orders and exact duplicate lines are excluded; split refunds are combined.", + "findings": ["The category chart identifies the largest revenue contribution after refunds.", + "Daily values reconcile to the total for the same delivered-order population."], + "caveats": ["Synthetic three-day demonstration; no production reliability claim.", + "No cost data is supplied, so revenue cannot establish profit or margin.", + "The chart-inspection tool is exercised by a scripted client, not a vision model."], + "tables": tables, + "metrics": [{{"id": "net-revenue", "label": "Net revenue", "table": "total", "row": 0, + "column": "Revenue", "definition": "Delivered line revenue less refunded units, " + "after exact-row deduplication, 1–3 November 2026."}}], + "charts": [{{"id": "category-revenue", "title": "Net revenue by category", "kind": "bar", + "table": "categories", "x": "Category", "y": "Revenue"}}, + {{"id": "daily-revenue", "title": "Net revenue by day", "kind": "line", + "table": "daily", "x": "Date", "y": "Revenue"}}], + "evidence": [{{"id": name + "-sql", "kind": "sql", "code": query, + "source_ids": ["assigned"]}} for name, query in queries.items()] +}} +Path("/workspace/report-spec.json").write_text(json.dumps(report)) +print(json.dumps({{"tables": tables, "spec_path": "/workspace/report-spec.json"}})) +""" + + +def scripted_model(*, profit: bool = False) -> ScriptedAnalysisModel: + steps = [ + ("list_tables", {}), + ("describe_table", {"table": "lines"}), + ] + if profit: + steps.append( + ( + "AnalysisCompletion", + { + "status": "cannot_answer", + "summary": "Profit cannot be calculated from sales alone.", + "reason": "Product costs and operating expenses are absent. Revenue is not profit.", + "caveats": ["No proxy profit calculation was substituted."], + }, + ) + ) + else: + steps.extend( + [ + ("describe_table", {"table": "orders"}), + ("describe_table", {"table": "catalog"}), + ("describe_table", {"table": "refunds"}), + ("run_shell", {"command": "cat /docs/artifact-contract.md"}), + ( + "write_analysis_file", + { + "path": "/workspace/analysis.py", + "content": "import os, sqlite3\n" + "c=sqlite3.connect('file:'+os.environ['BACKINTEL_SQLITE_PATH']+'?mode=ro',uri=True)\n" + "print(c.execute('SELECT amount FROM lines').fetchall())\n", + }, + ), + ("run_shell", {"command": "python /workspace/analysis.py"}), + ("write_analysis_file", {"path": "/workspace/analysis.py", "content": analysis_program()}), + ("run_shell", {"command": "python /workspace/analysis.py"}), + ("render_report", {"spec_path": "/workspace/report-spec.json"}), + ("inspect_chart", {"path": "/artifacts/report/charts/category-revenue.png"}), + ( + "AnalysisCompletion", + { + "status": "completed", + "summary": "The scripted run generated a reconciled report, charts and workbook.", + "caveats": ["Integration demonstration, not a measured live-model success."], + }, + ), + ] + ) + return ScriptedAnalysisModel(responses=[call(name, args, i) for i, (name, args) in enumerate(steps)]) + + +def independent_reference(path: Path) -> dict: + """Independent row-wise integer arithmetic; does not reuse the analyst's SQL.""" + with sqlite3.connect(f"file:{path}?mode=ro", uri=True) as connection: + orders = {row[0]: row[1:] for row in connection.execute("SELECT * FROM orders")} + categories = dict(connection.execute("SELECT * FROM catalog")) + refunds = defaultdict(int) + for _, line_id, quantity in connection.execute("SELECT * FROM refunds"): + refunds[line_id] += quantity + lines = set(connection.execute("SELECT * FROM lines")) + by_category, by_day = defaultdict(int), defaultdict(int) + for line_id, order_id, sku, quantity, unit_cents in lines: + day, status = orders[order_id] + if status != "delivered" or not "2026-11-01" <= day <= "2026-11-03": + continue + cents = (quantity - refunds[line_id]) * unit_cents + by_category[categories[sku]] += cents + by_day[day] += cents + return { + "categories": { + "columns": ["Category", "Revenue"], + "rows": [ + [key, value / 100] + for key, value in sorted(by_category.items(), key=lambda item: (-item[1], item[0])) + ], + }, + "daily": { + "columns": ["Date", "Revenue"], + "rows": [[key, value / 100] for key, value in sorted(by_day.items())], + }, + "total": {"columns": ["Revenue"], "rows": [[sum(by_category.values()) / 100]]}, + } + + +def verify_reference(report: dict, source: Path) -> None: + expected = independent_reference(source) + actual = {t["id"]: {"columns": t["columns"], "rows": t["rows"]} for t in report["tables"]} + # Row/table presentation order is not a business-correctness requirement. + if actual.keys() != expected.keys() or any( + actual[name]["columns"] != table["columns"] or sorted(actual[name]["rows"]) != sorted(table["rows"]) + for name, table in expected.items() + ): + raise ValueError("Report values differ from the independent raw-source reference.") + + +def verify_outputs(directory: Path, source: Path, *, scripted_trace: bool = True) -> dict: + """Check actual files and trace; this does not grade visual aesthetics or model reasoning.""" + from openpyxl import load_workbook + + record = json.loads((directory / "run.json").read_text()) + if record["status"] != "completed": + raise ValueError(f"Specialist did not complete: {record.get('error', record['status'])}") + bundle = directory / "artifacts" / "report" + manifest = verify_bundle(bundle) + report = json.loads((bundle / "report.json").read_text()) + verify_reference(report, source) + if not any( + metric["table"] == "total" and metric["column"] == "Revenue" and metric["row"] == 0 + for metric in report["metrics"] + ): + raise ValueError("The requested total-revenue metric is missing.") + if not {"categories", "daily"}.issubset({chart["table"] for chart in report["charts"]}): + raise ValueError("The requested category and daily charts are missing.") + workbook = load_workbook(bundle / "supporting.xlsx", read_only=True, data_only=True) + try: + for index, table in enumerate(report["tables"], 1): + with (bundle / "tables" / f"{table['id']}.csv").open(newline="") as handle: + csv_rows = list(csv.reader(handle)) + expected_csv = [[str(v) for v in row] for row in [table["columns"], *table["rows"]]] + if csv_rows != expected_csv: + raise ValueError("CSV disagrees with the canonical analytical table.") + sheet = workbook[f"T{index}-{table['id']}"[:31]] + if [list(row) for row in sheet.values] != [table["columns"], *table["rows"]]: + raise ValueError("Workbook disagrees with the canonical analytical table.") + finally: + workbook.close() + messages = record["messages"] + failed_program = any( + m.get("type") == "tool" + and m.get("name") == "run_shell" + and "no such column: amount" in str(m.get("content")) + for m in messages + ) + chart_returned = any( + m.get("type") == "tool" + and m.get("name") == "inspect_chart" + and "data:image/png;base64," in str(m.get("content")) + for m in messages + ) + if scripted_trace and (not failed_program or not chart_returned): + raise ValueError("The full trace did not exercise error recovery and image delivery.") + if scripted_trace: + for message in messages: + if message.get("type") != "tool": + continue + content = message.get("content", "") + if isinstance(content, str) and content.startswith("Error invoking tool"): + raise ValueError("The scripted integration contains an unintended tool-schema error.") + if message.get("name") in {"run_shell", "run_python", "write_analysis_file", "render_report"}: + result = json.loads(content) + if result.get("exit_code") != 0 and "no such column: amount" not in result.get("stderr", ""): + raise ValueError("The scripted integration contains an unintended command error.") + wrong = copy.deepcopy(report) + for table in wrong["tables"]: + if table["id"] in {"categories", "daily"}: + table["rows"][0][1] += 1 + else: + table["rows"][0][0] += 1 + wrong_tables = {table["id"]: table for table in wrong["tables"]} + for metric in wrong["metrics"]: + table = wrong_tables[metric["table"]] + metric["value"] = table["rows"][metric["row"]][table["columns"].index(metric["column"])] + validate_report(wrong) # A consistent shape must not be enough to pass factual grading. + try: + verify_reference(wrong, source) + except ValueError: + negative_rejected = True + else: + raise AssertionError("Independent verifier accepted the deliberately wrong control.") + return { + "independent_numbers": True, + "csv_workbook_consistency": True, + "failed_program_then_recovery": failed_program, + "png_returned_through_tool": chart_returned, + "wrong_result_control_rejected": negative_rejected, + "artifact_count": len(manifest["artifacts"]), + "report": str(bundle / "report.html"), + } + + +async def run_demo(output: Path, *, profile: str = "analysis") -> dict: + output = output.resolve() + output.mkdir(parents=True, exist_ok=False) + source = output / "sources" / "sales.sqlite" + create_source(source) + before = hashlib.sha256(source.read_bytes()).hexdigest() + store = SQLiteStore(source, dataset="analysis-demo") + try: + success = await run_analysis( + brief(), + model=scripted_model(), + model_id="scripted/offline", + store=store, + output=output / "report-run", + profile=profile, + max_model_calls=20, + max_tool_calls=30, + scripted=True, + ) + if success["status"] != "completed": + raise RuntimeError(f"Report run failed: {success.get('error', success['status'])}") + checks = verify_outputs(output / "report-run", source) + refusal = await run_analysis( + brief(profit=True), + model=scripted_model(profit=True), + model_id="scripted/offline", + store=store, + output=output / "missing-cost-run", + profile=profile, + max_model_calls=5, + max_tool_calls=5, + scripted=True, + ) + if refusal["status"] != "cannot_answer" or "cost" not in refusal["completion"]["reason"]: + raise ValueError("Missing-cost scenario did not produce the expected explanatory refusal.") + if hashlib.sha256(source.read_bytes()).hexdigest() != before: + raise ValueError("The read-only source changed during execution.") + summary = { + "execution": "scripted integration; not a model evaluation", + "provider_calls": 0, + "model_cost_usd": 0, + "checks": {**checks, "missing_cost_refusal": True, "source_unchanged": True}, + "resources": success["resources"], + "latency_s": success["latency_s"] + refusal["latency_s"], + "limits": [ + "Scripted decisions do not establish LLM competence.", + "Image delivery was exercised; visual aesthetics require separate inspection.", + "The fixture is small and synthetic, not a large-data benchmark.", + ], + } + (output / "evaluation.json").write_text(json.dumps(summary, indent=2)) + return summary + finally: + store.close() + + +def evaluation_examples(source: Path) -> list[dict]: + """Only inputs reach the analyst; outputs are evaluator-owned reference material.""" + report_brief = brief().model_dump() + report_brief["required_outputs"].append( + "Machine-readable tables: categories with columns Category, Revenue; daily with columns " + "Date, Revenue; total with column Revenue and one row. Revenue values are USD. " + "Use those table IDs and column names; presentation order is your choice." + ) + source_hash = hashlib.sha256(source.read_bytes()).hexdigest() + return [ + { + "inputs": {"case_id": "report", "brief": report_brief}, + "outputs": { + "status": "completed", + "source_sha256": source_hash, + "tables": independent_reference(source), + }, + }, + { + "inputs": {"case_id": "missing-cost", "brief": brief(profit=True).model_dump()}, + "outputs": {"status": "cannot_answer", "source_sha256": source_hash}, + }, + ] + + +def grade_case(case_id: str, record: dict, directory: Path, source: Path, reference: dict) -> list[dict]: + """Deterministic task feedback; failed runs require triage before a model-quality claim.""" + if hashlib.sha256(source.read_bytes()).hexdigest() != reference["source_sha256"]: + return [{"key": "evaluation_valid", "score": 0, "comment": "Assigned source changed."}] + if record.get("status") in {"failed", "cancelled"}: + return [ + {"key": "evaluation_valid", "score": 0, "comment": "Run requires failure classification."}, + {"key": "task_pass", "score": None, "comment": record.get("error", "Run did not finish.")}, + ] + if case_id == "report": + if independent_reference(source) != reference["tables"]: + raise ValueError("Frozen reference does not match the assigned source.") + try: + verify_outputs(directory, source, scripted_trace=False) + passed, comment = True, "Independent numerical reference and exported bundle agree." + except (ValueError, KeyError, OSError) as error: + passed, comment = False, str(error) + else: + reason = record.get("completion", {}).get("reason", "").lower() + passed = record.get("status") == "cannot_answer" and any( + term in reason for term in ("cost", "expense") + ) + comment = "Deterministic missing-cost refusal check; narrative nuance is not graded." + return [ + {"key": "evaluation_valid", "score": 1, "comment": "Run available for deterministic grading."}, + {"key": "task_pass", "score": int(passed), "comment": comment}, + { + "key": "cost_complete", + "score": int(record.get("usage", {}).get("cost_complete", False)), + "comment": "Missing provider costs are unknown, never zero.", + }, + ] + + +def run_live( + output: Path, + *, + model_id: str, + reasoning: str, + max_cost: float, + reservation: float, + profile: str = "analysis", + langsmith: bool = False, + dataset_name: str = "backintel-analysis-artifacts-v1", +) -> dict: + """Two real runs, one repetition, serial, with shared admission and no provider retries.""" + from langsmith import Client, tracing_context + + from backintel.bench.usage import RunBudget + from backintel.models import build_model + + if not model_id.startswith("openrouter/"): + raise ValueError("This live entry requires OpenRouter provider-reported cost accounting.") + budget = RunBudget(limit_usd=max_cost, max_calls=48, reservation_usd=reservation) + if not budget.can_admit(): + raise ValueError("The declared budget cannot admit even one reserved call.") + model = build_model( + model_id, + reasoning_effort=reasoning, + responses_api=True, + first_party_only=True, + max_retries=0, + timeout_s=300, + ) + output = output.resolve() + output.mkdir(parents=True, exist_ok=False) + source = output / "sources" / "sales.sqlite" + create_source(source) + examples = evaluation_examples(source) + by_id = {example["inputs"]["case_id"]: example for example in examples} + scores, records = {}, {} + + def target(inputs: dict) -> dict: + case_id = inputs["case_id"] + if case_id not in by_id or inputs != by_id[case_id]["inputs"]: + raise ValueError("Dataset inputs differ from this frozen two-case task.") + store = SQLiteStore(source, dataset="analysis-live-eval") + try: + record = asyncio.run( + run_analysis( + AnalysisBrief.model_validate(inputs["brief"]), + model=model, + model_id=model_id, + store=store, + output=output / (case_id + "-run"), + profile=profile, + budget=budget, + max_model_calls=24, + max_tool_calls=50, + timeout_s=900, + ) + ) + finally: + store.close() + records[case_id] = record + report_path = output / (case_id + "-run") / "artifacts" / "report" / "report.json" + return { + **{key: value for key, value in record.items() if key != "messages"}, + "report": json.loads(report_path.read_text()) if report_path.exists() else None, + "output_dir": str(output / (case_id + "-run")), + } + + def evaluator(inputs: dict, outputs: dict, reference_outputs: dict) -> list[dict]: + case_id = inputs["case_id"] + scores[case_id] = grade_case(case_id, outputs, output / (case_id + "-run"), source, reference_outputs) + return scores[case_id] + + summary = { + "execution": "live model evaluation", + "model": model_id, + "reasoning": reasoning, + "api": "responses", + "routing": "first party only; no fallbacks", + "cases": 2, + "repetitions": 1, + "max_model_calls_per_case": 24, + "max_tool_calls_per_case": 50, + "timeout_s_per_case": 900, + "source_sha256": hashlib.sha256(source.read_bytes()).hexdigest(), + "reference_sha256": hashlib.sha256( + json.dumps([example["outputs"] for example in examples], sort_keys=True).encode() + ).hexdigest(), + "evaluator_sha256": hashlib.sha256(Path(__file__).read_bytes()).hexdigest(), + "scope": "Initial two-case smoke evaluation, not the planned expanded model comparison.", + } + try: + if langsmith: + client = Client() + if client.has_dataset(dataset_name=dataset_name): + dataset = client.read_dataset(dataset_name=dataset_name) + else: + dataset = client.create_dataset( + dataset_name=dataset_name, + description="BackIntel analysis v1: joined/refunded/duplicate sales and missing costs.", + ) + client.create_examples(dataset_id=dataset.id, examples=examples) + stored = list(client.list_examples(dataset_id=dataset.id)) + frozen = [{"inputs": ex.inputs, "outputs": ex.outputs} for ex in stored] + if sorted(frozen, key=lambda ex: ex["inputs"]["case_id"]) != sorted( + examples, key=lambda ex: ex["inputs"]["case_id"] + ): + raise ValueError( + "Existing LangSmith dataset differs; create a new version, do not overwrite." + ) + results = client.evaluate( + target, + data=stored, + evaluators=[evaluator], + experiment_prefix="backintel-analysis-" + model_id.rsplit("/", 1)[-1], + metadata=summary, + max_concurrency=1, + num_repetitions=1, + blocking=False, + description="Actual specialist graph; independent code grading; no LLM judge.", + ) + summary["langsmith_runs"] = [ + {"case_id": row["example"].inputs["case_id"], "run_id": str(row["run"].id)} for row in results + ] + summary["experiment"] = results.experiment_name + summary["project_url"] = client.read_project(project_name=results.experiment_name).url + else: + with tracing_context(enabled=False): + for example in examples: + result = target(example["inputs"]) + evaluator(example["inputs"], result, example["outputs"]) + finally: + summary["budget"] = budget.report() + summary["scores"] = scores + summary["usage"] = {case: record["usage"] for case, record in records.items()} + (output / "evaluation.json").write_text(json.dumps(summary, indent=2, default=str)) + return summary + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--output", + type=Path, + default=Path("data/output") / ("analysis-demo-" + datetime.now(UTC).strftime("%Y%m%dT%H%M%S%fZ")), + ) + parser.add_argument("--profile", choices=["analysis", "large"], default="analysis") + parser.add_argument("--live", action="store_true", help="run an explicitly qualified live model") + parser.add_argument("--model", help="qualified model/endpoint; no default or policy approval implied") + parser.add_argument( + "--reasoning", choices=["none", "minimal", "low", "medium", "high", "xhigh"], default="medium" + ) + parser.add_argument( + "--max-cost-usd", type=float, help="shared approved admission allowance for both cases" + ) + parser.add_argument( + "--reservation-usd", type=float, help="approved per-call reservation, not a billing cap" + ) + parser.add_argument("--langsmith", action="store_true", help="upload live dataset, traces and experiment") + parser.add_argument("--dataset", default="backintel-analysis-artifacts-v1") + args = parser.parse_args() + if args.live: + if not args.model or args.max_cost_usd is None or args.reservation_usd is None: + parser.error("--live requires --model, --max-cost-usd and --reservation-usd") + result = run_live( + args.output, + model_id=args.model, + reasoning=args.reasoning, + max_cost=args.max_cost_usd, + reservation=args.reservation_usd, + profile=args.profile, + langsmith=args.langsmith, + dataset_name=args.dataset, + ) + else: + if args.langsmith or args.model or args.max_cost_usd is not None or args.reservation_usd is not None: + parser.error("Model, budget and LangSmith options require --live") + result = asyncio.run(run_demo(args.output, profile=args.profile)) + print(json.dumps(result, indent=2, default=str)) + + +if __name__ == "__main__": + main() diff --git a/scripts/verify_runtime_evidence.py b/scripts/verify_runtime_evidence.py index 0cacea1..8efabb2 100644 --- a/scripts/verify_runtime_evidence.py +++ b/scripts/verify_runtime_evidence.py @@ -18,6 +18,7 @@ from pydantic import ValidationError from backintel.answer_validation import answer_schema +from backintel.bench.run import git_sha, source_version, working_tree from backintel.sandbox import PythonSandbox from backintel.settings import Settings, repo_root from backintel.stores import open_store @@ -146,6 +147,10 @@ def main() -> int: sandbox = PythonSandbox(settings.sandbox, timeout_s=30) report = { "created_at": datetime.now(UTC).isoformat(), + "git_sha": git_sha(), + "source_version": source_version(), + "working_tree": working_tree(), + "runner_sha256": hashlib.sha256(Path(__file__).read_bytes()).hexdigest(), "scope": "Runtime evidence execution smoke; not a model quality or full data parity evaluation", "dataset": "olist", "model_calls": 0, diff --git a/tests/test_analysis_cases.py b/tests/test_analysis_cases.py new file mode 100644 index 0000000..23d4ce1 --- /dev/null +++ b/tests/test_analysis_cases.py @@ -0,0 +1,239 @@ +"""Task correctness controls, independent of model calls or Docker availability.""" + +import copy +import json + +import pytest + +from backintel.artifacts import validate_report +from scripts.analysis_cases import ( + CASE_IDS, + build_cases, + reference_queries, + scripted_spec, + verify_case_completion, + verify_case_report, +) + + +@pytest.fixture +def cases(tmp_path): + return {case.case_id: case for case in build_cases(tmp_path / "cases")} + + +def test_frozen_cases_are_reproducible_and_keep_references_out_of_runtime(tmp_path): + first, second = (build_cases(tmp_path / name) for name in ("first", "second")) + assert tuple(case.case_id for case in first) == CASE_IDS + assert len({case.source_sha256 for case in first}) == 3 + assert json.loads((tmp_path / "first" / "manifest.json").read_text()) == json.loads( + (tmp_path / "second" / "manifest.json").read_text() + ) + for case, repeat in zip(first, second, strict=True): + assert case.source_sha256 == repeat.source_sha256 + assert case.reference_sha256 == repeat.reference_sha256 + assert case.reference.parent.name == "private-references" + assert case.reference.stat().st_mode & 0o777 == 0o600 + assert "private-references" not in case.brief.model_dump_json() + assert "SELECT " not in case.brief.model_dump_json() + store = case.open_store() + try: + assert store.python_mounts() == {str(case.source): str(case.source)} + assert str(case.reference) not in store.python_environment().values() + finally: + store.close() + + +@pytest.mark.parametrize("case_id", [case for case in CASE_IDS if case != "missing-cost"]) +def test_executable_reference_solutions_match_raw_source_oracles(cases, case_id): + case = cases[case_id] + spec = scripted_spec(case) + verify_case_report(case, validate_report(spec)) + verify_case_completion(case, {"status": "completed"}, spec) + assert {table["id"] for table in spec["tables"]} == set(reference_queries(case_id)) + # The declaration describes the integration honestly; it is not a model success record. + assert "does not measure model reasoning" in spec["summary"] + + +def test_equivalent_ordering_and_numeric_roundoff_are_accepted(cases): + case = cases["irregular-energy"] + spec = scripted_spec(case) + for table in spec["tables"]: + table["columns"].reverse() + table["rows"] = [list(reversed(row)) for row in reversed(table["rows"])] + spec["tables"].reverse() + total = next(table for table in spec["tables"] if table["id"] == "energy_total") + energy_index = total["columns"].index("Energy") + total["rows"][0][energy_index] += 1e-10 + verify_case_report(case, validate_report(spec)) + + +def test_alternate_minute_grid_calculation_is_accepted(cases): + """Fixture endpoints are whole minutes, so an explicit minute grid is also exact. + + The shipped solution integrates LEAD intervals. This independently written query + samples the active observation at each minute and groups those valid minutes. + """ + case = cases["irregular-energy"] + query = """ + WITH RECURSIVE minute(n) AS (SELECT 0 UNION ALL SELECT n+1 FROM minute WHERE n<59), + grid AS ( + SELECT meter_id, window_start_utc, + CAST(strftime('%s',window_start_utc) AS INTEGER)+60*n AS instant + FROM meters CROSS JOIN windows CROSS JOIN minute + ), samples AS ( + SELECT g.*, r.power, r.unit, r.quality, + CAST(strftime('%s',r.observed_at) AS INTEGER) AS observed + FROM grid g LEFT JOIN readings r ON r.meter_id=g.meter_id + AND CAST(strftime('%s',r.observed_at) AS INTEGER)=( + SELECT MAX(CAST(strftime('%s',prior.observed_at) AS INTEGER)) + FROM readings prior WHERE prior.meter_id=g.meter_id + AND CAST(strftime('%s',prior.observed_at) AS INTEGER)<=g.instant + ) + ), valid AS ( + SELECT *, quality='good' AND power IS NOT NULL AND instant-observed<900 AS usable + FROM samples + ) + SELECT meter_id AS Meter,window_start_utc AS WindowStart, + SUM(CASE WHEN usable THEN power*CASE unit WHEN 'kW' THEN 1000 ELSE 1 END/60000 END) + AS Energy, + SUM(CASE WHEN usable THEN 1 ELSE 0 END) AS CoveredMinutes, + SUM(CASE WHEN usable THEN 1 ELSE 0 END)*100.0/60 AS CoveragePercent + FROM valid GROUP BY meter_id,window_start_utc ORDER BY window_start_utc DESC,meter_id DESC + """ + store = case.open_store() + try: + result = store.query(query) + finally: + store.close() + spec = scripted_spec(case) + hourly, total = spec["tables"] + hourly.update(columns=result.columns, rows=result.rows) + energy = sum(row[2] for row in result.rows if row[2] is not None) + minutes = sum(row[3] for row in result.rows) + total["rows"] = [[energy, minutes, 540, 100 * minutes / 540]] + assert energy == pytest.approx(2.1791666666666667) + assert minutes == 127 + verify_case_report(case, validate_report(spec)) + + +def test_plausible_wrong_business_calculations_fail_despite_valid_artifact_shape(cases): + # Joining raw refund and duplicate-line rows inflates delivered net revenue. + case = cases["sales-refunds"] + store = case.open_store() + try: + naive = store.query(""" + SELECT SUM((l.quantity-COALESCE(r.quantity,0))*l.unit_cents)/100.0 AS Revenue + FROM lines l JOIN orders o ON o.order_id=l.order_id + LEFT JOIN refunds r ON r.line_id=l.line_id WHERE o.status='delivered' + """) + finally: + store.close() + spec = scripted_spec(case) + next(t for t in spec["tables"] if t["id"] == "total")["rows"] = naive.rows + assert naive.rows == [[190.0]] + with pytest.raises(ValueError, match="independent raw-source reference"): + verify_case_report(case, validate_report(spec)) + + # Missing coverage cannot be silently rendered as measured zero consumption. + case = cases["irregular-energy"] + spec = scripted_spec(case) + for row in spec["tables"][0]["rows"]: + if row[2] is None: + row[2] = 0.0 + with pytest.raises(ValueError, match="independent raw-source reference"): + verify_case_report(case, validate_report(spec)) + + # Mean service percentages ignores unequal eligible exposure and customer weights. + case = cases["service-availability"] + spec = scripted_spec(case) + services, fleet = spec["tables"] + percentages = [row[3] for row in services["rows"] if row[3] is not None] + fleet["rows"][0][2] = sum(percentages) / len(percentages) + with pytest.raises(ValueError, match="independent raw-source reference"): + verify_case_report(case, validate_report(spec)) + + +def test_units_population_and_chart_binding_are_required(cases): + case = cases["service-availability"] + correct = scripted_spec(case) + mutations = [] + wrong_units = copy.deepcopy(correct) + wrong_units["tables"][0]["units"]["DowntimeMinutes"] = "hours" + mutations.append(wrong_units) + missing_empty_service = copy.deepcopy(correct) + missing_empty_service["tables"][0]["rows"] = [ + row for row in missing_empty_service["tables"][0]["rows"] if row[0] != "chat" + ] + mutations.append(missing_empty_service) + wrong_chart = copy.deepcopy(correct) + wrong_chart["charts"][0]["y"] = "EligibleMinutes" + mutations.append(wrong_chart) + for report in mutations: + with pytest.raises(ValueError): + verify_case_report(case, report) + + +def test_missing_cost_requires_explanatory_refusal_without_substitute_report(cases): + case = cases["missing-cost"] + completion = { + "status": "cannot_answer", + "reason": "Product costs are missing; they are required for profit.", + } + verify_case_completion(case, completion) + assert reference_queries(case.case_id) == {} + for invalid in ( + {"status": "completed", "reason": "Profit is revenue."}, + {"status": "cannot_answer", "reason": "I cannot answer."}, + {"status": "cannot_answer", "reason": "Costs are present."}, + {"status": "cannot_answer", "reason": "Costs are not missing."}, + {"status": "cannot_answer", "reason": "There is no cost issue."}, + {"status": "cannot_answer", "reason": "There is no issue with costs."}, + {"status": "cannot_answer", "reason": "Costs are missing. Costs are present."}, + {"status": "cannot_answer", "reason": "Costs aren't unavailable."}, + {"status": "cannot_answer", "reason": "It is not true that costs are missing."}, + {"status": "cannot_answer", "reason": "Costs are required to calculate profit."}, + ): + with pytest.raises(ValueError): + verify_case_completion(case, invalid) + with pytest.raises(ValueError, match="substitute"): + verify_case_completion(case, completion, scripted_spec(cases["sales-refunds"])) + + +@pytest.mark.parametrize( + "reason", + [ + "Product costs and operating expenses are unavailable.", + "The source lacks product costs and expense data.", + "No cost data is supplied.", + "Missing product cost inputs prevent calculating profit.", + "Costs are not provided in this source.", + "Required profit inputs are absent: cost of goods sold and applicable expenses.", + ], +) +def test_explicit_missing_cost_statements_are_accepted(cases, reason): + verify_case_completion(cases["missing-cost"], {"status": "cannot_answer", "reason": reason}) + + +@pytest.mark.parametrize("metric_id", ["arbitrary", "revenue", None]) +def test_correct_cell_with_wrong_metric_identity_is_rejected(cases, metric_id): + case = cases["irregular-energy"] + report = scripted_spec(case) + report["metrics"][0]["id"] = metric_id + with pytest.raises(ValueError, match="summary metric"): + verify_case_report(case, report) + + +@pytest.mark.parametrize("target", ["source", "reference"]) +def test_changed_inputs_invalidate_refusal_and_success(cases, target): + case = cases["sales-refunds"] + report = scripted_spec(case) + path = getattr(case, target) + path.chmod(0o600) + path.write_bytes(path.read_bytes() + b" ") + with pytest.raises(ValueError, match="changed after case preparation"): + verify_case_report(case, report) + if target == "source": + with pytest.raises(ValueError, match="changed after case preparation"): + verify_case_completion( + cases["missing-cost"], {"status": "cannot_answer", "reason": "Costs are absent."} + ) diff --git a/tests/test_analysis_comparison.py b/tests/test_analysis_comparison.py new file mode 100644 index 0000000..46547c9 --- /dev/null +++ b/tests/test_analysis_comparison.py @@ -0,0 +1,338 @@ +"""Comparison integrity and stop behavior, without model-provider requests.""" + +import json +from pathlib import Path + +import pytest + +from scripts.run_analysis_comparison import run_suite, summarize_arm + + +@pytest.mark.parametrize("model", [None, "chatgpt-plan/test", "openrouter/openai/test"]) +@pytest.mark.parametrize("all_tasks_passed,eligible", [(True, True), (True, False), (False, False)]) +def test_cli_success_requires_live_review_eligibility(monkeypatch, model, all_tasks_passed, eligible): + import scripts.run_analysis_comparison as suite + + monkeypatch.setattr("sys.argv", ["suite", *(["--model", model] if model else [])]) + monkeypatch.setattr( + suite, + "run_suite", + lambda *args, **kwargs: { + "arms": [ + {"summary": {"all_tasks_passed": all_tasks_passed, "eligible_for_live_review": eligible}} + ] + }, + ) + if eligible if model else all_tasks_passed: + suite.main() + else: + with pytest.raises(SystemExit) as error: + suite.main() + assert error.value.code == 1 + + +def test_selected_case_keeps_frozen_suite_and_limits_subscription_calls(tmp_path, monkeypatch): + import scripts.run_analysis_comparison as suite + + monkeypatch.setattr(suite, "build_model", lambda *args, **kwargs: object()) + seen = [] + + async def run(brief, **kwargs): + seen.append(kwargs) + return {"status": "failed", "error": "Controlled stop before provider call."} + + monkeypatch.setattr(suite, "run_analysis", run) + result = run_suite(tmp_path / "selected", model_ids=["chatgpt-plan/test"], case_ids=["irregular-energy"]) + assert result["case_ids"] == ["irregular-energy"] + assert len(seen) == 1 + assert seen[0]["budget"].max_calls == 24 + assert result["arms"][0]["summary"]["case_count"] == 1 + # SQLite file bytes include engine-version metadata. Freeze the full suite for this + # environment rather than pinning one host's file encoding across platforms. + full = run_suite(tmp_path / "full", model_ids=["chatgpt-plan/test"]) + assert full["case_ids"] == list(suite.CASE_IDS) + assert result["dataset_sha256"] == full["dataset_sha256"] + + +def test_unresolved_cases_and_unknown_costs_never_qualify_an_arm(): + records = {"a": {"status": "completed", "usage": {"cost_usd": 0.01, "cost_complete": True}}} + scores = {"a": [{"key": "task_pass", "score": 1}]} + result = summarize_arm(["a", "b"], records, scores, scripted=False) + assert result["unresolved"] == ["b"] + assert not result["eligible_for_live_review"] + records["b"] = {"status": "completed", "usage": {"cost_usd": None, "cost_complete": False}} + scores["b"] = [{"key": "task_pass", "score": 1}] + result = summarize_arm(["a", "b"], records, scores, scripted=False) + assert result["all_tasks_passed"] + assert result["provider_cost_usd"] is None + assert result["known_reported_cost_usd"] == 0.01 + assert not result["cost_complete"] + assert not result["eligible_for_live_review"] + assert not summarize_arm(["a", "b"], records, scores, scripted=True)["eligible_for_live_review"] + + +def test_plan_quality_review_requires_complete_tokens_without_invented_api_costs(): + records = { + "a": { + "status": "completed", + "usage": { + "billing_mode": "chatgpt_plan", + "usage_complete": True, + "cost_usd": None, + "cost_complete": False, + "cost_applicable": False, + "input_tokens": 100, + "output_tokens": 20, + "llm_calls": 1, + }, + } + } + scores = {"a": [{"key": "task_pass", "score": 1}]} + result = summarize_arm(["a"], records, scores, scripted=False, billing_mode="chatgpt_plan") + assert result["eligible_for_live_review"] + assert result["provider_cost_usd"] is None + assert result["known_reported_cost_usd"] is None + assert result["billing_mode"] == "chatgpt_plan" + assert not result["cost_applicable"] + assert not result["cost_complete"] + assert result["input_tokens"] == 100 + assert result["output_tokens"] == 20 + assert result["provider_calls"] == 1 + assert "no automatic model promotion" in result["scope"] + records["a"]["usage"]["usage_complete"] = False + assert not summarize_arm(["a"], records, scores, scripted=False, billing_mode="chatgpt_plan")[ + "eligible_for_live_review" + ] + records["a"]["usage"]["usage_complete"] = True + assert not summarize_arm(["a", "b"], records, scores, scripted=False, billing_mode="chatgpt_plan")[ + "eligible_for_live_review" + ] + scores["a"][0]["score"] = 0 + assert not summarize_arm(["a"], records, scores, scripted=False, billing_mode="chatgpt_plan")[ + "eligible_for_live_review" + ] + + +def test_plan_feedback_marks_cost_not_applicable(tmp_path): + from scripts.analysis_cases import build_cases + from scripts.run_analysis_comparison import grade + + case = build_cases(tmp_path / "cases")[1] + record = { + "status": "cannot_answer", + "completion": {"status": "cannot_answer", "reason": "Product costs are absent."}, + "usage": {"billing_mode": "chatgpt_plan", "usage_complete": True, "cost_complete": False}, + } + feedback = {item["key"]: item for item in grade(case, record, tmp_path)} + assert feedback["task_pass"]["score"] == 1 + assert feedback["cost_complete"]["score"] is None + assert feedback["usage_complete"]["score"] == 1 + + +def test_workbook_roundoff_is_accepted_but_changed_numbers_are_rejected(tmp_path, monkeypatch): + from openpyxl import Workbook, load_workbook + + import scripts.run_analysis_comparison as suite + from scripts.analysis_cases import build_cases, scripted_spec + + case = build_cases(tmp_path / "fixtures")[2] + report = scripted_spec(case) + output = tmp_path / "run" + bundle = output / "artifacts" / "report" + bundle.mkdir(parents=True) + (bundle / "report.json").write_text(json.dumps(report)) + workbook = Workbook() + workbook.remove(workbook.active) + for index, table in enumerate(report["tables"], 1): + sheet = workbook.create_sheet(f"T{index}-{table['id']}"[:31]) + for row in [table["columns"], *table["rows"]]: + sheet.append(row) + workbook.save(bundle / "supporting.xlsx") + workbook.close() + # Bundle integrity has separate renderer/integration checks; this exercises the grader + # against a real XLSX serialization and the case's independent numerical reference. + monkeypatch.setattr(suite, "verify_bundle", lambda path: {}) + record = {"status": "completed", "completion": {"status": "completed"}, "usage": {}} + assert suite.grade(case, record, output)[1]["score"] == 1 + workbook = load_workbook(bundle / "supporting.xlsx") + workbook["T1-energy_by_window"]["C2"] = 999 + workbook.save(bundle / "supporting.xlsx") + workbook.close() + assert suite.grade(case, record, output)[1]["score"] == 0 + + +def test_two_arms_share_budget_and_do_not_receive_private_references(tmp_path, monkeypatch): + import scripts.run_analysis_comparison as suite + + monkeypatch.setattr(suite, "build_model", lambda *args, **kwargs: object()) + seen = [] + + async def run(brief, **kwargs): + assert "reference" not in kwargs + assert "private-references" not in brief.model_dump_json() + assert all("private-references" not in path for path in kwargs["store"].python_mounts()) + seen.append((kwargs["model_id"], kwargs["budget"])) + kwargs["budget"].reserve() + kwargs["budget"].settle(0.1) + kwargs["output"].mkdir() + record = { + "status": "failed", + "error": "Controlled infrastructure failure", + "usage": {"cost_usd": 0.1, "cost_complete": True, "llm_calls": 1}, + } + (kwargs["output"] / "run.json").write_text(json.dumps(record)) + return record + + monkeypatch.setattr(suite, "run_analysis", run) + result = run_suite( + tmp_path / "comparison", + model_ids=["openrouter/openai/not-invoked-a", "openrouter/openai/not-invoked-b"], + max_cost=0.5, + reservation=0.1, + ) + assert len(seen) == 5 + assert len({id(budget) for _, budget in seen}) == 1 + assert seen[-1][0].endswith("-b") + assert result["budget"]["reported_spent_usd"] == 0.5 + assert len(result["arms"]) == 2 + assert all(len(arm["summary"]["unresolved"]) == 4 for arm in result["arms"]) + assert not any(arm["summary"]["eligible_for_live_review"] for arm in result["arms"]) + assert json.loads((tmp_path / "comparison" / "comparison.json").read_text())["budget"]["calls"] == 5 + assert ( + json.loads((tmp_path / "comparison" / "arm-1" / "service-availability" / "run.json").read_text())[ + "status" + ] + == "not_run" + ) + + +def test_subscription_arms_share_call_limit_without_paid_api_routing(tmp_path, monkeypatch): + import scripts.run_analysis_comparison as suite + + model_options, seen = [], [] + + def model(model_id, **kwargs): + model_options.append((model_id, kwargs)) + return object() + + async def run(brief, **kwargs): + assert "private-references" not in brief.model_dump_json() + assert all("private-references" not in path for path in kwargs["store"].python_mounts()) + assert kwargs["max_model_calls"] == 24 + assert kwargs["max_tool_calls"] == 50 + assert kwargs["timeout_s"] == 900 + budget = kwargs["budget"] + seen.append(budget) + # Use this case's full allowance to exercise the shared campaign limit. + for _ in range(kwargs["max_model_calls"]): + budget.reserve() + budget.settle({"prompt_tokens": 4, "completion_tokens": 1}) + kwargs["output"].mkdir() + record = { + "status": "completed", + "usage": { + "billing_mode": "chatgpt_plan", + "cost_usd": None, + "cost_complete": False, + "cost_applicable": False, + "usage_complete": True, + "llm_calls": 24, + "input_tokens": 96, + "output_tokens": 24, + }, + } + (kwargs["output"] / "run.json").write_text(json.dumps(record)) + return record + + monkeypatch.setattr(suite, "build_model", model) + monkeypatch.setattr(suite, "run_analysis", run) + # Numerical grading is covered separately. This test exercises selection/admission/accounting. + monkeypatch.setattr(suite, "grade", lambda *args: [{"key": "task_pass", "score": 1}]) + result = run_suite( + tmp_path / "comparison", + model_ids=["chatgpt-plan/not-invoked-a", "chatgpt-plan/not-invoked-b"], + ) + assert len(seen) == 8 + assert len({id(budget) for budget in seen}) == 1 + assert seen[0].max_calls == 192 + assert not seen[0].can_admit() + assert result["budget"]["calls"] == 192 + assert result["billing_mode"] == "chatgpt_plan" + assert result["provider_fallbacks"] is False + for model_id, options in model_options: + assert model_id.startswith("chatgpt-plan/") + assert options["first_party_only"] is False + assert options["max_retries"] == 0 + assert options["timeout_s"] == 300 + for arm in result["arms"]: + assert arm["summary"]["eligible_for_live_review"] + assert arm["summary"]["provider_cost_usd"] is None + assert arm["summary"]["known_reported_cost_usd"] is None + + +def test_unreported_plan_request_stops_all_remaining_cases_without_fallback(tmp_path, monkeypatch): + import scripts.run_analysis_comparison as suite + + models, calls = [], [] + + def model(model_id, **kwargs): + models.append(model_id) + return object() + + async def run(brief, **kwargs): + calls.append(kwargs["model_id"]) + kwargs["budget"].reserve() + kwargs["budget"].settle(None) + kwargs["output"].mkdir() + record = { + "status": "failed", + "error": "Controlled missing provider usage", + "usage": { + "billing_mode": "chatgpt_plan", + "usage_complete": False, + "cost_usd": None, + "cost_complete": False, + "llm_calls": 1, + }, + } + (kwargs["output"] / "run.json").write_text(json.dumps(record)) + return record + + monkeypatch.setattr(suite, "build_model", model) + monkeypatch.setattr(suite, "run_analysis", run) + result = run_suite( + tmp_path / "comparison", + model_ids=["chatgpt-plan/not-invoked-a", "chatgpt-plan/not-invoked-b"], + ) + assert calls == ["chatgpt-plan/not-invoked-a"] + assert all(model_id.startswith("chatgpt-plan/") for model_id in models) + assert result["budget"]["calls"] == 1 + statuses = [] + for arm in result["arms"]: + assert len(arm["summary"]["unresolved"]) == 4 + assert not arm["summary"]["eligible_for_live_review"] + assert arm["summary"]["provider_cost_usd"] is None + statuses.extend(json.loads(Path(path).read_text())["status"] for path in arm["records"].values()) + assert statuses.count("failed") == 1 + assert statuses.count("not_run") == 7 + + +@pytest.mark.parametrize( + "options", + [ + {"langsmith": True}, + {"max_cost": 1}, + {"model_ids": ["openrouter/openai/not-invoked"]}, + {"model_ids": ["openrouter/openai/a"], "max_cost": 0.1, "reservation": 1}, + {"model_ids": ["chatgpt-plan/a", "openrouter/openai/b"]}, + {"model_ids": ["chatgpt-plan/a"], "max_cost": 1}, + {"model_ids": ["chatgpt-plan/a"], "reservation": 0.1}, + {"model_ids": ["chatgpt-plan/a", "chatgpt-plan/a"]}, + {"model_ids": ["chatgpt-plan/"]}, + {"model_ids": ["openai/a"]}, + ], +) +def test_invalid_execution_options_fail_before_creating_outputs(tmp_path, options): + with pytest.raises(ValueError): + run_suite(tmp_path / "never-created", **options) + assert not (tmp_path / "never-created").exists() diff --git a/tests/test_analysis_workflow.py b/tests/test_analysis_workflow.py new file mode 100644 index 0000000..3dc49d0 --- /dev/null +++ b/tests/test_analysis_workflow.py @@ -0,0 +1,132 @@ +"""Independent analytical oracle checks; real graph/SmolVM run is explicitly opt-in.""" + +import asyncio +import copy +import os +import sqlite3 + +import pytest + +from scripts.run_analysis_demo import ( + ANALYSIS_QUERIES, + create_source, + independent_reference, + run_demo, + run_live, + verify_reference, +) + + +def test_independent_reference_catches_duplicate_refund_join_and_wrong_total(tmp_path): + source = tmp_path / "source.sqlite" + create_source(source) + expected = independent_reference(source) + assert expected["total"]["rows"] == [[135.0]] + assert expected["categories"]["rows"] == [["Home", 75.0], ["Outdoor", 40.0], ["Essentials", 20.0]] + with sqlite3.connect(source) as connection: + tables = [] + for name, query in ANALYSIS_QUERIES.items(): + cursor = connection.execute(query) + tables.append( + { + "id": name, + "columns": [c[0] for c in cursor.description], + "rows": [list(row) for row in cursor.fetchall()], + } + ) + verify_reference({"tables": tables}, source) + reordered = copy.deepcopy(tables) + reordered.reverse() + for table in reordered: + table["rows"].reverse() + verify_reference({"tables": reordered}, source) + wrong = copy.deepcopy(tables) + wrong[0]["rows"][0][1] += 1 + with pytest.raises(ValueError, match="independent"): + verify_reference({"tables": wrong}, source) + + +def test_failed_report_revision_cannot_publish_previous_success(tmp_path): + from types import SimpleNamespace + + from backintel.analysis import build_analysis_tools + from backintel.stores.sqlite import SQLiteStore + + source = tmp_path / "source.sqlite" + create_source(source) + retained = {"spec": {"title": "old successful report"}} + sandbox = SimpleNamespace(env={}, read_file=lambda _: "not valid JSON") + store = SQLiteStore(source, dataset="revision-check") + try: + tool = next( + t for t in build_analysis_tools(store, sandbox, {}, retained) if t.name == "render_report" + ) + result = tool.invoke({"spec_path": "/workspace/spec.json"}) + assert "Expecting value" in result + assert retained == {} + finally: + store.close() + + +def test_live_entry_keeps_references_outside_target_and_failed_runs_ungraded(tmp_path, monkeypatch): + import backintel.models + import scripts.run_analysis_demo as demo + + captured = [] + monkeypatch.setattr(backintel.models, "build_model", lambda *args, **kwargs: object()) + + async def failed_run(analysis_brief, **kwargs): + captured.append((analysis_brief.model_dump(), kwargs)) + return {"status": "failed", "error": "SmolVM unavailable", "usage": {"cost_complete": False}} + + monkeypatch.setattr(demo, "run_analysis", failed_run) + result = run_live( + tmp_path / "live", + model_id="openrouter/openai/test-model-not-invoked", + reasoning="medium", + max_cost=1, + reservation=0.1, + ) + assert len(captured) == 2 + assert captured[0][1]["budget"] is captured[1][1]["budget"] + assert captured[0][1]["budget"].max_calls == 48 + assert all("135" not in str(analysis_brief) for analysis_brief, _ in captured) + assert all("scripted" not in kwargs for _, kwargs in captured) + assert all(scores[1]["score"] is None for scores in result["scores"].values()) + assert result["budget"]["calls"] == 0 + + +@pytest.mark.parametrize("limit,reservation", [(float("nan"), 0.1), (1, float("inf")), (0.1, 1)]) +def test_live_entry_rejects_invalid_allowance_before_building_a_provider( + tmp_path, monkeypatch, limit, reservation +): + import backintel.models + + def forbidden(*args, **kwargs): + raise AssertionError("Provider construction must not precede budget validation") + + monkeypatch.setattr(backintel.models, "build_model", forbidden) + with pytest.raises(ValueError): + run_live( + tmp_path / "live", + model_id="openrouter/openai/test-model-not-invoked", + reasoning="medium", + max_cost=limit, + reservation=reservation, + ) + + +@pytest.mark.skipif( + os.environ.get("BACKINTEL_SMOLVM_TESTS") != "1", + reason="Set BACKINTEL_SMOLVM_TESTS=1 for the real SmolVM/Deep Agents integration.", +) +def test_full_scripted_graph_creates_verified_artifacts_and_refuses_missing_cost(tmp_path): + result = asyncio.run(run_demo(tmp_path / "demo")) + assert result["provider_calls"] == 0 + assert result["model_cost_usd"] == 0 + assert result["checks"]["independent_numbers"] + assert result["checks"]["csv_workbook_consistency"] + assert result["checks"]["failed_program_then_recovery"] + assert result["checks"]["wrong_result_control_rejected"] + assert result["checks"]["missing_cost_refusal"] + assert result["checks"]["source_unchanged"]