Clarify specialist skill access and refusal delivery

README.md

@@ -225,12 +225,17 @@ These are versioned development observations, not a production accuracy guarante
 | Historical TPC-H SF1 | 6/6 | Generated analytical workload; not real 10 GiB data or an official performance result |
 | Controlled Sol specialist on SmolVM | 3/4 | Retail refunds, missing-cost refusal, energy coverage and service availability; one trial each |
 | Energy after completion recovery fix | 1/1 | Targeted rerun delivered the correct report, heatmap, CSVs and workbook; other cases were not rerun on that candidate |
+| Fresh Sol guidance comparison on SmolVM, 2026-11-13 | Baseline 4/4; candidate 4/4 | Eight new trials; candidate removed virtual-skill shell lookup mistakes in all four cases, with three verified bundles per arm |
 
 The specialist cases use **three small synthetic SQLite fixtures**, with sales reused for
 the missing-cost question. The energy failure had correct intermediate numbers but no
 structured completion or exported bundle. The fix's live rerun completed directly without
 needing recovery; a deterministic SmolVM regression separately exercised recovery and export.
 The 3/4 baseline remains unchanged: a targeted success is not a new 4/4 experiment.
+The subsequent [fresh paired comparison](docs/results-analysis-sandbox.md#fresh-sol-guidance-comparison--2026-11-13-utc)
+tested every case on both versions. Both also pass the corrected metric/refusal checks;
+the candidate used 34 model calls versus 41. This supports the specific tool-guidance fix,
+not a general reliability rate or a faster-runtime claim.
 
 Historical results and failures are in the [capability report](docs/results.md) and
 [scorecard](results/scorecard.md). The [SmolVM baseline](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/datasets/d2e934c2-77e8-4926-93b3-0bcd644b200d/compare?selectedSessions=f667f6fc-4e8d-4bba-bcbf-fcde7d808d40)

docs/results-analysis-sandbox.md

@@ -1,5 +1,70 @@
 # Analysis sandbox implementation results
 
+## Fresh Sol guidance comparison — 2026-11-13 UTC
+
+Both versions passed **4/4** fresh live trials on SmolVM: sales/refunds, missing-cost refusal,
+irregular energy and service availability. Each arm exported three independently checked
+report/chart/CSV/XLSX bundles and one structured refusal. Both also passed the separately
+versioned evaluator correction described below. The earlier 3/4 run and selected energy
+rerun remain separate historical measurements.
+
+The candidate clarifies which tools read virtual skills versus VM files, requires a useful
+structured refusal when essential inputs are absent, and keeps temporary delivery claims
+out of completion summaries. This changes guidance and tool descriptions; source access,
+sandbox permissions, model, reasoning and output enforcement are unchanged.
+
+| Arm | Frozen commit | Method / source | Calls | Input / output tokens | Summed trial latency |
+| --- | --- | --- | ---: | ---: | ---: |
+| [Baseline](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/32ce081a-bcf3-4d2a-b7a3-f044e9c9e204) | `9b75233` | `a544130f2127` / `e903bc5341eb19cd` | 41 | 401,583 / 10,325 | 372.663 s |
+| [Candidate](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/0def1a04-7d51-4ba5-a29f-803789771039) | `3d0bb71` | `a2f72c30c929` / `5e52cbd9f01eaf26` | 34 | 387,065 / 11,650 | 408.103 s |
+
+Trace findings against the predeclared behavioral checks:
+
+- All four baseline cases first attempted to read three `/skills/` files with `run_shell`,
+  received missing-file errors, and then recovered with instruction `read_file`. All four
+  candidate cases used the correct instruction tools immediately, with no shell skill probes.
+- Both missing-cost trials refused without rendering or exporting a substitute report, and
+  neither claimed delivery. This pair does not demonstrate a refusal-quality improvement.
+- All eight trials returned structured completion directly, with zero recovery prompts.
+  The candidate's energy run corrected one workbook-check assertion after discovering the
+  renderer prefixes sheet names; the final workbook passed the independent host grader.
+- Candidate completion summaries report supported analytical findings. Clean host export,
+  manifest verification, independent numerical/unit/metric checks and workbook reconciliation
+  passed for each delivered bundle.
+
+The plan was frozen before inference: one trial per case per arm, serial execution,
+`chatgpt-plan/gpt-6.1-sol`, medium reasoning, 24 model calls / 50 tool calls / 900 seconds per
+case, 192 model calls total, no provider fallback, trial retry or LLM judge. The run consumed
+75 calls, 788,648 input tokens and 21,975 output tokens, with complete subscription usage.
+Per-request dollar charges are not reported by this route; this is not a claim of free
+inference. No paid API calls or credit purchases occurred. SmolVM 1.31.1 and the prepared
+image/archive identities match the migration receipt. Local verification work overlapped
+the experiment, so elapsed times are descriptive and are not a controlled speed comparison.
+
+The original dataset identity is
+`7294f8c0ca8ea4ef54e1667535e845a4d1cbd8591b82b5d235583cf9ff006615`.
+During execution, PR review found grader gaps in metric identity, negated missing-input
+explanations and live CLI eligibility. Runtime code stayed frozen. After both arms ended,
+their saved outputs were checked with the same corrected evaluator at `fd99f07`; source,
+brief, reference and tolerance identities matched. No new inference or threshold tuning was
+used for that regrade. Both remained 4/4. Original `task_pass` feedback stays untouched;
+`task_pass_evaluator_v2` and `evaluation_valid_evaluator_v2` record the correction. Separate
+`skill_access_correct` and `refusal_scope_clean` feedback expose the behavioral comparison.
+
+Corrected case-module SHA-256:
+`bed630e04f7b7769f2b252eeaf256e94477b893adf05df898422047bcc0b32f6`.
+The runner hash and all original identities are retained in `review-regrade.json`.
+LangSmith read-back verified all eight stored run identities/statuses/completions/usage and
+all 26 added feedback records. These are agent-reviewed results, not human calibration.
+
+Evidence: `data/output/specialist-guidance-pair-20261113T044726Z/` contains the frozen
+`plan.json`, both original `comparison.json` files, full trajectories and bundles,
+`review-regrade.json` and `langsmith-verification.json`. The prompt candidate is
+[PR #7](https://github.com/IntelIP/BackIntel/pull/7); the evaluator package is
+[PR #6](https://github.com/IntelIP/BackIntel/pull/6). This supports the specific guidance
+change. Unseen domains, real large data, the 92-response analyst campaign and release
+acceptance remain open.
+
 ## Reproducible evaluation package — 2026-11-13 UTC
 
 The expanded four-case runner, source builders, independent graders and original two-case

docs/roadmap.md

@@ -20,10 +20,13 @@ science covers descriptive analysis and calculation checks; forecasting, causal
 and general statistical-model validity remain unqualified. Native MongoDB deployments and
 vector databases are not established by the FerretDB benchmark.
 
-The controlled SmolVM specialist baseline passed 3/4 cases. The failed energy case now has
-one successful candidate rerun with verified artifact delivery, while the original failure
-remains retained. That is a targeted fix, not a new four-case candidate score or a reliability
-rate. See [current capabilities and measured outcomes](../README.md#what-has-been-measured).
+The original controlled SmolVM specialist baseline passed 3/4 cases, followed by a successful
+targeted energy fix. A fresh eight-trial guidance comparison on 2026-11-13 passed 4/4 on both
+versions, including the corrected grader. The candidate eliminated incorrect virtual-skill
+shell lookups in all four cases, with no measured task regression. Both arms refused missing
+costs without rendering a substitute report. These are small controlled trials; the broader
+92-response reliability comparison remains required. See
+[current capabilities and measured outcomes](../README.md#what-has-been-measured).
 
 ## Next achievable increments
 

evals/analysis/cases.md

@@ -5,6 +5,11 @@ The four-case live comparison completed on 2026-11-12 UTC; see
 [the measured results and limitations](../../docs/results-analysis-sandbox.md#live-subscription-comparison--completed-2026-11-12-utc).
 This specification does not establish broader reliability or release acceptance.
 
+A [fresh baseline/candidate guidance comparison](../../docs/results-analysis-sandbox.md#fresh-sol-guidance-comparison--2026-11-13-utc)
+on 2026-11-13 completed eight trials. Both arms passed 4/4 under the original and separately
+versioned corrected grader. The candidate eliminated virtual-skill shell probes; refusal
+behavior passed on both arms. Original feedback and the earlier failed trial are preserved.
+
 The 2026-11-12 [Sol rerun on SmolVM](../../docs/results-analysis-sandbox.md#live-sol-on-smolvm--2026-11-12-utc)
 retained these same inputs, reference answers and acceptance rules. It passed 3/4 after
 trace classification: the energy trial returned prose without structured completion, so

src/backintel/agent_files/analysis.md

@@ -22,13 +22,24 @@ Read the `analysis-specialist` skill and relevant store skill before analytical
 `/docs/artifact-contract.md` before constructing a report. Skills and documentation explain
 tools; they do not grant additional permissions.
 
+There are two file surfaces. The `ls`, `glob`, `grep` and `read_file` tools access the
+read-only instruction filesystem. Read the specialist with
+`read_file(file_path="/skills/analysis/analysis-specialist/SKILL.md")`; use the listed virtual
+paths for other skills. These `/skills/` paths are not files in the VM: do not use shell
+`cat` or `read_analysis_file` to read them. `run_shell`, `run_python` and the analysis-file
+tools access the VM's assigned sources, `/workspace`, `/artifacts` and packaged `/docs`.
+
 ## Working method
 
 1. Interpret the question: metric, grain, population, units, time window and decision the
    answer informs. Inspect schemas, formats, nulls and representative values before coding.
 2. Map required inputs to observed fields. If essential information is absent, stop that
    calculation and explain exactly what is missing. A proxy is allowed only when the brief
    permits it, and must be labelled. A tool error is not evidence that data does not exist.
+   When the requested result cannot be supported, call `AnalysisCompletion` with
+   `status="cannot_answer"` and the missing inputs or useful next step. A request for charts
+   or sheets does not require a substitute report about unrelated metrics. Do not render
+   a scope report merely to accompany a refusal; refusal runs export no artifacts.
 3. Use query pushdown or out-of-core/streaming processing for large data. Avoid dumping whole
    datasets into tool output or loading them into memory merely to discover their shape.
 4. Use the exposed query tools for assigned stores. Use `run_shell`, `run_python`,
@@ -49,8 +60,11 @@ tools; they do not grant additional permissions.
    the run. Set `status="completed"` only after `render_report` succeeds and the requested
    checks pass. Include the result and important assumptions/caveats in the tool arguments.
    If required inputs are absent or a constraint prevents completion, use `status="cannot_answer"`
-   with a specific `reason`. The host exports a completed report after this call; temporary
-   VM paths are not delivered files. Never fabricate an artifact or successful check.
+   with a specific `reason`. Summarize the supported analytical result, not file delivery.
+   The host exports and verifies a completed report after this call and supplies the actual
+   deliverable inventory. Temporary VM paths are previews, not user-accessible downloads.
+   A refusal has no delivered report, charts or workbook, even if a preview was rendered.
+   Never fabricate an artifact or successful check.
 
 ## Boundaries and stopping
 

src/backintel/agent_files/skills/analysis/analysis-specialist/SKILL.md

@@ -23,6 +23,11 @@ Never treat missing values as zero unless the definition supports it.
 
 ## Work in the sandbox
 
+Load skills with the instruction tools (`read_file`, `ls`, `glob`, `grep`). Their virtual
+`/skills/` paths belong to the read-only instruction filesystem, not the VM shell. Use
+`run_shell` and the analysis-file tools only for VM paths such as `/workspace`, `/artifacts`,
+assigned sources and `/docs`.
+
 Prefer an aggregate SQL query to moving entire tables into Python. Use DuckDB/Polars lazy
 scans or bounded chunks when the corpus is large. Inspect a small sample to discover formats;
 compute final answers over the full requested population, or explicitly label sampling.
@@ -90,8 +95,12 @@ verify the affected output once. If no image inspection tool exists, report that
 
 ## Finish
 
-Return the requested result, the important caveats and the artifact manifest/paths supplied
-by the tools. A `cannot_answer` result should name the missing input and useful next step.
+Call `AnalysisCompletion` with the supported result and important caveats. The manifest
+from `render_report` describes a temporary preview; the host delivers and verifies the
+final bundle after completion. Do not advertise VM paths as downloads or claim delivery
+before that export. A `cannot_answer` result should name the missing input and useful next
+step, then stop. Refusal runs export no artifacts; do not render a substitute report merely
+because the brief requests charts or sheets for an unsupported metric.
 A timeout or resource failure should say which part finished and which did not. Do not call
 an incomplete bundle complete or claim independent truth verification for an ordinary run.
 

src/backintel/analysis.py

@@ -57,7 +57,10 @@ class AnalysisBrief(BaseModel):
 class AnalysisCompletion(BaseModel):
     model_config = ConfigDict(extra="forbid")
     status: Literal["completed", "cannot_answer"]
-    summary: str = Field(min_length=1)
+    summary: str = Field(
+        min_length=1,
+        description="Supported analytical result. Do not claim file delivery or advertise VM paths.",
+    )
     reason: str | None = None
     caveats: list[str] = Field(default_factory=list)
 
@@ -195,10 +198,12 @@ def execute(result) -> str:
 
     @tool
     def run_shell(command: str) -> str:
-        """Run bash inside the isolated analytical container, never on the host.
+        """Run bash inside the isolated analytical VM, never on the host.
 
         /workspace persists between commands. Sources and /docs are read-only. /artifacts
-        holds deliverables. Use preinstalled tools only; there is no public network.
+        holds previews; the host exports the final bundle after completion. Virtual
+        /skills paths belong to read_file, not this shell. Use preinstalled tools only;
+        there is no public network.
         Returns bounded stdout/stderr, exit code, elapsed time and timeout status.
         """
         return execute(sandbox.run_shell(command))

tests/test_analysis_completion.py

@@ -82,6 +82,21 @@ def test_completion_recovery_respects_original_call_limit(store):
         asyncio.run(agent.ainvoke({"messages": [("user", "Analyze the data.")]}))
 
 
+def test_virtual_skill_read_uses_instruction_backend_without_vm_execution(store):
+    agent = graph(
+        store,
+        [
+            call("read_file", {"file_path": "/skills/analysis/analysis-specialist/SKILL.md"}, 0),
+            completion(),
+        ],
+    )
+    result = asyncio.run(agent.ainvoke({"messages": [("user", "Read your skill, then check inputs.")]}))
+    reads = [m for m in result["messages"] if m.type == "tool" and m.name == "read_file"]
+    assert len(reads) == 1
+    assert "# Analysis and visual reporting" in reads[0].content
+    assert result["structured_response"].status == "cannot_answer"
+
+
 @pytest.mark.skipif(os.environ.get("BACKINTEL_SMOLVM_TESTS") != "1", reason="Requires prepared SmolVM")
 def test_recovered_completion_exports_verified_bundle(tmp_path, store):
     spec = {