V0.9.2/one prompt eval

[gimlichael]

This pull request updates the AGENTS.md documentation to clarify and refine the rules around evaluation package execution, especially regarding explicit user-directed evals using the yolo or auto modifiers. The changes introduce a narrowly-scoped, one-shot external Eval Orchestrator handoff flow, and tighten the requirements for model/harness selection and agent roles during eval preparation and execution. The documentation now more clearly distinguishes between routine/manual and explicitly authorized automated eval handoffs, and details the technical and policy boundaries for each.

Explicit Eval Handoff Flow:

• Added a new section describing an optional, explicitly authorized one-shot external Eval Orchestrator handoff for eval requests with yolo or auto, detailing when and how this handoff is allowed, and the technical steps for invoking it using scripts/eval-request.ps1 and Invoke-EvalRequest.

Eval Preparation and Execution Rules:

• Clarified that normal eval requests result in package preparation and manual handoff, while explicit yolo/auto requests authorize the one-shot external handoff. Updated the preparation and execution sections to reflect this distinction and to specify that only the explicit modifier can trigger automatic handoff. [1] [2] [3]
• Updated agent role definitions and restrictions to specify that the preparer context cannot execute its own package, but a fresh external Orchestrator authorized by the user may do so for that package only.

Model/Harness Selection and Discovery:

• Tightened requirements for harness/model selection: discovery is only performed when the user has not supplied a harness, and never without an explicit runner. The documentation now prohibits reconfirmation of supplied runners and clarifies the handling of OpenCode selectors and repository defaults.

Skill-Creator Integration:

• Updated the skill-creator section to reflect the new explicit handoff flow, clarifying that repository-side validation remains deterministic, and only the explicitly user-directed external executor performs model-backed evaluation and reporting.

[greptile-apps]

Confidence Score: 1/5

The PR is not yet safe to merge because isolation can miss out-of-projection accesses, and two earlier one-shot lifecycle defects remain unresolved.

Findings

1.   Relative Paths Bypass Isolation ▶
2.  Forced Retry Erases Reservation ▶
3.  Terminal Result Is Unbound ▶
4.  Prohibited Temporary Git History ▶

Fix with agent prompt

### Issue 1
scripts/eval-runners/runner-common.ps1:440-444
If a captured path is home-relative, such as `~/secret.txt`, or Windows drive-relative, such as `C:..\..\Windows\System32`, it matches none of the absolute-path branches. This code instead appends its segments to the projection base, so the Copilot and OpenCode boundary checks treat an actual out-of-projection access as inside and do not record an isolation contradiction.

**How this was verified:** The parser handles only slash-rooted, drive-rooted, and UNC paths specially, then joins every other path to the projection base before the Copilot and OpenCode checks decide whether the access is inside.

### Issue 2
scripts/eval-request.ps1:undefined-327
When `Invoke-EvalRequest` is retried with the same `Iteration` and `Force` preparation options, it reruns preparation before checking the handoff reservation. Because `Force` deletes and recreates the iteration directory containing `.external-handoff-started`, `Get-EvalHandoff` returns `external_handoff` again for the same path. After a timeout or interrupted launch, this can delete active execution state and dispatch a second model-backed evaluation despite the one-shot guarantee.

### Issue 3
scripts/eval-request.ps1:239-265
`Update-ExternalEvalOrchestratorState` marks the lifecycle as terminal based only on the caller-supplied status. Although it checks the separate wait handle, it accepts a missing terminal result or one whose handle and status do not match the transition. A host adapter can therefore report completion for the wrong task-or without any result-and stop waiting on the retained handle before the real evaluation outcome arrives. Require terminal transitions to include a result bound to the retained handle and matching terminal status.

### Issue 4
scripts/eval-git-workspace.ps1:35-49
This helper renames a branch, checks out a temporary feature branch, stages changes, and creates throwaway commits. The repository directive explicitly forbids temporary Git repositories, test branches, and throwaway commits. This repository requirement must be satisfied before merging, either by representing the fixture without this prohibited workflow or by intentionally narrowing the directive.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

• Adds deterministic preparation, reservation, and external-Orchestrator lifecycle helpers.
• Extends runner isolation, grading, result finalization, and reporting contracts.
• Adds extensive PowerShell regression and conformance tests.
• Refines repository guidance for model selection, automated handoff authorization, and isolated Git fixtures.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Explicit eval request] --> B[Prepare and validate package]
  B --> C{Yolo or auto authorized?}
  C -- No --> D[Manual prompt handoff]
  C -- Yes --> E{Fresh Orchestrator available?}
  E -- No --> D
  E -- Yes --> F[Reserve one-shot handoff]
  F --> G[Delegate fresh Orchestrator]
  G --> H[Execute paired arms]
  H --> I[Freeze execution evidence]
  I --> J[Analyze and grade]
  J --> K[Finalize report]

Loading

_{Reviews (16) · Last reviewed commit: "♻️ extract path utilities and add contai..."}

[greptile-apps]

Forced Retry Erases Reservation

When Invoke-EvalRequest is retried with the same Iteration and Force preparation options, it reruns preparation before checking the handoff reservation. Because Force deletes and recreates the iteration directory containing .external-handoff-started, Get-EvalHandoff returns external_handoff again for the same path. After a timeout or interrupted launch, this can delete active execution state and dispatch a second model-backed evaluation despite the one-shot guarantee.

Prompt To Fix With AI

This is a comment left during a code review.
Path: scripts/eval-request.ps1
Line: 68

Comment:
**Forced Retry Erases Reservation**

When `Invoke-EvalRequest` is retried with the same `Iteration` and `Force` preparation options, it reruns preparation before checking the handoff reservation. Because `Force` deletes and recreates the iteration directory containing `.external-handoff-started`, `Get-EvalHandoff` returns `external_handoff` again for the same path. After a timeout or interrupted launch, this can delete active execution state and dispatch a second model-backed evaluation despite the one-shot guarantee.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Prohibited Temporary Git History

This helper renames a branch, checks out a temporary feature branch, stages changes, and creates throwaway commits. The repository directive explicitly forbids temporary Git repositories, test branches, and throwaway commits. This repository requirement must be satisfied before merging, either by representing the fixture without this prohibited workflow or by intentionally narrowing the directive.

Context Used: AGENTS.md (source)

Prompt To Fix With AI

This is a comment left during a code review.
Path: scripts/eval-git-workspace.ps1
Line: 35-49

Comment:
**Prohibited Temporary Git History**

This helper renames a branch, checks out a temporary feature branch, stages changes, and creates throwaway commits. The repository directive explicitly forbids temporary Git repositories, test branches, and throwaway commits. This repository requirement must be satisfied before merging, either by representing the fixture without this prohibited workflow or by intentionally narrowing the directive.

**Context Used:** AGENTS.md ([source](https://github.com/codebeltnet/agentic/blob/main/AGENTS.md))

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[greptile-apps]

Terminal Result Is Unbound

Update-ExternalEvalOrchestratorState marks the lifecycle as terminal based only on the caller-supplied status. Although it checks the separate wait handle, it accepts a missing terminal result or one whose handle and status do not match the transition. A host adapter can therefore report completion for the wrong task—or without any result—and stop waiting on the retained handle before the real evaluation outcome arrives. Require terminal transitions to include a result bound to the retained handle and matching terminal status.

Prompt To Fix With AI

This is a comment left during a code review.
Path: scripts/eval-request.ps1
Line: 239-265

Comment:
**Terminal Result Is Unbound**

`Update-ExternalEvalOrchestratorState` marks the lifecycle as terminal based only on the caller-supplied status. Although it checks the separate wait handle, it accepts a missing terminal result or one whose handle and status do not match the transition. A host adapter can therefore report completion for the wrong task—or without any result—and stop waiting on the retained handle before the real evaluation outcome arrives. Require terminal transitions to include a result bound to the retained handle and matching terminal status.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Relative Paths Bypass Isolation

If a captured path is home-relative, such as ~/secret.txt, or Windows drive-relative, such as C:..\..\Windows\System32, it matches none of the absolute-path branches. This code instead appends its segments to the projection base, so the Copilot and OpenCode boundary checks treat an actual out-of-projection access as inside and do not record an isolation contradiction.

How this was verified: The parser handles only slash-rooted, drive-rooted, and UNC paths specially, then joins every other path to the projection base before the Copilot and OpenCode checks decide whether the access is inside.

Prompt To Fix With AI

This is a comment left during a code review.
Path: scripts/eval-runners/runner-common.ps1
Line: 440-444

Comment:
**Relative Paths Bypass Isolation**

If a captured path is home-relative, such as `~/secret.txt`, or Windows drive-relative, such as `C:..\..\Windows\System32`, it matches none of the absolute-path branches. This code instead appends its segments to the projection base, so the Copilot and OpenCode boundary checks treat an actual out-of-projection access as inside and do not record an isolation contradiction.

**How this was verified:** The parser handles only slash-rooted, drive-rooted, and UNC paths specially, then joins every other path to the projection base before the Copilot and OpenCode checks decide whether the access is inside.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[gimlichael]

Closing this for now.

The work in this PR has been valuable in exposing several real challenges around portable, isolated, model-backed skill evaluation across Copilot, Codex, and OpenCode. We made meaningful progress on runner isolation, deterministic validation, execution evidence, model locking, and reproducibility.

However, the latest end-to-end runs have also shown that the orchestration and proof machinery is becoming disproportionately complex. We are starting to spend more effort making the evaluation framework evaluate itself than evaluating the skills it was created for.

Rather than continue adding more guards, state transitions, and runner-specific exceptions, I want to stop here and reassess the approach.

The next step will be to explore whether an existing, more scalable evaluation framework or established pattern can give us the guarantees we need without reinventing the wheel. If we return to a custom implementation, it should be materially simpler than the architecture in this PR and build on the lessons learned here.

So this is not necessarily the end of the idea, but it is the end of this iteration.

Closing PR #41 without merging.