behavioral packs: forbid tool-call syntax in the single-reply subject prompt - #97
Merged
Conversation
…ly on transport errors Under promptfoo 0.122.0 every assertion-failed row carries .error (the assertion message) AND failureReason: 1, so the old .error-based FAULT heuristic excluded REAL assertion failures from scoring as transport faults: a 4-pass/1-real-fail scenario read "[OK] 4/4 valid = 1.00 (1 FAULT excluded)" — the exact counterfeit green the gate exists to prevent (observed on PR #93's live routing run). Fix: classify on failureReason, the reliable discriminator. A row is a FAULT only when failureReason == 2/"error", or when it carries an error signal with NO failureReason recorded (legacy fallback for shapes that predate the field); failureReason == 1 is a real FAIL scored against the floor even when .error is present. All fail-closed properties preserved: non-pass with no signals stays FAIL, all-FAULT starvation stays STARVED, floors/min-runs/grouping untouched. Cheap-tier §18 fixtures move to the real 0.122 row shape and gain a new mutation check: a real assertion failure carrying .error must be scored as FAIL, not excluded as FAULT (proven red against the pre-fix script). docs/testing.md statistical-spine prose updated per the standing order; the machine LIVE-INVENTORY block is unchanged (no tier added/removed). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NsPQTo67oovz8ogirsH6uL
… prompt Every plugins/*/evals/promptfoo/prompt.txt now tells the subject model that it has NO tools in this reply and must never emit tool-call syntax, XML/JSON tool blocks, or a bare "let me read the file first" stop — say what it would look for and give the complete answer anyway. The six identical-template packs stay byte-identical; each variant keeps its own structure and voice. Why: PR #95 run 33592173756 (scope-fence leg) lost two of three rows to a fake `<tool name="read" args={"path": "src/list.js"} />` with no final response; the identical pack passed on PR #96 minutes later. Under the honest statistical gate those rows are real FAILs, so any pack could flip red on any run touching evals/paid/**. Cheap tier §21 asserts the load-bearing phrase is present in every pack prompt (proven red by removing it from one pack, then green); inert in the counterfeit synthetic root via the existing `.git` marker. docs/testing.md behavioral-tier prose gains one sentence; LIVE-INVENTORY unchanged. No rubric, assertion, repeat count, floor, or SKILL.md changed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NsPQTo67oovz8ogirsH6uL
|
You have reached your Codex usage limits for security reviews. Please try again later. |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
evals/paid/pass-rate.sh contains an implementation/comment mismatch around when .error is allowed to drive FAULT classification, which could make future failureReason shapes regress toward fail-open behavior.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Hardens the behavioral (promptfoo) eval subject prompts for this cross-harness plugin marketplace so single-reply packs don’t stochastically “tool-call and stop” (producing no final answer for the grader), and adds a cheap-tier guard to prevent prompt drift.
Changes:
- Add a “no tools / never emit tool-call syntax” clause to every
plugins/*/evals/promptfoo/prompt.txtbehavioral subject prompt. - Extend
evals/cheap/run.shwith a repo-level assertion that every behavioral pack prompt contains the load-bearing phrase (never emit tool-call), failing closed if no prompts are found. - Update
evals/paid/pass-rate.shdocumentation/comments and cheap-tier fixtures/docs aroundfailureReason-based FAULT vs FAIL classification.
File summaries
| File | Description |
|---|---|
| plugins/wayfinder/evals/promptfoo/prompt.txt | Adds explicit “no tools / never emit tool-call syntax” instruction for single-reply subject prompts. |
| plugins/voice/evals/promptfoo/prompt.txt | Adds explicit “no tools / never emit tool-call syntax” instruction for single-reply subject prompts. |
| plugins/verify-before-claim/evals/promptfoo/prompt.txt | Adds explicit “no tools / never emit tool-call syntax” instruction for single-reply subject prompts. |
| plugins/stop-rule/evals/promptfoo/prompt.txt | Adds explicit “no tools / never emit tool-call syntax” instruction for single-reply subject prompts. |
| plugins/semver-gate/evals/promptfoo/prompt.txt | Adds explicit “no tools / never emit tool-call syntax” instruction for single-reply subject prompts. |
| plugins/scope-fence/evals/promptfoo/prompt.txt | Adds explicit “no tools / never emit tool-call syntax” instruction for single-reply subject prompts. |
| plugins/redgate/evals/promptfoo/prompt.txt | Adds explicit “no tools / never emit tool-call syntax” instruction for single-reply subject prompts. |
| plugins/graveyard/evals/promptfoo/prompt.txt | Adds explicit “no tools / never emit tool-call syntax” instruction tailored to plan-shaped output. |
| plugins/fleet-playbook-curator/evals/promptfoo/prompt.txt | Adds explicit “no tools / never emit tool-call syntax” instruction tailored to plan-shaped output. |
| plugins/find-before-build/evals/promptfoo/prompt.txt | Adds explicit “no tools / never emit tool-call syntax” instruction for single-reply subject prompts. |
| plugins/agent-compiler/evals/promptfoo/prompt.txt | Adds explicit “no tools / never emit tool-call syntax” instruction for single-reply subject prompts. |
| evals/paid/pass-rate.sh | Updates FAULT/FAIL classification rationale to key on failureReason and avoid laundering real assertion failures as FAULTs. |
| evals/cheap/run.sh | Adds a cheap-tier assertion that every behavioral pack prompt includes the load-bearing no-tools phrase. |
| docs/testing.md | Documents the new no-tools clause expectation and the failureReason-keyed FAULT classification. |
Review details
- Files reviewed: 14/14 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e032b2e5ca
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Resolves the overlap in evals/paid/pass-rate.sh and the cheap self-test in evals/cheap/run.sh by taking main's version: the branch carried the #94 gate-fix cherry-pick, and main now carries that fix plus the narrower .error fallback (FAULT only when failureReason is absent) with its red-first fixture. This branch's own sections are unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NsPQTo67oovz8ogirsH6uL
On PR #97's first full run the voice pack's "gated path — plans the pipeline without fabricating its results" scenario fell from 3/3 to 0/3: every row printed per-claim verdict markers before any subagent could have run, and the grader read that as a fabricated validation. The clause's "give your complete best answer anyway" was nudging the subject toward verdict-shaped output. All 11 pack prompts now end the clause with "without inventing the results you did not obtain"; the six template packs stay byte-identical (one md5), the five variants keep their own wording, and the load-bearing phrase the cheap tier asserts ("never emit tool-call") is unchanged. docs/testing.md's description of the clause follows. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NsPQTo67oovz8ogirsH6uL
…pt file Codex review on PR #97: the cheap-tier gate (§21) scanned only plugins/*/evals/promptfoo/prompt.txt, so tailscale-wif — whose promptfooconfig.yaml renders the subject prompt from prompt.js — carried no no-tools clause while the gate reported green. The gate now reads every pack's promptfooconfig.yaml, takes each `file://` reference under `prompts:` (comments stripped), and asserts the phrase in that file; a config naming no prompt file, or one that does not exist, fails. prompt.js gains the clause in its own plan-shaped voice and still renders (require + call verified). Proven red first: with the phrase altered in prompt.js the cheap tier reported "plugins/tailscale-wif/evals/promptfoo/prompt.js is missing the no-tools clause" (1193 passed, 1 failed); restored, all 12 configured prompt files pass (1194 passed, 0 failed). docs/testing.md describes the discovery. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NsPQTo67oovz8ogirsH6uL
JRichlen
pushed a commit
that referenced
this pull request
Sep 2, 2026
Both sides appended a delimited section to the end of evals/cheap/run.sh: this branch's RQ-002 typed route/step contract tests and main's behavioral-pack no-tools clause gate (§21). Both are kept, in that order. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NsPQTo67oovz8ogirsH6uL
JRichlen
pushed a commit
that referenced
this pull request
Sep 3, 2026
…honest gate opens era 8 Daily refresh after main advanced with three merged PRs: - era 7 (window now Aug 29 – Sep 2): "Routing is a typed composition, not a single name" — RQ-002's specialist | envelope | guards | interaction_owner contract, forced by the S1 coin-flip the legacy single-name route could not resolve; the trajectory STEP contract; and the three live-run grader corrections (reasoning-trace read, specialist-only legacy pins, either-role disciplines). - new era 8, "The honest gate" (Sep 2): pass-rate.sh had excluded rows with .error as FAULTs even when a real assertion failed — fail-open green on the tier that grades everything else. Fixed red-first with a self-test fixture; three packs went red at once. Redgate's own pack then caught its driver scheduling landing and truncation unattended under a blanket approval: landing and destruction are now their own MAJOR gates, red-first is stated every time, and a coded allow is never gate consent. - inventory-only (curated:false, awaiting the owner's pass): #97's no-tools clause separating harness artifacts from skill failures. Regenerated index.html; cheap tier green on the merged head. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014XYwiWZicXu2CfDUKAXHpi
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The behavioral tier is a single-reply harness: the subject model never receives a tool result. A cheap subject model can still, stochastically, emit fake tool-call syntax and stop — leaving no final response for the grader.
Evidence — PR #95, run 33592173756, job 100128260502 (scope-fence leg). Two of three rows for the same scenario ended with a bare
<tool name="read"/>-style block and no final response, so the grader failed them. The identical pack passed on PR #96 minutes later. Under the honest statistical gate (the pass-rate fix that landed via #95), such rows are real FAILs scored against the floor — so any of the packs can flip red on any run that touchesevals/paid/**, with nothing about the skill having changed.The change
Every behavioral pack's subject prompt — the 11
plugins/*/evals/promptfoo/prompt.txtfiles plus theprompt.jsfunction tailscale-wif renders from — gains one clause, placed with its existing single-reply instruction, in that file's own voice. The six identical-template packs (agent-compiler, find-before-build, redgate, scope-fence, semver-gate, stop-rule) stay byte-identical to each other (md5sumconfirms: 6 × one hash); the variants (fleet-playbook-curator, graveyard, verify-before-claim, voice, wayfinder, tailscale-wif) keep their structure. Template form:The plan-shaped packs (fleet-playbook-curator, graveyard, tailscale-wif) say "give your complete plan anyway"; wayfinder says "read the map first". The load-bearing phrase
never emit tool-callis common to all.Second commit (76502d2) — "without inventing the results you did not obtain". The first full run (33592762256) was 10/12 legs green; the voice leg fell from 3/3 to 0/3 on "gated path — plans the pipeline without fabricating its results": every row printed per-claim verdict markers before any subagent could have run, which the grader correctly reads as a fabricated validation. The clause's "give your complete best answer anyway" was nudging the subject toward verdict-shaped output, so the clause now ends with the no-invention phrase everywhere (templates still one md5). The redgate miss on that run was the pre-existing main failure that #95 fixed; this branch now carries #95 via a merge of
main.Third commit (31fbd41) — the gate discovers each pack's configured prompt file. Codex found that §21 only globbed
prompt.txt, so tailscale-wif (file://prompt.js) was unprotected while the gate reported green. §21 now reads every pack'spromptfooconfig.yaml, takes eachfile://reference underprompts:, and asserts the phrase in that file; a config naming no prompt file, or a missing one, fails.prompt.jscarries the clause and still renders.Not changed: rubrics, assertions,
repeat:counts, floors, anySKILL.md.Cheap-tier guard (§21) — proven red, then green
Delimited section at the end of
evals/cheap/run.sh; fail-closed if zero packs are found; inert in the counterfeit tier's synthetic root via the existing.gitmarker (§20's pattern). FAIL substring:no-tools clause.Red — clause temporarily stripped from
plugins/scope-fence/evals/promptfoo/prompt.txt(first commit), and later the phrase altered inplugins/tailscale-wif/evals/promptfoo/prompt.js(third commit):Green — restored, at 31fbd41:
Docs
docs/testing.mdbehavioral-tier prose describes the no-tools clause (including the no-invention phrase) and the cheap tier's discovery of each pack's configured prompt file (standing order). LIVE-INVENTORY block verified byte-unchanged againstmain.Gates (all local, all exit 0, at 31fbd41)
evals/cheap/run.sh— 1194 passed, 0 failedevals/counterfeits/run.sh— 22 passed, 0 failed (baseline calibration green; §21 inert in the synthetic root)python3 ci/check_branch_protection.py --repo .— in syncpython3 ci/check_behavior_surfaces.py --repo .— all cases pass; pack prompts are already underplugins/${plugin}/evals/promptfoo/**, so no spec change was needed and every pack leg selects on this PRNo demonstration comment: this PR edits eval subject prompts and a cheap-tier check, not skill prose (
SKILL.md,references/, or a command), so the demonstration discipline does not apply. The behavioral legs running on this PR are the relevant evidence.🤖 Generated with Claude Code
https://claude.ai/code/session_01NsPQTo67oovz8ogirsH6uL