Repository navigation
content: adopt depth-first prompts (v3) for refactor, api-contract, and feedback agents - #29
Merged
Merged
Conversation
…nd feedback agents Batch 3 of the v3 rollout: each agent gets a new out-of-scope case and is tested on 3 cases x 2 samples, blind pairwise judging in both orders. - refactor-plan-agent v3: 6-0 vs baseline, 6-0 vs current. Adopted. - api-contract-agent v3: 6-0 vs baseline, 5-1 vs current. Adopted. - customer-feedback-clusterer v3: 4-1 vs baseline, 3-0 vs current. Adopted. - test-case-generator v3: 5-1 vs baseline but 2-2 vs current. Not adopted. Adopted agents' runbooks, outputs, and criteria are aligned with what the new prompts produce (evals/README.md step 6). All verdicts come from the same Sonnet judge; six GPT-judged pairs from a paused session were set aside and re-judged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…core at 4 Address review on #29: - api-contract-agent: the reviewer gave 5, but its own notes list invented example details and a case scored 4; the rubric reserves 5 for no invented facts. Record the reviewer's score and cap qualityScore at 4. - api-contract-agent, refactor-plan-agent: checklist, failure modes, and criteria promised refusals the prompts do not instruct; describe only what the prompts say (scope marking, big-bang rewrite flag, behavior changes allowed when asked). - Refresh stale realUseCases expectedResult for the three adopted agents. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Merged
5 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This is batch 3 of the v3 depth-rule rollout, after #26 and #27. For each of four agents I wrote a v3 prompt and added an out-of-scope case.
Each prompt ran on 3 cases × 2 samples, generated with claude-opus-5-5. Sonnet judged each pair blind, in both orders. A case run only counts when both orders agree.
outputs, andevaluationCriteria[2..]for the adopted agents now match what the new prompts produce, following step 6 inevals/README.md.evaluationCriteria[0]is unchanged.Test plan
npm run check:datanpm run lintnpm run typechecknpm test(34 pass)npm run build🤖 Generated with Claude Code