Repository navigation
content: adopt depth-first prompts (v3) for security, performance, and release-notes agents - #30
Merged
Merged
Conversation
…d release-notes agents Batch 4a of the v3 rollout: each agent gets a new out-of-scope case and is tested on 3 cases x 2 samples, blind pairwise judging in both orders. - security-checklist-agent v3: 5-1 vs baseline, 5-0 vs current. Adopted; reviewer's 5 capped at 4 (its notes cite very long output and a case scored 4). - performance-audit-agent v3: 5-0 vs baseline, 4-1 vs current. Adopted. - release-notes-writer v3: 5-0 vs baseline, 4-1 vs current. Adopted. Runbooks, outputs, use-case results, and criteria are aligned with what the new prompts instruct. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Address review on #30: - The first reviews graded the new prompts against the old "assumptions, risks, and next actions separately" criterion, because the runbook was aligned after the review. Re-review all three against the aligned criteria and re-apply. Performance's reviewer 5 is capped at 4 (its notes cite unmarked inferences, long output, a case scored 4); security re-reviewed at 4. - release-notes-writer: criterion promised [ASSUMPTION] tags the prompt never asks for; use its placeholder and question rules instead. Mark internal notes as produced only when requested. - evals/README.md: align the runbook before the reviewer grades (steps 5 and 6 swapped). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This is batch 4a of the v3 depth-rule rollout, after #26, #27, and #29. Each of the three agents got a v3 prompt and an out-of-scope case.
outputs, the runbook shape, checklist, and failure modes, the handoff starter,realUseCases[].expectedResult, andevaluationCriteria[2..]. They now describe only what the prompt instructs.evaluationCriteria[0]is unchanged.evals/README.md.Test plan
npm run check:datanpm run lintnpm run typechecknpm test(34 pass)npm run build🤖 Generated with Claude Code
Review fix: the first reviews used the old criteria, because the runbook was aligned after the review. All three agents were re-reviewed against the aligned criteria, and evals/README.md now aligns the runbook before the review.