Repository navigation
feat(aidd-qa): verify acceptance and show the journey, run both before the draft PR - #1012
Draft
blafourcade wants to merge 9 commits into
Draft
blafourcade wants to merge 9 commits into
blafourcade wants to merge 9 commits into
Conversation
Acceptance QA did two jobs: judging each scenario from its step results, and producing a short reviewer video. The video rules filled most of the runner reference, required ffmpeg, and fixed a 1280x720 frame that has nothing to do with a verdict. `01-acceptance-qa` now verifies only: `qa.md` with a verdict per scenario, the step results as its proof, no recording and no media dependency. The new `02-demo` takes a journey and a candidate and records one labelled video, with the recording, encoding and validation rules moved from verify. Measured with `claude -p` on a fixture app, against a copy of this plugin from `next` and from this tree: - Verify, 8 runs each side: verdicts by criterion, run verdict, out-of-interface and rejected sets identical; no video written; about 50 s a run instead of 85. - Demo, 4 runs: a passing journey labelled `pass`, a planted bug labelled `fail`, each VP9 1280x720 under 12 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VE16AxC1V9wAMGmGgefe4z
Real runs on a fixture app showed acceptance QA improvising where its contract was silent. It now: - runs one happy path per journey, where a second journey used to be rejected untested; - verifies a CLI criterion by its command, output and exit code, and leaves an API alone out of interface; - never asks when another skill or agent invoked it, returning `blocked` with the question, and always writes `qa.md`; - proves test-only storage in its own step before any start, by memory, an isolated fixture or the user, never by an environment name or inference; - allows one fallback entry only when memory or a browser test names it; - resolves viewport and permissions from the criterion, then memory, then the default; - defines one observable condition for a vague outcome, probes a timed boundary on both sides, and measures a short transition inside `run-code`; - defines what is destructive, and records `Limitations` without a new status. Measured with `claude -p` against this tree, each case graded against an expected result written before the run: - signup golden 3/3 unchanged; todo golden 3/3 with the planted bug failing; - API-only spec `skipped` 2/2; CLI pass and planted fail 2/2; - toast within 2 s pass and over 2 s fail 2/2, with measured times; - vague "flash" 3/3 measured as a style change and revert; - 390x844 viewport 2/2; mobile without dimensions `blocked` 2/2; - unproven storage `blocked` 3/3 with the app never started; - crashing entry `blocked` 2/2; documented fallback `pass` 2/2. The skill holds 1821 words, against 1521 after the split. Refs #918 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VE16AxC1V9wAMGmGgefe4z
The demo now shows a journey through its own interface: a browser journey as one video, a CLI journey as a transcript of the command, its output and its exit status, so a CLI needs no ffmpeg. A failing journey is shown as observed and labelled `fail`, never staged. The artifact lands in the folder the caller names, else beside the journey's source, and is named after its journey, never `happy-path` for a bug. Invoked by another skill or agent, the demo asks nothing and returns paths and labels; it offers to open a video only when the user invoked it directly. Both routers take the `| Action | Does |` form. Measured with `claude -p` against this tree, 2 runs a case: - a `qa.md` scenario: video in the `qa/` folder beside it, named after the journey, VP9 under 12 s, `pass`; - a bug, invoked by an agent: video in the caller's folder, `fail`, no question; - a CLI journey, invoked by an agent: transcript in the caller's folder, `fail` on output `3` against `2`, no video, no question. The demo holds 1059 words, against 1078 after the split. Refs #918 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VE16AxC1V9wAMGmGgefe4z
…aft PR Check reviewed the candidate and opened the draft pull request without ever running acceptance QA, so criteria only a run could prove reached the pull request unverified. After Review clears, the checker now runs acceptance QA on the same contract and candidate and routes its verdict: `pass` or `skipped` to Challenge, `fail` to Deliver, `blocked` by its criteria to Frame, `blocked` by the environment to a stop with the user action. On `pass` it runs the demo on the passing scenarios. Only `qa.md` and its demo artifacts may then be new, any other change re-enters Review, and a failed or blocked run is never committed. The draft pull request receives `qa.md` and the artifact paths. The checker may invoke both QA skills and writes only QA-owned paths. The pull-request skill links whatever evidence its caller passes and never presents a failed or blocked run as cleared. `aidd-qa` joins the recommended plugins. A guard pins each rule, red before the change; each mutation reddens its own test. Refs #919 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VE16AxC1V9wAMGmGgefe4z
The new handoff let only `qa.md` and its demo artifacts be new, and the checker wrote only QA-owned paths, yet the same checker writes the review round in that zone. Read literally, the review report was either never committed or sent the run back to Review. Check again commits everything it wrote, never a failed or blocked QA run, and a product or validator change still re-enters Review; the checker writes only the reports its skills own. Refs #919 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VE16AxC1V9wAMGmGgefe4z
With `aidd-qa` absent, an end-to-end run still produced a `qa.md` in the template's shape and a recording: the checker ran QA by other means instead of stopping, despite the general rule to verify each provider. Check now says it: without the QA skill, stop with the instruction to install `aidd-qa`, never run QA by other means. Rerun the same way, it stopped at the QA step with that instruction, wrote no `qa.md` and opened no pull request. Refs #919 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VE16AxC1V9wAMGmGgefe4z
Merge repeated rules and replace the verbose recording example with a compact protocol while keeping exact values, reducing the demo from 1059 to 691 words. Six real claude -p runs against frozen plugin copies retained observed outcomes: two qa.md scenarios produced qa/signup-with-email.webm labelled pass (vp9, 1280 wide, 4.75 s and 4.4 s); two agent-invoked bug journeys produced demo/delete-buy-milk.webm labelled fail (vp9, 1280 wide, 9.5 s and 9.67 s) without a question; two agent-invoked CLI journeys produced demo/count-a-b.txt labelled fail without a question. Refs #918 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VE16AxC1V9wAMGmGgefe4z
Keep procedure details in their owning skills and SDLC routing in its diagram. Reduce QA and demo from 2499 to 1204 words. Validation: 688 static tests pass; all six injected structural defects are detected. Codex demo CLI passes with the expected exit status 2. Full native QA and SDLC validation remains incomplete due to API and tooling failures; the skipped-route run is nonconforming. Refs #918 Refs #919
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Split acceptance QA (
qa.md) from demonstration (browser video or CLI transcript), and wire both into SDLC Check. Keep procedures in their owning skills; SDLC holds the flow diagram.QA and Demo total 1,209 words, down from 2,499 before this lean pass. Checker changes are limited to caller-required skill orchestration and two permissions.
Validation
denied, expected exit status2, verdictpass; candidate unchanged.skippedroute was nonconforming (different checkers, PR opened despite Challenge findings). No new browser video passed validation.fail→ Deliver remains unproven end to end.Linked issues
Closes #918
Closes #919
✅ I certify
🤖 Generated with Claude Code
https://claude.ai/code/session_01VE16AxC1V9wAMGmGgefe4z