From b752515a07542c82c3f8390416c6f9a49120baca Mon Sep 17 00:00:00 2001 From: Baptiste LAFOURCADE Date: Mon, 12 Oct 2026 22:01:01 +0200 Subject: [PATCH 1/9] feat(aidd-qa): split verification from the demo that shows a journey Acceptance QA did two jobs: judging each scenario from its step results, and producing a short reviewer video. The video rules filled most of the runner reference, required ffmpeg, and fixed a 1280x720 frame that has nothing to do with a verdict. `01-acceptance-qa` now verifies only: `qa.md` with a verdict per scenario, the step results as its proof, no recording and no media dependency. The new `02-demo` takes a journey and a candidate and records one labelled video, with the recording, encoding and validation rules moved from verify. Measured with `claude -p` on a fixture app, against a copy of this plugin from `next` and from this tree: - Verify, 8 runs each side: verdicts by criterion, run verdict, out-of-interface and rejected sets identical; no video written; about 50 s a run instead of 85. - Demo, 4 runs: a passing journey labelled `pass`, a planted bug labelled `fail`, each VP9 1280x720 under 12 s. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VE16AxC1V9wAMGmGgefe4z --- README.md | 4 +- .../2026_10_12_qa-verify-and-demo/phase-1.md | 21 ++++ .../2026_10_12_qa-verify-and-demo/phase-2.md | 25 +++++ .../2026_10_12_qa-verify-and-demo/phase-3.md | 19 ++++ .../2026_10_12_qa-verify-and-demo/phase-4.md | 17 ++++ .../2026_10_12_qa-verify-and-demo/plan.md | 30 ++++++ plugins/aidd-qa/.claude-plugin/plugin.json | 5 +- plugins/aidd-qa/CATALOG.md | 11 ++- plugins/aidd-qa/README.md | 5 +- .../aidd-qa/skills/01-acceptance-qa/SKILL.md | 6 +- .../actions/02-prerequisites.md | 6 +- .../actions/03-prepare-run.md | 6 +- .../actions/04-run-scenarios.md | 26 +++-- .../assets/qa-report-template.md | 4 +- .../interface-browser-playwright-cli.md | 79 ++------------- plugins/aidd-qa/skills/02-demo/SKILL.md | 29 ++++++ .../skills/02-demo/actions/01-record.md | 32 ++++++ .../interface-browser-playwright-cli.md | 98 +++++++++++++++++++ .../aidd-qa-verify-and-demo-split.test.js | 34 +++++++ scripts/__tests__/architecture-rules.test.js | 2 +- 20 files changed, 354 insertions(+), 105 deletions(-) create mode 100644 aidd_docs/tasks/2026_10/2026_10_12_qa-verify-and-demo/phase-1.md create mode 100644 aidd_docs/tasks/2026_10/2026_10_12_qa-verify-and-demo/phase-2.md create mode 100644 aidd_docs/tasks/2026_10/2026_10_12_qa-verify-and-demo/phase-3.md create mode 100644 aidd_docs/tasks/2026_10/2026_10_12_qa-verify-and-demo/phase-4.md create mode 100644 aidd_docs/tasks/2026_10/2026_10_12_qa-verify-and-demo/plan.md create mode 100644 plugins/aidd-qa/skills/02-demo/SKILL.md create mode 100644 plugins/aidd-qa/skills/02-demo/actions/01-record.md create mode 100644 plugins/aidd-qa/skills/02-demo/references/interface-browser-playwright-cli.md create mode 100644 scripts/__tests__/aidd-qa-verify-and-demo-split.test.js diff --git a/README.md b/README.md index 362d9c5e1..86ee80154 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ Unify **engineering teams** around **standardized workflows** and **shared best 🧱 **IDE agnostic** · 🏗️ **Legacy systems** · 🌱 **Token-optimized** · 🇫🇷 **Made in France**

- 9 plugins · 52 skills · 2 agents + 9 plugins · 53 skills · 2 agents

[![Open Source](https://img.shields.io/badge/Open_Source-Yes-yellow?logo=open-source-initiative&logoColor=white)](https://opensource.org/) @@ -324,7 +324,7 @@ Answers what a piece of work cost — tokens, models, and which skill spent them ### 🎬 [aidd-qa](plugins/aidd-qa/README.md) 🆕 -`1 skill` · **new** +`2 skills` · **new** Acceptance QA — locks browser scenarios from acceptance criteria and records reviewer evidence. diff --git a/aidd_docs/tasks/2026_10/2026_10_12_qa-verify-and-demo/phase-1.md b/aidd_docs/tasks/2026_10/2026_10_12_qa-verify-and-demo/phase-1.md new file mode 100644 index 000000000..e7ce6ecdb --- /dev/null +++ b/aidd_docs/tasks/2026_10/2026_10_12_qa-verify-and-demo/phase-1.md @@ -0,0 +1,21 @@ +--- +status: done +--- + +# Instruction: split + +## Architecture projection + +```txt +plugins/aidd-qa/ +├── .claude-plugin/plugin.json ✏️ lists 02-demo +└── skills/ + ├── 01-acceptance-qa/ ✏️ verify only: no recording, encoding, ffmpeg + └── 02-demo/ ✅ the recording rules, moved unchanged +``` + +## Acceptance criteria + +- Verify writes `qa.md` with no video, and needs no `ffmpeg`. +- On the golden fixtures, verdicts by criterion, run verdict, out-of-interface and rejected sets match `next`. +- The demo records a `qa.md` scenario with the moved rules: VP9, 1280 wide, at most 12 s. diff --git a/aidd_docs/tasks/2026_10/2026_10_12_qa-verify-and-demo/phase-2.md b/aidd_docs/tasks/2026_10/2026_10_12_qa-verify-and-demo/phase-2.md new file mode 100644 index 000000000..a0f1d4cc4 --- /dev/null +++ b/aidd_docs/tasks/2026_10/2026_10_12_qa-verify-and-demo/phase-2.md @@ -0,0 +1,25 @@ +--- +status: pending +--- + +# Instruction: verify-rules + +## Architecture projection + +```txt +plugins/aidd-qa/skills/01-acceptance-qa/ ✏️ one clause per rule, where the decision is made +``` + +## Acceptance criteria + +- One happy path per journey: a spec with two journeys runs both. +- A CLI criterion is verified by its command and expected output. +- Invoked by another skill or agent, it never asks: a missing decision yields `blocked` with the question. +- Test-only storage is proven before any start, by memory, an isolated fixture, or the user; an environment name is not proof. +- A crashed or UI-less entry gets one documented fallback, else `blocked`. +- Viewport and permissions come from the criterion, then `Browser QA` memory, then the default. +- A vague criterion becomes one observable condition, else it is rejected. +- A timed boundary is probed on both sides; measured values reach `Actual`. +- A short transition is measured inside `run-code`. +- Destructive means data outside the scenario's fixture or an irreversible external effect. +- `qa.md` holds `Limitations`; a limitation that weakens proof yields `blocked`. diff --git a/aidd_docs/tasks/2026_10/2026_10_12_qa-verify-and-demo/phase-3.md b/aidd_docs/tasks/2026_10/2026_10_12_qa-verify-and-demo/phase-3.md new file mode 100644 index 000000000..83ff1c6f2 --- /dev/null +++ b/aidd_docs/tasks/2026_10/2026_10_12_qa-verify-and-demo/phase-3.md @@ -0,0 +1,19 @@ +--- +status: pending +--- + +# Instruction: demo + +## Architecture projection + +```txt +plugins/aidd-qa/skills/02-demo/ ✏️ browser video and CLI transcript +``` + +## Acceptance criteria + +- A browser journey yields one video; a CLI journey yields the command and its output. +- A failing journey is shown as observed and labelled not expected. +- Invoked by another skill or agent, it asks nothing and returns its paths. +- A video lands in the folder the caller names, else beside the journey's source, never in a temporary directory. +- A video is named after its journey, never `happy-path` for a bug. diff --git a/aidd_docs/tasks/2026_10/2026_10_12_qa-verify-and-demo/phase-4.md b/aidd_docs/tasks/2026_10/2026_10_12_qa-verify-and-demo/phase-4.md new file mode 100644 index 000000000..a21ea33ed --- /dev/null +++ b/aidd_docs/tasks/2026_10/2026_10_12_qa-verify-and-demo/phase-4.md @@ -0,0 +1,17 @@ +--- +status: pending +--- + +# Instruction: sdlc-wiring + +## Architecture projection + +```txt +plugins/aidd-orchestrator/skills/01-sdlc/references/03-check.md ✏️ QA after review, demo for the pull request +plugins/aidd-dev/agents/checker.md ✏️ may invoke QA and demo +.claude-plugin/marketplace.json, README.md ✏️ aidd-qa on the recommended path +``` + +## Acceptance criteria + +- #919's acceptance criteria. diff --git a/aidd_docs/tasks/2026_10/2026_10_12_qa-verify-and-demo/plan.md b/aidd_docs/tasks/2026_10/2026_10_12_qa-verify-and-demo/plan.md new file mode 100644 index 000000000..78ce1f33a --- /dev/null +++ b/aidd_docs/tasks/2026_10/2026_10_12_qa-verify-and-demo/plan.md @@ -0,0 +1,30 @@ +--- +objective: "Acceptance QA verifies browser and CLI criteria without video, a demo skill shows any journey, and the SDLC runs both before the draft pull request." +status: in-progress +--- + +# Plan: QA verifies, demo shows + +## Overview + +| Field | Value | +| ---------- | ----- | +| **Goal** | Split `aidd-qa` into verify and show, harden verify for real journeys, wire both into the SDLC | +| **Source** | #918 (agreed contract in its body), #919 | + +## Phases + +| # | Phase | File | +| --- | ----------------- | ---------------------------- | +| 1 | split | [`phase-1.md`](./phase-1.md) | +| 2 | verify-rules | [`phase-2.md`](./phase-2.md) | +| 3 | demo | [`phase-3.md`](./phase-3.md) | +| 4 | sdlc-wiring | [`phase-4.md`](./phase-4.md) | + +## Decisions + +| Decision | Why | +| --- | --- | +| Verify carries no media; the demo owns it | a verdict is the step results; a missing `ffmpeg` must not block it | +| The demo never decides where its artifact goes | the caller (pull request, defect, user) does | +| Golden is verdicts by criterion, not bytes | an LLM run varies in wording, never in what it proves | diff --git a/plugins/aidd-qa/.claude-plugin/plugin.json b/plugins/aidd-qa/.claude-plugin/plugin.json index 6e94ad5cd..cce4a2e76 100644 --- a/plugins/aidd-qa/.claude-plugin/plugin.json +++ b/plugins/aidd-qa/.claude-plugin/plugin.json @@ -2,13 +2,14 @@ "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json", "name": "aidd-qa", "version": "1.0.0", - "description": "Acceptance QA: validates observable behavior against acceptance criteria and records reviewer evidence. Use when a reviewed candidate needs evidence that it meets its acceptance criteria. Do NOT use for diff review, unit or integration tests, or fixing the application.", + "description": "Acceptance QA: verifies observable behavior against acceptance criteria into a verdict report, and shows a journey as a short video. Use when a reviewed candidate needs a verdict on its acceptance criteria, or a journey needs a demonstration. Do NOT use for diff review, unit or integration tests, or fixing the application.", "author": { "name": "AI-Driven Dev", "url": "https://github.com/ai-driven-dev" }, "skills": [ - "./skills/01-acceptance-qa" + "./skills/01-acceptance-qa", + "./skills/02-demo" ], "keywords": [ "qa", diff --git a/plugins/aidd-qa/CATALOG.md b/plugins/aidd-qa/CATALOG.md index 189426589..6d403c053 100644 --- a/plugins/aidd-qa/CATALOG.md +++ b/plugins/aidd-qa/CATALOG.md @@ -9,6 +9,7 @@ Auto-generated index of skills, agents, references and assets shipped by the `ai - [`.claude-plugin`](#claude-plugin) - [`skills`](#skills) - [`skills/01-acceptance-qa`](#skills01-acceptance-qa) + - [`skills/02-demo`](#skills02-demo) --- @@ -30,5 +31,13 @@ Auto-generated index of skills, agents, references and assets shipped by the `ai | `actions` | [04-run-scenarios.md](skills/01-acceptance-qa/actions/04-run-scenarios.md) | - | | `assets` | [qa-report-template.md](skills/01-acceptance-qa/assets/qa-report-template.md) | - | | `references` | [interface-browser-playwright-cli.md](skills/01-acceptance-qa/references/interface-browser-playwright-cli.md) | - | -| `-` | [SKILL.md](skills/01-acceptance-qa/SKILL.md) | `Validate a reviewed candidate's observable behavior against its acceptance criteria and record short named videos as reviewer evidence. Use when acceptance criteria exist and a browser-observable journey needs proof it holds. Do NOT use for diff review, unit or integration tests, or application fixes.` | +| `-` | [SKILL.md](skills/01-acceptance-qa/SKILL.md) | `Validate a reviewed candidate's observable behavior against its acceptance criteria into a verdict report. Use when acceptance criteria exist and a browser-observable journey needs proof it holds. Do NOT use for diff review, unit or integration tests, or application fixes.` | + +#### `skills/02-demo` + +| Group | File | Description | +|-------|------|---| +| `actions` | [01-record.md](skills/02-demo/actions/01-record.md) | - | +| `references` | [interface-browser-playwright-cli.md](skills/02-demo/references/interface-browser-playwright-cli.md) | - | +| `-` | [SKILL.md](skills/02-demo/SKILL.md) | `Show a journey in a browser as one short named video labelled with the observed outcome. Use when a journey, such as a qa.md scenario, needs a demonstration a reviewer can watch. Do NOT use to verify acceptance criteria, review a diff, or fix the application.` | diff --git a/plugins/aidd-qa/README.md b/plugins/aidd-qa/README.md index 575ce2064..6348080b6 100644 --- a/plugins/aidd-qa/README.md +++ b/plugins/aidd-qa/README.md @@ -8,10 +8,11 @@ Acceptance QA concern for the AI-Driven Development framework. First time? Install with `/plugin install aidd-qa@aidd-framework`, then run `aidd-qa:01-acceptance-qa`. -Validates a reviewed candidate's observable behavior against its acceptance criteria and records reviewer evidence. Scenarios come only from acceptance criteria, never from the diff or the source code. Browser is the only supported interface today; a scenario without a browser-observable outcome is listed out of interface rather than tested. +Verifies a reviewed candidate's observable behavior against its acceptance criteria into a verdict report, and shows a journey as a short video. Scenarios come only from acceptance criteria, never from the diff or the source code. Browser is the only supported interface today; a scenario without a browser-observable outcome is listed out of interface rather than tested. ## Skills | Skill | Description | |---|---| -| [acceptance-qa](skills/01-acceptance-qa/SKILL.md) | Validate a reviewed candidate against its acceptance criteria and record one short named video per locked scenario. | +| [acceptance-qa](skills/01-acceptance-qa/SKILL.md) | Validate a reviewed candidate against its acceptance criteria into a verdict report. | +| [demo](skills/02-demo/SKILL.md) | Show a journey as one short video labelled with the observed outcome. | diff --git a/plugins/aidd-qa/skills/01-acceptance-qa/SKILL.md b/plugins/aidd-qa/skills/01-acceptance-qa/SKILL.md index 6d4f83e31..059380428 100644 --- a/plugins/aidd-qa/skills/01-acceptance-qa/SKILL.md +++ b/plugins/aidd-qa/skills/01-acceptance-qa/SKILL.md @@ -1,6 +1,6 @@ --- name: 01-acceptance-qa -description: Validate a reviewed candidate's observable behavior against its acceptance criteria and record short named videos as reviewer evidence. Use when acceptance criteria exist and a browser-observable journey needs proof it holds. Do NOT use for diff review, unit or integration tests, or application fixes. +description: Validate a reviewed candidate's observable behavior against its acceptance criteria into a verdict report. Use when acceptance criteria exist and a browser-observable journey needs proof it holds. Do NOT use for diff review, unit or integration tests, or application fixes. argument-hint: acceptance criteria | reviewed candidate --- @@ -19,9 +19,9 @@ Read only the next action's file before running it. | # | Action | Does | | --- | --------------- | ---------------------------------------------------------- | | 01 | `load-scope` | Lock at most one happy path and the edge cases its acceptance criteria name | -| 02 | `prerequisites` | Verify the browser runner and media dependencies | +| 02 | `prerequisites` | Verify the browser runner | | 03 | `prepare-run` | Resolve the shortest deterministic path to executable runs | -| 04 | `run-scenarios` | Record, normalize, verify, reset, and report every scenario | +| 04 | `run-scenarios` | Run, verdict, reset, and report every scenario | ## Transversal rules diff --git a/plugins/aidd-qa/skills/01-acceptance-qa/actions/02-prerequisites.md b/plugins/aidd-qa/skills/01-acceptance-qa/actions/02-prerequisites.md index dee1bf07f..e6a434347 100644 --- a/plugins/aidd-qa/skills/01-acceptance-qa/actions/02-prerequisites.md +++ b/plugins/aidd-qa/skills/01-acceptance-qa/actions/02-prerequisites.md @@ -8,17 +8,17 @@ The locked scope. ## Output -Verified `npx`, Playwright CLI `0.1.17`, `ffmpeg`, and `ffprobe`. +Verified `npx` and Playwright CLI `0.1.17`. ## Process -1. **Check.** In one pass, resolve `npx`, run `npx --yes @playwright/cli@0.1.17 --version`, and resolve `ffmpeg` and `ffprobe`. +1. **Check.** In one pass, resolve `npx` and run `npx --yes @playwright/cli@0.1.17 --version`. 2. **Continue.** When every check passes, continue without reporting it. 3. **Resolve.** When a dependency is missing, ask one concise question: the user installs the listed dependencies, or authorizes you to install them now. - If the user installs them, provide only the shortest platform-appropriate commands and stop until they confirm completion. - If authorized, install only the missing dependencies, then rerun every check. 4. **Stop.** Report the shortest decisive error when installation is declined or a recheck fails. -5. **Protect.** Never add runner or media dependencies to the application manifest. +5. **Protect.** Never add runner dependencies to the application manifest. ## Test diff --git a/plugins/aidd-qa/skills/01-acceptance-qa/actions/03-prepare-run.md b/plugins/aidd-qa/skills/01-acceptance-qa/actions/03-prepare-run.md index f2f9f495d..d6b057f12 100644 --- a/plugins/aidd-qa/skills/01-acceptance-qa/actions/03-prepare-run.md +++ b/plugins/aidd-qa/skills/01-acceptance-qa/actions/03-prepare-run.md @@ -1,6 +1,6 @@ # 03 - Prepare Run -Make every scenario executable before recording. +Make every scenario executable before running. ## Input @@ -15,7 +15,7 @@ Per scenario: fixture, initial URL, minimal steps, expected outcome, proven tear 1. **Reuse.** Resolve entry, auth, fixtures, and reset from the `Browser QA` section of `aidd_docs/memory/testing.md` when it exists, then a related browser test, then one targeted snapshot. Stop once executable. - Before starting the entry, prove the storage it mounts is test-only when its reset deletes data. Unproven: ask once, never guess. 2. **Preflight.** Check the application and fixed `1280×720` viewport. -3. **Authenticate.** Establish the role before recording; never show login discovery or secrets in evidence. +3. **Authenticate.** Establish the role before running; never put login discovery or secrets in the report. 4. **Fixture.** Use deterministic data per setup; never pick a live record by guesswork. 5. **Rehearse.** Only non-mutating steps and selectors; never the final state-changing action. 6. **Reset.** Resolve a verified, executable teardown per state-changing scenario; without one, reject the scenario and carry it forward with its reason. If preparation changed state, run the teardown and verify the baseline now, not at a restart. @@ -24,5 +24,5 @@ Per scenario: fixture, initial URL, minimal steps, expected outcome, proven tear - An entry on storage not proven test-only is never started when its reset deletes data; it produces one question. - A state-changing scenario without a verified teardown is rejected with its reason. -- No login discovery, secret, or guessed live record appears in evidence. +- No login discovery, secret, or guessed live record appears in the report. - Preparation that changed state is torn down and verified before the run is ready. diff --git a/plugins/aidd-qa/skills/01-acceptance-qa/actions/04-run-scenarios.md b/plugins/aidd-qa/skills/01-acceptance-qa/actions/04-run-scenarios.md index 99867f18d..f7334ae05 100644 --- a/plugins/aidd-qa/skills/01-acceptance-qa/actions/04-run-scenarios.md +++ b/plugins/aidd-qa/skills/01-acceptance-qa/actions/04-run-scenarios.md @@ -1,6 +1,6 @@ # 04 - Run Scenarios -Record, verify, and report one clean take per scenario. +Run, judge, and report every scenario. ## Input @@ -8,24 +8,22 @@ The prepared run and its rejections; from load-scope, the source label, candidat ## Output -`/qa.md` and one final WebM per scenario that ran. +`/qa.md`. ## Process -1. **Group.** Steps 1-7 cover executable scenarios only. At most two read-only ones concurrently, in isolated sessions; state-changing ones sequentially. -2. **Record.** Apply setup, then follow [interface-browser-playwright-cli.md](../references/interface-browser-playwright-cli.md). -3. **Verdict.** From each step's `expected`, `actual`, `ok`: any `ok: false` => `fail`, else `pass`. Keep the take of every `fail`. -4. **Recover.** A setup failure, throw, or non-zero exit is operational: discard the take, reset, retry once; a second one => `blocked`. -5. **Reset.** After each state-changing take, run its verified teardown, verify the baseline, then close the session. -6. **Normalize.** At most two independent raw files concurrently. Save only `qa/happy-path.webm` and `qa/edge-case-.webm` once `ffprobe` and frame inspection pass; record each final file's duration from `ffprobe`. -7. **Clean.** Delete raw takes and validation frames only after every final file passes codec, dimension, duration, path, cut-point, and frame checks. Never retain screenshots or alternate media. -8. **Report.** Fill [qa-report-template.md](../assets/qa-report-template.md), even when every scenario was rejected: one row per scenario (evidence `none` when `blocked`), Out of interface and Rejected (`none` when empty), Findings only for `fail` or `blocked`. Run verdict, in order: any `fail` => `fail`; any `blocked` or rejected => `blocked`; else `pass`. `skipped` belongs to load-scope only. -9. **Return.** Output the verdict and evidence paths; when `qa/happy-path.webm` exists, ask `Open happy-path.webm in the browser for review?` and open it when confirmed. +1. **Group.** Steps 1-5 cover executable scenarios only. At most two read-only ones concurrently, in isolated sessions; state-changing ones sequentially. +2. **Run.** Apply setup, then follow [interface-browser-playwright-cli.md](../references/interface-browser-playwright-cli.md). +3. **Verdict.** From each step's `expected`, `actual`, `ok`: any `ok: false` => `fail`, else `pass`. +4. **Recover.** A setup failure, throw, or non-zero exit is operational: reset, retry once; a second one => `blocked`. +5. **Reset.** After each state-changing scenario, run its verified teardown, verify the baseline, then close the session. +6. **Report.** Fill [qa-report-template.md](../assets/qa-report-template.md), even when every scenario was rejected: one row per scenario, Out of interface and Rejected (`none` when empty), Findings only for `fail` or `blocked`. Run verdict, in order: any `fail` => `fail`; any `blocked` or rejected => `blocked`; else `pass`. `skipped` belongs to load-scope only. +7. **Return.** Output the verdict and the `qa.md` path. ## Test -- Each row names its criteria, expected, actual, verdict (`pass`, `fail`, or `blocked`), duration, and evidence. +- Each row names its criteria, expected, actual, and verdict (`pass`, `fail`, or `blocked`). - Run verdict: `fail` over `blocked` or rejected over `pass`; a `pass` is never reported without the Out of interface list. -- A product mismatch yields `fail` from the returned results, never a thrown take; a second operational failure yields `blocked`. -- Raw takes and frames survive until every final file passes; then only `qa/happy-path.webm` and `qa/edge-case-.webm` remain beside `qa.md`. +- A product mismatch yields `fail` from the returned results, never a thrown run; a second operational failure yields `blocked`. +- Nothing but `qa.md` is written to the evidence folder. - Every earlier rejection appears in the report with its reason, even when it empties the executable set, and the report still runs. diff --git a/plugins/aidd-qa/skills/01-acceptance-qa/assets/qa-report-template.md b/plugins/aidd-qa/skills/01-acceptance-qa/assets/qa-report-template.md index a36355ceb..ed2655fff 100644 --- a/plugins/aidd-qa/skills/01-acceptance-qa/assets/qa-report-template.md +++ b/plugins/aidd-qa/skills/01-acceptance-qa/assets/qa-report-template.md @@ -11,8 +11,8 @@ ## Scenarios -| Scenario | Criteria | Expected | Actual | Verdict | Duration | Evidence | -| -------- | --------- | -------- | ------ | ------- | -------- | -------- | +| Scenario | Criteria | Expected | Actual | Verdict | +| -------- | -------- | -------- | ------ | ------- | ## Rejected diff --git a/plugins/aidd-qa/skills/01-acceptance-qa/references/interface-browser-playwright-cli.md b/plugins/aidd-qa/skills/01-acceptance-qa/references/interface-browser-playwright-cli.md index 0f4535309..8e19c2724 100644 --- a/plugins/aidd-qa/skills/01-acceptance-qa/references/interface-browser-playwright-cli.md +++ b/plugins/aidd-qa/skills/01-acceptance-qa/references/interface-browser-playwright-cli.md @@ -1,6 +1,6 @@ # Playwright CLI runner -Playwright Agent CLI produces WebM evidence without becoming an application dependency. +Playwright Agent CLI drives the browser without becoming an application dependency. ## Version and invocation @@ -12,89 +12,24 @@ npx --yes @playwright/cli@0.1.17 -s=qa- Run every command from a temporary directory outside the application repository. -`run-code` takes one `page` argument: pass `async page => { ... }`, never bare statements. Never throw on a product mismatch: check each expected outcome with a bounded `waitFor`, and return `{ step, expected, actual, ok }` per step. Keep stdout, stderr, and exit status visible; no redirects, pipes, command substitutions, or `|| true` in runner commands. A throw, `SyntaxError`, or non-zero exit is a tooling failure and invalidates the take. +## Run-code contract -## Recording +`run-code` takes one `page` argument: pass `async page => { ... }`, never bare statements. Never throw on a product mismatch: check each expected outcome with a bounded `waitFor`, and return `{ step, expected, actual, ok }` per step. Keep stdout, stderr, and exit status visible; no redirects, pipes, command substitutions, or `|| true` in runner commands. A throw, `SyntaxError`, or non-zero exit is a tooling failure and invalidates the run. -- One named session and raw WebM per scenario; viewport and frame `1280×720`. -- Reach the initial state before `video-start`; put every recorded interaction in one `run-code`. -- Never record separate click, fill, or scroll commands. -- Hold the initial and verified final states for `1000 ms`. -- Wait `300 ms` between actions and after each scroll. -- Run `video-stop` only after `run-code` succeeds. +## Scenario + +One named session per scenario, viewport `1280×720`. Reach the initial state, then put every step in one `run-code`. ```bash npx --yes @playwright/cli@0.1.17 -s=qa-- resize 1280 720 -npx --yes @playwright/cli@0.1.17 -s=qa-- video-start raw-.webm --size=1280x720 -# Replace the locators; omit the scroll block when irrelevant. +# Replace the locators and add one result per step. npx --yes @playwright/cli@0.1.17 -s=qa-- run-code 'async page => { - const pause = milliseconds => page.waitForTimeout(milliseconds); - await pause(1000); await page.getByRole("button", { name: "first action" }).click(); - await pause(300); - await page.mouse.wheel(0, 600); - await pause(300); await page.getByRole("button", { name: "final action" }).click(); const expected = "observable expected outcome"; const ok = await page.getByText(expected).waitFor({ state: "visible", timeout: 5000 }).then(() => true, () => false); - await pause(1000); return [{ step: "final action", expected, actual: ok ? expected : "not visible", ok }]; }' -npx --yes @playwright/cli@0.1.17 -s=qa-- video-stop # Run the teardown and verify the baseline before closing. npx --yes @playwright/cli@0.1.17 -s=qa-- close ``` - -`video-stop` writes the raw file to the current directory. Name raw files `raw-happy-path.webm` or `raw-edge-case-.webm`. - -## Duration - -Probe each raw take once. Maximum 12 seconds, never a target; do not pad or extend. - -```bash -ffprobe -v error -select_streams v:0 \ - -show_entries stream=codec_name,width,height:format=duration \ - -of json -``` - -Without `ffmpeg`, only an already-short raw take passes; otherwise report `blocked: media-postprocess-unavailable`. - -Never trim the head or infer the first action from scene changes. Inspect both sides of a tail cut; reject one that removes the result or final one-second hold. - -```bash -qa_cut_frames=$(mktemp -d) -ffmpeg -v error -ss -i raw-.webm \ - -frames:v 1 "$qa_cut_frames/before.png" -ffmpeg -v error -ss -i raw-.webm \ - -frames:v 1 "$qa_cut_frames/after.png" -``` - -## Normalization - -Re-encode without audio. Use only a visually verified tail cut; omit `-to` when unused. - -```bash -ffmpeg -y -to -i raw-.webm \ - -an -vf "fps=12,scale=1280:-2:force_original_aspect_ratio=decrease" \ - -c:v libvpx-vp9 -crf 36 -b:v 0 .webm -``` - -Above 12 seconds, remove non-scenario actions and record again. Never accelerate. Block when the required journey cannot fit. - -## Validation - -Probe the final file, then inspect four frames per second chronologically. - -```bash -qa_final_frames=$(mktemp -d) -ffmpeg -v error -i .webm -vf "fps=4" \ - "$qa_final_frames/frame-%03d.png" -``` - -Require: - -- codec `vp9`, width `1280`, duration at most 12 seconds; -- initial state, every action and transition, and final result visible; -- initial and final holds spanning at least four sampled frames. - -`ffprobe` alone is insufficient. Delete raw takes, cut-point frames, and validation frames only after every final WebM passes both checks. diff --git a/plugins/aidd-qa/skills/02-demo/SKILL.md b/plugins/aidd-qa/skills/02-demo/SKILL.md new file mode 100644 index 000000000..533061431 --- /dev/null +++ b/plugins/aidd-qa/skills/02-demo/SKILL.md @@ -0,0 +1,29 @@ +--- +name: 02-demo +description: Show a journey in a browser as one short named video labelled with the observed outcome. Use when a journey, such as a qa.md scenario, needs a demonstration a reviewer can watch. Do NOT use to verify acceptance criteria, review a diff, or fix the application. +argument-hint: journey | candidate +--- + +# Demo + +```mermaid +flowchart LR + journey["journey + candidate"] --> record["record"] --> videos["one video per journey"] +``` + +## Actions + +Read only the next action's file before running it. + +| # | Action | Does | +| --- | -------- | --------------------------------------------------------- | +| 01 | `record` | Record, normalize, validate, and label one video per journey | + +## Transversal rules + +- Run against a candidate and never patch the application. +- Never delete data not proven test-only. +- Never invent a step or an expected outcome; both come from the journey. +- Never decide where a video goes; the caller does. +- Never spawn agents. Keep state-changing browser work sequential. +- Do not narrate transitions or successful checks. Report only a blocker, a required decision, or the final paths. diff --git a/plugins/aidd-qa/skills/02-demo/actions/01-record.md b/plugins/aidd-qa/skills/02-demo/actions/01-record.md new file mode 100644 index 000000000..459b05b05 --- /dev/null +++ b/plugins/aidd-qa/skills/02-demo/actions/01-record.md @@ -0,0 +1,32 @@ +# 01 - Record + +Record one clean, short, validated take per journey. + +## Input + +A journey (steps and expected outcome, such as a `qa.md` scenario) and a candidate (branch, commit, or running URL). + +## Output + +One final WebM per journey, named `happy-path.webm` or `edge-case-.webm`, labelled with the observed outcome (`pass`, `fail`, or `blocked`). + +## Process + +1. **Check.** In one pass, resolve `npx`, run `npx --yes @playwright/cli@0.1.17 --version`, and resolve `ffmpeg` and `ffprobe`. Continue silently when all pass. +2. **Resolve.** A dependency is missing: ask once whether the user installs it or authorizes you to. If authorized, install only the missing ones and recheck. If declined or a recheck fails, stop with the shortest decisive error. Never add them to the application manifest. +3. **Prepare.** Reach the journey's initial state with deterministic data and the role authenticated off-camera. A state-changing journey without a verified teardown is not recorded. +4. **Record.** Follow [interface-browser-playwright-cli.md](../references/interface-browser-playwright-cli.md). Keep the take of a mismatch. +5. **Label.** From the returned step results: any `ok: false` => `fail`, else `pass`. +6. **Recover.** A setup failure, throw, or non-zero exit discards the take; reset and retry once. A second one => `blocked`. +7. **Reset.** After each state-changing take, run its teardown, verify the baseline, then close the session. +8. **Normalize.** At most two independent raw files concurrently. Keep only a file whose `ffprobe` and frame inspection pass. +9. **Clean.** Delete raw takes and validation frames only after every final file passes codec, dimension, duration, path, cut-point, and frame checks. Never retain screenshots or alternate media. +10. **Return.** Output each video path, its label and duration. When `happy-path.webm` exists, ask `Open happy-path.webm in the browser for review?` and open it when confirmed. + +## Test + +- Passing checks produce no message; a missing dependency produces one choice and no unapproved installation. +- A product mismatch yields a `fail` label from the returned results, never a thrown take; a second operational failure yields `blocked`. +- Raw takes and frames survive until every final file passes; then only the final WebM files remain. +- A take over 12 seconds is re-recorded without non-journey actions, never accelerated. +- No login discovery, secret, or guessed live record appears in a video. diff --git a/plugins/aidd-qa/skills/02-demo/references/interface-browser-playwright-cli.md b/plugins/aidd-qa/skills/02-demo/references/interface-browser-playwright-cli.md new file mode 100644 index 000000000..e597dc2c2 --- /dev/null +++ b/plugins/aidd-qa/skills/02-demo/references/interface-browser-playwright-cli.md @@ -0,0 +1,98 @@ +# Playwright CLI recorder + +Playwright Agent CLI produces WebM takes without becoming an application dependency. + +## Invocation + +Execute the pin `@playwright/cli@0.1.17`, from a temporary directory outside the application repository; never `latest`. + +```bash +npx --yes @playwright/cli@0.1.17 -s=demo- +``` + +Pass `run-code` one function, `async page => { ... }`. A product mismatch never throws: bound each expected outcome with a `waitFor` and return `{ step, expected, actual, ok }` per step. A throw, `SyntaxError`, or non-zero exit invalidates the take. Leave stdout, stderr, and exit status visible; no redirects, pipes, command substitutions, or `|| true`. + +## Recording + +- One named session and raw WebM per journey; viewport and frame `1280×720`. +- Reach the initial state before `video-start`; put every recorded interaction in one `run-code`. +- Never record separate click, fill, or scroll commands. +- Hold the initial and verified final states for `1000 ms`. +- Wait `300 ms` between actions and after each scroll. +- Run `video-stop` only after `run-code` succeeds. + +```bash +npx --yes @playwright/cli@0.1.17 -s=demo-- resize 1280 720 +npx --yes @playwright/cli@0.1.17 -s=demo-- video-start raw-.webm --size=1280x720 +# Replace the locators; omit the scroll block when irrelevant. +npx --yes @playwright/cli@0.1.17 -s=demo-- run-code 'async page => { + const pause = milliseconds => page.waitForTimeout(milliseconds); + await pause(1000); + await page.getByRole("button", { name: "first action" }).click(); + await pause(300); + await page.mouse.wheel(0, 600); + await pause(300); + await page.getByRole("button", { name: "final action" }).click(); + const expected = "observable expected outcome"; + const ok = await page.getByText(expected).waitFor({ state: "visible", timeout: 5000 }).then(() => true, () => false); + await pause(1000); + return [{ step: "final action", expected, actual: ok ? expected : "not visible", ok }]; +}' +npx --yes @playwright/cli@0.1.17 -s=demo-- video-stop +# Run the teardown and verify the baseline before closing. +npx --yes @playwright/cli@0.1.17 -s=demo-- close +``` + +`video-stop` writes the raw file to the current directory. Name raw files `raw-happy-path.webm` or `raw-edge-case-.webm`. + +## Duration + +Probe each raw take once. Maximum 12 seconds, never a target; do not pad or extend. + +```bash +ffprobe -v error -select_streams v:0 \ + -show_entries stream=codec_name,width,height:format=duration \ + -of json +``` + +Without `ffmpeg`, only an already-short raw take passes; otherwise report `blocked: media-postprocess-unavailable`. + +Never trim the head or infer the first action from scene changes. Inspect both sides of a tail cut; reject one that removes the result or final one-second hold. + +```bash +qa_cut_frames=$(mktemp -d) +ffmpeg -v error -ss -i raw-.webm \ + -frames:v 1 "$qa_cut_frames/before.png" +ffmpeg -v error -ss -i raw-.webm \ + -frames:v 1 "$qa_cut_frames/after.png" +``` + +## Normalization + +Re-encode without audio. Use only a visually verified tail cut; omit `-to` when unused. + +```bash +ffmpeg -y -to -i raw-.webm \ + -an -vf "fps=12,scale=1280:-2:force_original_aspect_ratio=decrease" \ + -c:v libvpx-vp9 -crf 36 -b:v 0 .webm +``` + +Above 12 seconds, remove non-scenario actions and record again. Never accelerate. Block when the required journey cannot fit. + +## Validation + +Probe the final file, then inspect four frames per second chronologically. + +```bash +qa_final_frames=$(mktemp -d) +ffmpeg -v error -i .webm -vf "fps=4" \ + "$qa_final_frames/frame-%03d.png" +``` + +Require: + +- codec `vp9`, width `1280`, duration at most 12 seconds; +- initial state, every action and transition, and final result visible; +- initial and final holds spanning at least four sampled frames. + +`ffprobe` alone is insufficient. Delete raw takes, cut-point frames, and validation frames only after every final WebM passes both checks. diff --git a/scripts/__tests__/aidd-qa-verify-and-demo-split.test.js b/scripts/__tests__/aidd-qa-verify-and-demo-split.test.js new file mode 100644 index 000000000..838c8654a --- /dev/null +++ b/scripts/__tests__/aidd-qa-verify-and-demo-split.test.js @@ -0,0 +1,34 @@ +const assert = require("node:assert/strict"); +const fs = require("node:fs"); +const path = require("node:path"); +const test = require("node:test"); + +const qa = path.resolve(__dirname, "../../plugins/aidd-qa"); +const verify = path.join(qa, "skills/01-acceptance-qa"); +const demo = path.join(qa, "skills/02-demo"); + +const filesOf = (dir) => + fs.readdirSync(dir, { recursive: true, withFileTypes: true }) + .filter((entry) => entry.isFile()) + .map((entry) => path.join(entry.parentPath, entry.name)); + +const textOf = (dir) => filesOf(dir).map((file) => fs.readFileSync(file, "utf8")).join("\n"); + +test("verify carries no media", () => { + const offenders = filesOf(verify).filter((file) => + /ffmpeg|ffprobe|webm|video/i.test(fs.readFileSync(file, "utf8")), + ); + assert.deepEqual(offenders.map((file) => path.relative(qa, file)), []); +}); + +test("demo owns the duration cap, the VP9 encode and frame validation", () => { + const text = textOf(demo); + assert.match(text, /12 seconds/); + assert.match(text, /libvpx-vp9/); + assert.match(text, /fps=4/); +}); + +test("the plugin lists verify and demo", () => { + const manifest = JSON.parse(fs.readFileSync(path.join(qa, ".claude-plugin/plugin.json"), "utf8")); + assert.deepEqual(manifest.skills, ["./skills/01-acceptance-qa", "./skills/02-demo"]); +}); diff --git a/scripts/__tests__/architecture-rules.test.js b/scripts/__tests__/architecture-rules.test.js index 33f5666fb..d4cba7e32 100644 --- a/scripts/__tests__/architecture-rules.test.js +++ b/scripts/__tests__/architecture-rules.test.js @@ -229,7 +229,7 @@ test("sweeping the repository's own plugins/ tree yields zero violations", () => assert.deepEqual(violations, []); // A sweep that never actually exercised a skill with action files would pass the same way — // this pins the sweep to the measured count so it cannot go vacuously green. - assert.equal(skillsWithActions, 49); + assert.equal(skillsWithActions, 50); }); test("a table that declares no action column cites nothing, whatever its cells read", () => { From d0cbf3f781604543b183d298fb3a82c693a58d38 Mon Sep 17 00:00:00 2001 From: Baptiste LAFOURCADE Date: Mon, 12 Oct 2026 22:28:23 +0200 Subject: [PATCH 2/9] feat(aidd-qa): verify real journeys without improvising Real runs on a fixture app showed acceptance QA improvising where its contract was silent. It now: - runs one happy path per journey, where a second journey used to be rejected untested; - verifies a CLI criterion by its command, output and exit code, and leaves an API alone out of interface; - never asks when another skill or agent invoked it, returning `blocked` with the question, and always writes `qa.md`; - proves test-only storage in its own step before any start, by memory, an isolated fixture or the user, never by an environment name or inference; - allows one fallback entry only when memory or a browser test names it; - resolves viewport and permissions from the criterion, then memory, then the default; - defines one observable condition for a vague outcome, probes a timed boundary on both sides, and measures a short transition inside `run-code`; - defines what is destructive, and records `Limitations` without a new status. Measured with `claude -p` against this tree, each case graded against an expected result written before the run: - signup golden 3/3 unchanged; todo golden 3/3 with the planted bug failing; - API-only spec `skipped` 2/2; CLI pass and planted fail 2/2; - toast within 2 s pass and over 2 s fail 2/2, with measured times; - vague "flash" 3/3 measured as a style change and revert; - 390x844 viewport 2/2; mobile without dimensions `blocked` 2/2; - unproven storage `blocked` 3/3 with the app never started; - crashing entry `blocked` 2/2; documented fallback `pass` 2/2. The skill holds 1821 words, against 1521 after the split. Refs #918 Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VE16AxC1V9wAMGmGgefe4z --- .../2026_10_12_qa-verify-and-demo/phase-2.md | 2 +- plugins/aidd-qa/CATALOG.md | 2 +- .../aidd-qa/skills/01-acceptance-qa/SKILL.md | 7 +++-- .../01-acceptance-qa/actions/01-load-scope.md | 12 ++++---- .../actions/02-prerequisites.md | 2 +- .../actions/03-prepare-run.md | 18 +++++------ .../actions/04-run-scenarios.md | 4 +-- .../assets/qa-report-template.md | 6 +++- .../interface-browser-playwright-cli.md | 10 +++++-- .../__tests__/aidd-qa-verify-rules.test.js | 30 +++++++++++++++++++ 10 files changed, 66 insertions(+), 27 deletions(-) create mode 100644 scripts/__tests__/aidd-qa-verify-rules.test.js diff --git a/aidd_docs/tasks/2026_10/2026_10_12_qa-verify-and-demo/phase-2.md b/aidd_docs/tasks/2026_10/2026_10_12_qa-verify-and-demo/phase-2.md index a0f1d4cc4..b5715c2fe 100644 --- a/aidd_docs/tasks/2026_10/2026_10_12_qa-verify-and-demo/phase-2.md +++ b/aidd_docs/tasks/2026_10/2026_10_12_qa-verify-and-demo/phase-2.md @@ -1,5 +1,5 @@ --- -status: pending +status: done --- # Instruction: verify-rules diff --git a/plugins/aidd-qa/CATALOG.md b/plugins/aidd-qa/CATALOG.md index 6d403c053..1883940a3 100644 --- a/plugins/aidd-qa/CATALOG.md +++ b/plugins/aidd-qa/CATALOG.md @@ -31,7 +31,7 @@ Auto-generated index of skills, agents, references and assets shipped by the `ai | `actions` | [04-run-scenarios.md](skills/01-acceptance-qa/actions/04-run-scenarios.md) | - | | `assets` | [qa-report-template.md](skills/01-acceptance-qa/assets/qa-report-template.md) | - | | `references` | [interface-browser-playwright-cli.md](skills/01-acceptance-qa/references/interface-browser-playwright-cli.md) | - | -| `-` | [SKILL.md](skills/01-acceptance-qa/SKILL.md) | `Validate a reviewed candidate's observable behavior against its acceptance criteria into a verdict report. Use when acceptance criteria exist and a browser-observable journey needs proof it holds. Do NOT use for diff review, unit or integration tests, or application fixes.` | +| `-` | [SKILL.md](skills/01-acceptance-qa/SKILL.md) | `Validate a reviewed candidate's observable behavior against its acceptance criteria into a verdict report. Use when acceptance criteria exist and a journey observable through a browser or a command needs proof it holds. Do NOT use for diff review, unit or integration tests, or application fixes.` | #### `skills/02-demo` diff --git a/plugins/aidd-qa/skills/01-acceptance-qa/SKILL.md b/plugins/aidd-qa/skills/01-acceptance-qa/SKILL.md index 059380428..476ff4e06 100644 --- a/plugins/aidd-qa/skills/01-acceptance-qa/SKILL.md +++ b/plugins/aidd-qa/skills/01-acceptance-qa/SKILL.md @@ -1,6 +1,6 @@ --- name: 01-acceptance-qa -description: Validate a reviewed candidate's observable behavior against its acceptance criteria into a verdict report. Use when acceptance criteria exist and a browser-observable journey needs proof it holds. Do NOT use for diff review, unit or integration tests, or application fixes. +description: Validate a reviewed candidate's observable behavior against its acceptance criteria into a verdict report. Use when acceptance criteria exist and a journey observable through a browser or a command needs proof it holds. Do NOT use for diff review, unit or integration tests, or application fixes. argument-hint: acceptance criteria | reviewed candidate --- @@ -18,7 +18,7 @@ Read only the next action's file before running it. | # | Action | Does | | --- | --------------- | ---------------------------------------------------------- | -| 01 | `load-scope` | Lock at most one happy path and the edge cases its acceptance criteria name | +| 01 | `load-scope` | Lock one happy path per journey and the edge cases its acceptance criteria name | | 02 | `prerequisites` | Verify the browser runner | | 03 | `prepare-run` | Resolve the shortest deterministic path to executable runs | | 04 | `run-scenarios` | Run, verdict, reset, and report every scenario | @@ -26,7 +26,8 @@ Read only the next action's file before running it. ## Transversal rules - Run against a reviewed change and never patch the application. -- Never delete data not proven test-only. +- Invoked by another skill or agent, never ask: return `blocked` with the question. +- Every run ends with `qa.md`, a blocked one included. - Never derive a scenario from the diff or the source code; every scenario traces to an acceptance criterion. - Never spawn agents. Batch independent reads and tool checks, but keep state-changing browser work sequential. - Do not narrate action transitions, searches, fixtures, selectors, or successful checks. Report only a blocker, a required decision, or the final verdict and paths. diff --git a/plugins/aidd-qa/skills/01-acceptance-qa/actions/01-load-scope.md b/plugins/aidd-qa/skills/01-acceptance-qa/actions/01-load-scope.md index 94f0fad70..fb9d249fb 100644 --- a/plugins/aidd-qa/skills/01-acceptance-qa/actions/01-load-scope.md +++ b/plugins/aidd-qa/skills/01-acceptance-qa/actions/01-load-scope.md @@ -8,20 +8,20 @@ Acceptance criteria (issue, spec, user story, or the user) and the reviewed cand ## Output -- Scope: at most 1 happy path and its edge cases, each with the criteria it proves. -- Out of interface: criteria, or quoted parts, with no browser-observable outcome. +- Scope: one happy path per journey and its edge cases, each with the criteria it proves. +- Out of interface: criteria, or quoted parts, not observable through a browser or a command. - Rejected: scenario, criterion, reason. - Evidence folder, source label (criteria path), and candidate reference. ## Process 1. **Resolve.** Take criteria from the issue, spec, story, or user; never a plan. Note the candidate reference. -2. **Filter.** Keep browser-observable criteria; split a partial one and quote the rest out of interface. Never test by reading code. +2. **Filter.** Keep criteria observable through a browser or a command, never an API alone; split a partial one and quote the rest out of interface. Never test by reading code. 3. **Locate.** Use the source's feature folder, else `aidd_docs/tasks//_/`. 4. **Skip.** Nothing kept: write `qa.md` from [qa-report-template.md](../assets/qa-report-template.md) with verdict `skipped`, then stop. -5. **Lock.** 1 happy path from the primary journey; edge cases only when a criterion names them. An edge from the diff, code, tests, or an unmapped plan edge is never a candidate. Ask once when journeys conflict or an edge is external or destructive. -6. **Validate.** Reject a scenario missing a trigger or an observable outcome; keep the reason. -7. **Show.** Evidence folder, happy path, `Edge case | Criteria | Decision` table, Out of interface and Rejected when non-empty. Never repeat steps. +5. **Lock.** One happy path per journey; edge cases only when a criterion names them. An edge from the diff, code, tests, or an unmapped plan edge is never a candidate. Ask once when journeys conflict or an edge is external or destructive (alters data outside the scenario's fixture, or has an irreversible external effect; a verified teardown of scenario-owned test data is not). +6. **Validate.** Reject a scenario missing a trigger or an observable outcome; keep the reason. Define for a vague outcome ("flashes", "jumps") one condition the page shows, such as a style that changes then reverts; reject it only when no page change can express it. +7. **Show.** Only when a decision is needed: the happy path, `Edge case | Criteria | Decision` table, Out of interface and Rejected when non-empty. ## Test diff --git a/plugins/aidd-qa/skills/01-acceptance-qa/actions/02-prerequisites.md b/plugins/aidd-qa/skills/01-acceptance-qa/actions/02-prerequisites.md index e6a434347..21ba1d1ef 100644 --- a/plugins/aidd-qa/skills/01-acceptance-qa/actions/02-prerequisites.md +++ b/plugins/aidd-qa/skills/01-acceptance-qa/actions/02-prerequisites.md @@ -12,7 +12,7 @@ Verified `npx` and Playwright CLI `0.1.17`. ## Process -1. **Check.** In one pass, resolve `npx` and run `npx --yes @playwright/cli@0.1.17 --version`. +1. **Check.** When a scenario uses a browser, in one pass, resolve `npx` and run `npx --yes @playwright/cli@0.1.17 --version`. 2. **Continue.** When every check passes, continue without reporting it. 3. **Resolve.** When a dependency is missing, ask one concise question: the user installs the listed dependencies, or authorizes you to install them now. - If the user installs them, provide only the shortest platform-appropriate commands and stop until they confirm completion. diff --git a/plugins/aidd-qa/skills/01-acceptance-qa/actions/03-prepare-run.md b/plugins/aidd-qa/skills/01-acceptance-qa/actions/03-prepare-run.md index d6b057f12..9678e8c78 100644 --- a/plugins/aidd-qa/skills/01-acceptance-qa/actions/03-prepare-run.md +++ b/plugins/aidd-qa/skills/01-acceptance-qa/actions/03-prepare-run.md @@ -8,21 +8,21 @@ Verified prerequisites and the locked scope. ## Output -Per scenario: fixture, initial URL, minimal steps, expected outcome, proven teardown, and isolated session id. Plus every scenario rejected here, with its reason. +Per scenario: fixture, initial URL, minimal steps, expected outcome, proven teardown, and isolated session id, viewport, and permissions. Plus every scenario rejected here, with its reason. ## Process -1. **Reuse.** Resolve entry, auth, fixtures, and reset from the `Browser QA` section of `aidd_docs/memory/testing.md` when it exists, then a related browser test, then one targeted snapshot. Stop once executable. - - Before starting the entry, prove the storage it mounts is test-only when its reset deletes data. Unproven: ask once, never guess. -2. **Preflight.** Check the application and fixed `1280×720` viewport. -3. **Authenticate.** Establish the role before running; never put login discovery or secrets in the report. -4. **Fixture.** Use deterministic data per setup; never pick a live record by guesswork. -5. **Rehearse.** Only non-mutating steps and selectors; never the final state-changing action. -6. **Reset.** Resolve a verified, executable teardown per state-changing scenario; without one, reject the scenario and carry it forward with its reason. If preparation changed state, run the teardown and verify the baseline now, not at a restart. +1. **Reuse.** Read [the runner reference](../references/interface-browser-playwright-cli.md), then resolve entry, auth, fixtures, and reset from the `Browser QA` section of `aidd_docs/memory/testing.md` when it exists, then a related browser test. +2. **Prove.** A documented reset that deletes data: start nothing until its storage is proven test-only by `Browser QA` memory, an isolated fixture or config, or the user; never by an environment name or your own inference, whether this run resets or not. +3. **Preflight.** Start the entry in the background for the whole run and check it; resolve what is still missing with one targeted snapshot. An entry that crashes or serves no browser UI gets one fallback entry the same memory or browser test names (a file found in the tree is not one), else `blocked` with the shortest decisive error; never invent a start command. Take viewport and permissions from the criterion, then `Browser QA` memory, then `1280×720` and browser defaults, and record them in the prepared run; a mobile criterion without dimensions asks. +4. **Authenticate.** Establish the role before running; never put login discovery or secrets in the report. +5. **Fixture.** Use deterministic data per setup; never pick a live record by guesswork. +6. **Rehearse.** Only non-mutating steps and selectors; never the final state-changing action. +7. **Reset.** Resolve a verified, executable teardown per state-changing scenario; without one, reject the scenario and carry it forward with its reason. If preparation changed state, run the teardown and verify the baseline now, not at a restart. ## Test -- An entry on storage not proven test-only is never started when its reset deletes data; it produces one question. +- Storage not proven test-only: no entry starts and no snapshot is taken. - A state-changing scenario without a verified teardown is rejected with its reason. - No login discovery, secret, or guessed live record appears in the report. - Preparation that changed state is torn down and verified before the run is ready. diff --git a/plugins/aidd-qa/skills/01-acceptance-qa/actions/04-run-scenarios.md b/plugins/aidd-qa/skills/01-acceptance-qa/actions/04-run-scenarios.md index f7334ae05..79603753a 100644 --- a/plugins/aidd-qa/skills/01-acceptance-qa/actions/04-run-scenarios.md +++ b/plugins/aidd-qa/skills/01-acceptance-qa/actions/04-run-scenarios.md @@ -15,9 +15,9 @@ The prepared run and its rejections; from load-scope, the source label, candidat 1. **Group.** Steps 1-5 cover executable scenarios only. At most two read-only ones concurrently, in isolated sessions; state-changing ones sequentially. 2. **Run.** Apply setup, then follow [interface-browser-playwright-cli.md](../references/interface-browser-playwright-cli.md). 3. **Verdict.** From each step's `expected`, `actual`, `ok`: any `ok: false` => `fail`, else `pass`. -4. **Recover.** A setup failure, throw, or non-zero exit is operational: reset, retry once; a second one => `blocked`. +4. **Recover.** A setup failure, throw, or unexpected non-zero exit is operational: reset, retry once; a second one => `blocked`. 5. **Reset.** After each state-changing scenario, run its verified teardown, verify the baseline, then close the session. -6. **Report.** Fill [qa-report-template.md](../assets/qa-report-template.md), even when every scenario was rejected: one row per scenario, Out of interface and Rejected (`none` when empty), Findings only for `fail` or `blocked`. Run verdict, in order: any `fail` => `fail`; any `blocked` or rejected => `blocked`; else `pass`. `skipped` belongs to load-scope only. +6. **Report.** Fill [qa-report-template.md](../assets/qa-report-template.md), even when every scenario was rejected: one row per scenario, Out of interface and Rejected (`none` when empty), Findings only for `fail` or `blocked`. Run verdict, in order: any `fail` => `fail`; any `blocked` or rejected => `blocked`; else `pass`; a limitation that weakens the proof => `blocked`, any other stays listed under `Limitations`. `skipped` belongs to load-scope only. 7. **Return.** Output the verdict and the `qa.md` path. ## Test diff --git a/plugins/aidd-qa/skills/01-acceptance-qa/assets/qa-report-template.md b/plugins/aidd-qa/skills/01-acceptance-qa/assets/qa-report-template.md index ed2655fff..23a580c38 100644 --- a/plugins/aidd-qa/skills/01-acceptance-qa/assets/qa-report-template.md +++ b/plugins/aidd-qa/skills/01-acceptance-qa/assets/qa-report-template.md @@ -7,7 +7,7 @@ ## Out of interface -{{criteria, or quoted parts, with no browser-observable outcome, or "none"}} +{{criteria, or quoted parts, not observable through a browser or a command, or "none"}} ## Scenarios @@ -18,4 +18,8 @@ {{scenarios rejected in load-scope or prepare-run, criterion — reason, or "none"}} +## Limitations + +{{what the run could not cover, or "none"}} + {{findings-section-when-needed}} diff --git a/plugins/aidd-qa/skills/01-acceptance-qa/references/interface-browser-playwright-cli.md b/plugins/aidd-qa/skills/01-acceptance-qa/references/interface-browser-playwright-cli.md index 8e19c2724..5b6486f94 100644 --- a/plugins/aidd-qa/skills/01-acceptance-qa/references/interface-browser-playwright-cli.md +++ b/plugins/aidd-qa/skills/01-acceptance-qa/references/interface-browser-playwright-cli.md @@ -14,14 +14,14 @@ Run every command from a temporary directory outside the application repository. ## Run-code contract -`run-code` takes one `page` argument: pass `async page => { ... }`, never bare statements. Never throw on a product mismatch: check each expected outcome with a bounded `waitFor`, and return `{ step, expected, actual, ok }` per step. Keep stdout, stderr, and exit status visible; no redirects, pipes, command substitutions, or `|| true` in runner commands. A throw, `SyntaxError`, or non-zero exit is a tooling failure and invalidates the run. +`run-code` takes `page` as its only binding: pass `async page => { ... }`, never bare statements; browser globals such as `URL` go inside `page.evaluate`. Never throw on a product mismatch: check each expected outcome with a bounded `waitFor`, and return `{ step, expected, actual, ok }` per step. Probe a timed boundary on both sides and report the measured values in `actual`. Measure a transition shorter than any sampling inside the single `run-code` (bounded observation), never infer it. Keep stdout, stderr, and exit status visible; no redirects, pipes, command substitutions, or `|| true` in runner commands. A throw, `SyntaxError`, or non-zero exit is a tooling failure and invalidates the run. ## Scenario -One named session per scenario, viewport `1280×720`. Reach the initial state, then put every step in one `run-code`. +One named session per scenario, at the resolved viewport. Reach the initial state, then put every step in one `run-code`. ```bash -npx --yes @playwright/cli@0.1.17 -s=qa-- resize 1280 720 +npx --yes @playwright/cli@0.1.17 -s=qa-- resize # Replace the locators and add one result per step. npx --yes @playwright/cli@0.1.17 -s=qa-- run-code 'async page => { await page.getByRole("button", { name: "first action" }).click(); @@ -33,3 +33,7 @@ npx --yes @playwright/cli@0.1.17 -s=qa-- run-code 'async # Run the teardown and verify the baseline before closing. npx --yes @playwright/cli@0.1.17 -s=qa-- close ``` + +## Command scenario + +Run the command from the same temporary directory, stdout, stderr, and exit status visible. Compare stdout and exit code to the expected; return the same `{ step, expected, actual, ok }` row. diff --git a/scripts/__tests__/aidd-qa-verify-rules.test.js b/scripts/__tests__/aidd-qa-verify-rules.test.js new file mode 100644 index 000000000..bb49f8460 --- /dev/null +++ b/scripts/__tests__/aidd-qa-verify-rules.test.js @@ -0,0 +1,30 @@ +const assert = require("node:assert/strict"); +const fs = require("node:fs"); +const path = require("node:path"); +const test = require("node:test"); + +const verify = path.resolve(__dirname, "../../plugins/aidd-qa/skills/01-acceptance-qa"); +const read = (file) => fs.readFileSync(path.join(verify, file), "utf8"); + +const rules = [ + ["one happy path per journey", "actions/01-load-scope.md", /One happy path per journey/], + ["a criterion is observable through a browser or a command", "actions/01-load-scope.md", /Keep criteria observable through a browser or a command, never an API alone/], + ["a command scenario compares stdout and exit code", "references/interface-browser-playwright-cli.md", /Compare stdout and exit code/], + ["invoked by another skill or agent, never ask", "SKILL.md", /another skill or agent, never ask: return `blocked` with the question/], + ["test-only proof precedes the entry and any snapshot", "actions/03-prepare-run.md", /\*\*Prove\.\*\* A documented reset that deletes data: start nothing until its storage is proven test-only.*never by an environment name or your own inference[\s\S]*\*\*Preflight\.\*\* Start the entry/], + ["a crashed or UI-less entry gets one fallback, else blocked", "actions/03-prepare-run.md", /one fallback entry the same memory or browser test names \(a file found in the tree is not one\), else `blocked`.*never invent a start command/], + ["viewport and permissions resolve criterion, memory, default", "actions/03-prepare-run.md", /from the criterion, then `Browser QA` memory, then `1280×720`/], + ["a vague outcome becomes one observable condition", "actions/01-load-scope.md", /Define for a vague outcome .* one condition the page shows/], + ["a timed boundary is probed on both sides", "references/interface-browser-playwright-cli.md", /Probe a timed boundary on both sides/], + ["a short transition is measured inside run-code", "references/interface-browser-playwright-cli.md", /shorter than any sampling inside the single `run-code`/], + ["destructive is defined by the fixture boundary", "actions/01-load-scope.md", /verified teardown of scenario-owned test data is not/], + ["prepare-run reads the runner reference first", "actions/03-prepare-run.md", /1\. \*\*Reuse\.\*\* Read \[the runner reference\]/], + ["load-scope shows the scope only when a decision is needed", "actions/01-load-scope.md", /\*\*Show\.\*\* Only when a decision is needed/], + ["page is the only run-code binding", "references/interface-browser-playwright-cli.md", /`page` as its only binding.*inside `page\.evaluate`/], + ["a limitation that weakens the proof yields blocked", "actions/04-run-scenarios.md", /limitation that weakens the proof => `blocked`/], + ["the report holds Limitations", "assets/qa-report-template.md", /## Limitations/], +]; + +for (const [name, file, pattern] of rules) { + test(name, () => assert.match(read(file), pattern)); +} From 513731c0e26e856e4d8a9ba3bc8c520c9fc9abf5 Mon Sep 17 00:00:00 2001 From: Baptiste LAFOURCADE Date: Mon, 12 Oct 2026 22:36:03 +0200 Subject: [PATCH 3/9] feat(aidd-qa): show a browser or CLI journey, a bug included The demo now shows a journey through its own interface: a browser journey as one video, a CLI journey as a transcript of the command, its output and its exit status, so a CLI needs no ffmpeg. A failing journey is shown as observed and labelled `fail`, never staged. The artifact lands in the folder the caller names, else beside the journey's source, and is named after its journey, never `happy-path` for a bug. Invoked by another skill or agent, the demo asks nothing and returns paths and labels; it offers to open a video only when the user invoked it directly. Both routers take the `| Action | Does |` form. Measured with `claude -p` against this tree, 2 runs a case: - a `qa.md` scenario: video in the `qa/` folder beside it, named after the journey, VP9 under 12 s, `pass`; - a bug, invoked by an agent: video in the caller's folder, `fail`, no question; - a CLI journey, invoked by an agent: transcript in the caller's folder, `fail` on output `3` against `2`, no video, no question. The demo holds 1059 words, against 1078 after the split. Refs #918 Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VE16AxC1V9wAMGmGgefe4z --- .../2026_10_12_qa-verify-and-demo/phase-3.md | 2 +- plugins/aidd-qa/CATALOG.md | 2 +- .../aidd-qa/skills/01-acceptance-qa/SKILL.md | 12 ++--- plugins/aidd-qa/skills/02-demo/SKILL.md | 18 ++++---- .../skills/02-demo/actions/01-record.md | 39 ++++++++-------- .../interface-browser-playwright-cli.md | 2 +- .../aidd-qa-verify-and-demo-split.test.js | 45 +++++++++++++++++++ 7 files changed, 84 insertions(+), 36 deletions(-) diff --git a/aidd_docs/tasks/2026_10/2026_10_12_qa-verify-and-demo/phase-3.md b/aidd_docs/tasks/2026_10/2026_10_12_qa-verify-and-demo/phase-3.md index 83ff1c6f2..9f51c3361 100644 --- a/aidd_docs/tasks/2026_10/2026_10_12_qa-verify-and-demo/phase-3.md +++ b/aidd_docs/tasks/2026_10/2026_10_12_qa-verify-and-demo/phase-3.md @@ -1,5 +1,5 @@ --- -status: pending +status: done --- # Instruction: demo diff --git a/plugins/aidd-qa/CATALOG.md b/plugins/aidd-qa/CATALOG.md index 1883940a3..543b817e3 100644 --- a/plugins/aidd-qa/CATALOG.md +++ b/plugins/aidd-qa/CATALOG.md @@ -39,5 +39,5 @@ Auto-generated index of skills, agents, references and assets shipped by the `ai |-------|------|---| | `actions` | [01-record.md](skills/02-demo/actions/01-record.md) | - | | `references` | [interface-browser-playwright-cli.md](skills/02-demo/references/interface-browser-playwright-cli.md) | - | -| `-` | [SKILL.md](skills/02-demo/SKILL.md) | `Show a journey in a browser as one short named video labelled with the observed outcome. Use when a journey, such as a qa.md scenario, needs a demonstration a reviewer can watch. Do NOT use to verify acceptance criteria, review a diff, or fix the application.` | +| `-` | [SKILL.md](skills/02-demo/SKILL.md) | `Show a journey as one short named video (browser) or text transcript (CLI), labelled with the observed outcome. Use when a journey, such as a qa.md scenario, needs a demonstration a reviewer can watch or read. Do NOT use to verify acceptance criteria, review a diff, or fix the application.` | diff --git a/plugins/aidd-qa/skills/01-acceptance-qa/SKILL.md b/plugins/aidd-qa/skills/01-acceptance-qa/SKILL.md index 476ff4e06..8cef52306 100644 --- a/plugins/aidd-qa/skills/01-acceptance-qa/SKILL.md +++ b/plugins/aidd-qa/skills/01-acceptance-qa/SKILL.md @@ -16,12 +16,12 @@ flowchart LR Read only the next action's file before running it. -| # | Action | Does | -| --- | --------------- | ---------------------------------------------------------- | -| 01 | `load-scope` | Lock one happy path per journey and the edge cases its acceptance criteria name | -| 02 | `prerequisites` | Verify the browser runner | -| 03 | `prepare-run` | Resolve the shortest deterministic path to executable runs | -| 04 | `run-scenarios` | Run, verdict, reset, and report every scenario | +| Action | Does | +| --- | --- | +| load-scope | lock one happy path per journey and the edge cases its criteria name | +| prerequisites | verify the browser runner | +| prepare-run | resolve the shortest deterministic path to executable runs | +| run-scenarios | run, verdict, reset and report every scenario | ## Transversal rules diff --git a/plugins/aidd-qa/skills/02-demo/SKILL.md b/plugins/aidd-qa/skills/02-demo/SKILL.md index 533061431..616dcfaaf 100644 --- a/plugins/aidd-qa/skills/02-demo/SKILL.md +++ b/plugins/aidd-qa/skills/02-demo/SKILL.md @@ -1,6 +1,6 @@ --- name: 02-demo -description: Show a journey in a browser as one short named video labelled with the observed outcome. Use when a journey, such as a qa.md scenario, needs a demonstration a reviewer can watch. Do NOT use to verify acceptance criteria, review a diff, or fix the application. +description: Show a journey as one short named video (browser) or text transcript (CLI), labelled with the observed outcome. Use when a journey, such as a qa.md scenario, needs a demonstration a reviewer can watch or read. Do NOT use to verify acceptance criteria, review a diff, or fix the application. argument-hint: journey | candidate --- @@ -8,22 +8,22 @@ argument-hint: journey | candidate ```mermaid flowchart LR - journey["journey + candidate"] --> record["record"] --> videos["one video per journey"] + journey["journey + candidate"] --> record["record"] --> artifact["one artifact per journey"] ``` ## Actions -Read only the next action's file before running it. +Read only the action's file before running it. -| # | Action | Does | -| --- | -------- | --------------------------------------------------------- | -| 01 | `record` | Record, normalize, validate, and label one video per journey | +| Action | Does | +| --- | --- | +| record | record, validate and label one artifact per journey | ## Transversal rules - Run against a candidate and never patch the application. - Never delete data not proven test-only. - Never invent a step or an expected outcome; both come from the journey. -- Never decide where a video goes; the caller does. -- Never spawn agents. Keep state-changing browser work sequential. -- Do not narrate transitions or successful checks. Report only a blocker, a required decision, or the final paths. +- Invoked by another skill or agent, never ask: return `blocked` with the question. +- Never spawn agents. Keep state-changing work sequential. +- Do not narrate. Report only a blocker, a required decision, or the final paths and labels. diff --git a/plugins/aidd-qa/skills/02-demo/actions/01-record.md b/plugins/aidd-qa/skills/02-demo/actions/01-record.md index 459b05b05..6cbabbe14 100644 --- a/plugins/aidd-qa/skills/02-demo/actions/01-record.md +++ b/plugins/aidd-qa/skills/02-demo/actions/01-record.md @@ -1,32 +1,35 @@ # 01 - Record -Record one clean, short, validated take per journey. +Record one validated artifact per journey. ## Input -A journey (steps and expected outcome, such as a `qa.md` scenario) and a candidate (branch, commit, or running URL). +A journey (steps and expected outcome, such as a `qa.md` scenario), a candidate (branch, commit, or running URL), and optionally a destination folder. ## Output -One final WebM per journey, named `happy-path.webm` or `edge-case-.webm`, labelled with the observed outcome (`pass`, `fail`, or `blocked`). +One artifact named after its `journey slug`, never `happy-path` for a bug repro: a browser journey yields a WebM, a CLI journey a `.txt` transcript of each command, its output and its exit status. Labelled `pass`, `fail` (shown as observed, not expected) or `blocked`. + +It lands in the caller's folder, else beside the journey's source (the `qa/` folder beside `qa.md`), never a temporary directory. ## Process -1. **Check.** In one pass, resolve `npx`, run `npx --yes @playwright/cli@0.1.17 --version`, and resolve `ffmpeg` and `ffprobe`. Continue silently when all pass. -2. **Resolve.** A dependency is missing: ask once whether the user installs it or authorizes you to. If authorized, install only the missing ones and recheck. If declined or a recheck fails, stop with the shortest decisive error. Never add them to the application manifest. -3. **Prepare.** Reach the journey's initial state with deterministic data and the role authenticated off-camera. A state-changing journey without a verified teardown is not recorded. -4. **Record.** Follow [interface-browser-playwright-cli.md](../references/interface-browser-playwright-cli.md). Keep the take of a mismatch. -5. **Label.** From the returned step results: any `ok: false` => `fail`, else `pass`. -6. **Recover.** A setup failure, throw, or non-zero exit discards the take; reset and retry once. A second one => `blocked`. -7. **Reset.** After each state-changing take, run its teardown, verify the baseline, then close the session. -8. **Normalize.** At most two independent raw files concurrently. Keep only a file whose `ffprobe` and frame inspection pass. -9. **Clean.** Delete raw takes and validation frames only after every final file passes codec, dimension, duration, path, cut-point, and frame checks. Never retain screenshots or alternate media. -10. **Return.** Output each video path, its label and duration. When `happy-path.webm` exists, ask `Open happy-path.webm in the browser for review?` and open it when confirmed. +1. **Check.** Resolve `npx` and the pinned `@playwright/cli` for a browser journey, plus `ffmpeg` and `ffprobe` only for a browser video. +2. **Resolve.** A dependency is missing: ask once whether to install it, never in the application manifest. Declined or a recheck fails => stop with the shortest decisive error. +3. **Prepare.** Reach the journey's initial state with deterministic data, authenticated off-camera. A state-changing journey without a verified teardown is not recorded. +4. **Record.** Browser: follow [interface-browser-playwright-cli.md](../references/interface-browser-playwright-cli.md). CLI: write each command, its output and exit status verbatim. Keep a mismatch as observed; never stage or fake it. +5. **Label.** Any failed step => `fail`, else `pass`. +6. **Recover.** A setup failure or tooling error discards the take; reset and retry once. A second one => `blocked`. +7. **Reset.** After a state-changing take, run its teardown and verify the baseline. +8. **Normalize.** Browser only: keep a file only when `ffprobe` and frame inspection pass. +9. **Clean.** Delete raw takes and validation frames once every final file passes. +10. **Return.** Output each path and label. Only when the user invoked it directly, ask `Open