test: pre-1.0 coverage — provider config, flows, surface authoring, governance, parity - #83
Merged
Merged
Conversation
The audit found 62 of 134 composer testids in no spec, all of them on the paths a NEW team touches first. The Build/Preview path was well covered; provider configuration, surface authoring, flow decomposition, and governance authoring were not covered at all. This closes those four gaps with behaviour-level tests — state changes, persisted results, honest error text, the thing actually rendering — never "the element exists". No product source is touched. No spec makes a model call. Provider configuration (e2e/composer-settings.spec.ts, agent config) Ollama and OpenAI-compatible endpoints configured through the real agent against e2e/serve-provider.mjs — a real HTTP server speaking both discovery protocols, added as a third webServer. It serves discovery only: a build against it would be a model call. Covers discovery (including the agent's embedding-model filter), model choice, persistence and re-open, manual model entry when a server does not enumerate, and the honest failure text from an unreachable endpoint (which configures nothing). The credential invariant is proved end to end rather than asserted cosmetically: the fixture's /keyed endpoint 401s unless the exact key arrives, so the model list only appears if the key travelled browser → agent → provider — and localStorage/sessionStorage are then asserted to contain no trace of it. The agent-ABSENT half lives where it is true by construction, in composer-prod-smoke.spec.ts: with the agent probe blocked, "Agent not running" is a setup step with real instructions, and both provider forms are inert rather than dead-looking. Flows (e2e/composer-flows.spec.ts, agent-free config) Plan editing before anything is built (rename, retitle, reorder with a real swap, add, trim) and that planning creates nothing; the drive creating the flow immediately with PENDING steps, the plan freezing into per-step rebuilds, Preview's outline state and flow-lint's matching warning; a rebuilt step binding on accept; the editor's cancel creating nothing; a walk completing on the surface's own emitted action; and a step over a surface the emitter refuses showing the emitter's own reason. Surface authoring (e2e/composer-surfaces.spec.ts, agent-free config) Author → live gates → live preview → save → listed under its HUMAN title → rendered in Preview → survives reload → reported by Checks against that surface. Plus both honest-failure directions: a `must` rule blocks the save and nothing is written (before or after a reload), and fixing the violation unblocks it; a `should` rule warns without blocking. Governance (e2e/composer-governance.spec.ts, agent-free config) Intent authoring gated on a real description, then governing Build and the surface editor; the rationale gate on rules; a saved rule visibly firing in the impact panel and in Checks, and its removal undoing exactly that; the typed-rule form projections; and the session-scope honesty a browser project states about governance edits. Browser/agent parity (e2e/composer-parity.spec.ts, agent config) The product-level twin of the emit-seam unit equivalence: one repository project is EXPORTED and imported back as a browser project, so both carry byte-equal vocabulary, and the same authored surface must get the same verdict — same refusal text, same clean state, same surface-scoped findings in Checks (compared non-vacuously). Scoped to the surface on purpose: a repository additionally emits its surfacesDir, a documented corpus asymmetry that would make whole-table equality a lie. Config hygiene: the exhibit config ignored composer specs by hand-typed name — the same shape as the CI filter list of #81, where adding a spec silently opted it into the wrong suite. It now ignores them by pattern. Counts: composer-smoke 14 → 28, composer-agent 44 → 49, composer-production 14 → 15, exhibit 110 → 110 (unchanged, verified). Uncovered testids 62 → 22, and the 22 are all on surfaces outside this milestone (Catalog, Mapper, Components, Repository, hub). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
dspack-studio-composer | 63816ab | Aug 12 2026, 04:42 PM |
There was a problem hiding this comment.
Pull request overview
Expands Playwright E2E coverage for the pre-1.0 milestone by adding new agent-free and agent-mode Composer suites (provider configuration, flows, surfaces, governance, and browser/agent parity), plus a local provider fixture and configuration hygiene improvements—without changing product source.
Changes:
- Add new Composer E2E specs covering provider configuration, flows, surface authoring, governance, and browser/agent parity.
- Introduce a local “provider fixture” HTTP server used by agent-mode tests for real discovery/probing without any model calls.
- Tighten suite separation by ignoring Composer specs in the main Studio Playwright config via a pattern and updating CI workflow step naming/comments accordingly.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| playwright.config.ts | Ignores all composer-*.spec.ts in the Studio config via pattern to prevent suite bleed. |
| playwright.composer-smoke.config.ts | Adds a second project (composer-product) to run new agent-free product suites on the hosted/demo artifact. |
| playwright.composer-agent.config.ts | Expands agent-mode coverage (settings + parity) and starts the provider fixture server. |
| e2e/support/composer-browser.ts | New shared helpers for agent-free Composer suites (project creation, scripted build, surface authoring). |
| e2e/serve-provider.mjs | New local provider fixture implementing Ollama/OpenAI discovery endpoints and credential-gated discovery. |
| e2e/composer-surfaces.spec.ts | New surface authoring coverage: lint/preview/save/list/reload, warnings vs must-fail refusal, emitter refusal UX. |
| e2e/composer-settings.spec.ts | New provider configuration coverage through the agent, including discovery, manual model entry, unreachable endpoint UX, and “no secret in storage” invariant. |
| e2e/composer-prod-smoke.spec.ts | Adds an agent-absent settings test asserting Local AI is honestly unavailable and forms are inert. |
| e2e/composer-parity.spec.ts | New product-level parity check between repository-backed (agent) and browser-backed projects. |
| e2e/composer-governance.spec.ts | New governance authoring coverage for intents and typed rules, including impact and Checks round-trips. |
| e2e/composer-flows.spec.ts | New flows coverage: editable plans, pending steps, rebuild/binding, cancel-no-write, completion, and unrenderable step refusal UX. |
| .github/workflows/ci.yml | Updates CI step naming/comments to reflect the expanded Composer hosted/demo and agent-mode suites (incl. fixture server). |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+28
to
+30
| /** Must match FIXTURE_KEY in e2e/composer-settings.spec.ts. */ | ||
| const EXPECTED_KEY = "sk-fixture-credential-0123456789"; | ||
|
|
Comment on lines
+22
to
+26
| const FIXTURE = "http://localhost:3314"; | ||
| /** Must match EXPECTED_KEY in e2e/serve-provider.mjs. */ | ||
| const FIXTURE_KEY = "sk-fixture-credential-0123456789"; | ||
| /** Nothing listens here — the honest "unreachable endpoint" case. */ | ||
| const DEAD_ENDPOINT = "http://localhost:3399"; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
D13 of the pre-1.0 milestone: close the e2e gaps on the surfaces a new team touches first. Zero product source changed (
git diff --name-only main -- apps packagesis empty)./models404s; unreachable-endpoint failure text asserted to configure nothing; agent-absent honesty. The credential invariant is proved end-to-end, not cosmetically — a fixture endpoint 401s unless the exact key arrives, so the model chip can only appear if the key travelled browser → agent → provider, and storage is then asserted to hold endpoint + model with no credential.mustrule blocks and writes nothing; ashouldrule warns without blocking).Testids uncovered: 62 → 22. Suites: smoke 14 → 28, agent 44 → 49, production 14 → 15, exhibit unchanged at 110. No sleeps — every wait is on an outcome.
Also fixed config hygiene: the exhibit config excluded composer specs by hand-typed name (the same brittle shape as the CI filter list), silently pulling new specs into the wrong suite; now excluded by pattern.
🤖 Generated with Claude Code