chore: Add workflow to override visual regression test results - #4954
chore: Add workflow to override visual regression test results#4954jperals wants to merge 5 commits into
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4954 +/- ##
=======================================
Coverage 97.66% 97.66%
=======================================
Files 959 959
Lines 31345 31345
Branches 11579 11579
=======================================
Hits 30614 30614
Misses 724 724
Partials 7 7 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
8ea1878 to
a92f7c8
Compare
fba0b42 to
85f92b8
Compare
There was a problem hiding this comment.
🟡 Changes recommended
The override workflow is missing required token permissions for its Pulls API call, and the documentation currently claims PR commenting behavior that the workflow does not implement.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds a per-commit override mechanism for visual regression results so maintainers can explicitly approve intentional visual diffs and allow the required “Visual regression result” check to pass without rebuilding baselines.
Changes:
- Adds a new Override visual regression workflow that can be triggered via PR comment or manual dispatch and posts a
visual-regression-overridecommit status. - Updates the reusable visual regression workflow to check for the override status and skip expensive screenshot comparison when overridden.
- Updates the deploy workflow to surface a dedicated required “Visual regression result” gate and documents the new override flow.
File summaries
| File | Description |
|---|---|
| docs/RUNNING_TESTS.md | Documents the override-based approval flow for intentional visual diffs. |
| .github/workflows/visual-regression.yml | Adds a commit-status override check and skips visual shards when overridden. |
| .github/workflows/visual-regression-override.yml | Introduces the workflow that applies the per-commit override status and reruns failed deploy jobs. |
| .github/workflows/deploy.yml | Adds statuses read permission, passes commit SHA into the visual workflow, and adds a required “Visual regression result” gate. |
Review details
- Files reviewed: 4/4 changed files
- Comments generated: 3
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Description
This is a prerequisite to making the visual regression test workflow mandatory.
Ticket:
D417180663Doc:
J5RmAnz6YwPaHow has this been tested?
On this PR, two new checks are run:
On a PR without visual changes like this one, there are no functional differences.
Once this change is merged, it will be possible to test the actual override workflow (GitHub workflows can only be triggered from comments or actions if they are in main).
Review checklist
The following items are to be evaluated by the author(s) and the reviewer(s).
Correctness
CONTRIBUTING.md.CONTRIBUTING.md.Security
checkSafeUrlfunction.Testing
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.