ci: PR smoke test for a11y-scan SPM plugin (end-to-end scan) - #35
ci: PR smoke test for a11y-scan SPM plugin (end-to-end scan)#35Crash0v3rrid3 wants to merge 3 commits into
Conversation
Runs an end-to-end accessibility scan on every PR: builds the plugin and executes a real scan against the tests/spm harness (sample SwiftUI sources with intentional a11y issues), reusing the repo's own gated integration test (testA11yScanPluginRuns) so the invocation stays in one place. The scan downloads the BrowserStack CLI and makes authenticated calls, so it is gated to same-repo PRs (secrets are never exposed to fork PRs) and manual dispatch. Without the BROWSERSTACK_USERNAME / BROWSERSTACK_ACCESS_KEY secrets configured the e2e test XCTSkips and the job still passes. actions/checkout pinned by SHA to match existing workflows (DEVA11Y-476). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The repo root is a plugin-only package with no buildable target, so
`swift build` there fails ("does not contain a buildable target").
Building the tests/spm harness compiles the a11y-scan command plugin via
the path dependency plus the sample sources, so use that as the build step.
Verified locally on Swift 6.2: tests/spm `swift build` compiles the plugin,
and `swift test` passes with the e2e scan test skipping when RUN_A11Y_SCAN
is unset.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
run-a11y-scan.sh passed `--allow-network-connections 'all(ports: [])'` — that is PackageDescription API syntax, not a valid CLI value, and the empty port list did not satisfy the a11y-scan plugin's declared need for ports 80/443. SwiftPM therefore refused the scan: error: Plugin 'a11y-scan' wants permission to allow all network connections on ports: 80, 443. Use `--allow-network-connections all:80,443` to allow this. Surfaced by the new PR smoke-test job, which is the first thing to run the scan in CI. Fix per SwiftPM's own guidance: `all:80,443`. Applied to the SwiftPM and Xcode harness scripts and the tests/spm README (same bug in all three). Verified locally: the CLI now clears the permission gate and runs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Smoke gate is green ✅ — and it caught a real pre-existing bugThe first CI run failed at the scan step:
Root cause (not in the workflow): the harness scripts passed
After the fix, the end-to-end scan runs and passes (42s) with the repo secrets. (Non-blocking) The Node 20 deprecation warning is from |
What
Adds
.github/workflows/spm-smoke-test.yml— a GitHub Actions workflow that smoke-tests thea11y-scanSwiftPM command plugin end-to-end on every PR.On a macOS runner it:
cd tests/spm && swift build— compiles thea11y-scancommand plugin (via the path dependency) and the sample sources. (The repo root is a plugin-only package with no buildable target, so it is not built directly.)cd tests/spm && swift testwithRUN_A11Y_SCAN=1+ credentials — this un-gates the repo's existingtestA11yScanPluginRuns, which runsscripts/run-a11y-scan.sh --non-strict(a real scan: downloads the BrowserStack CLI, authenticates, scans the sample SwiftUI sources with intentional a11y issues) and asserts exit 0.It reuses the harness the repo already ships (
tests/spm/) rather than duplicating theswift package plugin … scaninvocation, so there's a single source of truth for how the plugin is run.Design notes
BROWSERSTACK_USERNAME/BROWSERSTACK_ACCESS_KEY, and secrets are never exposed to fork PRs. The job is gated withgithub.event.pull_request.head.repo.fork == false(plusworkflow_dispatchfor manual runs). Fork PRs skip the job instead of failing.testA11yScanPluginRunsXCTSkips and the job still passes (build + unit test only) — so merging this doesn't turn the tree red before secrets are added.--non-strictis used by the underlying test, so planted a11y issues in the sample views don't fail the run — a clean exit means the whole download → auth → scan pipeline worked.actions/checkoutis pinned by SHA (v4.2.2) to matchSemgrep.yml/verify-selfupdate-checksums.yml(supply-chain hardening, DEVA11Y-476).Verified locally (macOS, Swift 6.2)
tests/spmswift build→Compiling plugin a11y-scan+ sample sources, build complete.tests/spmswift test(noRUN_A11Y_SCAN) →testLibraryIdentitypasses,testA11yScanPluginRunsskips, exit 0. Confirms the no-secrets path stays green.Add two repo secrets (Settings → Secrets and variables → Actions):
BROWSERSTACK_USERNAMEBROWSERSTACK_ACCESS_KEYUntil then the job passes but the e2e scan self-skips.
Notes / open questions
tests/spm/). Thetests/xcode-app/build-phase harness is not wired up here — happy to add an Xcode job in a follow-up if you want CI to cover that integration path too.chore/spm-pr-smoke-test; rename/link as needed.🤖 Generated with Claude Code