Skip to content

ci: PR smoke test for a11y-scan SPM plugin (end-to-end scan) - #35

Draft
Crash0v3rrid3 wants to merge 3 commits into
mainfrom
chore/spm-pr-smoke-test
Draft

ci: PR smoke test for a11y-scan SPM plugin (end-to-end scan)#35
Crash0v3rrid3 wants to merge 3 commits into
mainfrom
chore/spm-pr-smoke-test

Conversation

@Crash0v3rrid3

@Crash0v3rrid3 Crash0v3rrid3 commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

What

Adds .github/workflows/spm-smoke-test.yml — a GitHub Actions workflow that smoke-tests the a11y-scan SwiftPM command plugin end-to-end on every PR.

On a macOS runner it:

  1. cd tests/spm && swift build — compiles the a11y-scan command plugin (via the path dependency) and the sample sources. (The repo root is a plugin-only package with no buildable target, so it is not built directly.)
  2. cd tests/spm && swift test with RUN_A11Y_SCAN=1 + credentials — this un-gates the repo's existing testA11yScanPluginRuns, which runs scripts/run-a11y-scan.sh --non-strict (a real scan: downloads the BrowserStack CLI, authenticates, scans the sample SwiftUI sources with intentional a11y issues) and asserts exit 0.

It reuses the harness the repo already ships (tests/spm/) rather than duplicating the swift package plugin … scan invocation, so there's a single source of truth for how the plugin is run.

Design notes

  • Fork-safe. The scan needs BROWSERSTACK_USERNAME / BROWSERSTACK_ACCESS_KEY, and secrets are never exposed to fork PRs. The job is gated with github.event.pull_request.head.repo.fork == false (plus workflow_dispatch for manual runs). Fork PRs skip the job instead of failing.
  • Degrades gracefully. If the secrets aren't configured yet, testA11yScanPluginRuns XCTSkips and the job still passes (build + unit test only) — so merging this doesn't turn the tree red before secrets are added.
  • --non-strict is used by the underlying test, so planted a11y issues in the sample views don't fail the run — a clean exit means the whole download → auth → scan pipeline worked.
  • actions/checkout is pinned by SHA (v4.2.2) to match Semgrep.yml / verify-selfupdate-checksums.yml (supply-chain hardening, DEVA11Y-476).

Verified locally (macOS, Swift 6.2)

  • tests/spm swift buildCompiling plugin a11y-scan + sample sources, build complete.
  • tests/spm swift test (no RUN_A11Y_SCAN) → testLibraryIdentity passes, testA11yScanPluginRuns skips, exit 0. Confirms the no-secrets path stays green.
  • The full authenticated scan path was not run locally (no creds on this machine); it exercises on CI once the secrets are set.

⚠️ Required before this actually runs the scan

Add two repo secrets (Settings → Secrets and variables → Actions):

  • BROWSERSTACK_USERNAME
  • BROWSERSTACK_ACCESS_KEY

Until then the job passes but the e2e scan self-skips.

Notes / open questions

  • Covers the SwiftPM harness (tests/spm/). The tests/xcode-app/ build-phase harness is not wired up here — happy to add an Xcode job in a follow-up if you want CI to cover that integration path too.
  • No DEVA11Y ticket linked yet — branch is chore/spm-pr-smoke-test; rename/link as needed.

🤖 Generated with Claude Code

Crash0v3rrid3 and others added 3 commits August 3, 2026 18:22
Runs an end-to-end accessibility scan on every PR: builds the plugin and
executes a real scan against the tests/spm harness (sample SwiftUI sources
with intentional a11y issues), reusing the repo's own gated integration
test (testA11yScanPluginRuns) so the invocation stays in one place.

The scan downloads the BrowserStack CLI and makes authenticated calls, so
it is gated to same-repo PRs (secrets are never exposed to fork PRs) and
manual dispatch. Without the BROWSERSTACK_USERNAME / BROWSERSTACK_ACCESS_KEY
secrets configured the e2e test XCTSkips and the job still passes.

actions/checkout pinned by SHA to match existing workflows (DEVA11Y-476).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The repo root is a plugin-only package with no buildable target, so
`swift build` there fails ("does not contain a buildable target").
Building the tests/spm harness compiles the a11y-scan command plugin via
the path dependency plus the sample sources, so use that as the build step.

Verified locally on Swift 6.2: tests/spm `swift build` compiles the plugin,
and `swift test` passes with the e2e scan test skipping when RUN_A11Y_SCAN
is unset.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
run-a11y-scan.sh passed `--allow-network-connections 'all(ports: [])'` —
that is PackageDescription API syntax, not a valid CLI value, and the empty
port list did not satisfy the a11y-scan plugin's declared need for ports
80/443. SwiftPM therefore refused the scan:

  error: Plugin 'a11y-scan' wants permission to allow all network
  connections on ports: 80, 443. Use `--allow-network-connections
  all:80,443` to allow this.

Surfaced by the new PR smoke-test job, which is the first thing to run the
scan in CI. Fix per SwiftPM's own guidance: `all:80,443`. Applied to the
SwiftPM and Xcode harness scripts and the tests/spm README (same bug in all
three). Verified locally: the CLI now clears the permission gate and runs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@Crash0v3rrid3

Copy link
Copy Markdown
Collaborator Author

Smoke gate is green ✅ — and it caught a real pre-existing bug

The first CI run failed at the scan step:

error: Plugin 'a11y-scan' wants permission to allow all network connections on ports: 80, 443. Use `--allow-network-connections all:80,443` to allow this.

Root cause (not in the workflow): the harness scripts passed --allow-network-connections 'all(ports: [])' — that's PackageDescription API syntax pasted into a CLI flag, and the empty port list didn't satisfy the plugin's declared need for ports 80/443. This had never been exercised in CI before, so it went unnoticed. Fixed to all:80,443 in:

  • tests/spm/scripts/run-a11y-scan.sh
  • tests/xcode-app/scripts/run-a11y-scan.sh
  • tests/spm/README.md

After the fix, the end-to-end scan runs and passes (42s) with the repo secrets. BROWSERSTACK_USERNAME / BROWSERSTACK_ACCESS_KEY are clearly already configured on the repo — the scan authenticated and completed.

(Non-blocking) The Node 20 deprecation warning is from actions/checkout v4.2.2, which the repo's other workflows also pin — left as-is for consistency.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant