ci: accessibility job, Playwright + axe-core WCAG 2.1 AA scans (#168)

.github/workflows/quality-gate.yml

@@ -164,17 +164,20 @@ jobs:
       # the build on any skip.
       # The API reads PGDATABASE (api/config.py). The schema lives in
       # te_mgmt_dev, not the job-level default "postgres" (issue #45).
+      # Accessibility scans need a browser and the frontend, not this database;
+      # they run in the `accessibility` job below (#168) and are reported here as
+      # NOT RUN, not as skipped.
       - name: Full test suite — final result with skip accounting
         env:
           PGDATABASE: te_mgmt_dev
-        run: python3 scripts/test_report.py --strict
+        run: python3 scripts/test_report.py --markers "not accessibility" --strict
 
       # #194: measure api/services coverage with per-loader floors.
       - name: api/services coverage gate
         env:
           PGDATABASE: te_mgmt_dev
         run: |
-          python -m pytest tests/ \
+          python -m pytest tests/ -m "not accessibility" \
             --cov=api/services \
             --cov-report=json:coverage-services.json \
             --cov-report=term-missing \
@@ -189,3 +192,62 @@ jobs:
           path: evals/reports/
           if-no-files-found: ignore
           retention-days: 30
+
+  # #168: WCAG 2.1 A and AA scans of the frontend (csv-table-hub-main) with
+  # Playwright + axe-core in Chromium, against the Vite dev server. The API is
+  # stubbed, so this job needs no database and a backend-only change cannot
+  # alter what it scans. Known violations are recorded in
+  # tests/accessibility/known_violations.json: the job fails on a NEW violation
+  # and when a recorded one is fixed (so the list only shrinks). Not yet a
+  # required check; make it one after a few green runs (see the issue).
+  accessibility:
+    name: accessibility (Playwright + axe-core, WCAG 2.1 AA)
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v7
+
+      - name: Setup Python
+        uses: actions/setup-python@v7
+        with:
+          python-version: '3.11'
+          cache: 'pip'
+          cache-dependency-path: |
+            requirements-dev.txt
+            requirements-a11y.txt
+            api/requirements.txt
+
+      - name: Setup Node
+        uses: actions/setup-node@v6
+        with:
+          node-version: '22'
+          cache: 'npm'
+          cache-dependency-path: csv-table-hub-main/package-lock.json
+
+      - name: Install dev dependencies
+        # api/requirements.txt is needed too: tests/conftest.py imports the API
+        # app, so pytest collection fails without FastAPI.
+        run: pip install -r requirements-dev.txt -r api/requirements.txt -r requirements-a11y.txt
+
+      - name: Install frontend dependencies
+        working-directory: csv-table-hub-main
+        run: npm ci
+
+      - name: Install Chromium for Playwright
+        run: python -m playwright install --with-deps chromium
+
+      # Prints the same final block as the other jobs. --strict fails on any skip.
+      # The scans start `npm run dev` themselves (tests/accessibility/conftest.py).
+      - name: Accessibility scans — final result with skip accounting
+        env:
+          A11Y_REPORT_DIR: accessibility-report
+        run: python3 scripts/test_report.py --markers accessibility --strict
+
+      - name: Upload accessibility report
+        if: always()
+        uses: actions/upload-artifact@v7
+        with:
+          name: accessibility-report
+          path: accessibility-report/
+          if-no-files-found: ignore
+          retention-days: 30

Makefile

@@ -27,10 +27,11 @@ test-evals:
 test-e2e:
 	pytest -m "e2e or integration or parity" tests/ -v
 
-# All: every test tier including evals.
+# All: every test tier including evals. Accessibility scans need a browser and the
+# frontend, so they run on their own: `make test-accessibility` (tests/accessibility/README.md).
 # Mirrors: bun run test:evals:all
 test-all:
-	pytest tests/ --cov=build/csv --cov=evals --cov-report=term-missing
+	pytest tests/ -m "not accessibility" --cov=build/csv --cov=evals --cov-report=term-missing
 	python3 evals/runner.py --tiers p,i,s,e --verbose
 
 # ── One category at a time (#167) ─────────────────────────────────────────────

docs/TEST_CASES.md

@@ -1,10 +1,36 @@
 # Test Catalogue
 
 > Auto-generated by `scripts/test_catalogue.py` — do not edit by hand.
-> 732 tests across 38 files.
+> 779 tests across 40 files.
 
 To regenerate: `python3 scripts/test_catalogue.py`
 
+## `tests/accessibility/test_accessibility.py`
+
+| Test | Markers | Objective |
+|---|---|---|
+| `test_sign_in_page` | — | The sign-in page, before anyone is signed in. |
+
+| `test_sign_in_page_after_a_failed_attempt` | — | The sign-in page showing the server's refusal. |
+
+| `test_upload_page_with_nothing_uploaded_yet` | — | The upload page for a new tenant: no migrated files and no import history. |
+
+| `test_upload_page_with_files_and_import_history` | — | The upload page with a migrated file and one import of each kind (done, failed, duplicate). |
+
+| `test_upload_page_failed_upload_attempts` | — | The upload page's list of every attempt, with a failed, a partly imported and a rejected one. |
+
+| `test_upload_page_column_mapping_dialog` | — | The dialog that opens when a CSV is chosen: column names and the type of each. |
+
+| `test_upload_page_rejection_toast` | — | The toast shown when a file that is not a CSV is chosen. |
+
+| `test_audit_log_with_history` | — | The audit log listing one import of each kind. |
+
+| `test_audit_log_report_menu` | — | The audit log with its report-download menu open. |
+
+| `test_review_page` | — | The review screen for one stored upload, with a row that failed validation. |
+
+| `test_signed_out_visitors_are_sent_to_sign_in` | — | A page that needs a session sends a visitor with none to the sign-in page. |
+
 ## `tests/system/test_accounts.py`
 
 | Test | Markers | Objective |
@@ -1105,6 +1131,66 @@ To regenerate: `python3 scripts/test_catalogue.py`
 
 | `test_dev_row_counts_stable_after_second_deploy` | — | — |
 
+## `tests/unit/test_a11y_baseline.py`
+
+| Test | Markers | Objective |
+|---|---|---|
+| `Compare.test_nothing_found_and_nothing_recorded_is_ok` | — | — |
+
+| `Compare.test_a_violation_not_in_the_baseline_is_new_and_fails` | — | — |
+
+| `Compare.test_the_baseline_is_per_scenario` | — | — |
+
+| `Compare.test_exactly_as_recorded_is_known_and_passes` | — | — |
+
+| `Compare.test_more_elements_than_recorded_is_a_regression` | — | — |
+
+| `Compare.test_fewer_elements_is_a_notice_not_a_failure` | — | — |
+
+| `Compare.test_a_recorded_rule_that_no_longer_fails_must_be_removed` | — | — |
+
+| `Compare.test_new_and_fixed_are_reported_together` | — | — |
+
+| `Describe.test_lists_a_few_elements_and_counts_the_rest` | — | — |
+
+| `LoadBaseline.test_rejects_a_rule_listed_twice_for_a_scenario` | — | — |
+
+| `LoadBaseline.test_rejects_an_entry_without_a_positive_count` | — | — |
+
+| `LoadBaseline.test_rejects_an_entry_without_a_note` | — | — |
+
+| `RecordedBaseline.test_is_valid_and_names_real_scenarios` | — | — |
+
+| `RecordedBaseline.test_scans_wcag_21_a_and_aa` | — | — |
+
+| `test_nothing_found_and_nothing_recorded_is_ok` | — | — |
+
+| `test_a_violation_not_in_the_baseline_is_new_and_fails` | — | — |
+
+| `test_the_baseline_is_per_scenario` | — | — |
+
+| `test_exactly_as_recorded_is_known_and_passes` | — | — |
+
+| `test_more_elements_than_recorded_is_a_regression` | — | — |
+
+| `test_fewer_elements_is_a_notice_not_a_failure` | — | — |
+
+| `test_a_recorded_rule_that_no_longer_fails_must_be_removed` | — | — |
+
+| `test_new_and_fixed_are_reported_together` | — | — |
+
+| `test_lists_a_few_elements_and_counts_the_rest` | — | — |
+
+| `test_rejects_a_rule_listed_twice_for_a_scenario` | — | — |
+
+| `test_rejects_an_entry_without_a_positive_count` | — | — |
+
+| `test_rejects_an_entry_without_a_note` | — | — |
+
+| `test_is_valid_and_names_real_scenarios` | — | — |
+
+| `test_scans_wcag_21_a_and_aa` | — | — |
+
 ## `tests/unit/test_api_hardening.py`
 
 | Test | Markers | Objective |
@@ -1277,6 +1363,14 @@ To regenerate: `python3 scripts/test_catalogue.py`
 
 | `TestAllJobsHaveConsistentStructure.test_all_jobs_install_dev_deps` | — | — |
 
+| `TestAccessibilityJob.test_has_accessibility_job_that_runs_only_the_accessibility_marker` | — | — |
+
+| `TestAccessibilityJob.test_accessibility_job_installs_a_browser_and_the_frontend` | — | — |
+
+| `TestAccessibilityJob.test_integration_job_excludes_accessibility_everywhere_it_runs_the_whole_tree` | — | — |
+
+| `TestAccessibilityJob.test_free_tier_markers_cannot_select_accessibility_tests` | — | — |
+
 | `TestGapAnalysisG2Accepted.test_g2_row_exists` | — | — |
 
 | `TestGapAnalysisG2Accepted.test_g2_is_accepted_or_deferred` | — | — |
@@ -1301,6 +1395,14 @@ To regenerate: `python3 scripts/test_catalogue.py`
 
 | `test_all_jobs_install_dev_deps` | — | — |
 
+| `test_has_accessibility_job_that_runs_only_the_accessibility_marker` | — | — |
+
+| `test_accessibility_job_installs_a_browser_and_the_frontend` | — | — |
+
+| `test_integration_job_excludes_accessibility_everywhere_it_runs_the_whole_tree` | — | — |
+
+| `test_free_tier_markers_cannot_select_accessibility_tests` | — | — |
+
 | `test_g2_row_exists` | — | — |
 
 | `test_g2_is_accepted_or_deferred` | — | — |

requirements-a11y.txt

@@ -0,0 +1,10 @@
+# Accessibility scans (tests/accessibility/, the `accessibility` job in
+# .github/workflows/quality-gate.yml). Kept apart from requirements-dev.txt so the
+# other jobs, and developers who never scan, do not install a browser driver.
+#
+#   pip install -r requirements-a11y.txt
+#   python -m playwright install chromium
+#   (cd csv-table-hub-main && npm ci)
+#   python -m pytest tests/accessibility -v
+playwright>=1.64.1
+axe-playwright-python>=0.1.8

scripts/run_qa.ps1

@@ -34,7 +34,7 @@ switch ($Target) {
         pytest -m "e2e or integration or parity" tests/ -v
     }
     "test-all" {
-        pytest tests/ --cov=build/csv --cov=evals --cov-report=term-missing
+        pytest tests/ -m "not accessibility" --cov=build/csv --cov=evals --cov-report=term-missing
         python evals/runner.py --tiers p,i,s --verbose
     }
     "lint" {

scripts/select_tests.py

@@ -36,6 +36,7 @@
     ("tests/system/",            "integration",                      "system test changed → integration"),
     ("tests/system_integration/", "e2e or integration or parity",    "system-integration test changed → e2e + integration + parity"),
     ("tests/accessibility/",     "accessibility",                    "accessibility test changed → accessibility"),
+    ("csv-table-hub-main/src/", "accessibility",                    "frontend changed → accessibility scans (CI job only)"),
     ("scripts/",                 "security",                         "scripts changed → security scan"),
     ("build/adapters/",          "security",                         "adapter changed → security scan"),
     ("build/csv/loader_",        "e2e",                              "loader changed → e2e"),

tests/README.md

@@ -10,7 +10,7 @@ automatically.
 | `system/` | The API and a live PostgreSQL | `system` (automatic) | `make test-system` |
 | `system_integration/` | The whole pipeline: shell loaders, every deployed environment, cross-environment parity | `sit` (automatic) | `make test-sit` |
 | `performance/` | Load and timing tests (none yet) | `performance` (automatic) | `make test-performance` |
-| `accessibility/` | A real browser against the running app, WCAG scans (#168) | `accessibility` (automatic) | `make test-accessibility` |
+| `accessibility/` | Chromium, the frontend's dev server and Node; the API is stubbed. WCAG 2.1 AA scans, own CI job ([README](accessibility/README.md), #168) | `accessibility` (automatic) | `make test-accessibility` |
 | `uat/` | User-acceptance scenarios (none yet) | `uat` (automatic) | `make test-uat` |
 
 Which folder does a new test go in? Ask what is the least it needs: nothing →

tests/accessibility/README.md

@@ -0,0 +1,85 @@
+# Accessibility scans (#168)
+
+WCAG 2.1 level A and AA checks of the frontend (`csv-table-hub-main`), run with
+[axe-core](https://github.com/dequelabs/axe-core) in Chromium through Playwright, against the
+Vite dev server. They run in their own CI job, `accessibility` in
+`.github/workflows/quality-gate.yml`, and nowhere else: the integration job runs with
+`-m "not accessibility"`.
+
+## Run them
+
+```bash
+pip install -r requirements-a11y.txt
+python -m playwright install chromium
+(cd csv-table-hub-main && npm ci)
+
+python -m pytest tests/accessibility -v        # or: make test-accessibility
+```
+
+The scans start `npm run dev` themselves on a free port and stop it afterwards. Missing
+Playwright, Chromium or `node_modules` fails the run with the command to fix it; it is never a
+skip.
+
+| Variable | Effect |
+|---|---|
+| `A11Y_BASE_URL` | Scan a frontend that is already running instead of starting one |
+| `A11Y_CHROMIUM_PATH` | Use this Chromium instead of the one `playwright install` fetched |
+| `A11Y_REPORT_DIR` | Write one JSON file per scenario with every violation axe-core found (CI uploads these as the `accessibility-report` artifact) |
+
+## What is scanned
+
+The API is stubbed in `stub_api.py`: no database, no API process. Every `/api/**` request gets a
+fixed answer and a signed-in session is planted in the browser. A request the stub has no answer for fails the
+test, so a new endpoint in the frontend is noticed here instead of silently scanning a half-rendered page. A backend-only change therefore
+cannot alter what is scanned, and the job needs no PostgreSQL service.
+
+| Scenario | State |
+|---|---|
+| `sign_in`, `sign_in_failed` | the sign-in page, and after the server refuses a login |
+| `upload_page_empty` | a new tenant: no files, no import history |
+| `upload_page_with_history` | a migrated file and one import of each kind (done, failed, duplicate) |
+| `upload_page_failed_attempts` | the list of every upload attempt, with a failed, a partly imported and a rejected one |
+| `upload_mapping_dialog` | the dialog that opens when a CSV is chosen |
+| `upload_page_toast` | the toast shown when a non-CSV file is chosen |
+| `audit_log_with_history`, `audit_log_report_menu` | the audit log, and with its report menu open |
+| `review_page` | the review screen for one stored upload |
+
+Each test waits for content that proves the page rendered before it scans. A page that
+redirected to sign-in or showed an error would otherwise scan clean and pass for the wrong
+reason. `test_signed_out_visitors_are_sent_to_sign_in` guards the redirect itself.
+
+Not covered: dark mode, mobile widths, keyboard-only flows (axe-core checks markup, not whether
+a task can be completed), and the Streamlit spike (#154).
+
+## Known violations, and why the job passes today
+
+The frontend has accessibility problems, and fixing them is separate work (out of scope for
+#168). A scan that fails on any violation would keep the job red until then, and a job that is
+always red gets ignored. So `known_violations.json` records what is wrong now, per scenario and
+rule with an element count, and each scan fails on a **change**:
+
+* a rule that fails where it is not recorded, or on more elements than recorded: **fails**
+  (a regression);
+* a recorded rule that no longer fails: **fails** until its entry is deleted, so the list can only
+  get shorter. The message names the entry;
+* fewer elements than recorded: reported only. Lower the number when convenient.
+
+When a frontend change fixes a violation, delete its entry in the same pull request. To see the
+whole picture, including what is recorded, set `A11Y_REPORT_DIR` and read the JSON, or run with
+`-s` to print the known violations.
+
+**To make the job strict instead** (red until the frontend is fixed), empty the `violations`
+list. That is the only change needed.
+
+The recorded violations are the ones the closed PR #41 bundled with its scan: unnamed selects and
+icon-only buttons, unnamed upload progress bars, and low-contrast text (including the error
+toast).
+
+## A frontend bug these scans work around
+
+In development, React StrictMode runs the effects in
+`csv-table-hub-main/src/hooks/use-local-storage.ts` twice: the first pass writes the hook's empty
+default over the stored import history, and the second reads that back, so saved history is never
+shown after a reload. Production builds do not double-run effects. `stub_api.seed_script`
+drops writes of `[]` to that key so the audit page can be scanned with rows in it; without that, the
+page was scanned empty and passed without checking a single row. Remove the guard when the hook is fixed.