Skip to content

Commit e44cefe

Browse files
committed
Merge main into the Tier E control branch (picks up #221 and #222)
The tests moved to tests/unit/ with the reorganisation; Git carried the control's tests across the rename. One path in them computed the repo root with parent.parent and is now parents[2]. The generated test catalogue was the only conflict: taken from main and regenerated with scripts/test_catalogue.py. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VBrxqChRJtxdvSpFhiUWUy
2 parents 9036305 + 2141213 commit e44cefe

86 files changed

Lines changed: 2188 additions & 945 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/quality-gate.yml‎

Lines changed: 64 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -164,17 +164,20 @@ jobs:
164164
# the build on any skip.
165165
# The API reads PGDATABASE (api/config.py). The schema lives in
166166
# te_mgmt_dev, not the job-level default "postgres" (issue #45).
167+
# Accessibility scans need a browser and the frontend, not this database;
168+
# they run in the `accessibility` job below (#168) and are reported here as
169+
# NOT RUN, not as skipped.
167170
- name: Full test suite — final result with skip accounting
168171
env:
169172
PGDATABASE: te_mgmt_dev
170-
run: python3 scripts/test_report.py --strict
173+
run: python3 scripts/test_report.py --markers "not accessibility" --strict
171174

172175
# #194: measure api/services coverage with per-loader floors.
173176
- name: api/services coverage gate
174177
env:
175178
PGDATABASE: te_mgmt_dev
176179
run: |
177-
python -m pytest tests/ \
180+
python -m pytest tests/ -m "not accessibility" \
178181
--cov=api/services \
179182
--cov-report=json:coverage-services.json \
180183
--cov-report=term-missing \
@@ -189,3 +192,62 @@ jobs:
189192
path: evals/reports/
190193
if-no-files-found: ignore
191194
retention-days: 30
195+
196+
# #168: WCAG 2.1 A and AA scans of the frontend (csv-table-hub-main) with
197+
# Playwright + axe-core in Chromium, against the Vite dev server. The API is
198+
# stubbed, so this job needs no database and a backend-only change cannot
199+
# alter what it scans. Known violations are recorded in
200+
# tests/accessibility/known_violations.json: the job fails on a NEW violation
201+
# and when a recorded one is fixed (so the list only shrinks). Not yet a
202+
# required check; make it one after a few green runs (see the issue).
203+
accessibility:
204+
name: accessibility (Playwright + axe-core, WCAG 2.1 AA)
205+
runs-on: ubuntu-latest
206+
steps:
207+
- name: Checkout
208+
uses: actions/checkout@v7
209+
210+
- name: Setup Python
211+
uses: actions/setup-python@v7
212+
with:
213+
python-version: '3.11'
214+
cache: 'pip'
215+
cache-dependency-path: |
216+
requirements-dev.txt
217+
requirements-a11y.txt
218+
api/requirements.txt
219+
220+
- name: Setup Node
221+
uses: actions/setup-node@v6
222+
with:
223+
node-version: '22'
224+
cache: 'npm'
225+
cache-dependency-path: csv-table-hub-main/package-lock.json
226+
227+
- name: Install dev dependencies
228+
# api/requirements.txt is needed too: tests/conftest.py imports the API
229+
# app, so pytest collection fails without FastAPI.
230+
run: pip install -r requirements-dev.txt -r api/requirements.txt -r requirements-a11y.txt
231+
232+
- name: Install frontend dependencies
233+
working-directory: csv-table-hub-main
234+
run: npm ci
235+
236+
- name: Install Chromium for Playwright
237+
run: python -m playwright install --with-deps chromium
238+
239+
# Prints the same final block as the other jobs. --strict fails on any skip.
240+
# The scans start `npm run dev` themselves (tests/accessibility/conftest.py).
241+
- name: Accessibility scans — final result with skip accounting
242+
env:
243+
A11Y_REPORT_DIR: accessibility-report
244+
run: python3 scripts/test_report.py --markers accessibility --strict
245+
246+
- name: Upload accessibility report
247+
if: always()
248+
uses: actions/upload-artifact@v7
249+
with:
250+
name: accessibility-report
251+
path: accessibility-report/
252+
if-no-files-found: ignore
253+
retention-days: 30

‎Makefile‎

Lines changed: 33 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
.PHONY: test-free test-gate test-evals test-e2e test-all \
2+
test-unit test-system test-sit test-performance test-accessibility test-uat \
23
lint lint-diff health \
34
eval-list eval-compare eval-summary select-tests \
45
csv-load csv-list csv-demo
@@ -26,12 +27,42 @@ test-evals:
2627
test-e2e:
2728
pytest -m "e2e or integration or parity" tests/ -v
2829

29-
# All: every test tier including evals.
30+
# All: every test tier including evals. Accessibility scans need a browser and the
31+
# frontend, so they run on their own: `make test-accessibility` (tests/accessibility/README.md).
3032
# Mirrors: bun run test:evals:all
3133
test-all:
32-
pytest tests/ --cov=build/csv --cov=evals --cov-report=term-missing
34+
pytest tests/ -m "not accessibility" --cov=build/csv --cov=evals --cov-report=term-missing
3335
python3 evals/runner.py --tiers p,i,s,e --verbose
3436

37+
# ── One category at a time (#167) ─────────────────────────────────────────────
38+
# tests/ is split by what a test needs: unit (nothing), system (the API and
39+
# PostgreSQL), system_integration (whole pipeline, all environments), then
40+
# performance, accessibility and uat. See tests/README.md.
41+
42+
# pytest exits 5 when a folder holds no tests yet; that is not a failure for a
43+
# category that has none so far.
44+
define run_category
45+
python3 -m pytest $(1) -v; rc=$$?; if [ $$rc -eq 5 ]; then echo "No tests in $(1) yet."; exit 0; fi; exit $$rc
46+
endef
47+
48+
test-unit:
49+
$(call run_category,tests/unit)
50+
51+
test-system:
52+
$(call run_category,tests/system)
53+
54+
test-sit:
55+
$(call run_category,tests/system_integration)
56+
57+
test-performance:
58+
$(call run_category,tests/performance)
59+
60+
test-accessibility:
61+
$(call run_category,tests/accessibility)
62+
63+
test-uat:
64+
$(call run_category,tests/uat)
65+
3566
# ── Code quality ──────────────────────────────────────────────────────────────
3667

3768
# Full flake8 + bandit scan over all Python source.

‎README.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -244,8 +244,8 @@ Interactive docs: `http://localhost:8000/docs`
244244
python -m pytest tests\ -m "unit or regression or security or snapshot" -q
245245
246246
# Integration tests — requires running API + provisioned database
247-
python -m pytest tests\test_api_integration.py -v
248-
python -m pytest tests\test_te_loader_integration.py -v
247+
python -m pytest tests\system\test_api_integration.py -v
248+
python -m pytest tests\system\test_te_loader_integration.py -v
249249
250250
# Per-environment contract tests
251251
python -m pytest tests\environments\ -v
@@ -338,7 +338,7 @@ organisations ──< personnel
338338
| Connection Limit | 10 | 15 | 25 | 50 |
339339
| Seed Data | ✅ | ✅ | ❌ | ❌ |
340340

341-
Staging and Prod are **never seeded**. The `tests/environments/` contract tests
341+
Staging and Prod are **never seeded**. The `tests/system_integration/environments/` contract tests
342342
verify this automatically.
343343

344344
---

‎api/auth.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -78,7 +78,7 @@ async def tenant_scope(authorization: str = Header(default=""),
7878

7979

8080
# M2c (#94): which roles may perform each action. docs/ROLES.md shows the same
81-
# table, and tests/test_roles.py checks that the two agree.
81+
# table, and tests/system/test_roles.py checks that the two agree.
8282
PERMISSIONS: dict[str, frozenset[str]] = {
8383
"preview": frozenset({"uploader", "tenant_admin"}),
8484
"upload": frozenset({"uploader", "tenant_admin"}),

‎api/routers/audit_routes.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@
55
O-04 update: the table is no longer delete-only. Successful uploads also write a
66
row, through the shared writer in api/services/audit.py, so this feed now covers
77
creations as well as destructions. Integration coverage lives in
8-
tests/test_api_integration.py::AuditLogTrail."""
8+
tests/system/test_api_integration.py::AuditLogTrail."""
99

1010
from __future__ import annotations
1111

‎docs/API_INTEGRATION.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -237,8 +237,8 @@ Per the repository's no-skip policy, the integration group **fails** with
237237
remediation text when the database is unreachable rather than skipping.
238238

239239
```bash
240-
python -m pytest tests/test_api_unit.py -m unit # no database
241-
python -m pytest tests/test_api_unit.py tests/test_api_integration.py # both files
240+
python -m pytest tests/unit/test_api_unit.py -m unit # no database
241+
python -m pytest tests/unit/test_api_unit.py tests/system/test_api_integration.py # both files
242242
python scripts/test_report.py --strict # whole suite
243243
```
244244

@@ -254,8 +254,8 @@ with remediation text rather than skipping, per the no-skip policy.
254254

255255
## Also required
256256

257-
CI installs only `requirements-dev.txt`. Because `tests/test_api_unit.py` and
258-
`tests/test_api_integration.py` import FastAPI at module level, both
257+
CI installs only `requirements-dev.txt`. Because `tests/unit/test_api_unit.py` and
258+
`tests/system/test_api_integration.py` import FastAPI at module level, both
259259
workflows need:
260260

261261
```yaml

‎docs/ARCHITECTURE.md‎

Lines changed: 10 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -50,15 +50,15 @@ The three layers can break independently, so we keep them physically separate. T
5050
| `tests/run_tests.sh` | Bash wrapper that sources `config.local.env` |
5151
| `tests/run_python_tests.ps1` | Windows runner — invoked by the GitHub Actions workflow |
5252
| `tests/conftest.py` | pytest env-var isolation between tests |
53-
| `tests/test_csv_validator.py` | unittest for `build/csv/validator.py` |
54-
| `tests/test_csv_utilise.py` | unit tests for `build/csv_utilise.sh` argument parsing |
55-
| `tests/test_csv_loader_arbitrary_shapes.py` | integration: arbitrary CSV shapes through loader → PG (skips without PG) |
56-
| `tests/test_e2e_pipeline.py` | e2e: CSV → validate → load → verify (DB half skips without PG) |
57-
| `tests/test_parity.py` | cross-environment row-count / schema parity (skips without PG) |
58-
| `tests/test_regression.py` | pinned tests for previously found bug classes |
59-
| `tests/test_security.py` | static credential/SQL-pattern scans |
60-
| `tests/test_snapshot.py` | golden-file output comparisons (`tests/snapshots/`) |
61-
| `tests/test_evals_runner.py` | unittest for `evals/runner.py` itself |
53+
| `tests/unit/test_csv_validator.py` | unittest for `build/csv/validator.py` |
54+
| `tests/unit/test_csv_utilise.py` | unit tests for `build/csv_utilise.sh` argument parsing |
55+
| `tests/system_integration/test_csv_loader_arbitrary_shapes.py` | integration: arbitrary CSV shapes through loader → PG (skips without PG) |
56+
| `tests/system_integration/test_e2e_pipeline.py` | e2e: CSV → validate → load → verify (DB half skips without PG) |
57+
| `tests/system_integration/test_parity.py` | cross-environment row-count / schema parity (skips without PG) |
58+
| `tests/unit/test_regression.py` | pinned tests for previously found bug classes |
59+
| `tests/unit/test_security.py` | static credential/SQL-pattern scans |
60+
| `tests/unit/test_snapshot.py` | golden-file output comparisons (`tests/snapshots/`) |
61+
| `tests/unit/test_evals_runner.py` | unittest for `evals/runner.py` itself |
6262

6363
### `evals/` — data-driven scenarios
6464

@@ -110,7 +110,7 @@ These are left at the repo root and excluded from the layer model. They can move
110110
## Path conventions inside the layers
111111

112112
- Code inside `evals/runner.py` computes `PROJECT_ROOT = Path(__file__).resolve().parent.parent` and reaches into `build/csv/validator.py`, `build/environments/env_dev.sql`, `tests/run_all_tests.sql` from there.
113-
- Code inside `tests/test_csv_validator.py` does `Path(__file__).resolve().parents[1] / "build" / "csv" / "validator.py"`.
113+
- Code inside `tests/unit/test_csv_validator.py` does `Path(__file__).resolve().parents[2] / "build" / "csv" / "validator.py"`.
114114
- Shell scripts inside `build/` cd into their own directory and use relative paths (`./adapters/...`, `./schema/...`).
115115
- Nothing in `build/` references `tests/` or `evals/`.
116116

‎docs/ENGINE_PARITY_TODO.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -96,7 +96,7 @@ temp_documents -> references test_programs (independent of the chain above)
9696
Then, in the same commit, move each table from `known_missing_tables` into
9797
`expected_tables` in
9898
`evals/expected/tier_x/01_sqlite_schema_structure.json`. Tier X will fail until
99-
you do — `tests/test_evals_tier_e_x.py` asserts that the two lists together
99+
you do — `tests/unit/test_evals_tier_e_x.py` asserts that the two lists together
100100
account for all 12 tables, and that everything listed as missing is genuinely
101101
absent. That is the reminder working as designed.
102102

‎docs/ENV_VARIABLES.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -69,7 +69,7 @@ the environment SQL templates. Copy `build/config.env.example` to
6969

7070
> **Staging and prod must never be seeded.** The `SEED_STAGING` and `SEED_PROD`
7171
> variables default to `false`. Changing them to `true` will insert test data
72-
> into production. The `tests/test_parity.py` contract tests verify this.
72+
> into production. The `tests/system_integration/test_parity.py` contract tests verify this.
7373
7474
### Additional build-script variables
7575

‎docs/GAP_ANALYSIS.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -117,9 +117,9 @@ scenario still fails on drift in either direction. It also executes the DDL
117117
rather than `adapter_sqlite.sh`, so the adapter scripts themselves remain
118118
unexercised — a smaller, separate gap.
119119

120-
Also mitigated: `tests/test_parity.py::TestAllEnvironmentsHaveRequiredTables`
120+
Also mitigated: `tests/system_integration/test_parity.py::TestAllEnvironmentsHaveRequiredTables`
121121
runs against all four PostgreSQL environments, and
122-
`tests/test_evals_tier_e_x.py` unit-tests both tiers' comparison logic with
122+
`tests/unit/test_evals_tier_e_x.py` unit-tests both tiers' comparison logic with
123123
negative controls for added columns, changed types, changed nullability,
124124
reordered columns, and missing or extra tables.
125125

0 commit comments

Comments
 (0)