Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
.PHONY: test-free test-gate test-evals test-e2e test-all \
test-unit test-system test-sit test-performance test-accessibility test-uat \
lint lint-diff health \
eval-list eval-compare eval-summary select-tests \
csv-load csv-list csv-demo
Expand Down Expand Up @@ -32,6 +33,35 @@ test-all:
pytest tests/ --cov=build/csv --cov=evals --cov-report=term-missing
python3 evals/runner.py --tiers p,i,s,e --verbose

# ── One category at a time (#167) ─────────────────────────────────────────────
# tests/ is split by what a test needs: unit (nothing), system (the API and
# PostgreSQL), system_integration (whole pipeline, all environments), then
# performance, accessibility and uat. See tests/README.md.

# pytest exits 5 when a folder holds no tests yet; that is not a failure for a
# category that has none so far.
define run_category
python3 -m pytest $(1) -v; rc=$$?; if [ $$rc -eq 5 ]; then echo "No tests in $(1) yet."; exit 0; fi; exit $$rc
endef

test-unit:
$(call run_category,tests/unit)

test-system:
$(call run_category,tests/system)

test-sit:
$(call run_category,tests/system_integration)

test-performance:
$(call run_category,tests/performance)

test-accessibility:
$(call run_category,tests/accessibility)

test-uat:
$(call run_category,tests/uat)

# ── Code quality ──────────────────────────────────────────────────────────────

# Full flake8 + bandit scan over all Python source.
Expand Down
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -244,8 +244,8 @@ Interactive docs: `http://localhost:8000/docs`
python -m pytest tests\ -m "unit or regression or security or snapshot" -q

# Integration tests — requires running API + provisioned database
python -m pytest tests\test_api_integration.py -v
python -m pytest tests\test_te_loader_integration.py -v
python -m pytest tests\system\test_api_integration.py -v
python -m pytest tests\system\test_te_loader_integration.py -v

# Per-environment contract tests
python -m pytest tests\environments\ -v
Expand Down Expand Up @@ -338,7 +338,7 @@ organisations ──< personnel
| Connection Limit | 10 | 15 | 25 | 50 |
| Seed Data | ✅ | ✅ | ❌ | ❌ |

Staging and Prod are **never seeded**. The `tests/environments/` contract tests
Staging and Prod are **never seeded**. The `tests/system_integration/environments/` contract tests
verify this automatically.

---
Expand Down
2 changes: 1 addition & 1 deletion api/auth.py
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ async def tenant_scope(authorization: str = Header(default=""),


# M2c (#94): which roles may perform each action. docs/ROLES.md shows the same
# table, and tests/test_roles.py checks that the two agree.
# table, and tests/system/test_roles.py checks that the two agree.
PERMISSIONS: dict[str, frozenset[str]] = {
"preview": frozenset({"uploader", "tenant_admin"}),
"upload": frozenset({"uploader", "tenant_admin"}),
Expand Down
2 changes: 1 addition & 1 deletion api/routers/audit_routes.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
O-04 update: the table is no longer delete-only. Successful uploads also write a
row, through the shared writer in api/services/audit.py, so this feed now covers
creations as well as destructions. Integration coverage lives in
tests/test_api_integration.py::AuditLogTrail."""
tests/system/test_api_integration.py::AuditLogTrail."""

from __future__ import annotations

Expand Down
8 changes: 4 additions & 4 deletions docs/API_INTEGRATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -237,8 +237,8 @@ Per the repository's no-skip policy, the integration group **fails** with
remediation text when the database is unreachable rather than skipping.

```bash
python -m pytest tests/test_api_unit.py -m unit # no database
python -m pytest tests/test_api_unit.py tests/test_api_integration.py # both files
python -m pytest tests/unit/test_api_unit.py -m unit # no database
python -m pytest tests/unit/test_api_unit.py tests/system/test_api_integration.py # both files
python scripts/test_report.py --strict # whole suite
```

Expand All @@ -254,8 +254,8 @@ with remediation text rather than skipping, per the no-skip policy.

## Also required

CI installs only `requirements-dev.txt`. Because `tests/test_api_unit.py` and
`tests/test_api_integration.py` import FastAPI at module level, both
CI installs only `requirements-dev.txt`. Because `tests/unit/test_api_unit.py` and
`tests/system/test_api_integration.py` import FastAPI at module level, both
workflows need:

```yaml
Expand Down
20 changes: 10 additions & 10 deletions docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,15 +50,15 @@ The three layers can break independently, so we keep them physically separate. T
| `tests/run_tests.sh` | Bash wrapper that sources `config.local.env` |
| `tests/run_python_tests.ps1` | Windows runner — invoked by the GitHub Actions workflow |
| `tests/conftest.py` | pytest env-var isolation between tests |
| `tests/test_csv_validator.py` | unittest for `build/csv/validator.py` |
| `tests/test_csv_utilise.py` | unit tests for `build/csv_utilise.sh` argument parsing |
| `tests/test_csv_loader_arbitrary_shapes.py` | integration: arbitrary CSV shapes through loader → PG (skips without PG) |
| `tests/test_e2e_pipeline.py` | e2e: CSV → validate → load → verify (DB half skips without PG) |
| `tests/test_parity.py` | cross-environment row-count / schema parity (skips without PG) |
| `tests/test_regression.py` | pinned tests for previously found bug classes |
| `tests/test_security.py` | static credential/SQL-pattern scans |
| `tests/test_snapshot.py` | golden-file output comparisons (`tests/snapshots/`) |
| `tests/test_evals_runner.py` | unittest for `evals/runner.py` itself |
| `tests/unit/test_csv_validator.py` | unittest for `build/csv/validator.py` |
| `tests/unit/test_csv_utilise.py` | unit tests for `build/csv_utilise.sh` argument parsing |
| `tests/system_integration/test_csv_loader_arbitrary_shapes.py` | integration: arbitrary CSV shapes through loader → PG (skips without PG) |
| `tests/system_integration/test_e2e_pipeline.py` | e2e: CSV → validate → load → verify (DB half skips without PG) |
| `tests/system_integration/test_parity.py` | cross-environment row-count / schema parity (skips without PG) |
| `tests/unit/test_regression.py` | pinned tests for previously found bug classes |
| `tests/unit/test_security.py` | static credential/SQL-pattern scans |
| `tests/unit/test_snapshot.py` | golden-file output comparisons (`tests/snapshots/`) |
| `tests/unit/test_evals_runner.py` | unittest for `evals/runner.py` itself |

### `evals/` — data-driven scenarios

Expand Down Expand Up @@ -110,7 +110,7 @@ These are left at the repo root and excluded from the layer model. They can move
## Path conventions inside the layers

- Code inside `evals/runner.py` computes `PROJECT_ROOT = Path(__file__).resolve().parent.parent` and reaches into `build/csv/validator.py`, `build/environments/env_dev.sql`, `tests/run_all_tests.sql` from there.
- Code inside `tests/test_csv_validator.py` does `Path(__file__).resolve().parents[1] / "build" / "csv" / "validator.py"`.
- Code inside `tests/unit/test_csv_validator.py` does `Path(__file__).resolve().parents[2] / "build" / "csv" / "validator.py"`.
- Shell scripts inside `build/` cd into their own directory and use relative paths (`./adapters/...`, `./schema/...`).
- Nothing in `build/` references `tests/` or `evals/`.

Expand Down
2 changes: 1 addition & 1 deletion docs/ENGINE_PARITY_TODO.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ temp_documents -> references test_programs (independent of the chain above)
Then, in the same commit, move each table from `known_missing_tables` into
`expected_tables` in
`evals/expected/tier_x/01_sqlite_schema_structure.json`. Tier X will fail until
you do — `tests/test_evals_tier_e_x.py` asserts that the two lists together
you do — `tests/unit/test_evals_tier_e_x.py` asserts that the two lists together
account for all 12 tables, and that everything listed as missing is genuinely
absent. That is the reminder working as designed.

Expand Down
2 changes: 1 addition & 1 deletion docs/ENV_VARIABLES.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ the environment SQL templates. Copy `build/config.env.example` to

> **Staging and prod must never be seeded.** The `SEED_STAGING` and `SEED_PROD`
> variables default to `false`. Changing them to `true` will insert test data
> into production. The `tests/test_parity.py` contract tests verify this.
> into production. The `tests/system_integration/test_parity.py` contract tests verify this.

### Additional build-script variables

Expand Down
4 changes: 2 additions & 2 deletions docs/GAP_ANALYSIS.md
Original file line number Diff line number Diff line change
Expand Up @@ -117,9 +117,9 @@ scenario still fails on drift in either direction. It also executes the DDL
rather than `adapter_sqlite.sh`, so the adapter scripts themselves remain
unexercised — a smaller, separate gap.

Also mitigated: `tests/test_parity.py::TestAllEnvironmentsHaveRequiredTables`
Also mitigated: `tests/system_integration/test_parity.py::TestAllEnvironmentsHaveRequiredTables`
runs against all four PostgreSQL environments, and
`tests/test_evals_tier_e_x.py` unit-tests both tiers' comparison logic with
`tests/unit/test_evals_tier_e_x.py` unit-tests both tiers' comparison logic with
negative controls for added columns, changed types, changed nullability,
reordered columns, and missing or extra tables.

Expand Down
4 changes: 2 additions & 2 deletions docs/QUICKSTART.md
Original file line number Diff line number Diff line change
Expand Up @@ -198,10 +198,10 @@ Or use the convenience script from the repo root:
python -m pytest tests\ -m "unit or regression or security or snapshot" -q

# Integration tests — requires running API + provisioned database
python -m pytest tests\test_api_integration.py -v
python -m pytest tests\system\test_api_integration.py -v

# T&E mode integration tests
python -m pytest tests\test_te_loader_integration.py -v
python -m pytest tests\system\test_te_loader_integration.py -v

# Per-environment contract tests
python -m pytest tests\environments\ -v
Expand Down
2 changes: 1 addition & 1 deletion docs/ROLES.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Roles and permissions (BR-36, #94)

Every member of a tenant has one role (ADR-0003). This table is enforced by `PERMISSIONS` in `api/auth.py`. `tests/test_roles.py` fails if the two disagree, and it tries every role against every route.
Every member of a tenant has one role (ADR-0003). This table is enforced by `PERMISSIONS` in `api/auth.py`. `tests/system/test_roles.py` fails if the two disagree, and it tries every role against every route.

| Action | Route | uploader | reviewer | tenant_admin | auditor |
|---|---|---|---|---|---|
Expand Down
6 changes: 3 additions & 3 deletions docs/STARTER_GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ Both modes go through the same `POST /api/csv/upload` endpoint, with a
The same codebase runs against four isolated databases — Dev, Test, Staging,
Prod — each with its own database name, schema, and app user (table in the
main `README.md` under "Multi-Environment Setup"). Staging and Prod are never
seeded with sample data; `tests/environments/` enforces that automatically.
seeded with sample data; `tests/system_integration/environments/` enforces that automatically.
If you're not sure which environment a command is touching, check
`docs/DATABASE_ENVIRONMENTS.md` and `docs/ENV_VARIABLES.md` before running
anything destructive.
Expand Down Expand Up @@ -135,14 +135,14 @@ because of a specific past incident:

- **All SQL identifiers (table/column names) go through
`psycopg2.sql.Identifier()`** — never f-string interpolation. This is
AST-checked by `tests/test_security.py`.
AST-checked by `tests/unit/test_security.py`.
- **pytest markers matter**: `unit`, `integration`, `e2e`, `regression`,
`security`, `snapshot`, `parity`. Run `unit` (and friends) for a fast,
DB-free sanity check; `integration` and `e2e` need a live, provisioned
PostgreSQL. See `pytest.ini` for the full marker list.
- **Staging and Prod are never seeded** — if you're touching provisioning
scripts, don't make that possible even accidentally; there's a contract
test for it in `tests/environments/`.
test for it in `tests/system_integration/environments/`.

## If something doesn't work

Expand Down
Loading
Loading