[amar-python]

Summary

Ports a Tier E scenario that existed only in the archived PostgreDataMigrationApp repo (its commit f902ae5, "add Tier E negative control for empty-schema detection") into this repo, rewritten against the current evals/runner.py. A small, separate change; it touches none of the open PRs (#213, #221, #222).

Why

Scenario 01_envs_have_identical_structure passes when Test, Staging and Prod match Dev. That is only worth anything if a deploy that wrote nothing is something it can see. Nothing in the repo showed that against a real server: the existing unit tests cover the comparison logic with a mocked psql.

Scenario 02_empty_schema_is_detected

Against the real database it:

1. reads Dev's real schema signature, and fails if Dev isn't deployed (fewer than min_reference_tables), because comparing nothing with nothing proves nothing;
2. creates a throwaway empty schema _te_empty_neg_ctrl beside Dev's and requires the real schema query to read it as having no tables. A failed query is a failure, not an empty schema (the confusion behind issue CI: integration job has never passed - T&E tables not found in te_dev #45);
3. requires the real comparison of Dev against the empty schema to report the missing tables, so scenario 01 would fail on it;
4. always drops the throwaway schema, and never with CASCADE: if something else is found under that name, the scenario fails instead of deleting what isn't its own.

Scenario 01 deploys Dev first, so --tiers e runs them in the right order. It already runs in the integration (postgres service) job (--tiers p,i,s,e), so no workflow change is needed; the count there becomes 27.

What changed from the original commit (why it is a port, not a cherry-pick)

• The original calls _get_schema_fingerprint, which no longer exists. The current helper is _column_signature, which returns a table→columns map and raises on a failed query; the original's fp is None branch is obsolete.
• The original only checked that an empty schema returns 0 tables. It did not check what its own description says it proves, that the parity comparison fails. Step 3 above adds that.
• Added the precondition (step 1) and the no-CASCADE cleanup (step 4).

Files

• evals/runner.py: dispatch, _psql_run, _run_empty_schema_negative_control (+109 lines)
• evals/datasets/tier_e/02_empty_schema_is_detected/NOTES.txt, evals/expected/tier_e/02_empty_schema_is_detected.json
• tests/test_evals_tier_e_x.py: 15 new unit tests (mocked psql), one per branch, including that the control fails when the comparison is neutralised
• README.md (scenario count 26 → 27), evals/HANDOFF.md, evals/FAILURE_MODES.md, regenerated docs/TEST_CASES.md
• Not touched: docs/VCRM.md's dated entries (e.g. "26 of 26 passed … on its first real run") are history.

Verified (local, PostgreSQL 16, all four environments deployed)

• python3 evals/runner.py --tiers p,i,s,e in CI order: 27 passed, 0 failed; the throwaway schema is gone afterwards.
• Each of three attempts to break the control turned it red, then reverted:
  • comparison neutralised → "comparing dev (12 tables) with an empty schema reported no missing tables, so scenario 01 would pass a deploy that wrote nothing";
  • a non-empty schema planted under the control's name → the scenario fails and the planted row survives (nothing was dropped);
  • the CI: integration job has never passed - T&E tables not found in te_dev #45 failure mode reintroduced (a failed schema query read as "empty") → fails with "Deploy it first … compare nothing with nothing".
• tests/test_evals_tier_e_x.py 37 passed; make lint clean (flake8 + bandit); make health 90/0; test_catalogue.py --check current; free tier passes.
• On GitHub Actions, all three jobs are green on this head, including integration (postgres service), whose eval step runs --tiers p,i,s,e and fails the job on any failed scenario, so the new scenario passed against the CI database too.

Not verified

• I did not read the new scenario's own line in the CI step log; its pass is inferred from the job's green result (the runner exits non-zero on any failed scenario).

• The control creates and drops a schema in Dev's database, so it needs the privilege to do that. The CI job and local setup connect as postgres; a restricted role would see a clear "could not create the throwaway schema" failure, not a silent pass.

• CI green on the current head (9036305): free-tier, integration (postgres service) and the Windows python-validator-tests.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VBrxqChRJtxdvSpFhiUWUy