[amar-python]

Summary

Three independent changes on one branch, one commit each: #195, #172, #184. They touch no common files except build/te_core_schema.sql (both copies), where #172 changes the password-echo line and #184 adds the version stamp, in different places. Each can be reviewed or reverted on its own.

This branch was restarted from main after #202 merged. main has since been merged in twice (see Merged with main below). All three were chosen or confirmed by the owner on 9 Nov.

---

#195 — record whether a dynamic table's column types were confirmed

Implements the 4 Nov product-owner decision: dynamic-mode tables are created from types a person confirmed, and the registry says which.

• alembic/versions/0009_type_provenance.py (new): csv_files.types_provenance with a CHECK (inferred_unconfirmed | user_confirmed). Existing dynamic rows are backfilled inferred_unconfirmed, since nothing recorded a confirmation. T&E rows stay NULL. Runs per tenant.
• UploadRequest.typesConfirmed: only types sent together with it count as user_confirmed. It is a 422 without types, and is rejected in T&E mode rather than ignored. The upload response returns typesProvenance; GET /api/csv/files returns types_provenance.
• Frontend: the uploader sends typesConfirmed for the user's confirmed mapping; the preview notes when types were never confirmed.
• test_migrations.py / test_tenant_isolation.py read the Alembic head from the script directory instead of pinning it.
• Tests (tests/test_type_provenance.py): preview creates no table; confirming creates it with the confirmed types; provenance stored and returned; a T&E import rejects a dynamic table as its target; migration backfill, CHECK and downgrade.

Decision for the owner: the API still accepts a dynamic upload without confirmed types (recorded inferred_unconfirmed) so existing clients keep working. Making that a hard 422 is small, but breaks every caller that omits types.

---

#172 — Terraform, deployer and entrypoint agree on the production database

Decision (9 Nov): the Azure production database is te_prod owned by pgadmin; scope is Azure only. The on-prem installer, prod tests, CI and docs keep te_mgmt_prod. Recorded in DECISIONS/2026-11-09-prod-db-naming.md.

• build/deploy_all.sh reads db_name / db_owner from the launcher instead of hard-coding te_mgmt_<env> / postgres; refuses unsafe identifiers; new --dry-run.
• infra/entrypoint.sh generates env_<env>.sql at run time from PGDATABASE, PGUSER and APP_DB_PASSWORD, so the image carries templates only (RV-S4: Docker image can bake build/config.local.env and build/environments/env_*.sql from configured checkout #174). The password goes to psql via \getenv and is never written. A mounted launcher wins. New render action.
• build/te_core_schema.sql (both copies): the set_config SELECT printed the application password into deployment output, which a container job logs. It now uses \gset. Found while verifying; not in the issue.
• infra/terraform-prod/main.tf: allow-lists dblink, pg_trgm, uuid-ossp (azure.extensions; Azure refuses CREATE EXTENSION otherwise, so the first deployment would have failed partway), and passes the application role's password to the job from Key Vault.
• tests/test_prod_db_naming.py (17 tests); 13 fail against main's versions of the five files. They run under Git Bash on Windows (the Windows job's bash is the WSL launcher, which has no distribution), and fail rather than skip without a working bash.

Fresh-server verification (PostgreSQL 16 whose only role is pgadmin, te_prod pre-created as Terraform does, an "image" of tracked files only, then PGUSER=pgadmin PGDATABASE=te_prod TARGET_ENV=prod APP_DB_PASSWORD=… ./entrypoint.sh deploy):

main                                          this branch
entrypoint exit code: 1                       entrypoint exit code: 0
[x] Environment file not found:               --- Generated .../env_prod.sql (db te_prod, owner pgadmin) ---
    .../build/environments/env_prod.sql       [ok] Succeeded : prod
T&E tables in te_prod.te_prod: 0              T&E tables in te_prod.te_prod: 12
roles: pgadmin                                roles: pgadmin, te_prod_user
                                              application password in job output: 0 occurrences

Not proven: no Docker daemon here, so no image was built (the entrypoint and deployer were run directly with the image's layout). The local pgadmin is a superuser; Azure's admin login is not, so a privilege difference could show on the first real run. terraform init + validate pass (v1.9.8); no plan or apply (no Azure access).

---

#184 — tenant T&E tables have an upgrade path

Decision (9 Nov): an Alembic chain with a parity test. Recorded in DECISIONS/2026-11-09-tenant-te-migrations.md; how to make the next change is in docs/TENANT_SCHEMA_MIGRATIONS.md.

Before, provision_te_tables cloned only missing tables, so a template change never reached an existing tenant. The issue's reproduction, run on a tenant that predates #177:

tenant from before #177          : program_id columns on the 4 tables: 0/4, composite FKs: 0/7
template (te_dev)                 : program_id columns on the 4 tables: 4/4, composite FKs: 7/7
main: provision_te_tables -> 0 tables created -> program_id columns on the 4 tables: 0/4, composite FKs: 0/7
this branch: migrate_te_schema -> 'adopted'    -> program_id columns on the 4 tables: 4/4, composite FKs: 7/7

• alembic_te/ (new): a second chain with its version table alembic_version_te inside each tenant schema. te0001 is the baseline (the shape before RV-F2 (BR-07) + RV-F9 (BR-08) + RV-F8 (BR-05): composite FKs for VCRM, results, defects, coverage #177). te0002 ports phase 4B: program_id backfill (with updated_at triggers paused so it doesn't look like an edit), composite FKs, fill triggers. Idempotent, one transaction; if cross-program rows exist it aborts, leaves the tenant unchanged, and lists them.
• api/tenancy.py migrate_te_schema: new tenant → cloned from the template, stamped at the template's version, brought forward; tenant from before versioning → stamped at the baseline, upgraded; otherwise upgraded. Serialised per tenant by an advisory lock (and in-process, since Alembic's op/context are process-global). Runs from migrate_tenant, so at API start for every tenant.
• Template stamp: COMMENT ON SCHEMA ... 'te_schema_version=te0002' in both copies of te_core_schema.sql. A template with no stamp is refused with the way out; one at a lower version is cloned then brought forward; an unknown version is refused.
• Found by the parity test: CREATE TABLE ... LIKE ... INCLUDING ALL names copied indexes and unique constraints by PostgreSQL's defaults, so every tenant already differed from its template in index names, and te0002's uq_* constraints would have been added a second time. Provisioning now renames the copies to the template's names; te0002 adopts an equivalent unique constraint instead of duplicating it.
• alembic_version_te is read-only for the runtime role.
• tests/test_tenant_te_migrations.py (16): a tenant reached by any route equals the template on columns, constraints, indexes, triggers and functions; the old-shape fixture is asserted rather than assumed; data and updated_at survive; cross-program rows stop the upgrade and change nothing; the next schema change (a throwaway te9001) reaches an existing tenant and a new one cloned from the older template; concurrent starts leave one correct tenant; the stamp is checked against the head in both schema files.

Trade-offs, stated in the decision note

• Every T&E change is written twice (template and migration); the parity test is the guard, there is no generator.
• A tenant whose upgrade fails stops API start, as the uploads chain does for duplicate registry rows. The failed upgrade rolls back and the error names the tenant and the rows. A per-tenant "needs attention" state was not built.
• The old-shape fixture comes from the chain's own downgrade, not an independent dump of the pre-RV-F2 (BR-07) + RV-F9 (BR-08) + RV-F8 (BR-05): composite FKs for VCRM, results, defects, coverage #177 schema; the test asserts what that shape is.
• Tenants cloned before this keep default index names on indexes the migrations don't touch (names only).

---

Merged with main

main moved twice while this was open, and each time the PR stopped merging cleanly (which also stops GitHub from starting CI for it). Both times it was merged in, not rebased.

1. RV-F10 (BR-25): failed uploads unreachable through normal browser review flow #187 / RV-F11: Cancel does not interrupt an in-progress import (UI misleads) #188 (frontend). Three conflicts, in csv.functions.ts, index.tsx and job-outcome.test.ts. Both sides were independent additions to the same lines (typesProvenanceSchema next to uploadIdField; typesProvenanceNote next to canStop in the import lists), so both are kept. tsc --noEmit clean; vitest 168 passed.
2. T&E re-upload that fails every row still overwrites the earlier load's registry record #215 / fix: a T&E re-upload that fails every row changes nothing (#215) #220 (T&E re-upload). One conflict, the generated docs/TEST_CASES.md: taken from main and regenerated with scripts/test_catalogue.py (831 tests).

On the merged tree the full pytest suite passes against a freshly deployed PostgreSQL 16.

---

CI-equivalent run (local, PostgreSQL 16, all four environments deployed from the changed schema)

• Full pytest suite: pass (test_oidc.py excluded locally only: unrelated cffi import problem in this sandbox)

• Coverage floors: csv_parse 82.1% (>=79), dynamic_loader 85.6% (>=84), te_loader 97.3% (>=96). dynamic_loader varies by run: its lost-race branch is covered only when the concurrency tests actually collide (87.4% in an earlier run of the same code).

• On a clean database, in CI order: evals/runner.py --tiers p,i,s,e 26/26; tests/run_tests.sh dev test 159/159 each

• make lint clean, make health 89 passed / 0 failed, bandit clean, test_catalogue.py --check current

• Frontend: vitest 168 passed, tsc --noEmit clean

• CI green on the current head (b6e59a5): free tier, integration (postgres service) and the Windows python-validator-tests job.

---

Generated by Claude Code