feat: type provenance (#195), prod DB naming (#172), tenant T&E migrations (#184)

DECISIONS/2026-11-09-prod-db-naming.md

@@ -0,0 +1,42 @@
+# Decision: Production database name and owner (#172)
+
+Status:  Decided and applied
+Date:    2026-11-09
+Review:  When the on-prem and Azure targets are unified, or Azure setup (#31, #32, #34) changes the server layout
+
+## Choice
+
+The Azure production database is **`te_prod`, owned by `pgadmin`**. Terraform creates it, and nothing else names it.
+
+- `infra/terraform-prod/main.tf` creates the database `te_prod` on a server whose administrator login is `pgadmin`.
+- The migration job receives them as `PGDATABASE` and `PGUSER`. `infra/entrypoint.sh` writes them into the launcher (`env_prod.sql`) it deploys with, as `db_name` and `db_owner`.
+- `build/deploy_all.sh` reads `db_name` and `db_owner` from the launcher instead of hard-coding `te_mgmt_<env>` and `postgres`. A launcher that sets neither keeps the old names.
+
+The on-prem installer is unchanged: it still deploys `te_mgmt_prod`, owned by `postgres`, from the committed template. So there are two production names, one per target, and each is set in one place.
+
+## Context
+
+#172: Terraform created `te_prod` owned by `pgadmin`; the deployer created `te_mgmt_prod` owned by `postgres`; the entrypoint expected an `env_prod.sql` that nothing in the image produced. A fresh image failed at the first step (`Environment file not found`), and had it got past that, it would have created a second database on Azure, where no `postgres` role exists.
+
+## Reasoning
+
+- Azure Database for PostgreSQL has no `postgres` superuser. The deployer's `OWNER = postgres` cannot work there; Terraform's `pgadmin` is the only owner available.
+- The launcher is already the file that names the database. Reading the name from it makes the deployer agree with whatever the launcher says, rather than with a second copy of the name.
+- The launcher holds a credential, so it must not be in the image (#174). Generating it at run time from the container's own settings removes the missing file without baking anything in. The application password is read by psql from `APP_DB_PASSWORD` (`\getenv`), so it never reaches a file.
+- Renaming prod to `te_prod` everywhere was considered and rejected for now. `te_mgmt_prod` is used by the offline installer (`scripts/offline/install.sh`, #60 has a 15 Nov deadline), the prod environment tests, the evals runner, CI and several docs. The change would touch about 15 files and a procedure that is meant to ship.
+
+## Also changed, because the first deployment needs it
+
+- Terraform allow-lists `dblink`, `pg_trgm` and `uuid-ossp` (`azure.extensions`). Azure refuses `CREATE EXTENSION` for anything not on that list, and the schema creates all three, so the first deployment would otherwise have failed partway through.
+- Terraform creates the application role's password (`pg-app-password` in Key Vault) and passes it to the job as `APP_DB_PASSWORD`. The runner identity already reads secrets from that vault.
+- `build/te_core_schema.sql` no longer prints the application password. A `SELECT set_config(..., :'app_password', ...)` echoed it into the deployment output, which a container job sends to its logs. It now uses `\gset`, which keeps the row out of the output.
+
+## Trade-offs accepted
+
+- Two production names exist until the targets are unified.
+- `te_prod` is also the schema name inside the database, so the Azure database and schema share a name (`te_prod.te_prod`). It is harmless, and matches what Terraform already created.
+- Verified without a Docker daemon: the entrypoint and deployer were run directly against a fresh PostgreSQL 16 whose only role is `pgadmin`. An image build and a run on Azure itself have not been done.
+
+## Escape hatch
+
+Mount a concrete `env_prod.sql` over the generated one; the entrypoint leaves an existing launcher alone.

DECISIONS/2026-11-09-tenant-te-migrations.md

@@ -0,0 +1,38 @@
+# Decision: Versioned migrations for tenant T&E tables (#184)
+
+Status:  Decided and applied
+Date:    2026-11-09
+Review:  When there are live tenants, or when a second T&E schema change is made (does the two-copy cost hold up?)
+
+## Choice
+
+The twelve T&E tables in each tenant schema get their own Alembic chain, `alembic_te/`, with its version table `alembic_version_te` inside the tenant schema. `migrate_tenant` clones a new tenant from the template and stamps it, adopts a tenant from before versioning at a baseline (`te0001`), and upgrades every tenant to head. A parity test proves a migrated tenant equals the template.
+
+Chosen by the owner on 9 Nov from three options:
+
+- **Alembic chain + parity test** (this)
+- Apply `te_core_schema.sql` to tenants (one source of truth, but it is a psql script that creates databases and roles, and the API would need psql at run time)
+- Reconcile from the template (smallest, but cannot backfill `program_id` or replace foreign keys safely)
+
+## Context
+
+`provision_te_tables` cloned only missing tables, so a template change never reached a tenant that already had them. #177 (composite foreign keys, `program_id`) made that concrete: every tenant created before it kept the old shape and the old protection against cross-program rows.
+
+## Reasoning
+
+- #177 is a data-aware change (backfill, a check that can fail on existing rows, replacing constraints). Only a migration can do that safely and say what it found.
+- A tenant needs to know which version it is at. A version table per tenant schema matches how the uploads chain already works.
+- The template is still what new tenants are cloned from, so provisioning stays cheap; the chain only has to carry *changes*. The baseline adopts what exists rather than re-creating the tables, so there is no second copy of the original DDL.
+
+## Trade-offs accepted
+
+- **Each T&E change is written twice**: in `te_core_schema.sql` (both copies) and as a migration. The parity test fails on any difference, which is the guard; there is no generator. See `docs/TENANT_SCHEMA_MIGRATIONS.md`.
+- A failing tenant stops API start (as the uploads chain does for duplicate registry rows). One tenant's bad data takes the API down until it is fixed. The upgrade rolls back, so nothing is half-applied, and the error names the tenant and the rows. A per-tenant "needs attention" state was not built.
+- The "old shape" test fixture comes from the chain's own `downgrade`, not an independent dump of the pre-#177 schema. The test asserts what that shape is, so the downgrade is not taken on trust.
+- Tenants cloned by the old provisioning keep default index names on indexes the migrations don't touch (listed in the doc). Only names.
+
+## Also changed, found by the parity test
+
+- `CREATE TABLE ... LIKE ... INCLUDING ALL` names copied indexes and unique constraints by PostgreSQL's defaults, not the template's, so every tenant already differed from its template in index names, and `te0002`'s `uq_*` constraints would have been added a second time beside the default-named ones. Provisioning now renames the copies to the template's names, and `te0002` adopts an equivalent unique constraint instead of adding another.
+- The runtime role cannot rewrite `alembic_version_te` (added to `APPEND_ONLY`, like `alembic_version`).
+- The template schema carries `COMMENT ON SCHEMA ... 'te_schema_version=te0002'`; a test fails if a migration is added and the stamp is left behind.

alembic/versions/0009_type_provenance.py

@@ -0,0 +1,41 @@
+"""Record whether a dynamic table's column types were confirmed by a person (#195).
+
+Decided 4 Nov: a dynamic-mode table is created from types a person confirmed,
+and the registry says which. ``types_provenance`` is ``user_confirmed`` or
+``inferred_unconfirmed``. T&E rows keep NULL: their columns are the fixed
+schema's, so there are no inferred types to confirm.
+
+Dynamic rows registered before this revision are marked ``inferred_unconfirmed``:
+nothing recorded whether anyone confirmed their types, and the flag must not
+claim a confirmation that can't be shown. Runs per tenant.
+
+Revision ID: 0009
+Revises: 0008
+Create Date: 2026-11-09
+"""
+
+from alembic import op
+
+from api.config import settings
+
+revision = "0009"
+down_revision = "0008"
+branch_labels = None
+depends_on = None
+
+
+def _schema() -> str:
+    return op.get_bind().dialect.identifier_preparer.quote(settings.UPLOADS_SCHEMA)
+
+
+def upgrade() -> None:
+    s = _schema()
+    op.execute(f"ALTER TABLE {s}.csv_files ADD COLUMN IF NOT EXISTS types_provenance TEXT "
+               "CHECK (types_provenance IN ('inferred_unconfirmed', 'user_confirmed'))")
+    # s is the dialect-quoted schema identifier from settings, never request input.
+    op.execute(f"UPDATE {s}.csv_files SET types_provenance = 'inferred_unconfirmed' "  # nosec B608
+               "WHERE mode = 'dynamic' AND types_provenance IS NULL")
+
+
+def downgrade() -> None:
+    op.execute(f"ALTER TABLE {_schema()}.csv_files DROP COLUMN IF EXISTS types_provenance")

alembic_te/env.py

@@ -0,0 +1,46 @@
+"""Alembic environment for the T&E tables inside ONE tenant schema (#184).
+
+A second chain, separate from ``alembic/``: that one versions the uploads objects
+(``csv_files``, ``audit_log``, ...); this one versions the twelve fixed T&E tables
+that api/tenancy.py clones into each tenant from the deployment's template schema.
+Its version table is ``alembic_version_te``, inside the tenant schema, so each tenant
+carries its own T&E schema version.
+
+It never runs from a command line: the target schema comes from the caller
+(``cfg.attributes["schema"]``), and api/tenancy.py serialises callers per tenant.
+"""
+
+import sys
+from pathlib import Path
+
+from alembic import context
+from sqlalchemy import create_engine
+from sqlalchemy.engine import URL
+from sqlalchemy.pool import NullPool
+
+sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
+
+from api.config import settings  # noqa: E402
+
+VERSION_TABLE = "alembic_version_te"
+
+if context.is_offline_mode():
+    raise RuntimeError("Offline (--sql) migrations are not supported; run against a database.")
+
+schema = context.config.attributes.get("schema")
+if not schema:
+    raise RuntimeError("T&E migrations run against one tenant schema. Use api.tenancy.migrate_tenant(slug).")
+
+url = URL.create(
+    "postgresql+psycopg2",
+    username=settings.PG_USER,
+    password=settings.PG_PASSWORD or None,
+    host=settings.PG_HOST,
+    port=settings.PG_PORT,
+    database=settings.PG_DATABASE,
+)
+engine = create_engine(url, poolclass=NullPool)
+with engine.connect() as connection:
+    context.configure(connection=connection, version_table=VERSION_TABLE, version_table_schema=schema)
+    with context.begin_transaction():
+        context.run_migrations()

alembic_te/script.py.mako

@@ -0,0 +1,21 @@
+"""${message}
+
+Revision ID: ${up_revision}
+Revises: ${down_revision | comma,n}
+Create Date: ${create_date}
+"""
+
+from alembic import op
+
+revision = ${repr(up_revision)}
+down_revision = ${repr(down_revision)}
+branch_labels = None
+depends_on = None
+
+
+def upgrade() -> None:
+    ${upgrades if upgrades else "pass"}
+
+
+def downgrade() -> None:
+    ${downgrades if downgrades else "pass"}

alembic_te/versions/te0001_baseline.py

@@ -0,0 +1,24 @@
+"""Baseline: the T&E tables as they were before versioned tenant migrations (#184).
+
+Nothing to run. This revision names the shape every tenant had until now: the
+twelve tables cloned from the template by api/tenancy.py's ``provision_te_tables``,
+before #177 added ``program_id`` and the composite foreign keys. A tenant that has
+T&E tables but no version table is stamped here, then upgraded to head.
+
+Revision ID: te0001
+Revises:
+Create Date: 2026-11-09
+"""
+
+revision = "te0001"
+down_revision = None
+branch_labels = None
+depends_on = None
+
+
+def upgrade() -> None:
+    pass
+
+
+def downgrade() -> None:
+    pass