Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 8 additions & 4 deletions .github/workflows/quality-gate.yml
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,8 @@ jobs:
PGPORT: "5432"
PGUSER: postgres
PGPASSWORD: postgres
PGDATABASE: postgres
PGDATABASE: te_mgmt_dev
API_ALLOW_DESTRUCTIVE: "true"
steps:
- name: Checkout
uses: actions/checkout@v7
Expand Down Expand Up @@ -138,15 +139,18 @@ jobs:
fi
done

- name: Evals — Tiers P, I, S
run: python3 evals/runner.py --tiers p,i,s --verbose

- name: Deploy all environments (for cross-env parity)
run: |
for env in dev test staging prod; do
bash build/deploy_all.sh "$env"
done

- name: Evals — Tiers X, E (post-deploy, before test artifacts)
run: python3 evals/runner.py --tiers x,e --verbose

- name: Evals — Tiers P, I, S
run: python3 evals/runner.py --tiers p,i,s --verbose

# Prints a final block accounting for every test: PASSED / FAILED /
# ERROR / SKIPPED (with reasons) / NOT RUN (deselected). --strict fails
# the build on any skip.
Expand Down
10 changes: 10 additions & 0 deletions evals/datasets/tier_e/01_all_envs_same_tables/NOTES.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
Tier E — Cross-environment structural parity.

After all four environments (dev, test, staging, prod) have been deployed,
their te_core_schema tables must be structurally identical: same table names,
same column names, same column types, same column order.

This scenario queries information_schema.columns for each environment and
asserts the structural fingerprints match.

Requires: PostgreSQL reachable, all four environment databases deployed.
3 changes: 3 additions & 0 deletions evals/datasets/tier_e/02_empty_schema_is_detected/NOTES.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Negative control: verify that _get_schema_fingerprint returns zero tables
for an empty schema, so 01_all_envs_same_tables would correctly fail if a
deploy silently wrote nothing.
11 changes: 11 additions & 0 deletions evals/datasets/tier_x/01_csv_round_trip_postgresql/NOTES.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
Tier X — CSV round-trip through PostgreSQL.

Loads each sample CSV (build/csv/samples/*.csv) into the dev database via
csv_loader.sh, exports it back via csv_utilise.sh export, and diffs the
data columns against the original. Marker columns (_csv_row_id, _loaded_at)
are excluded from the diff.

Proves that the loader → DB → export pipeline preserves data fidelity for
arbitrary CSV shapes.

Requires: PostgreSQL reachable via psql, config.local.env present.
9 changes: 9 additions & 0 deletions evals/expected/tier_e/01_all_envs_same_tables.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
{
"scenario": "01_all_envs_same_tables",
"description": "All four environments must have identical table structure (names, columns, types).",
"expected": {
"all_envs_match": true,
"min_envs_compared": 4,
"min_tables_checked": 12
}
}
7 changes: 7 additions & 0 deletions evals/expected/tier_e/02_empty_schema_is_detected.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{
"scenario": "02_empty_schema_is_detected",
"description": "Negative control: an empty schema must report 0 tables so the parity eval cannot false-positive on missing deploys.",
"expected": {
"empty_schema_tables": 0
}
}
8 changes: 8 additions & 0 deletions evals/expected/tier_x/01_csv_round_trip_postgresql.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{
"scenario": "01_csv_round_trip_postgresql",
"description": "Load sample CSVs into PostgreSQL, export back, and diff data columns against originals.",
"expected": {
"all_round_trips_match": true,
"min_csvs_tested": 3
}
}
Loading
Loading