Skip to content

fix: a T&E re-upload that fails every row changes nothing (#215) - #220

Merged
amar-python merged 2 commits into
mainfrom
fix/215-te-reupload-registry
Nov 9, 2026
Merged

amar-python merged 2 commits into
mainfrom
fix/215-te-reupload-registry

Conversation

@amar-python

Copy link
Copy Markdown
Owner

What

A T&E re-upload that fails every row now changes nothing: the earlier load's registry record survives, and any prior table of the same name is kept. Closes #215.

The bug

upload_te deleted the earlier load's registry record, and dropped a same-named dynamic table, before inserting any rows. It also skipped the #170 all-failed rollback for re-uploads (and not is_reupload). So a re-upload where every row failed replaced "6 loaded" with "0 loaded, 10 failed" while the 6 rows stayed in the table, and returned status: ok.

#170's original scenario still reproduced on main (same root cause, so fixed here). Before this change, on main: load a dynamic file, then upload an all-invalid T&E file with the same filename:

before (main at ad39ddf) after
T&E response ok, inserted 0 error, inserted 0
prior dynamic table dropped (404) readable (200)
registry replaced by te / 0 rows still dynamic / 1 row

The fix

In api/services/te_loader.py (net -3 lines):

  • The delete-permission check for overwriting stays up front, so a role without delete rights still cannot insert through a re-upload.
  • _cleanup_prior_registry_entries now runs only after at least one row has landed.
  • The all-failed rollback applies to re-uploads too.

Behaviour change (approved on #215)

A re-upload where every row fails now returns status: error (the same response a first-time all-failed upload already gets), not ok with 0 inserted. Re-uploads that insert at least one row behave as before.

Tests

  • New tests/test_te_reupload_registry.py (3 tests, live PostgreSQL): all-failed re-upload keeps the registry record and the table agrees with it; a re-upload that inserts rows still replaces the record (guards against over-correcting); RV-F3 (BR-34): all-failed T&E import runs registry cleanup and removes prior dataset #170's prior-dynamic-table case. Two of the three failed on main for the stated reason before the fix; the third (replace-on-insert) is a guard that passes both ways.
  • Two existing tests changed, deliberately: test_te_reupload_same_file_replaces_registry (test_api_coverage.py) and test_reupload_cleans_prior_registry_entry (test_te_loader_integration.py). Both re-uploaded an identical one-row file whose unique name made every row fail, so they were asserting the old behaviour by accident. They now re-upload a file that can succeed, which keeps their actual purpose (a re-upload replaces the record rather than adding a second).

Verification

Full suite against PostgreSQL 16 with all four environments deployed, as CI does: 391 tests, 388 passed, 0 failed, 0 skipped, 3 errors. The 3 errors are test_csv_loader_arbitrary_shapes (build/config.local.env not present); they fail identically on main without this change. Not run: CI (this is a draft).

Heads-up

#214 moves the test files into category folders, including the two I edited. Whichever merges second will need a small conflict resolution in those files, and my new test file will need moving into tests/system_integration/.


Generated by Claude Code

upload_te deleted the earlier load's registry record (and dropped a prior
dynamic table of the same name) before inserting any rows, and skipped the
#170 all-failed rollback for re-uploads. A re-upload where every row failed
therefore replaced "6 loaded" with "0 loaded, 10 failed" while the rows stayed
in the table, returned status "ok", and, for a same-named dynamic table, dropped
it: #170's original scenario still reproduced on main.

The permission check for overwriting stays up front. The cleanup now runs only
after at least one row has landed, and the all-failed rollback applies to
re-uploads too, so a fully failed re-upload returns status "error" and leaves
the registry and any prior table alone.

Tests: tests/test_te_reupload_registry.py (all-failed re-upload keeps the
record; a re-upload that inserts rows still replaces it; #170's prior-dynamic-
table case). Two older tests re-uploaded an identical single-row file whose
unique name made every row fail, so they encoded the old behaviour; they now
re-upload a file that can succeed.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011T6Wqm1jLdq3ZEfuKdp9aM
The "Test catalogue check" CI step failed because docs/TEST_CASES.md did not
list the new T&E re-upload tests. Regenerated with scripts/test_catalogue.py.

Also gives the three new tests a one-line docstring, and makes the edited
test's docstring open with a complete sentence: the catalogue shows only the
first docstring line, which was cutting it off mid-sentence.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011T6Wqm1jLdq3ZEfuKdp9aM

Copy link
Copy Markdown
Owner Author

CI: free-tier failed at Test catalogue check (docs/TEST_CASES.md is stale), not at any test. This PR adds tests, and the catalogue is generated and checked for freshness. Reproduced locally with python3 scripts/test_catalogue.py --check, fixed by regenerating it in b1593c4, check now exits 0. The lint step and the unit/regression suite in the same job had already passed.

While there I gave the three new tests a one-line docstring and made the edited test's docstring open with a complete sentence, since the catalogue shows only the first line. Waiting on the re-run.


Claude Code


Generated by Claude Code

@amar-python
amar-python marked this pull request as ready for review November 9, 2026 10:42
@amar-python
amar-python merged commit 12a515b into main Nov 9, 2026
3 checks passed
amar-python pushed a commit that referenced this pull request Nov 9, 2026
#220 added tests/test_te_reupload_registry.py at the top of tests/, where this
layout would leave it unclassified; moved to tests/system/ (it drives the API
against PostgreSQL) and the docstring that names its path updated. The test
catalogue was the only conflict: taken from main and regenerated.

Full pytest suite passes against a freshly deployed PostgreSQL 16.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VBrxqChRJtxdvSpFhiUWUy
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

T&E re-upload that fails every row still overwrites the earlier load's registry record

2 participants