ART M3: immutable source and append-only audit (BR-23, BR-24, BR-35) #107

Activity

1. amar-python

   added this to the ART extension milestone on Oct 26, 2026
2. amar-python

   added

   artART framework extension workART framework extension work

   on Oct 26, 2026
3. amar-python commented on Oct 27, 2026

   

   amar-python

   on Oct 27, 2026 – with Claude

   OwnerAuthor

   More actions

   Scope added by the product owner, 27 Oct 2026: keep the failure reasons with each upload on the server.

   Today the server stores only an upload's counts (total, imported, duplicate, failed) and an "upload" audit row. The reasons rows failed, such as row 1, column "date": "27 Oct 26" is not a valid timestamp, and the step-by-step upload log go only to the browser that uploaded the file. They live in that browser's Import history. Other users, other browsers and auditors can't see them, and they're lost if site data is cleared. This came up on #123, where the cause had to be rebuilt with scripts run on the dev machine.

   For M3:

   • Store each upload's row errors (capped by MAX_ROW_ERRORS_REPORTED) and its step log with the upload record or its audit event.
   • Show them from the Migrated files Preview, and include them in audit reports.
   • Verified by: an integration test showing that a partly failed upload's reasons can be read back from another session.

   ---

   Generated by Claude Code
4. amar-python

   added a commit that references this issue on Oct 27, 2026

   docs(readme): 27 Oct refresh: test count and limitations

   ...

   e8df6d6
5. amar-python commented on Oct 28, 2026

   

   amar-python

   on Oct 28, 2026 – with Claude

   OwnerAuthor

   More actions

   Decisions by the product owner, 28 Oct 2026. M3 is delivered in parts, the way M2 was.

   1. Audit store: upgrade the existing per-tenant audit_log rather than add a new audit_events table. Existing rows join the hash chain, and the Import history feed keeps working. The plan's audit_events name now refers to this table.
   2. Least-privilege database role: a separate M3 part. The first part relies on a trigger, which blocks UPDATE and DELETE for every role, including the owner, plus the hash chain. A later part adds a runtime role with INSERT and SELECT only, and updates the installer and local setup to match. M3 isn't closed until both parts are done.
   3. Retiring overwrite moves to M5, where versioned dataset names arrive. In M3, overwrite stays, but the original upload and its rows are stored before the table is replaced, so nothing is lost.

   Parts

   • M3a: append-only audit_log: actor, tenant, before and after values, a trigger rejecting UPDATE, DELETE and TRUNCATE, a prev_hash chain, and a chain check.
   • M3b: original file bytes and source rows stored, insert-only. The registry records the uploader and processing status (BR-23).
   • M3c: row errors and the step log stored with each upload, readable from another session, and shown in Preview and the audit reports.
   • M3d: the runtime database role, with INSERT and SELECT only on the audit table, plus installer and setup changes.

   ---

   Generated by Claude Code
6. amar-python

   added 4 commits that reference this issue on Oct 28, 2026

   Merge pull request #137 from amar-python/feat/107-m3a-append-only-audit

   ...

   b21737b

   feat(uploads): keep each upload's row errors and step log on the serv…

   ...

   472f9f8

   Merge pull request #138 from amar-python/feat/107-m3b-source-preserved

   ...

   1c5db31

   Merge pull request #139 from amar-python/feat/107-m3c-frontend

   ...

   adbd34f
7. amar-python commented on Oct 28, 2026

   

   amar-python

   on Oct 28, 2026 – with Claude

   OwnerAuthor

   More actions

   Progress, 28 Oct. All three merged at the product owner's direction:

   Part	PR	Merge commit
   M3a: append-only, hash-chained audit log	#137	b21737b
   M3b, and M3c on the server: originals, source rows, row errors and log kept insert-only	#138	1c5db31
   M3c in the frontend: stored failure reasons shown in Preview	#139	adbd34f

   Decision by the product owner, 28 Oct: option (b). Including the stored failure reasons in the audit reports moves to M7 (#111). That's where the reports are rebuilt on server data; today the Import history page reads only the uploading browser's local storage.

   What's left in M3: M3d, the least-privilege runtime role.

   ---

   Generated by Claude Code
8. amar-python

   added a commit that references this issue on Oct 28, 2026

   Merge pull request #140 from amar-python/feat/107-m3d-runtime-role

   ...

   18ff57e
9. amar-python commented on Oct 28, 2026

   

   amar-python

   on Oct 28, 2026 – with Claude

   OwnerAuthor

   More actions

   Evidence: M3 done, verified on the product owner's machine (Windows 10, PostgreSQL 18), 28 Oct.

   All four parts are merged at the product owner's direction:

   Part	PR	Merge commit
   M3a	#137	b21737b
   M3b, and M3c on the server	#138	1c5db31
   M3c in the frontend	#139	adbd34f
   M3d	#140	18ff57e

   CI passed 3 of 3 on main at 18ff57e.

   Local setup. After git pull to 18ff57e, the product owner ran create_app_role.py:

   Role 'te_app' is ready (created). Its settings are in C:\Users\User\.te-migration\app-db.env; start-api.ps1 loads them. Restart the API.
   

   API startup. start-api.ps1 started the API with no APP_DB_USER not set warning from the script, and none from the API about running as the owner:

   API starting on http://localhost:8000  (docs: /docs)
   INFO:     Application startup complete.
   

   App check. The product owner signed in to tenant dev and opened Preview on an existing migrated file. Its rows loaded through te_app, which means the startup grants and the hand-over of tables the owner created before M3d both worked on a real upgraded install.

   The screenshot is kept off this public issue because it shows the product owner's personal financial data.

   Verified by, from the M0 plan:

   • UPDATE and DELETE fail for both the app role and the owner: test_runtime_role.py and test_audit_append_only.py.
   • The hash chain is checked: test_audit_append_only.py, plus GET /api/audit/verify.
   • The uploader is on the registry row: test_preserved_uploads.py.
   • Failure reasons can be read from another session: test_preserved_uploads.py, plus the browser check on feat(frontend): show stored failure reasons in the Migrated files Preview (M3c, #107) #139.

   Carried forward by decision:

   • Retiring overwrite moves to M5 (ART M5: dedup and naming — per-tenant file dedup, record dedup, versioned datasets (BR-28..31) #109).
   • Stored failure reasons in the audit reports move to M7 (ART M7: sovereignty and reports — egress-blocked test run, classification, filtered audit reports, performance (BR-35, BR-37) #111).
   • The API holding the owner's password at startup is noted on feat(db): serve requests as a runtime role that can't change the audit or upload records (M3d, #107) #140.

   Closing.

   ---

   Generated by Claude Code
10. amar-python

    closed this as completedon Oct 28, 2026