RV-F5 (BR-24): preserved originals may differ from uploaded bytes (UTF-8 re-encoding) #185

What happened

External repo review at commit c7bebd1: the browser decodes CSV to text, and the API re-encodes that text as UTF-8 before hashing and storage. BOM or invalid UTF-8 byte sequences change the retained bytes and the digest. Node decoder probes demonstrated both transformations.

Code paths:

• csv-table-hub-main/src/routes/_authenticated/index.tsx L151–195 (browser decode)
• api/services/originals.py L39–67 (API re-encode and store)

Note: touches csv-table-hub-main/* (Lovable frontend code), partially covered by the pending #154 decision. The backend-side fix is independent.

Why this matters

BR-24 calls for preservation of the original file. If the stored "original" is not byte-identical to what was uploaded, the hash cannot be used as evidence of integrity. A client auditor who requests the original file back and verifies against their copy will see a mismatch on any file with a BOM or non-UTF-8 bytes.

Proposed

1. Backend: upload path accepts raw bytes, not decoded text. Hash and store the original bytes unchanged.
2. Backend: keep a separately-stored decoded/normalised representation with its encoding metadata, used for the loaders.
3. Frontend: when Spike: rebuild M4 review screen in Streamlit (1-day time-box) #154 decides, the client-side decode goes away (if frontend replaced) or the client sends raw bytes alongside the decoded form (if Lovable retained).
4. Reproduction tests: BOM-prefixed CSV and a CSV with a single 0x80 byte must round-trip byte-identical.

Done when

• Reproduction tests pass against the fix.
• Original hash matches uploader's local hash for BOM-prefixed and invalid-UTF-8 inputs.
• Decoded view still available for the loaders and the review screen.

Reference

Tracker row G48. Repo review finding RV-F5. BR-24. Partially blocked by #154 (frontend decision affects client-side decode behaviour).