F1 (High): distinct CSV rows silently discarded as duplicates — ambiguous serialization before hashing #178

What happened

ChatGPT-generated testing report at commit c7bebd1 reproduced live (isolated PG 17 at 127.0.0.1:55439) that distinct CSV rows can be silently discarded as duplicates due to ambiguous serialization before hashing.

Reproduction: upload a 2-row CSV with 2 text columns:

• Row 1: x\u001fy, z (unit separator in column 1)
• Row 2: x, y\u001fz (unit separator in column 2)

Response: status: ok, totalRows: 2, insertedRows: 1, duplicateRowsSkipped: 1, failedRows: 0.

Expected: 2 distinct rows persisted. The two rows have genuinely different content (same characters, different placement across columns) but serialize to the same join-with-separator string, so they hash identically.

Code path: api/services/dynamic_loader.py:307 joins columns with \u001f before hashing for dedup.

Why this matters

This is silent data loss. The user sees "1 inserted, 1 duplicate skipped" with no indication which row was dropped, no way to recover it, and no clue that the "duplicate" was not actually a duplicate.

This is not a cryptographic hash collision — it is an ambiguous serialization bug. Any character chosen as a join delimiter can appear in cell data, so switching from \u001f to another control char just moves the attack surface.

Blast radius:

• ASCII \u001f and other control chars appear in CSVs from database exports, log aggregators and Unicode-aware text processors.
• A client CSV containing any \u001f makes that upload's dedup results untrustworthy.
• Deliberate exploitation: an attacker who controls one row could craft a second row that collides, silently dropping a legitimate row.

ChatGPT rates this High severity. Agreed.

Decision taken 3 Nov 2026

Fix standalone pre-M5 (not folded into M5 #109) with option (a) of 3 considered: JSON serialization before hashing, with sort_keys=True, ensure_ascii=False and a hash-version marker so new hashes coexist with any persisted old hashes.

Options rejected:

• Length-prefixed concatenation (b): equally correct, less readable
• Hash-each-column-separately-and-combine (c): more expensive, Merkle-tree pattern overkill for row dedup
• Fold into M5 (ART M5: dedup and naming — per-tenant file dedup, record dedup, versioned datasets (BR-28..31) #109): dedup redesign is months away; silent data loss cannot wait

Proposed

1. In api/services/dynamic_loader.py around line 307, change the pre-hash serialization from \u001f.join(row) to json.dumps(row, sort_keys=True, ensure_ascii=False, separators=(",", ":")).
2. Prefix the stored hash with a version marker (v2: or similar) so old hashes (prefix absent or v1:) can still match legacy stored dedup state. Dedup check compares both.
3. Document the hash versioning in a comment at the serialization site and in DECISIONS/<date>-dedup-hash-v2.md.
4. Reproduction test added matching ChatGPT's two-row case; must fail on main at c7bebd1, pass with fix.
5. Regression tests covering: all 32 ASCII control chars as cell content, Unicode cells, empty cells, cells with embedded quotes, cells with embedded newlines.

Done when

• ChatGPT's exact reproduction case persists both rows after the fix (verified-open becomes verified-fixed).
• All 32 ASCII control chars tested as cell content; none causes a false duplicate.
• The hash-version test: upload a row, record its stored hash, re-upload the same row, verify dedup still triggers (backward compatibility).
• The hash-version test: upload a row that would have falsely collided under the old serialization, verify it does NOT dedup against a pre-existing colliding row with the old hash.
• CI replica run quoted on the PR (counts in passed / failed / skipped form).
• No change to the dedup response schema (frontend unaffected).

Hash compatibility note

If dedup state is persisted anywhere (dedup registry table, filename-to-hash mapping, audit log row hashes), the version marker is critical. Without it, every pre-existing uploaded dataset would appear to be made of entirely new rows, triggering duplicate inserts on re-upload. Implementation must check both old and new hash formats.

Reference

Tracker row G63 (main tab). ChatGPT testing report F1, 3 Nov 2026. Related: M5 #109 (dedup redesign — this bug is now an input for that redesign). Not related to any of the earlier RV-* findings; this is a separate bug class.