Skip to content

fix(csv-parse): align UTF-16LE comments, quotes and escapes - #550

Open
Sskift wants to merge 3 commits into
adaltas:masterfrom
Sskift:fix/utf16-comment-alignment
Open

Sskift wants to merge 3 commits into
adaltas:masterfrom
Sskift:fix/utf16-comment-alignment

Conversation

@Sskift

@Sskift Sskift commented Nov 2, 2026 •

Copy link
Copy Markdown

UTF-16LE characters can contain the bytes of comment, quote or escape markers across adjacent code units. The parser currently recognizes those bytes as syntax even when no such character is present.

For example, \u2301\u0100 contains the bytes 23 00 at offset 1 and is truncated when comment: "#" is enabled. Similarly, \u2201\u0100 can cause an opening/closing quote error, and an unaligned custom escape marker can discard field bytes.

Match these markers only at UTF-16LE code-unit boundaries. The shared check uses the absolute buffered byte offset and current encoding, so it also handles encoding aliases, odd chunk boundaries and BOM discovery. UTF-8 and binary input take the existing unrestricted path directly.

This complements #545, which covers field and record delimiters and leaves other matchers outside its scope.

Validation:

  • Comment regressions: eight failures and one passing binary control before the fix; all nine pass after it.
  • Quote/escape regressions: twelve failures and one passing control before their fix; all thirteen pass after it.
  • csv-parse suite: 582 passing, two pending, excluding the existing Windows sample-path collection failure tracked in fix: resolve sample file URLs as native paths #512.
  • Additional independent checks cover encoding aliases, String/Buffer markers, BOM discovery, every two-chunk split, byte-sized writes and binary/other-encoding controls.
  • TypeScript, changed-file ESLint/Prettier and git diff --check pass.

AI-assisted implementation and local verification using Codex.

@Sskift Sskift changed the title fix(csv-parse): align UTF-16LE comment matching to code units fix(csv-parse): align UTF-16LE comments, quotes and escapes Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant