Skip to content

feat(sql-parser)!: reimplement the parser by hand in TypeScript - #266

Merged
joe-re merged 1 commit into
releasefrom
feat/v2-parser
Sep 27, 2026
Merged

joe-re merged 1 commit into
releasefrom
feat/v2-parser

Conversation

@joe-re

@joe-re joe-re commented Sep 27, 2026 •

Copy link
Copy Markdown
Owner

Summary

Reimplements @joe-re/sql-parser as a hand-written parser in TypeScript.

Why

The PEG parser stops at the first syntax error and returns no AST, so completion relies on workarounds (keywords from the error's expected, and a second parser only for the FROM clause). A hand-written parser gives full control over error recovery, locations and tokens, which is needed for cursor-aware completion and for linting incomplete SQL.

Design

  • src/lexer.ts: tokens with offsets, including whitespace and comments (tokenize() is exported)
  • src/parser.ts: recursive descent; expressions use a Pratt parser (OR < AND < NOT < comparison < || < + - < * / %)
  • Every node has an exact location (statements, *, IN (...) lists, VALUES included)
  • Types are generated from the source (dist/index.d.ts)
  • The error shape (name: 'SyntaxError', expected, found, location, EXPECTED COLUMN NAME) is kept, because sql-language-server builds keyword completion from it; the expectations mirror what the old parser reported

Compatibility

  • All existing tests pass unchanged: sql-parser 68, sqlint 36, server 128 (index.js now re-exports dist)
  • Completion was compared over every typing prefix of all SQL used by the test suites (10,334 prefixes, candidates computed by the real complete() with the old vs new parser in separate processes; a deliberately broken parser was used as a negative control): 82% identical, 10% same candidates in a different order, 7% different. The remaining differences were reviewed; most are improvements, e.g. || is now parsed (the old parser stopped there and suggested unrelated keywords), and multi-word keywords like GROUP BY / ALTER TABLE complete their next word. Quirks users are used to are kept, e.g. WITH / SELECT offered right after FROM.
  • LSP sessions over stdio (syntax error / lint + fix) give the same diagnostics positions, completion counts and code actions as release; the syntax error message is simpler now (no PEG character classes)
  • Intentional AST fixes: keywords are always keyword nodes, IS NOT NULL is operator: 'IS NOT' + null node, aggregate names are upper case, INSERT columns is string[], extra VALUES rows are in more_rows
  • Speed is on par (500 statements / 130k chars: 97 ms vs 107 ms for 5 runs)

New syntax

||, CASE, RIGHT / FULL / CROSS / NATURAL JOIN, JOIN ... USING, LIMIT ... OFFSET and LIMIT a, b, INTERSECT / EXCEPT, :: casts, INSERT ... SELECT, INSERT ... SET, GENERATED column clauses

Test plan

  • build / lint / typecheck / vsc-compile
  • sql-parser 99 (68 existing + 31 new in test/v2.test.ts), sqlint 36, server 128
  • Completion differential over 10,334 prefixes (see above)
  • LSP stdio sessions vs release
  • VS Code e2e (CI)

Follow-ups

  • Move sql-language-server completion off the expected / parseFromClause workaround to a cursor-context API on top of this parser (error recovery), then drop the compatibility expectations

Replace the PEG grammar with a hand-written tokenizer and recursive descent
parser, as the base for editor features such as error recovery and
cursor-aware completion.

- parse / parseAll / parseFromClause keep their API. Errors keep the
  SyntaxError shape (expected / found / location, EXPECTED COLUMN NAME)
  because sql-language-server builds keyword completion from them.
- Expressions use a Pratt parser (OR < AND < NOT < comparison < || <
  + - < * / %), so AND / OR stay left associative.
- Every node has a location; ranges are exact except column_list_item
  without alias, which keeps trailing whitespace for sqlint rules.
- Types are generated from src (dist/index.d.ts).
- New syntax: ||, CASE, RIGHT/FULL/CROSS/NATURAL JOIN, USING, LIMIT
  OFFSET, INTERSECT/EXCEPT, :: casts, INSERT ... SELECT / SET.
- tokenize() exposes tokens including whitespace and comments.
- Remove parser.pegjs, the generated base/*.js and the hand-written
  index.d.ts.
@joe-re joe-re changed the title feat(sql-parser)!: reimplement the parser by hand in TypeScript (MIT) feat(sql-parser)!: reimplement the parser by hand in TypeScript Sep 27, 2026
@joe-re
joe-re merged commit ac7e845 into release Sep 27, 2026
5 checks passed
@joe-re
joe-re deleted the feat/v2-parser branch September 27, 2026 23:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant