Repository navigation
feat(sql-parser)!: reimplement the parser by hand in TypeScript - #266
Merged
Merged
Conversation
Replace the PEG grammar with a hand-written tokenizer and recursive descent parser, as the base for editor features such as error recovery and cursor-aware completion. - parse / parseAll / parseFromClause keep their API. Errors keep the SyntaxError shape (expected / found / location, EXPECTED COLUMN NAME) because sql-language-server builds keyword completion from them. - Expressions use a Pratt parser (OR < AND < NOT < comparison < || < + - < * / %), so AND / OR stay left associative. - Every node has a location; ranges are exact except column_list_item without alias, which keeps trailing whitespace for sqlint rules. - Types are generated from src (dist/index.d.ts). - New syntax: ||, CASE, RIGHT/FULL/CROSS/NATURAL JOIN, USING, LIMIT OFFSET, INTERSECT/EXCEPT, :: casts, INSERT ... SELECT / SET. - tokenize() exposes tokens including whitespace and comments. - Remove parser.pegjs, the generated base/*.js and the hand-written index.d.ts.
joe-re
force-pushed
the
feat/v2-parser
branch
from
September 27, 2026 22:06
a052a23 to
da45d49
Compare
2 of 3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Reimplements
@joe-re/sql-parseras a hand-written parser in TypeScript.Why
The PEG parser stops at the first syntax error and returns no AST, so completion relies on workarounds (keywords from the error's
expected, and a second parser only for the FROM clause). A hand-written parser gives full control over error recovery, locations and tokens, which is needed for cursor-aware completion and for linting incomplete SQL.Design
src/lexer.ts: tokens with offsets, including whitespace and comments (tokenize()is exported)src/parser.ts: recursive descent; expressions use a Pratt parser (OR < AND < NOT < comparison <||<+ -<* / %)location(statements,*,IN (...)lists,VALUESincluded)dist/index.d.ts)name: 'SyntaxError',expected,found,location,EXPECTED COLUMN NAME) is kept, because sql-language-server builds keyword completion from it; the expectations mirror what the old parser reportedCompatibility
index.jsnow re-exportsdist)complete()with the old vs new parser in separate processes; a deliberately broken parser was used as a negative control): 82% identical, 10% same candidates in a different order, 7% different. The remaining differences were reviewed; most are improvements, e.g.||is now parsed (the old parser stopped there and suggested unrelated keywords), and multi-word keywords likeGROUP BY/ALTER TABLEcomplete their next word. Quirks users are used to are kept, e.g.WITH/SELECToffered right afterFROM.release; the syntax error message is simpler now (no PEG character classes)IS NOT NULLisoperator: 'IS NOT'+nullnode, aggregate names are upper case, INSERTcolumnsisstring[], extra VALUES rows are inmore_rowsNew syntax
||,CASE,RIGHT/FULL/CROSS/NATURAL JOIN,JOIN ... USING,LIMIT ... OFFSETandLIMIT a, b,INTERSECT/EXCEPT,::casts,INSERT ... SELECT,INSERT ... SET,GENERATEDcolumn clausesTest plan
test/v2.test.ts), sqlint 36, server 128releaseFollow-ups
expected/parseFromClauseworkaround to a cursor-context API on top of this parser (error recovery), then drop the compatibility expectations