Skip to content

feat(sql,core): FROM-less SELECT handshake executed against Elasticsearch (story 20.9) - #268

Merged
fupelaqu merged 3 commits into
mainfrom
feature/20.9
Sep 2, 2026
Merged

feat(sql,core): FROM-less SELECT handshake executed against Elasticsearch (story 20.9)#268
fupelaqu merged 3 commits into
mainfrom
feature/20.9

Conversation

@fupelaqu

@fupelaqu fupelaqu commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Closes #251

FROM-less SELECT — the connection handshake (story 20.9, 0.22.0 train)

SELECT 1 is the connection/health idiom of the JDBC/SQLAlchemy ecosystem: Tableau re-issues it on every interaction (measured in the Epic 19 capture — each one aborted browse), Superset's connect test sends it, its engine probe and the Flight sidecar's own schema probe send SELECT 1 LIMIT 100, and connection pools use it as connectionTestQuery. Until now every one of these failed with a parse error.

What this PR does

  • New statement kind FromlessSelect (sql): select ~ limit.?, inserted in dqlStatement immediately after searchStatement (| commits to the first succeeding alternative; searchStatement fails on FROM-less input because single requires from). General FROM-less select-list: literals and Painless-translatable constant scalars, multiple items, AS/bare aliases, LIMIT/OFFSET.
  • Executed AGAINST Elasticsearch (lead direction 2026-09-02): the statement is rewritten to SELECT <items> FROM softclient4es_handshake LIMIT 1 and rides the unmodified FROM-ful SQL→Painless script_fields pipeline — FROM-less and FROM-ful semantics agree by construction (one code path, one params.__now__ clock per request). A locally-answered handshake would report "connected" against a dead cluster; with ES unreachable, SELECT 1 now fails with the propagated client error — a pool's connectionTestQuery becomes a real test.
  • The handshake index: softclient4es_handshake (1 shard / 0 replicas; index.hidden: true on ES ≥ 7.7; single dummy keyword mapping; one seeded doc), created lazily at first use — probe-before-act, race-safe, memoized per client, bounded self-heal (one retry on index_not_found / empty page). Behind a seam (HandshakeEvaluator) so the future Painless-execute-API backend swaps in without touching the statement kind, executor, or tests.
  • Invisible to browse: an explicit name filter in TableExecutor's ShowTables arm — the ONE seam both BI browse surfaces traverse (jdbc getTables and Flight GET_TABLES both execute SHOW TABLES through gateway.run), so zero driver/sidecar changes. DESCRIBE TABLE softclient4es_handshake still works, deliberately.
  • Guarded: named-reason rejects (post-Every parse rejection from GatewayApi.run is labelled a schema DDL error #262 shape) for column references (own name or embedded), SELECT *, aggregates, window functions (checked before aggregation — WindowFunction extends AggregateFunction), EXCEPT, duplicate output names, unbound ? (top-level and nested — a nested params.paramValue would be a silent NULL at ES), array literals, negative/overflow-negative LIMIT/OFFSET. Guards re-run on the programmatic path (run(statement) never calls validate()). Clauses (WHERE/GROUP BY/HAVING/ORDER BY) and UNION stay grammar-rejected.
  • Quota/routing: FromlessSelect takes CoreDqlExtension's structural no-cap arm (result ≤ 1 row; pinned — no CapHitKind.QueryResults increment); the internal rewrite always carries LIMIT 1, so it can never reach scroll/PIT, and the statement's own LIMIT/OFFSET are applied engine-side on the assembled row (LIMIT 0 still connection-checks).
  • Docs: documentation/sql/dql_statements.md §"FROM-less SELECT (connection handshake)" — semantics, index disclosure, read-only pre-creation guidance, reject list. Web twin rides the 0.22.0 doc sweep.

Tests

  • FromlessSelectParserSpec (accept/reject/round-trip — every accepted form's .sql re-parses to an equal AST, fix(sql): make ALTER COLUMN ... SET|ADD FIELD parse and actually apply #218 fixed point) + FromlessSelectValidateSpec (guard arms parse-first; rewrite shape pins incl. the LIMIT-isolation pin) — sql, Docker-free.
  • FromlessSelectGatewaySpec (core, Docker-free, NopeClientApi fixtures): bare no-op client ⇒ handshake fails (nothing answers locally); stubbed client ⇒ row assembly, array unwrap, ensure-flow order + keyword mapping, memoization, LIMIT semantics, multi-statement; failing client ⇒ connection error propagates; probe-403 AND create-403 ⇒ original status preserved + pre-creation guidance appended; SHOW TABLES filter.
  • CoreDqlExtensionSpec: FromlessSelect no-cap routing pin (the recorded search statement IS the LIMIT-1 rewrite on the handshake index).
  • Testkit template (GatewayApiIntegrationSpec + ReplGatewayIntegrationSpec): value assertions on real ES — SELECT 1 ⇒ one row, column "1", Integer (class-pinned; Scala's cooperative equality would let a Long pass shouldBe 1); UPPER/arithmetic/'125'::BIGINT; NULL; RANDOM fresh per execution; one clock per statement; 1/0 fails loudly; invisibility on every SHOW TABLES pattern + DESCRIBE still working.
  • Integration green on all four majors (real Docker clusters, sbt17): ES 6.8 (rest + jest), 7.17, 8.18, 9.0 — GatewayApi + REPL legs each.

Release notes (0.22.0)

  1. New statement form: FROM-less SELECT of constant scalar expressions (Support FROM-less SELECT of literal expressions — SELECT 1 is the connection/health idiom of Tableau and Superset and is rejected #251) — executed against Elasticsearch as Painless (connection-check semantics: with the cluster unreachable, SELECT 1 fails). Unblocks Tableau/Superset/DBeaver connection tests and connectionTestQuery pools — which now genuinely test the connection.
  2. The driver creates an index in your cluster at first FROM-less SELECT: softclient4es_handshake (1 shard / 0 replicas, one seeded {"dummy":"dummy"} doc, index.hidden on ES ≥ 7.7). Never listed by SHOW TABLES (nor JDBC/Flight browse); never auto-deleted. Read-only deployments: pre-create + seed it as an administrator (docs §FROM-less SELECT) and lazy creation becomes a no-op probe — the BI service account then only needs read on softclient4es_handshake. Without pre-creation, a read-only session's first SELECT 1 fails with the cluster's own security error plus the pre-creation guidance appended.
  3. Behaviour change: statements like SELECT 1 that previously failed with a parse error now succeed and return one row (column named "1", INTEGER — matching what the 19.4 shim reported).
  4. SELECT <expr> rejections now name the reason (... requires a FROM clause) instead of a grammar-combinator error.
  5. Not included (still rejected): column references / * / aggregates / window functions without FROM; WHERE/GROUP BY/HAVING/ORDER BY without FROM; FROM-less UNION ALL; backtick-quoted aliases (Identifier quoting coverage: backticks rejected in every position; double-quoted table names unparseable in FROM #252); CAST(<bare literal> AS T) — pre-existing grammar gap (CAST/TRY_CAST/CONVERT reject bare literal operands — CAST('125' AS BIGINT) does not parse #267), use '<literal>'::T.

Coordination

🤖 Generated with Claude Code

fupelaqu added a commit that referenced this pull request Sep 2, 2026
…ke index

The read-only-account guidance showed only the raw Elasticsearch PUTs; an
administrator connected through SoftClient4ES itself can equivalently run
CREATE TABLE IF NOT EXISTS + INSERT (lead review of PR #268). Both
statements parse-probed through the real parser; CREATE TABLE IF NOT
EXISTS probes via indexExists (unaffected by the SHOW TABLES filter) and
INSERT executes with refresh=true, so the seeded doc is immediately
searchable.

Refs #251

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fupelaqu added a commit that referenced this pull request Sep 2, 2026
…s; 403 guidance names both routes

Lead review of PR #268, round 2: the SQL pre-creation snippet now pins the
same settings the lazy creation applies (1 shard / 0 replicas — without
OPTIONS the created index takes ES defaults: 1 replica = yellow on
single-node, 5 shards on ES 6.8), and the appended 403 guidance names the
SQL route (CREATE TABLE IF NOT EXISTS ... OPTIONS + INSERT) alongside the
REST PUTs. Snippet parse-probed; rendered settings verified byte-equal to
the lazy creation's ({"index":{"number_of_shards":"1","number_of_replicas":"0"}}).
FromlessSelectGatewaySpec 9/9; 2.12 core compile green.

Refs #251

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fupelaqu and others added 3 commits September 2, 2026 16:42
…arch (story 20.9)

SELECT 1 is the connection/health idiom of the JDBC/SQLAlchemy ecosystem
(Tableau re-issues it on every interaction; Superset's connect test and the
Flight sidecar's schema probe send it too). It now parses as the new
FromlessSelect statement kind and executes AGAINST the cluster: rewritten to
SELECT <items> FROM softclient4es_handshake LIMIT 1 through the unmodified
FROM-ful script_fields pipeline behind the HandshakeEvaluator seam, so a
green handshake genuinely means connected and connectionTestQuery pools
become real tests. The handshake index (1 shard/0 replicas, dummy keyword,
one seeded doc, index.hidden on ES >= 7.7) is created lazily, race-safely,
once per client, and is never listed by SHOW TABLES (covering jdbc getTables
and Flight GET_TABLES through the one TableExecutor seam). Constant-only
guards (window > aggregation > star > name > dependencies > placeholder
walk) re-run on the programmatic path; the statement's LIMIT/OFFSET apply
engine-side on the assembled row and never leak into the rewrite. searchAs
now aborts cleanly on a FROM-less statement instead of a macro MatchError.

Integration green on real ES 6.8 (rest+jest), 7.17, 8.18, 9.0.

Closes #251

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ke index

The read-only-account guidance showed only the raw Elasticsearch PUTs; an
administrator connected through SoftClient4ES itself can equivalently run
CREATE TABLE IF NOT EXISTS + INSERT (lead review of PR #268). Both
statements parse-probed through the real parser; CREATE TABLE IF NOT
EXISTS probes via indexExists (unaffected by the SHOW TABLES filter) and
INSERT executes with refresh=true, so the seeded doc is immediately
searchable.

Refs #251

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…s; 403 guidance names both routes

Lead review of PR #268, round 2: the SQL pre-creation snippet now pins the
same settings the lazy creation applies (1 shard / 0 replicas — without
OPTIONS the created index takes ES defaults: 1 replica = yellow on
single-node, 5 shards on ES 6.8), and the appended 403 guidance names the
SQL route (CREATE TABLE IF NOT EXISTS ... OPTIONS + INSERT) alongside the
REST PUTs. Snippet parse-probed; rendered settings verified byte-equal to
the lazy creation's ({"index":{"number_of_shards":"1","number_of_replicas":"0"}}).
FromlessSelectGatewaySpec 9/9; 2.12 core compile green.

Refs #251

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@fupelaqu
fupelaqu marked this pull request as ready for review September 2, 2026 15:47
@fupelaqu
fupelaqu merged commit 79df7df into main Sep 2, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support FROM-less SELECT of literal expressions — SELECT 1 is the connection/health idiom of Tableau and Superset and is rejected

1 participant