Skip to content

Add Queryable nodes for the new LINQ operators - #15

Open
davidanthoff wants to merge 3 commits into
mainfrom
add-missing-linq-operators
Open

davidanthoff wants to merge 3 commits into
mainfrom
add-missing-linq-operators

Conversation

@davidanthoff

Copy link
Copy Markdown
Member

Second of four PRs adding the LINQ operators Query.jl is missing. Depends on QueryOperators.jl#56.

CI will stay red until QueryOperators 1.2.0 is registered. The [compat] bound is an ordinary QueryOperators = "1.2"; no feature guards. Locally the full suite passes (22 items, 14 new) against the QueryOperators branch.

What it does

Adds a Queryable node for each operator introduced in QueryOperators 1.2, so a backend sees them in the query tree instead of having the in-memory implementation run behind its back — outer joins, the seven set operators, the four partitioning operators, reverse/shuffle/index, append/prepend/zip, count_by/aggregate_by/chunk, and of_type/cast. Each gets a describe_node method so @queryplan() renders it rather than falling through to the type-name fallback (a test asserts that).

Three things worth reviewing

A QueryableBinary abstract type. get_source and _collect_nodes! needed a method per node that combines two sources, since those carry outer/inner rather than source. Outer joins, set operators and zip would have added eleven more pairs of near-identical methods. QueryableJoin and QueryableGroupJoin are now subtypes of QueryableBinary, the tree walk has one method covering every two-input node, and a new node only has to pick the right supertype. QueryableBinary <: Queryable and the concrete types are unchanged, so node isa QueryableJoin and anything dispatching on Queryable behave exactly as before — QueryDuckDB needs no change for this.

Terminal-operator support, and a bug it fixes. The tree had no notion of an operator that returns a value, and QueryOperators.count had no method taking a Queryable at all — so df |> @duckdb() |> @count() failed outright today. QueryableScalar now captures one, and execute_scalar dispatches on the root source type:

execute_scalar(q::QueryableScalar) = _execute_scalar(get_source(q.source), q)

_execute_scalar(::QueryableSource, q::QueryableScalar) = q.f(_materialize(q.source), q.args...)

A backend specialises _execute_scalar on its own source type to push the operator down; the fallback materialises and delegates to the in-memory implementation, so every backend is correct for free and push-down is a pure optimisation. count is fixed as a side effect and a regression test pins it. QueryDuckDB will specialise this in the fourth PR.

order adds no node. It lowers to orderby with an identity key selector, so thenby can still follow it and a backend can recognise the identity selector via the is_identity_lambda helper QueryDuckDB already has, and emit ORDER BY ALL.

Smaller: the plain and _by forms of the set operators share a node, with a nothing key selector marking the plain form; zip does the same with its result selector.

Tests call QueryOperators.* directly rather than through Query macros, since those land in the Query.jl PR.

🤖 Generated with Claude Code

davidanthoff and others added 2 commits September 16, 2026 22:25
get_source and _collect_nodes! needed a method per node type that combines two
sources, because those carry `outer`/`inner` rather than `source`. With outer
joins, seven set operators and zip arriving, that is eleven more pairs of
near-identical methods.

Introduce QueryableBinary instead, and subtype QueryableJoin and
QueryableGroupJoin under it. The tree walk now has one method covering every
two-input node, and a new one only has to pick the right supertype.

QueryableBinary <: Queryable and the concrete types are unchanged, so
`node isa QueryableJoin` and anything dispatching on Queryable still behave
exactly as before.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds a node per operator introduced in QueryOperators 1.2, so a backend sees
them in the query tree rather than having the in-memory implementation run
behind its back: the outer joins, the seven set operators, the four
partitioning operators, reverse/shuffle/index, append/prepend/zip,
count_by/aggregate_by/chunk, and of_type/cast. Each gets a describe_node
method so @queryplan() renders it.

order and order_descending add no node — they lower to orderby with an
identity key selector, so thenby can still follow them and a backend can
recognise the identity selector and sort by every column.

The plain and _by forms of the set operators share a node, with a `nothing`
key selector marking the plain form; zip does the same with its result
selector.

Terminal operators needed new machinery. The tree had no notion of an
operator that returns a value, and QueryOperators.count had no method taking
a Queryable at all, so a query ending in @count() failed outright.
QueryableScalar now captures one, and execute_scalar dispatches on the root
source type so a backend can specialise _execute_scalar and translate the
operator; the fallback materialises the source and delegates to the in-memory
implementation, which every backend gets for free. count is fixed as a side
effect, and a regression test pins it.

Requires QueryOperators 1.2, which is not registered yet, so CI here stays
red until it is.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@davidanthoff
davidanthoff force-pushed the add-missing-linq-operators branch from b5c6379 to f781a67 Compare September 17, 2026 05:35
A backend that translates only some terminal operators needs a way to hand
the rest back. Splitting _execute_scalar_fallback out of the QueryableSource
method gives it one, so its coverage can stay partial without affecting
correctness.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant