Push the new LINQ operators down into SQL

[davidanthoff]

Last of four PRs adding the LINQ operators Query.jl was missing. Depends on QueryableBackend.jl#15, which depends on QueryOperators.jl#56; the front-end macros are Query.jl#360.

CI will stay red until QueryableBackend 1.1.0 is registered. The [compat] bound is an ordinary QueryableBackend = "1.1"; no feature guards. Locally all 147 test items pass — 24 new, and all 123 pre-existing ones still green.

Four separable concerns

1. Nullability — please review this one first

DuckDB hands back columns typed Union{Missing,T}, but Query's operators use DataValue for an absent value and never Missing. The two backends therefore disagreed the moment a null appeared, which every outer join guarantees. src/result.jl now converts rows on the way out, so piping a DuckDB result back through query operators — or comparing it element-wise against the in-memory backend — sees the same thing either way.

The column interfaces still speak Missing, because that is what the TableTraits _using_missing protocol is defined in terms of and what sinks like DataFrame expect. So |> DataFrame output is completely unchanged, which all 123 pre-existing tests confirm.

This was the option-(b)-shaped change from the plan, but scoped to the row boundary rather than converting column storage to DataValueArray — smaller, and it leaves the sink path untouched. Tests cover both directions: rows carry DataValue, DataFrames carry missing.

2. Errors that explain themselves

build_sql used to fail with a bare Unsupported query operation: <type>. Each node with no SQL equivalent now gets a message naming the operator, the obstacle, and a way forward:

@chunk has no SQL equivalent — SQL has no batching construct. Materialize the query first, e.g. … |> DataFrame |> @chunk(3).

This also covers @groupjoin, @mapmany, @summarize and the pivots, which were already unsupported and previously gave the bare message.

3. Terminal operators

@count, @any, @all, @first, @element_at, @min_by and @max_by become part of the SQL instead of pulling every row into Julia. Anything else — @last, @single, @contains, @aggregate, @sequence_equal — falls through to QueryableBackend's default, which materializes and runs the in-memory implementation. Correctness never depends on that list being complete, so the list can grow later without risk.

This is also what makes @count work against DuckDB at all: QueryOperators.count had no Queryable method before, so df |> @duckdb() |> @count() failed outright. There's a regression test.

4. The SQL itself

LEFT/RIGHT/FULL OUTER JOIN; UNION ALL/UNION/EXCEPT/INTERSECT plus DISTINCT ON forms for the _by variants; ORDER BY ALL for @order (detected via the existing is_identity_lambda, which is why @order needed no new node type); ORDER BY random() for @shuffle; GROUP BY with COUNT(*) for @count_by; and QUALIFY ROW_NUMBER() OVER () … COUNT(*) OVER () for @take_last/@drop_last.

That last one takes its row count from a window function rather than repeating the subquery — repeating it would have desynced the inner query's positional parameters.

Bug fixed along the way: two-input nodes hardcoded source_tbl_2 for their right-hand source, so two joins in one query would have collided. The name is now derived from the node's position in the walked tree, and register_query_sources walks it the same way. A single join still resolves to source_tbl_2, so nothing changes for existing queries.

Known gaps, documented in the README

• Set operations return rows in whatever order DuckDB produces; the in-memory implementation preserves first-seen order. Tests compare sorted.
• @shuffle(rng=...) cannot be pushed down — DuckDB has its own generator — and says so.
• @index is unsupported: it yields (index, item) pairs whose item is a whole row, and SQL has no nested row values.
• The second operand of a join or set operation must itself be a @duckdb() source, as was already true for @join.

🤖 Generated with Claude Code