chore(deps): bump substrait from 0.63.0 to 0.65.0 - #24861
Open
alexandrefimov wants to merge 2 commits into
Open
Conversation
The 0.65.0 release of the substrait crate moves its embedded spec from 0.87.0 to 0.102.0, so fifteen spec releases of removals arrive at once. API changes adapted: - FetchRel: the offset/count oneofs are gone, the expression fields are the only form left - window function: Bound and its Preceding/Following are boxed, and a bound carries offset_expr next to the deprecated offset - interval literals: the precision_mode oneof became a plain precision field, and the deprecated microseconds form is gone - aggregates: a grouping set references relation-level expressions by index rather than embedding them - virtual tables: rows are expressions, values was removed - function calls: the deprecated args field was removed in favour of arguments - types and literals: Timestamp, TimestampTz and Time gave way to the precision-carrying variants, and UserDefinedTypeReference to UserDefined - OuterReference: steps_out moved into a oneof alongside a relation reference - Plan requires execution_behavior, RelCommon requires rel_anchor, and RelType gained LateralJoin and TopN - the enum expression form was removed The consumer paths that read the removed fields are deleted rather than kept: those fields are reserved in the spec, so they cannot appear on a decoded plan. The checked-in JSON fixtures that used the old grouping and fetch shapes are rewritten accordingly. The pbjson-types pin moves to 0.9.0 to match the version substrait uses.
…expr as CurrentRow, and read RelCommon from UpdateRel
alexandrefimov
force-pushed
the
substrait-0.65
branch
from
September 1, 2026 19:50
c73279d to
f204495
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
None. Following #20750 and #20876, which carried the previous substrait bumps together with the code changes, this supersedes the two dependabot bumps: #24255 (0.64.1, open since 11 August) and #24842 (0.65.0), both red with no diagnosis.
Rationale for this change
datafusion-substraitdoes not compile againstsubstrait0.64.1 or 0.65.0, so the dependency is stuck at 0.63.0. The two bumps fail for different reasons. 0.64.1 still carries spec 0.87.0 and costs six errors, three of which are not about Substrait at all —pbjson_types::Anymoved when the crate was restructured. 0.65.0 is the real step: it moves the embedded spec from 0.87.0 to 0.102.0 in one release (substrait-io/substrait-rs#531), so fifteen spec releases of removals arrive together, andcargo check -p datafusion-substraitreports 65 errors.What changes are included in this PR?
Grouped by what the spec did rather than by file:
offset_modeandcount_modeare gone;offset_exprandcount_exprare the only form. Thecount == -1sentinel disappears with them — an unset count already means ALL.Boundand itsPreceding/Followingare boxed, and a bound now carriesoffset_exprbeside the deprecatedoffset. The producer writes both, which the field docs allow, and emitsCurrentRowfor a zero distance, sinceoffsetcannot represent zero and setting neither field is invalid. The consumer readsoffset_exprwhen it is set, as the spec requires, maps a zero there toCurrentRow, and supports only an int64 literal: a DataFusion frame bound holds aScalarValue, not an expression.precision_modeoneof became a plainprecisionfield, and the deprecatedmicrosecondsform is gone. The producer already emittedPrecision(9)andPrecision(3), so the arms map across unchanged.parse_flat_grouping_exprsalready filledexpression_referencesnext to the deprecated field, so the producer only stops writing the latter;from_substrait_groupingloses the branch that preferred it, and with it theconsumerandinput_schemaparameters and itsasync.values, the deprecatedargs,Kind::Timestamp/TimestampTz/Time,LiteralType::Timestamp,Kind::UserDefinedTypeReferenceand the enum expression form are all removed.Kind::UserDefinedalready existed and is unchanged.Planrequiresexecution_behavior;RelCommongained an optionalrel_anchorthat its struct literal must now name;RelTypegainedLateralJoinandTopN, andUpdateRelgained acommonthatretrieve_rel_commonnow reads;RexTypegainedExecutionContextVariable, andKindgainedUnbound.OuterReference.steps_outmoved into a oneof next to a relation reference.The consumer branches that read the removed fields are deleted rather than kept. Those fields are
reservedin spec 0.102 —read_rel::VirtualTablecarriesreserved 1; reserved "values";— so they cannot appear on a decoded plan and there is nothing left to stay compatible with. The checked-in fixtures had to move with them: 23 files carriedgroupings[].groupingExpressions, 21 of them with keys inside, and five TPC-H plans usedfetch.count, so aLIMITread back asfetch=None. Both groups are rewritten, with the grouping keys moved to the relation and referenced by index; 24 files change in all.Worth knowing when reading a plan written before 0.102: a lost
fetch.countturns intofetch=None, but lost grouping keys turn an aggregate into a global one — a wrong answer rather than a missing limit, and indistinguishable from a legitimateGROUP BY (), so a consumer cannot detect it.pbjson-typesmoves to 0.9.0 to match the versionsubstraituses; the pin exists to track it.Plan.execution_behavioris documented as required, so it is set, withVARIABLE_EVALUATION_MODE_UNSPECIFIED: DataFusion emits no execution context variables, andPER_PLANwould claim semantics it does not implement.Two hand-written fixtures,
test_plans/multiple_joins.jsonandtest_plans/multilayer_aggregate.substrait.json, are reformatted rather than edited in place: they mix printer styles within one file, so no serializer reproduces them and their diffs are noisier than the rest.What is the testing strategy for this PR?
cargo test -p datafusion-substraitpasses 207 integration and 51 unit tests.cargo clippy -p datafusion-substrait --all-targets -- -D warningsandcargo fmt --all -- --checkare clean, as is clippy fordatafusion-sqllogictestwith thesubstraitfeature.The fixture rewrite was checked for meaning, not just for parsing: for every edited file the number of grouping keys per aggregate is the same before and after, and the referenced indices resolve to the expressions that were previously inline.
cargo test --test sqllogictests --features substrait -- --substrait-round-tripreports 120 failures, the same count asmain, with an identical distribution across failure classes — so the migration moves neither producer nor consumer behaviour on the plans DataFusion builds itself.Are there any user-facing changes?
A consumer no longer reads plans that use the removed fields: grouping sets with inline expressions, virtual tables with
values, function calls withargs, interval literals in themicrosecondsform, the oldTimestamp/TimestampTz/Timetypes and literals, and enum expressions. This is not a choice the crate can make differently — the fields are reserved in the spec version the crate now targets.Two source-level breaks for downstream code:
SubstraitConsumer::consume_enumis gone, so an implementor that overrides it stops compiling, and the re-exported proto types change shape —RexType::WindowFunctionand the windowBoundvariants are boxed.