Skip to content

chore(deps): bump substrait from 0.63.0 to 0.65.0 - #24861

Open
alexandrefimov wants to merge 2 commits into
apache:mainfrom
alexandrefimov:substrait-0.65
Open

chore(deps): bump substrait from 0.63.0 to 0.65.0#24861
alexandrefimov wants to merge 2 commits into
apache:mainfrom
alexandrefimov:substrait-0.65

Conversation

@alexandrefimov

@alexandrefimov alexandrefimov commented Sep 1, 2026

Copy link
Copy Markdown

Which issue does this PR close?

None. Following #20750 and #20876, which carried the previous substrait bumps together with the code changes, this supersedes the two dependabot bumps: #24255 (0.64.1, open since 11 August) and #24842 (0.65.0), both red with no diagnosis.

Rationale for this change

datafusion-substrait does not compile against substrait 0.64.1 or 0.65.0, so the dependency is stuck at 0.63.0. The two bumps fail for different reasons. 0.64.1 still carries spec 0.87.0 and costs six errors, three of which are not about Substrait at all — pbjson_types::Any moved when the crate was restructured. 0.65.0 is the real step: it moves the embedded spec from 0.87.0 to 0.102.0 in one release (substrait-io/substrait-rs#531), so fifteen spec releases of removals arrive together, and cargo check -p datafusion-substrait reports 65 errors.

What changes are included in this PR?

Grouped by what the spec did rather than by file:

  • Fetch. offset_mode and count_mode are gone; offset_expr and count_expr are the only form. The count == -1 sentinel disappears with them — an unset count already means ALL.
  • Window bounds. Bound and its Preceding / Following are boxed, and a bound now carries offset_expr beside the deprecated offset. The producer writes both, which the field docs allow, and emits CurrentRow for a zero distance, since offset cannot represent zero and setting neither field is invalid. The consumer reads offset_expr when it is set, as the spec requires, maps a zero there to CurrentRow, and supports only an int64 literal: a DataFusion frame bound holds a ScalarValue, not an expression.
  • Interval literals. The precision_mode oneof became a plain precision field, and the deprecated microseconds form is gone. The producer already emitted Precision(9) and Precision(3), so the arms map across unchanged.
  • Grouping. A grouping set references relation-level expressions by index. parse_flat_grouping_exprs already filled expression_references next to the deprecated field, so the producer only stops writing the latter; from_substrait_grouping loses the branch that preferred it, and with it the consumer and input_schema parameters and its async.
  • Virtual tables, function arguments, types. values, the deprecated args, Kind::Timestamp / TimestampTz / Time, LiteralType::Timestamp, Kind::UserDefinedTypeReference and the enum expression form are all removed. Kind::UserDefined already existed and is unchanged.
  • New surface. Plan requires execution_behavior; RelCommon gained an optional rel_anchor that its struct literal must now name; RelType gained LateralJoin and TopN, and UpdateRel gained a common that retrieve_rel_common now reads; RexType gained ExecutionContextVariable, and Kind gained Unbound. OuterReference.steps_out moved into a oneof next to a relation reference.

The consumer branches that read the removed fields are deleted rather than kept. Those fields are reserved in spec 0.102 — read_rel::VirtualTable carries reserved 1; reserved "values"; — so they cannot appear on a decoded plan and there is nothing left to stay compatible with. The checked-in fixtures had to move with them: 23 files carried groupings[].groupingExpressions, 21 of them with keys inside, and five TPC-H plans used fetch.count, so a LIMIT read back as fetch=None. Both groups are rewritten, with the grouping keys moved to the relation and referenced by index; 24 files change in all.

Worth knowing when reading a plan written before 0.102: a lost fetch.count turns into fetch=None, but lost grouping keys turn an aggregate into a global one — a wrong answer rather than a missing limit, and indistinguishable from a legitimate GROUP BY (), so a consumer cannot detect it.

pbjson-types moves to 0.9.0 to match the version substrait uses; the pin exists to track it.

Plan.execution_behavior is documented as required, so it is set, with VARIABLE_EVALUATION_MODE_UNSPECIFIED: DataFusion emits no execution context variables, and PER_PLAN would claim semantics it does not implement.

Two hand-written fixtures, test_plans/multiple_joins.json and test_plans/multilayer_aggregate.substrait.json, are reformatted rather than edited in place: they mix printer styles within one file, so no serializer reproduces them and their diffs are noisier than the rest.

What is the testing strategy for this PR?

cargo test -p datafusion-substrait passes 207 integration and 51 unit tests. cargo clippy -p datafusion-substrait --all-targets -- -D warnings and cargo fmt --all -- --check are clean, as is clippy for datafusion-sqllogictest with the substrait feature.

The fixture rewrite was checked for meaning, not just for parsing: for every edited file the number of grouping keys per aggregate is the same before and after, and the referenced indices resolve to the expressions that were previously inline.

cargo test --test sqllogictests --features substrait -- --substrait-round-trip reports 120 failures, the same count as main, with an identical distribution across failure classes — so the migration moves neither producer nor consumer behaviour on the plans DataFusion builds itself.

Are there any user-facing changes?

A consumer no longer reads plans that use the removed fields: grouping sets with inline expressions, virtual tables with values, function calls with args, interval literals in the microseconds form, the old Timestamp / TimestampTz / Time types and literals, and enum expressions. This is not a choice the crate can make differently — the fields are reserved in the spec version the crate now targets.

Two source-level breaks for downstream code: SubstraitConsumer::consume_enum is gone, so an implementor that overrides it stops compiling, and the re-exported proto types change shape — RexType::WindowFunction and the window Bound variants are boxed.

The 0.65.0 release of the substrait crate moves its embedded spec from
0.87.0 to 0.102.0, so fifteen spec releases of removals arrive at once.
API changes adapted:

- FetchRel: the offset/count oneofs are gone, the expression fields are
  the only form left
- window function: Bound and its Preceding/Following are boxed, and a
  bound carries offset_expr next to the deprecated offset
- interval literals: the precision_mode oneof became a plain precision
  field, and the deprecated microseconds form is gone
- aggregates: a grouping set references relation-level expressions by
  index rather than embedding them
- virtual tables: rows are expressions, values was removed
- function calls: the deprecated args field was removed in favour of
  arguments
- types and literals: Timestamp, TimestampTz and Time gave way to the
  precision-carrying variants, and UserDefinedTypeReference to UserDefined
- OuterReference: steps_out moved into a oneof alongside a relation
  reference
- Plan requires execution_behavior, RelCommon requires rel_anchor, and
  RelType gained LateralJoin and TopN
- the enum expression form was removed

The consumer paths that read the removed fields are deleted rather than
kept: those fields are reserved in the spec, so they cannot appear on a
decoded plan. The checked-in JSON fixtures that used the old grouping and
fetch shapes are rewritten accordingly.

The pbjson-types pin moves to 0.9.0 to match the version substrait uses.
@github-actions github-actions Bot added the substrait Changes to the substrait crate label Sep 1, 2026
…expr as CurrentRow, and read RelCommon from UpdateRel
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

substrait Changes to the substrait crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant