fix!: replace assert-based validation with real exceptions - #1058
fix!: replace assert-based validation with real exceptions#1058nielspardon wants to merge 3 commits into
Conversation
ffc0d0e to
cb11d79
Compare
andrew-coleman
left a comment
There was a problem hiding this comment.
One blocker: the NestedList check now rejects SELECT ARRAY[nullable_col, notnull_col], which converts on main. Comments inline.
Three of the new checks are unreachable (VirtualTableScan null elements, toLiteral, requireTable) — the PR description cites them as motivating bugs.
Untested on the isthmus side: every new throw except requireCatalogReader.
| RelNode input = write.getInput().accept(this, context); | ||
| assert relBuilder.getRelOptSchema() != null; | ||
| final RelOptTable targetTable = | ||
| relBuilder.getRelOptSchema().getTableForMember(write.getNames()); | ||
| final RelOptSchema relOptSchema = requireRelOptSchema(); | ||
| final RelOptTable targetTable = relOptSchema.getTableForMember(write.getNames()); |
There was a problem hiding this comment.
Pre-existing, not this PR — flagging for a separate issue. Line 715 converts the input, then case CTAS re-converts it in handleCreateTableAs. With a rel_anchor on the input that throws UnsupportedOperationException: Duplicate rel_anchor=1. Base behaves identically.
| <rule ref="category/java/design.xml/AvoidThrowingRawExceptionTypes" /> | ||
| <rule ref="category/java/errorprone.xml/MissingSerialVersionUID" /> | ||
|
|
||
| <rule name="AvoidAssertStatement" language="java" |
There was a problem hiding this comment.
Please add AvoidAssertStatement to the rule list in AGENTS.md line 126-128.
| } | ||
|
|
||
| /** A {@link RelOptTable} that delegates everything but the schema it claims to belong to. */ | ||
| private static final class DelegatingRelOptTable implements RelOptTable { |
There was a problem hiding this comment.
RelOptTableImpl.create(RelOptSchema, RelDataType, List<String>, Expression) returns the schema you pass from getRelOptSchema() — replaces this class. Also extend()/toRel() currently forward to the undecorated delegate.
Java assertions only run when the host JVM is started with -ea, which is true for Gradle's test JVMs and for almost nothing else. The assert-based invariant checks in :core and :isthmus were therefore enforced in CI and silently skipped in every real deployment. What that costs today: a non-literal where a literal is required is converted to an empty literal (a protobuf oneof getter returns the default instance when its case is not set), an empty NestedList throws IndexOutOfBoundsException from getType() instead of pointing at ExpressionCreator.emptyList(), and a missing Calcite catalog entry becomes an NPE inside Calcite rather than the "Table not found in Calcite catalog" message the same file already produces. Caller-facing invariants -- builder input, proto messages, Calcite RelNodes -- now throw IllegalArgumentException; internal and configuration invariants throw IllegalStateException. One check is dropped rather than converted: the null check on a value just assigned from a cast in SubstraitRelNodeConverter. A custom PMD rule (AvoidAssertStatement) keeps new asserts out of both main and test sources, which is why the one assert in isthmus test code is converted too. BREAKING CHANGE: validation that previously used `assert` now throws unconditionally. Callers catching AssertionError must catch IllegalArgumentException or IllegalStateException instead, and code running without -ea -- i.e. most deployments -- will now see these checks fire: VirtualTableScan and Expression.NestedList reject malformed input both from their builders and from ProtoRelConverter / ProtoExpressionConverter, and VariadicParameterConsistencyValidator throws IllegalArgumentException rather than AssertionError. Closes substrait-io#1047
…ions `LogicalTableModify` needs a `Prepare.CatalogReader`, and neither site that builds one was checking that it had a usable value: - The UPDATE path casts `RelOptTable.getRelOptSchema()`, which Calcite declares `@Nullable`. `TableModify` stores the reader unchecked and only dereferences it later (`getExpectedInputRowType()` for UPDATE), so a null survives conversion and fails as an NPE somewhere else entirely. Dropping the `catalogReader != null` assert here was wrong: the cast succeeds for null, and this is a different schema from the one `requireRelOptSchema()` validates. - Both paths cast a `RelOptSchema` to `Prepare.CatalogReader` without checking the type, so a schema that is not a catalog reader fails as a `ClassCastException` rather than a reportable error. `requireCatalogReader` now narrows the schema at both sites. It uses `instanceof`, so one check covers the null and the wrong-type case, and the message names the actual class. The schema sources are unchanged: UPDATE still reads it from the resolved table, the write path from the RelBuilder.
- NestedList's homogeneity check compares element types modulo nullability. The assert it replaced compared them exactly, which rejected `SELECT ARRAY[not_null_col, nullable_col]` under -ea; SQL list constructors do not cast their values to a common type, so enforcing that in production would break queries that convert today. - Drop the two null-element checks in VirtualTableScan: the generated Immutables builder already requireNonNulls every element, so they were unreachable. Hoist the schema field types out of the per-row loop. - Check the map operand count before the cast that would otherwise throw ClassCastException first, and build the map literal insertion-ordered so it keeps the key order written in the query. - Make requireRelOptSchema/requireCatalogReader protected so subclasses overriding the write and update visits can reuse them. - Drop the @throws tag for the TableModify guard, which today's call sites cannot reach, and add the missing copy() @throws tags on the DDL relations. - Cover the reachable new throws: DDL copy input counts, CASE and map operand counts, the missing RelOptSchema and unknown-table paths, and a list literal of mixed nullability. - Document AvoidAssertStatement in the CONTRIBUTING PMD tripwire list.
45780f2 to
38201af
Compare
assertcompiles behind$assertionsDisabled, so the invariant checks in:coreand:isthmusfired in Gradle's test JVMs (which enable-eaby default) and nowhere else. The invariants were documented by the code and enforced in CI, but not in the place where a malformed plan actually causes damage. And when they do run,AssertionErroris anError, not anException, so a host that wraps plan conversion incatch (Exception e)to report a bad plan does not catch it — it propagates as a hard failure.Two of these checks cost a consumer something today:
Expression.NestedList—getType()readsvalues().get(0), so an empty nested list throwsIndexOutOfBoundsExceptionfrom an unrelated place instead of the assert's "useExpressionCreator.emptyList()" hint.SubstraitRelNodeConvertertargetTable— a missing catalog entry passesnullintoLogicalTableModify.create, which NPEs inside Calcite, rather than producing the clear"Table not found in Calcite catalog"message the same file already produces 150 lines earlier.The rest are invariants that today's call sites cannot violate. They stay as explicit guards — an assert that genuinely cannot fail costs nothing to write as a throw — but they are not fixes for anything.
The rule applied
RelNode) →IllegalArgumentExceptionIllegalStateExceptionassertleft, anywhere.:coreVirtualTableScan.check()— name count vs depth-first named-field count, row shape, rows not nullable, row field types match schemaIllegalArgumentExceptionExpression.NestedList.check()— non-empty, all values the same typeIllegalArgumentExceptionExpressionProtoConverter.toLiteral()— guard; all call sites are staticallyExpression.LiteralIllegalArgumentExceptionVariadicParameterConsistencyValidator(already threwAssertionErrorunconditionally)IllegalArgumentExceptionVirtualTableScan.check()'s compound assert is split into one check per invariant so the message names the mismatched counts, and the schema's field types are now derived once per relation instead of once per row.NestedList's homogeneity check compares element types modulo nullability. Theassertit replaces compared types exactly, which under-earejectedSELECT ARRAY[not_null_col, nullable_col]: SQL list constructors do not cast their values to a common type, so such a list legitimately holds values that differ only in nullability. Enforcing the old comparison in production would have broken queries that convert today.:isthmusSubstraitRelNodeConverter×2 —relBuilder.getRelOptSchema()IllegalStateExceptionSubstraitRelNodeConverter—targetTableIllegalStateException, reusing the existing messageSubstraitRelNodeConverter—catalogReader, asserted on a value just assigned from a castSqlMapValueConstructorCallConverter— operand count evenIllegalArgumentExceptionCallConverters.CASE— operand count oddIllegalArgumentExceptionCreateTable.copy(),CreateView.copy()—inputs.size() == 1IllegalArgumentExceptionSubstraitRelVisitor×2 (INSERT/DELETE, UPDATE) — guard; Calcite'sTableModifyconstructor already dereferences the tableIllegalArgumentExceptionFunctionConverter.matchKeys()— guard; both lists come from the same operand streamIllegalStateExceptionTwo small refactors fall out of this: a
requireTable(TableModify)helper inSubstraitRelVisitor(which also collapses the repeatedmodify.getTable()calls) andrequireRelOptSchema()/requireCatalogReader()helpers inSubstraitRelNodeConverter, shared by the write and update paths andprotectedso subclasses overriding thosevisitmethods can reuse them. ThetargetTablenull check deliberately stays after theswitch— the CTAS branch returns earlier and legitimately has no pre-existing table, so hoisting it to the lookup would breakhandleCreateTableAs.SqlMapValueConstructorCallConverteralso grows two fixes the operand-count check exposed: it ran after the cast that would already have thrownClassCastException, and the map it builds is now insertion-ordered so a map literal keeps the key order written in the query.The stale
@throws AssertionErrorJavadoc tags are updated.Guarding against regressions
A custom PMD rule
AvoidAssertStatement(//AssertStatement) insubstrait-pmd.xmlfails the build on any newassert, and its violation message states theIllegalArgumentException/IllegalStateExceptionrule above at the offending line. PMD scans test source sets too, so the one assert in isthmus test code (RepeatRel.copy()) is converted as well.Two calls worth a second opinion
Plan.Root.check()is the precedent — hardIllegalArgumentExceptionfor the invariant,LOGGER.warnonly for its one legacy allowance.VirtualTableScanrow/schema types still compare with exactType.equals, so nullability must match precisely there. That is the strictest reading of the spec and the most likely thing to reject another producer's plan, but relaxing it would be a semantic change rather than part of this one.:sparkneeded no changes: its Scala sources userequire(...), not the Javaassertkeyword.BREAKING CHANGE: validation that previously used
assertnow throws unconditionally. Callers catchingAssertionErrormust catchIllegalArgumentExceptionorIllegalStateExceptioninstead, and code running without-ea— i.e. most deployments — will now see these checks fire:VirtualTableScanandExpression.NestedListreject malformed input both from their builders and fromProtoRelConverter/ProtoExpressionConverter, andVariadicParameterConsistencyValidatorthrowsIllegalArgumentExceptionrather thanAssertionError.Closes #1047
🤖 Generated with AI