W-23708419: Actually exclude TCK conformance on PR builds via scalatest tag - #155
Merged
Conversation
The -PskipTCKTests=true flag used Gradle's Test.exclude('**/TCKCliTest.class')
to keep the master-only TCK conformance suite off PR builds. That filter is a
no-op under the maiflai scalatest runner, which ignores Gradle include/exclude
class patterns — so all 708 TCK scenarios still ran on every PR inside the
build-foundation step (~3-4 min/OS), defeating the master-only intent.
Tag the TCK scenarios with a ScalaTest Tag and exclude that tag via the
plugin's `tags { exclude }` DSL (maps to scalatest -l), which the runner does
honor. On PR builds the tagged scenarios are now filtered out while the
module's other IT specs still run; the master-only "Run CLI TCK Conformance"
step is unaffected.
Verified: native-cli-integration-tests:test -PskipTCKTests=true now runs 5
tests instead of 713 (TCK scenarios excluded).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
andres-rad
approved these changes
Aug 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The project convention is that the DataWeave TCK conformance suite runs only on master, not on PRs (see
CLAUDE.md→ CI). CI enforces this two ways:Run CLI TCK Conformancestep is gated ongithub.ref == 'refs/heads/master'✅ (correctly skipped on PRs).build-foundationstep runs on every branch and passes-PskipTCKTests=true, which was supposed to exclude the TCK suite from the aggregatebuild.But
-PskipTCKTests=trueused Gradle'sTest.exclude('**/TCKCliTest.class'), and that filter is a no-op under thecom.github.maiflai.scalatestrunner — the plugin replaces Gradle's test runner and ignores its include/exclude class-pattern filters. So all 708 TCK scenarios still ran on every PR insidebuild-foundation(~3–4 min per OS × 3 OSes), silently defeating the master-only intent.Fix
Use ScalaTest tag exclusion, which the maiflai runner does honor:
Tag(TckConformance).-PskipTCKTests=true, exclude that tag via the plugin'stags { exclude }DSL (maps to scalatest's-l <tag>runner arg — verified against the plugin source ingradle-scalatest0.33).On PR builds the tagged scenarios are now filtered out while the module's other IT specs still run. The master-only
Run CLI TCK Conformancestep is unaffected.Verification
./gradlew native-cli-integration-tests:test -PskipTCKTests=truelocally:Total number of tests run: 5(the non-TCK IT specs), TCK scenarios excluded,BUILD SUCCESSFUL.Note
The
****** Running with weaveSuiteVersion ******log line and TCK zip extraction still appear, because they run in theTCKCliTestclass constructor when the class is instantiated. Tag exclusion filters the test bodies (the expensive nativedw runinvocations), not class construction — so the conformance work no longer runs, but the constructor-level logging remains. Eliminating that entirely would require gating the wholetesttask (a larger behavioral change that also drops the other IT specs from PR builds).🤖 Generated with Claude Code