Skip to content

W-23708419: Actually exclude TCK conformance on PR builds via scalatest tag - #155

Merged
mlischetti merged 1 commit into
masterfrom
skip-tck-on-pr-builds
Aug 5, 2026
Merged

W-23708419: Actually exclude TCK conformance on PR builds via scalatest tag#155
mlischetti merged 1 commit into
masterfrom
skip-tck-on-pr-builds

Conversation

@mlischetti

Copy link
Copy Markdown
Contributor

Problem

The project convention is that the DataWeave TCK conformance suite runs only on master, not on PRs (see CLAUDE.md → CI). CI enforces this two ways:

  • The dedicated Run CLI TCK Conformance step is gated on github.ref == 'refs/heads/master' ✅ (correctly skipped on PRs).
  • The earlier build-foundation step runs on every branch and passes -PskipTCKTests=true, which was supposed to exclude the TCK suite from the aggregate build.

But -PskipTCKTests=true used Gradle's Test.exclude('**/TCKCliTest.class'), and that filter is a no-op under the com.github.maiflai.scalatest runner — the plugin replaces Gradle's test runner and ignores its include/exclude class-pattern filters. So all 708 TCK scenarios still ran on every PR inside build-foundation (~3–4 min per OS × 3 OSes), silently defeating the master-only intent.

Fix

Use ScalaTest tag exclusion, which the maiflai runner does honor:

  • Tag the TCK scenarios with a ScalaTest Tag (TckConformance).
  • When -PskipTCKTests=true, exclude that tag via the plugin's tags { exclude } DSL (maps to scalatest's -l <tag> runner arg — verified against the plugin source in gradle-scalatest 0.33).

On PR builds the tagged scenarios are now filtered out while the module's other IT specs still run. The master-only Run CLI TCK Conformance step is unaffected.

Verification

./gradlew native-cli-integration-tests:test -PskipTCKTests=true locally:

  • Before: 708 TCK scenarios executed.
  • After: Total number of tests run: 5 (the non-TCK IT specs), TCK scenarios excluded, BUILD SUCCESSFUL.

Note

The ****** Running with weaveSuiteVersion ****** log line and TCK zip extraction still appear, because they run in the TCKCliTest class constructor when the class is instantiated. Tag exclusion filters the test bodies (the expensive native dw run invocations), not class construction — so the conformance work no longer runs, but the constructor-level logging remains. Eliminating that entirely would require gating the whole test task (a larger behavioral change that also drops the other IT specs from PR builds).

🤖 Generated with Claude Code

The -PskipTCKTests=true flag used Gradle's Test.exclude('**/TCKCliTest.class')
to keep the master-only TCK conformance suite off PR builds. That filter is a
no-op under the maiflai scalatest runner, which ignores Gradle include/exclude
class patterns — so all 708 TCK scenarios still ran on every PR inside the
build-foundation step (~3-4 min/OS), defeating the master-only intent.

Tag the TCK scenarios with a ScalaTest Tag and exclude that tag via the
plugin's `tags { exclude }` DSL (maps to scalatest -l), which the runner does
honor. On PR builds the tagged scenarios are now filtered out while the
module's other IT specs still run; the master-only "Run CLI TCK Conformance"
step is unaffected.

Verified: native-cli-integration-tests:test -PskipTCKTests=true now runs 5
tests instead of 713 (TCK scenarios excluded).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@mlischetti
mlischetti requested a review from a team as a code owner August 5, 2026 21:25
@mlischetti mlischetti changed the title fix(ci): actually exclude TCK conformance on PR builds via scalatest tag W-23708419: Actually exclude TCK conformance on PR builds via scalatest tag Aug 5, 2026
@mlischetti
mlischetti merged commit a18d2a1 into master Aug 5, 2026
4 checks passed
@mlischetti
mlischetti deleted the skip-tck-on-pr-builds branch August 5, 2026 22:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants