Skip to content

Compile CDAP with --release 11 - #16331

Open
sahusanket wants to merge 1 commit into
developfrom
feat/java11-compile-dp21-develop
Open

sahusanket wants to merge 1 commit into
developfrom
feat/java11-compile-dp21-develop

Conversation

@sahusanket

@sahusanket sahusanket commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Move CDAP's compile target from Java 8 to Java 11 (--release 11) on develop (kept aligned with release/6.11 for clean cherry-picking). Dataproc image version validation is unchanged: the existing 1.5 minimum stays as it is.

Flipping <release>11</release> by itself breaks CDAP startup and error reporting on Java 11 (v55) class files. This PR pairs the compile target change with the bytecode, DNS cache, Guice, JAXB, script, and toolchain updates needed to build and run on Java 11.

Why Each Change Is Needed

  1. Java 11 compile target and build toolchain (pom.xml, archetype, docs, bundle, javadoc, spotbugs, CI):

    • Switches maven-compiler-plugin from source/target 1.8 to <release>11</release> in root pom.xml, the data-pipeline plugins archetype (cdap.version 6.12.0-SNAPSHOT), and user-guide plugin samples; keeps Scala bytecode at -target:jvm-1.8.
    • Bumps maven-bundle-plugin to 5.1.9 so bnd can inspect v55 class files, unifies maven-javadoc-plugin at 3.6.3 (<doclint>none</doclint>, <release>11</release>), replaces findbugs-maven-plugin (3.0.5) with spotbugs-maven-plugin (4.8.6.4), and sets Temurin 11 via actions/setup-java@v4 in .github/workflows/build.yml.
    • Disables the -P dist maven-jar-plugin jar execution (<phase>none</phase>, <inherited>false</inherited>) on the four <packaging>pom</packaging> aggregator POMs (cdap, cdap-app-templates, cdap-etl, cdap-etl-archetypes) so Maven 3.9's maven-install-plugin 3.1.4 does not overwrite .pom files in .m2 with empty .jar archives.
  2. Opcodes.ASM7 across bytecode rewriters (asm.version kept at 7.1):

    • javac --release 11 emits NestHost and NestMembers attributes for every nested class, enum, and anonymous class. Visitors initialized with Opcodes.ASM5 fail with UnsupportedOperationException: This feature requires ASM7 when MainClassLoader, ProgramClassLoader, or HttpHandlerGenerator loads CDAP or user classes.
    • Keeps asm.version at 7.1 (managing asm, asm-commons, asm-tree, asm-analysis, and asm-util uniformly) so the diff stays minimal and identical for release/6.11 cherry-picking, and updates all 37 Opcodes.ASM5 visitor sites (AuthEnforceRewriter, DatasetClassRewriter, HttpHandlerGenerator, Classes, SparkClassRewriter, and DefaultArtifactInspector) to Opcodes.ASM7.
  3. Java 11-compatible DNS cache TTL initialization (AbstractProgramTwillRunnable):

    • --release 11 forbids importing sun.net.InetAddressCachePolicy directly (and does not allow --add-exports for system modules), so the old InetAddressCachePolicy.get() check no longer compiles.
    • Without an explicit TTL, InetAddressCachePolicy caches DNS lookups forever once a SecurityManager is installed (CDAP-20781). initializeDnsCachePolicy() now sets the public networkaddress.cache.ttl security property to 30 before System.setSecurityManager, unless the user already set networkaddress.cache.ttl or sun.net.inetaddr.ttl. The JDK reads this property regardless of the SecurityManager, so no internal-class access is needed.
  4. Guice 4.0:no_aop classifier and transitive exclusions:

    • Plain guice-4.0.jar bundles shaded ASM 5 ($ClassReader), which causes ConfigurationException.getMessage() and printStackTrace() to throw UncheckedExecutionException: IllegalArgumentException whenever a Guice binding error references a v55 class.
    • Switches CDAP modules to com.google.inject:guice:4.0:no_aop and excludes plain com.google.inject:guice from transitive pullers in root dependencyManagement so only guice-4.0-no_aop.jar is packaged, with zero Guava changes required today.
    • TODO (Guava 32 follow-up): Once the Guava 32 upgrade lands, upgrade guice.version to 5.1.0 (which bundles ASM 9.2) and remove the no_aop classifier and managed com.google.inject:guice exclusions.
  5. Java 11+ launch checks for SSH and CDAP scripts:

    • Updates RemoteExecutionTwillPreparer to prefer ${JAVA_HOME}/bin/java and check for Java 11+ (handling JEP 322 + pre-release strings such as 11+28 alongside 11.0.x and filtering Picked up ... JVM stderr banners) before launching remote jobs over SSH, and updates functions.sh (fixing the missing leading space before -Xlog:gc* and enforcing Java 11+ in cdap_set_java while filtering Picked up ... lines) and cdap.bat (skipping Picked up JAVA_TOOL_OPTIONS lines, rejecting non-numeric versions, and stopping cli and tx-debugger when the check fails).
  6. Amazon EMR runtime extension JAXB runtime and extension TCCL on Java 11:

    • Adds javax.xml.bind:jaxb-api:2.3.1 and org.glassfish.jaxb:jaxb-runtime:2.3.8 to cdap-runtime-ext-emr and sets the EMR extension classloader as thread context classloader in ElasticMapReduceProvisioner (one private helper wraps each AWS call) so AWS SDK 1.11 XML marshalling (ImportKeyPairRequest, Base64) resolves JAXB from the EMR extension on Java 11 without NoClassDefFoundError: javax/xml/bind/JAXBException.
  7. Java 11 platform classloader, AppClassLoader classpath resolution, and JEP 252 locale test fixes:

    • Replaces null parent classloaders in LogPipelineCheck, AbstractArtifactManager, and PackageFilterClassLoader with ClassLoader.getPlatformClassLoader() so JDK 11 platform modules (java.sql, org.ietf.jgss, java.naming, java.xml) remain visible.
    • On Java 9+, ClassLoader.getSystemClassLoader() (AppClassLoader) is no longer a URLClassLoader, so ClassPath.getClassPathEntries and ClassLoaders.getClassLoaderURLs returned empty entries whenever called outside MainClassLoader. Both helpers now fall back to parsing java.class.path (shared ClassLoaders.parseJavaClassPath(), preserving trailing / on directories) when encountering the system classloader.
    • Replaces DateFormat.getDateTimeInstance(SHORT, SHORT, Locale.US) with a per-call SimpleDateFormat("M/d/yy h:mm a", Locale.US) (parseDateTime) in TimePartitionedFileSetTest and MapReduceWithPartitionedTest so hardcoded non-CLDR date strings parse identically under JDK 11's default CLDR locale provider (JEP 252).
    • Adds @PowerMockIgnore for java.xml packages in AutoInstallTest so PowerMockRunner's MockClassLoader delegates javax.xml.parsers.FactoryFinder to the boot/platform module on JDK 11.

Verification

Area Verification
Reactor build & unit tests Full mvn -P templates,dist,k8s,!unit-tests clean install and targeted module unit tests on Temurin 11 (0 Checkstyle violations)
Bytecode version javap -v confirms major version: 55 on compiled CDAP jars
ASM7 nestmates HttpHandlerGeneratorTest, DatasetClassRewriterTest, AuthEnforceRewriterTest, ClassesTest, and SparkRunnerClassLoaderTest on Java 11 v55 classes (asm-7.1)
Platform CL & AppClassLoader LogPipelineCheckTest, AbstractArtifactManagerTest, and ClassLoaderTest verify java.sql.Driver, org.ietf.jgss.GSSManager, and java.class.path fallback
Guice no_aop GuiceErrorMessageTest on v55 class; zero plain guice-4.0.jar in lib/
DNS TTL & scripts AbstractProgramTwillRunnableTest plus functions.sh shell checks (including JAVA_TOOL_OPTIONS banner handling)
EMR JAXB & TCCL ElasticMapReduceProvisionerTest (JAXB, Base64, ImportKeyPairRequestMarshaller, and extension TCCL set and restored around validateProperties) + JAXBException.class in packaged EMR ext
Downstream plugins hydrator-plugins, google-cloud, and wrangler compile cleanly on JDK 11 against v55 cdap-api

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request upgrades the CDAP codebase to support Java 11. Key changes include upgrading ASM to version 9, updating Maven plugins and compiler configurations to target Java 11, adding JAXB dependencies for runtime compatibility, and updating startup scripts to enforce a minimum Java version of 11. Additionally, a minimum Dataproc image version floor of 2.1 is introduced with enforcement and warning modes. Feedback on the changes highlights a syntax issue in the Windows batch script (cdap.bat) regarding double-quote escaping within a for loop, as well as opportunities to simplify verbose null-safety and ternary expressions in AbstractProgramTwillRunnable.

Comment thread cdap-standalone/bin/cdap.bat Outdated
endlocal
EXIT /B 1
set "ver="
for /f "tokens=2 delims=\"" %%v in ("!line!") do set "ver=%%v"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

In Windows batch files, trying to escape double quotes using \" inside for /f options terminates the options string early, causing the delimiter to be set to \ instead of ". Since there is no backslash in the Java version string, ver remains empty, and the script fails with a version detection error. To correctly use double quotes as a delimiter without enclosing the options in double quotes, use caret escaping for the spaces and the delimiter.

for /f tokens^=2^ delims^=^" %%v in ("!line!") do set "ver=%%v"

Comment on lines +182 to +183
if (Strings.isNullOrEmpty(securityTtl == null ? null : securityTtl.trim())
&& Strings.isNullOrEmpty(systemTtl == null ? null : systemTtl.trim())) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The null-safety check and trimming logic here is unnecessarily verbose and can be simplified using standard Java idiomatic guards. This improves readability and maintainability.

Suggested change
if (Strings.isNullOrEmpty(securityTtl == null ? null : securityTtl.trim())
&& Strings.isNullOrEmpty(systemTtl == null ? null : systemTtl.trim())) {
if ((securityTtl == null || securityTtl.trim().isEmpty())
&& (systemTtl == null || systemTtl.trim().isEmpty())) {

Comment on lines +194 to +196
String effectiveStr = !Strings.isNullOrEmpty(securityTtl == null ? null : securityTtl.trim())
? securityTtl.trim()
: (systemTtl == null ? null : systemTtl.trim());

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

This ternary expression can be simplified for better readability by using standard null/empty checks.

Suggested change
String effectiveStr = !Strings.isNullOrEmpty(securityTtl == null ? null : securityTtl.trim())
? securityTtl.trim()
: (systemTtl == null ? null : systemTtl.trim());
String effectiveStr = (securityTtl != null && !securityTtl.trim().isEmpty())
? securityTtl.trim()
: (systemTtl != null ? systemTtl.trim() : null);

@sahusanket
sahusanket force-pushed the feat/java11-compile-dp21-develop branch 6 times, most recently from a0dd1b9 to 40ebf6d Compare November 18, 2026 09:02
@sahusanket sahusanket changed the title Compile CDAP with --release 11 and enforce Dataproc 2.1+ minimum Compile CDAP with --release 11 Nov 18, 2026
@sahusanket sahusanket added the build Triggers github actions build label Nov 18, 2026
@sahusanket sahusanket self-assigned this Nov 18, 2026
@sahusanket
sahusanket force-pushed the feat/java11-compile-dp21-develop branch 3 times, most recently from 4285a5d to 0f8cf74 Compare November 18, 2026 11:06
Move CDAP's compile target from Java 8 to Java 11 (--release 11) on
develop (aligned with release/6.11):

- Build & toolchain: set <release>11</release> in root pom.xml, the
  data-pipeline plugins archetype (6.12.0-SNAPSHOT), and user-guide plugin
  sample; bump maven-bundle-plugin to 5.1.9, unify maven-javadoc-plugin
  at 3.6.3 (doclint=none, release=11), replace findbugs-maven-plugin 3.0.5
  with spotbugs-maven-plugin 4.8.6.4, add Temurin 11 setup in
  .github/workflows/build.yml, and disable the dist maven-jar-plugin jar
  execution on pom-packaging aggregator modules for Maven 3.9 install.
- Bytecode rewriters: keep asm.version at 7.1 (managing asm, asm-commons,
  asm-tree, asm-analysis, and asm-util) and update all 37 Opcodes.ASM5
  visitor sites to Opcodes.ASM7 so Java 11 NestHost and NestMembers
  attributes are handled cleanly without bumping ASM; add Java 11
  nestmate coverage in ClassesTest and SparkRunnerClassLoaderTest.
- DNS cache TTL (CDAP-20781): replace the direct sun.net.InetAddressCachePolicy
  import in AbstractProgramTwillRunnable with initializeDnsCachePolicy(), which
  sets the public networkaddress.cache.ttl security property to 30 before
  installing SecurityManager unless the user already set a TTL.
- Guice 4.0:no_aop: switch CDAP modules to com.google.inject:guice:4.0:no_aop
  and exclude plain com.google.inject:guice on transitive dependencies so
  Guice error formatting on Java 11 (v55) classes does not fail inside
  Guice 4.0's shaded ASM 5 (without keeping an orphaned unclassified
  guice dependency entry).
  TODO(Guava-32): Upgrade guice.version to 5.1.0 and remove no_aop
  classifier and exclusions once Guava 32 lands.
- Java 11+ launch checks: check for Java 11+ (including JEP 322
  + delimiter strings and filtering "Picked up ..." lines) in
  RemoteExecutionTwillPreparer, functions.sh (fixing missing leading space
  before -Xlog:gc*), and cdap.bat (stopping cli and tx-debugger when
  the check fails).
- EMR Java 11 JAXB & extension TCCL: package jaxb-api 2.3.1 and
  jaxb-runtime 2.3.8 in cdap-runtime-ext-emr and set the EMR extension
  classloader as thread context classloader in
  ElasticMapReduceProvisioner through one helper.
- Java 11 platform classloader, AppClassLoader, & test compatibility:
  replace null parent classloaders in LogPipelineCheck,
  AbstractArtifactManager, and PackageFilterClassLoader with
  ClassLoader.getPlatformClassLoader(); fall back to java.class.path
  (shared ClassLoaders.parseJavaClassPath()) in ClassPath and
  ClassLoaders when encountering ClassLoader.getSystemClassLoader();
  use per-call SimpleDateFormat via parseDateTime() in
  TimePartitionedFileSetTest and MapReduceWithPartitionedTest for JEP 252
  CLDR compatibility; and add @PowerMockIgnore for java.xml in
  AutoInstallTest.
@sahusanket
sahusanket force-pushed the feat/java11-compile-dp21-develop branch from 0f8cf74 to 7959981 Compare November 19, 2026 10:01

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build Triggers github actions build

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant