Repository navigation
Compile CDAP with --release 11 - #16331
sahusanket wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
Code Review
This pull request upgrades the CDAP codebase to support Java 11. Key changes include upgrading ASM to version 9, updating Maven plugins and compiler configurations to target Java 11, adding JAXB dependencies for runtime compatibility, and updating startup scripts to enforce a minimum Java version of 11. Additionally, a minimum Dataproc image version floor of 2.1 is introduced with enforcement and warning modes. Feedback on the changes highlights a syntax issue in the Windows batch script (cdap.bat) regarding double-quote escaping within a for loop, as well as opportunities to simplify verbose null-safety and ternary expressions in AbstractProgramTwillRunnable.
| endlocal | ||
| EXIT /B 1 | ||
| set "ver=" | ||
| for /f "tokens=2 delims=\"" %%v in ("!line!") do set "ver=%%v" |
There was a problem hiding this comment.
In Windows batch files, trying to escape double quotes using \" inside for /f options terminates the options string early, causing the delimiter to be set to \ instead of ". Since there is no backslash in the Java version string, ver remains empty, and the script fails with a version detection error. To correctly use double quotes as a delimiter without enclosing the options in double quotes, use caret escaping for the spaces and the delimiter.
for /f tokens^=2^ delims^=^" %%v in ("!line!") do set "ver=%%v"
| if (Strings.isNullOrEmpty(securityTtl == null ? null : securityTtl.trim()) | ||
| && Strings.isNullOrEmpty(systemTtl == null ? null : systemTtl.trim())) { |
There was a problem hiding this comment.
The null-safety check and trimming logic here is unnecessarily verbose and can be simplified using standard Java idiomatic guards. This improves readability and maintainability.
| if (Strings.isNullOrEmpty(securityTtl == null ? null : securityTtl.trim()) | |
| && Strings.isNullOrEmpty(systemTtl == null ? null : systemTtl.trim())) { | |
| if ((securityTtl == null || securityTtl.trim().isEmpty()) | |
| && (systemTtl == null || systemTtl.trim().isEmpty())) { |
| String effectiveStr = !Strings.isNullOrEmpty(securityTtl == null ? null : securityTtl.trim()) | ||
| ? securityTtl.trim() | ||
| : (systemTtl == null ? null : systemTtl.trim()); |
There was a problem hiding this comment.
This ternary expression can be simplified for better readability by using standard null/empty checks.
| String effectiveStr = !Strings.isNullOrEmpty(securityTtl == null ? null : securityTtl.trim()) | |
| ? securityTtl.trim() | |
| : (systemTtl == null ? null : systemTtl.trim()); | |
| String effectiveStr = (securityTtl != null && !securityTtl.trim().isEmpty()) | |
| ? securityTtl.trim() | |
| : (systemTtl != null ? systemTtl.trim() : null); |
a0dd1b9 to
40ebf6d
Compare
4285a5d to
0f8cf74
Compare
Move CDAP's compile target from Java 8 to Java 11 (--release 11) on develop (aligned with release/6.11): - Build & toolchain: set <release>11</release> in root pom.xml, the data-pipeline plugins archetype (6.12.0-SNAPSHOT), and user-guide plugin sample; bump maven-bundle-plugin to 5.1.9, unify maven-javadoc-plugin at 3.6.3 (doclint=none, release=11), replace findbugs-maven-plugin 3.0.5 with spotbugs-maven-plugin 4.8.6.4, add Temurin 11 setup in .github/workflows/build.yml, and disable the dist maven-jar-plugin jar execution on pom-packaging aggregator modules for Maven 3.9 install. - Bytecode rewriters: keep asm.version at 7.1 (managing asm, asm-commons, asm-tree, asm-analysis, and asm-util) and update all 37 Opcodes.ASM5 visitor sites to Opcodes.ASM7 so Java 11 NestHost and NestMembers attributes are handled cleanly without bumping ASM; add Java 11 nestmate coverage in ClassesTest and SparkRunnerClassLoaderTest. - DNS cache TTL (CDAP-20781): replace the direct sun.net.InetAddressCachePolicy import in AbstractProgramTwillRunnable with initializeDnsCachePolicy(), which sets the public networkaddress.cache.ttl security property to 30 before installing SecurityManager unless the user already set a TTL. - Guice 4.0:no_aop: switch CDAP modules to com.google.inject:guice:4.0:no_aop and exclude plain com.google.inject:guice on transitive dependencies so Guice error formatting on Java 11 (v55) classes does not fail inside Guice 4.0's shaded ASM 5 (without keeping an orphaned unclassified guice dependency entry). TODO(Guava-32): Upgrade guice.version to 5.1.0 and remove no_aop classifier and exclusions once Guava 32 lands. - Java 11+ launch checks: check for Java 11+ (including JEP 322 + delimiter strings and filtering "Picked up ..." lines) in RemoteExecutionTwillPreparer, functions.sh (fixing missing leading space before -Xlog:gc*), and cdap.bat (stopping cli and tx-debugger when the check fails). - EMR Java 11 JAXB & extension TCCL: package jaxb-api 2.3.1 and jaxb-runtime 2.3.8 in cdap-runtime-ext-emr and set the EMR extension classloader as thread context classloader in ElasticMapReduceProvisioner through one helper. - Java 11 platform classloader, AppClassLoader, & test compatibility: replace null parent classloaders in LogPipelineCheck, AbstractArtifactManager, and PackageFilterClassLoader with ClassLoader.getPlatformClassLoader(); fall back to java.class.path (shared ClassLoaders.parseJavaClassPath()) in ClassPath and ClassLoaders when encountering ClassLoader.getSystemClassLoader(); use per-call SimpleDateFormat via parseDateTime() in TimePartitionedFileSetTest and MapReduceWithPartitionedTest for JEP 252 CLDR compatibility; and add @PowerMockIgnore for java.xml in AutoInstallTest.
0f8cf74 to
7959981
Compare
Summary
Move CDAP's compile target from Java 8 to Java 11 (
--release 11) ondevelop(kept aligned withrelease/6.11for clean cherry-picking). Dataproc image version validation is unchanged: the existing1.5minimum stays as it is.Flipping
<release>11</release>by itself breaks CDAP startup and error reporting on Java 11 (v55) class files. This PR pairs the compile target change with the bytecode, DNS cache, Guice, JAXB, script, and toolchain updates needed to build and run on Java 11.Why Each Change Is Needed
Java 11 compile target and build toolchain (
pom.xml, archetype, docs, bundle, javadoc, spotbugs, CI):maven-compiler-pluginfromsource/target 1.8to<release>11</release>in rootpom.xml, the data-pipeline plugins archetype (cdap.version6.12.0-SNAPSHOT), and user-guide plugin samples; keeps Scala bytecode at-target:jvm-1.8.maven-bundle-pluginto5.1.9so bnd can inspectv55class files, unifiesmaven-javadoc-pluginat3.6.3(<doclint>none</doclint>,<release>11</release>), replacesfindbugs-maven-plugin(3.0.5) withspotbugs-maven-plugin(4.8.6.4), and sets Temurin 11 viaactions/setup-java@v4in.github/workflows/build.yml.-P distmaven-jar-pluginjarexecution (<phase>none</phase>,<inherited>false</inherited>) on the four<packaging>pom</packaging>aggregator POMs (cdap,cdap-app-templates,cdap-etl,cdap-etl-archetypes) so Maven 3.9'smaven-install-plugin3.1.4 does not overwrite.pomfiles in.m2with empty.jararchives.Opcodes.ASM7across bytecode rewriters (asm.versionkept at7.1):javac --release 11emitsNestHostandNestMembersattributes for every nested class, enum, and anonymous class. Visitors initialized withOpcodes.ASM5fail withUnsupportedOperationException: This feature requires ASM7whenMainClassLoader,ProgramClassLoader, orHttpHandlerGeneratorloads CDAP or user classes.asm.versionat7.1(managingasm,asm-commons,asm-tree,asm-analysis, andasm-utiluniformly) so the diff stays minimal and identical forrelease/6.11cherry-picking, and updates all 37Opcodes.ASM5visitor sites (AuthEnforceRewriter,DatasetClassRewriter,HttpHandlerGenerator,Classes,SparkClassRewriter, andDefaultArtifactInspector) toOpcodes.ASM7.Java 11-compatible DNS cache TTL initialization (
AbstractProgramTwillRunnable):--release 11forbids importingsun.net.InetAddressCachePolicydirectly (and does not allow--add-exportsfor system modules), so the oldInetAddressCachePolicy.get()check no longer compiles.InetAddressCachePolicycaches DNS lookups forever once aSecurityManageris installed (CDAP-20781).initializeDnsCachePolicy()now sets the publicnetworkaddress.cache.ttlsecurity property to30beforeSystem.setSecurityManager, unless the user already setnetworkaddress.cache.ttlorsun.net.inetaddr.ttl. The JDK reads this property regardless of theSecurityManager, so no internal-class access is needed.Guice
4.0:no_aopclassifier and transitive exclusions:guice-4.0.jarbundles shaded ASM 5 ($ClassReader), which causesConfigurationException.getMessage()andprintStackTrace()to throwUncheckedExecutionException: IllegalArgumentExceptionwhenever a Guice binding error references av55class.com.google.inject:guice:4.0:no_aopand excludes plaincom.google.inject:guicefrom transitive pullers in rootdependencyManagementso onlyguice-4.0-no_aop.jaris packaged, with zero Guava changes required today.guice.versionto5.1.0(which bundles ASM 9.2) and remove theno_aopclassifier and managedcom.google.inject:guiceexclusions.Java 11+ launch checks for SSH and CDAP scripts:
RemoteExecutionTwillPreparerto prefer${JAVA_HOME}/bin/javaand check for Java 11+ (handling JEP 322+pre-release strings such as11+28alongside11.0.xand filteringPicked up ...JVM stderr banners) before launching remote jobs over SSH, and updatesfunctions.sh(fixing the missing leading space before-Xlog:gc*and enforcing Java 11+ incdap_set_javawhile filteringPicked up ...lines) andcdap.bat(skippingPicked up JAVA_TOOL_OPTIONSlines, rejecting non-numeric versions, and stoppingcliandtx-debuggerwhen the check fails).Amazon EMR runtime extension JAXB runtime and extension TCCL on Java 11:
javax.xml.bind:jaxb-api:2.3.1andorg.glassfish.jaxb:jaxb-runtime:2.3.8tocdap-runtime-ext-emrand sets the EMR extension classloader as thread context classloader inElasticMapReduceProvisioner(one private helper wraps each AWS call) so AWS SDK 1.11 XML marshalling (ImportKeyPairRequest,Base64) resolves JAXB from the EMR extension on Java 11 withoutNoClassDefFoundError: javax/xml/bind/JAXBException.Java 11 platform classloader,
AppClassLoaderclasspath resolution, and JEP 252 locale test fixes:nullparent classloaders inLogPipelineCheck,AbstractArtifactManager, andPackageFilterClassLoaderwithClassLoader.getPlatformClassLoader()so JDK 11 platform modules (java.sql,org.ietf.jgss,java.naming,java.xml) remain visible.ClassLoader.getSystemClassLoader()(AppClassLoader) is no longer aURLClassLoader, soClassPath.getClassPathEntriesandClassLoaders.getClassLoaderURLsreturned empty entries whenever called outsideMainClassLoader. Both helpers now fall back to parsingjava.class.path(sharedClassLoaders.parseJavaClassPath(), preserving trailing/on directories) when encountering the system classloader.DateFormat.getDateTimeInstance(SHORT, SHORT, Locale.US)with a per-callSimpleDateFormat("M/d/yy h:mm a", Locale.US)(parseDateTime) inTimePartitionedFileSetTestandMapReduceWithPartitionedTestso hardcoded non-CLDR date strings parse identically under JDK 11's default CLDR locale provider (JEP 252).@PowerMockIgnoreforjava.xmlpackages inAutoInstallTestsoPowerMockRunner'sMockClassLoaderdelegatesjavax.xml.parsers.FactoryFinderto the boot/platform module on JDK 11.Verification
mvn -P templates,dist,k8s,!unit-tests clean installand targeted module unit tests on Temurin 11 (0Checkstyle violations)javap -vconfirmsmajor version: 55on compiled CDAP jarsHttpHandlerGeneratorTest,DatasetClassRewriterTest,AuthEnforceRewriterTest,ClassesTest, andSparkRunnerClassLoaderTeston Java 11v55classes (asm-7.1)AppClassLoaderLogPipelineCheckTest,AbstractArtifactManagerTest, andClassLoaderTestverifyjava.sql.Driver,org.ietf.jgss.GSSManager, andjava.class.pathfallbackno_aopGuiceErrorMessageTestonv55class; zero plainguice-4.0.jarinlib/AbstractProgramTwillRunnableTestplusfunctions.shshell checks (includingJAVA_TOOL_OPTIONSbanner handling)ElasticMapReduceProvisionerTest(JAXB,Base64,ImportKeyPairRequestMarshaller, and extension TCCL set and restored aroundvalidateProperties) +JAXBException.classin packaged EMR exthydrator-plugins,google-cloud, andwranglercompile cleanly on JDK 11 againstv55cdap-api