Repository navigation
Bump netty 4.1.75.Final to 4.1.137.Final via netty-bom - #16306
deepika-lalwani123 wants to merge 7 commits into
Conversation
There was a problem hiding this comment.
Code Review
This pull request updates and centralizes various dependency versions across the project, including upgrading Jetty, Netty, Logback, Gson, and RESTEasy, introducing BOM imports for Netty and Jetty, and moving HTTP client/core versions to the parent POM's dependency management. Feedback on the changes suggests further simplifying child POM configurations by removing redundant <version> tags for logback-classic, logback-core, httpclient, and httpcore, as these are now centrally managed.
| <dependency> | ||
| <groupId>ch.qos.logback</groupId> | ||
| <artifactId>logback-classic</artifactId> | ||
| <version>1.2.8</version> | ||
| <version>${logback.version}</version> | ||
| <scope>runtime</scope> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>ch.qos.logback</groupId> | ||
| <artifactId>logback-core</artifactId> | ||
| <version>1.2.8</version> | ||
| <version>${logback.version}</version> | ||
| <scope>runtime</scope> | ||
| </dependency> |
There was a problem hiding this comment.
Since logback-classic and logback-core are already managed in the parent POM's <dependencyManagement> section, specifying the <version> tag here is redundant. Removing the explicit version tags simplifies the configuration and ensures the version is centrally managed.
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-core</artifactId>
<scope>runtime</scope>
</dependency>
| <dependency> | ||
| <groupId>org.apache.httpcomponents</groupId> | ||
| <artifactId>httpclient</artifactId> | ||
| <version>4.5.2</version> | ||
| <version>${httpclient.version}</version> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>org.apache.httpcomponents</groupId> | ||
| <artifactId>httpcore</artifactId> | ||
| <version>4.4.4</version> | ||
| <version>${httpcore.version}</version> | ||
| <scope>compile</scope> | ||
| </dependency> |
There was a problem hiding this comment.
Since httpclient and httpcore are now managed in the parent POM's <dependencyManagement> section, specifying the <version> tag here is redundant. Removing the explicit version tags simplifies the configuration and ensures the version is centrally managed.
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpcore</artifactId>
<scope>compile</scope>
</dependency>
| <dependency> | ||
| <groupId>org.apache.httpcomponents</groupId> | ||
| <artifactId>httpclient</artifactId> | ||
| <version>4.5.2</version> | ||
| <version>${httpclient.version}</version> | ||
| </dependency> |
There was a problem hiding this comment.
Since httpclient is now managed in the parent POM's <dependencyManagement> section, specifying the <version> tag here is redundant. Removing the explicit version tag simplifies the configuration and ensures the version is centrally managed.
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
</dependency>
| <dependency> | ||
| <groupId>org.apache.httpcomponents</groupId> | ||
| <artifactId>httpcore</artifactId> | ||
| <version>4.4.4</version> | ||
| <version>${httpcore.version}</version> | ||
| <scope>compile</scope> | ||
| </dependency> |
There was a problem hiding this comment.
Since httpcore is now managed in the parent POM's <dependencyManagement> section, specifying the <version> tag here is redundant. Removing the explicit version tag simplifies the configuration and ensures the version is centrally managed.
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpcore</artifactId>
<scope>compile</scope>
</dependency>
cdap-data-streams3_2.12 was the only system-artifact module still on maven-bundle-plugin 3.3.0. cdap-data-pipeline3_2.12, cdap-etl-batch and cdap-program-report already use 3.5.0. The bnd in 3.3.0 can't parse the root-level module-info.class shipped in gson 2.8.9 and jackson-annotations 2.18.12 (the CVE bumps in this stack). Packaging fails with "Invalid class file module-info.class (java.lang.ArrayIndexOutOfBoundsException: 19)", and cdap-master -Pdist can't produce the data-streams system artifact. 3.5.0 handles it. Checked locally (JDK 8): - On develop, the data-streams bundle built with 3.3.0 and with 3.5.0 is the same: same 28 embedded jars, same Export-Package and Import-Package, same Require-Capability. - On this stack, packaging fails with 3.3.0 and succeeds with 3.5.0. The only bundle differences from develop are the intended version bumps (gson 2.8.9, logback 1.2.13). maven-bundle-plugin 5.1.9 was also tried. It builds, but changes what gets embedded: 11 to 17 extra jars per bundle, including cdap-api-common, cdap-runtime-spi, scala-library, jetty 9.4.51 and jackson 2.15.1. So it was not used here.
Root pom.xml: - jgit 5.9.0.202009080501-r -> 5.13.5.202508271544-r (stays on 5.x; 6.x needs Java 11). 5.13.4.202507202350-r fixes the same CVEs but was built on JDK 17 without --release 8: it calls the Java 9+ ByteBuffer.mark() overload and fails on Java 8 with NoSuchMethodError (all RepositoryManagerTest cases fail). 5.13.5 is the 5.13.4 fix release for this. - gson.version 2.3.1 -> 2.8.9. - log4j2.slf4j.version (log4j-to-slf4j / log4j-api) 2.20.0 -> 2.25.5. - commons-lang3 3.12.0 -> 3.18.0 (must precede commons-configuration2 2.15.0 in CL-07). cdap-master/pom.xml: remove the commons-io 2.12.0 override in the k8s profile so master/lib uses the managed commons-io 2.15.1. The quartz 2.3.2 bump needs a code change, so it is in the next commit (vuln/cl-09b-code) together with that change; every commit builds. Why: CVE fixes from image scanning of gcr.io/cdapio/cdap. Closes jgit b/578567823 (High) and b/578495338 (Medium). Platform side of log4j-api b/578584043 and commons-io b/578659719. Risk: low to medium. gson is used in many modules and is visible to plugins. Testing: JDK 8, Checkstyle on: cdap-source-control (all tests), cdap-proto (all tests), cdap-common LocationsTest, cdap-app-fabric compile and TimeSchedulerTest; mvn dependency:tree for the bumped artifacts.
Versions: - jackson.version 2.15.1 -> 2.18.12 (jackson-core, jackson-databind, jackson-annotations, jackson-dataformat-cbor, jackson-module-scala_2.12) - jackson-jaxrs-json-provider, jackson-jaxrs-base and jackson-module-jaxb-annotations 2.12.7 -> 2.18.12. These come from hadoop-yarn-common and were not managed; importing com.fasterxml.jackson:jackson-bom after the explicit entries aligns them with databind. Why: image scanning of gcr.io/cdapio/cdap reports Jackson CVEs. 2.18.12 is the minimum version that fixes all of them and runs on Java 8. Bugs: closes b/578521062 and b/578660123; part of the shared Jackson bug set (with CL-04 and CL-10). Merge gate (paired with the SnakeYAML 2.0 commit stacked directly on this one): this commit moves jackson-dataformat-yaml in master/lib (from elasticsearch-x-content) from 2.8.11 to 2.18.12. 2.18.12 needs SnakeYAML 1.33 or later, and master/lib has 1.17 until the next commit; the old 2.8.11 does not work with SnakeYAML 2.0. So the two commits must merge in the same push (one PR with two commits), also on release/6.11. Risk: medium-low. CDAP src/main does not import Jackson (CDAP uses Gson). Users are libraries: AWS SDK (emr), avro (program-report), Hadoop YARN, google-cloud-storage (dataproc), google-http-client (cloudkms). Spark programs keep using Spark's own Jackson. Testing (JDK 8, Checkstyle on): cdap-runtime-ext-emr, cdap-runtime-ext-dataproc, cdap-securestore-ext-cloudkms mvn package pass; ReportGenerationAppTest, cdap-spark-core3_2.12 (43 tests), DataPipelineTest Spark methods on cdap-data-pipeline3_2.12 and Spark2Test on cdap-unit-test-spark3_2.12 pass, except testPySpark, which fails because the local Python 3.13 cannot import pyspark 3.3 (typing.io was removed); unrelated to Jackson. dependency:tree shows every com.fasterxml.jackson artifact at 2.18.12.
- cdap-kubernetes, cdap-credential-ext-gcp-wi: k8s.version (client-java, client-java-api-fluent) 16.0.2 -> 17.0.2, and declare org.yaml:snakeyaml directly. - Root pom.xml: new snakeyaml.version 2.0, managed in dependencyManagement, so every module (including master/lib, where SnakeYAML also arrives through elasticsearch-x-content) resolves to exactly 2.0 instead of depending on Maven nearest-wins mediation. SnakeYAML cannot move to 2.x alone: client-java 16 uses the SafeConstructor/Constructor APIs removed in SnakeYAML 2. client-java 17 is built for SnakeYAML 2.0. No CDAP source change is needed (CDAP does not use SnakeYAML directly; both modules compile unchanged). Why: CVE-2022-1471 (Critical) in SnakeYAML 1.33 from image scanning of gcr.io/cdapio/cdap. Closes b/578585839. Risk: medium. - client-java 17 is built on gson 2.10 and okhttp 4.10; these modules keep their gson 2.8.9 and okhttp 4.9.1 pins (KubeMasterEnvironmentTest and the module tests pass). - master/lib gets jackson-dataformat-yaml through elasticsearch-x-content 6.5.3. Its own 2.8.11 calls SnakeYAML 1.x APIs that 2.0 removed (ParserImpl(StreamReader) and others). The Jackson 2.18.12 commit directly below this one already moves it to 2.18.12, which works with SnakeYAML 2.0. Merge gate: the two commits must merge in the same push (2.18.12 alone does not work with the SnakeYAML 1.17 that master/lib has before this commit), also on release/6.11. Testing (JDK 1.8.0_382, Checkstyle 0 violations): - cdap-kubernetes: mvn package -Pdist, 73 tests, 0 failures (5 skipped), including KubeMasterEnvironmentTest. - cdap-credential-ext-gcp-wi: mvn package -Pdist, 3/3 tests. - cdap-elastic: test-compile. - target/libexec of both modules: client-java* 17.0.2, snakeyaml 2.0. - dependency:tree -Dincludes=org.yaml: snakeyaml 2.0 in cdap-kubernetes, gcp-wi, cdap-elastic, cdap-app-fabric, cdap-standalone and cdap-master -Pdist,k8s.
… commons-text Hadoop 3.3.6 (hadoop-common, hadoop-auth, hadoop-yarn-client) pulls these jars into master/lib, ui/lib and the extensions. Image scanning of gcr.io/cdapio/cdap reports CVEs in all of them. Pin each one in the root dependencyManagement: - org.codehaus.jettison:jettison 1.1 -> 1.5.4 - com.nimbusds:nimbus-jose-jwt 9.8.1 -> 9.37.4 (also drops json-smart, which 9.37.x shades) - commons-beanutils:commons-beanutils 1.9.4 -> 1.11.0 - org.apache.commons:commons-configuration2 2.10.1 -> 2.15.0 - org.apache.commons:commons-text 1.10.0 -> 1.15.0 - commons-io:commons-io 2.15.1 -> 2.17.0 (needed by configuration2 2.15.0) Why commons-io moves: configuration2 2.15.0 builds its file location strategies on org.apache.commons.io.build.AbstractSupplier and calls IOSupplier.getUnchecked(), which commons-io adds in 2.17.0. With commons-io 2.15.1, Hadoop's MetricsConfig (MiniDFSCluster, MetricsSystem) fails with NoSuchMethodError in FileLocatorUtils.<clinit>; UGIProviderTest catches it. 2.17.0 is the lowest commons-io that works and still runs on Java 8. The other commons-io APIs that configuration2 uses (IOUtils.closeQuietly(Closeable, Consumer), AbstractSupplier) are already in 2.17.0. commons-lang3 stays at 3.18.0 (from the jgit/gson/log4j2/commons-lang3 commit). Bugs: b/578523779 (commons-configuration2, CVE-2026-45205), plus the jettison, nimbus-jose-jwt / json-smart and commons-beanutils image-scanning bugs (b/578572706 for beanutils). Some of these also need the cdap-etl-tools removal and the events-writer checkstyle-dependency removal. Risk: low. CDAP src/main does not import any of these packages; only Hadoop uses them. commons-beanutils-core 1.8.3 (via netty-http) is a different artifact and is not changed here. Testing (JDK 1.8.0_382, Checkstyle 0 violations): - cdap-common: mvn package, 321 tests, 0 failures, 2 skipped (incl. LocationsTest). - cdap-security: mvn package, 125 tests, 0 failures (incl. UGIProviderTest with MiniDFSCluster). - cdap-app-fabric: LocalizationUtilsTest, DefaultUGIProviderTest, DefaultSecureStoreServiceTest, 10 tests, 0 failures. - dependency:tree for cdap-common, cdap-security, cdap-app-fabric and cdap-master -Pdist,k8s shows the pinned versions; json-smart is gone. - Still to run: S1 on Dataproc, S4 with security on, S5.
Versions: - cdap-kubernetes, cdap-credential-ext-gcp-wi: protobuf-java 3.21.10 -> 3.25.5 (pulled in by K8s client-java 17.0.2) - cdap-securestore-ext-gcp-secretstore: protobuf-java and protobuf-java-util 3.21.12 -> 3.25.5 (pulled in by google-cloud-secretmanager 2.8.0; -util is kept on the same version as the runtime) Why: image scanning of gcr.io/cdapio/cdap reports protobuf-java CVEs for the jars in ext/environments/k8s, ext/credentialproviders and ext/securestores. 3.25.5 is the minimum version that fixes them and runs on Java 8. The pins are per module only. A root pin would downgrade the Dataproc provisioner (4.33.6) and the Spanner extensions (4.29.0) and break them at runtime. Bugs: the protobuf bug set for the Platform image (CL-11, together with CL-01, CL-04 and the metrics-writer CL-11b). Risk: low-medium. Minor runtime jump within 3.x; CDAP code only uses runtime types (ByteString, FieldMask, Duration, Timestamps). Testing (JDK 8, Checkstyle on): mvn package on cdap-kubernetes (73 tests, 5 skipped), cdap-credential-ext-gcp-wi (3), and cdap-securestore-ext-gcp-secretstore (23) passes. Regression: cdap-runtime-ext-dataproc (85), cdap-storage-ext-spanner (19, 2 skipped) and cdap-messaging-ext-spanner (7, 1 skipped) pass and dependency:tree shows they stay on protobuf 4.33.6 / 4.29.0.
Updates netty.version from 4.1.75.Final to 4.1.137.Final and replaces the three managed io.netty entries (netty-buffer, netty-codec-http, netty-all) in the root dependencyManagement with an import of io.netty:netty-bom. The BOM aligns every netty 4 artifact, including the transitive ones (codec, handler, transport, resolver, epoll/kqueue native classifiers), on one version. Why: image scanning of gcr.io/cdapio/cdap reports netty 4.1.75 CVEs in master/lib. This is the Platform side of the 64 netty bugs (7 Critical); they are fully closed only together with CL-01 (cdap-etl-tools), CL-04 (events-writer Pub/Sub) and the Plugins / Dockerfile tracks. Risk: low to medium. Netty 4.1.x is API compatible; newer releases parse HTTP/1.1 more strictly in the router and netty-http services. CDAP uses the JDK SSL provider, so the OpenSSL hostname change does not apply. Testing: - cdap-common: HttpsEnablerTest, AuthenticationChannelHandlerTest, SpillableBodyConsumerTest, CombineInputStreamTest, RemoteTaskExecutorTest (31/31 pass). - cdap-gateway: NettyRouterHttpTest, NettyRouterHttpsTest, NettyRouterPipelineTest, RouterPathLookupTest, RoutingToDataSetsTest (78 run, 0 failures, 1 skipped by @ignore). - cdap-app-fabric compiles (test-compile); Checkstyle clean. - mvn dependency:tree -Dincludes=io.netty on cdap-common, cdap-gateway, cdap-app-fabric, cdap-watchdog, cdap-data-fabric, cdap-standalone and cdap-master -Pdist,k8s: every io.netty 4 artifact is 4.1.137.Final.
netty-http 1.7.0 creates HttpContentCompressor with the no-arg constructor. With Netty 4.1.137.Final (netty-bom bump), that constructor turns on zstd whenever zstd-jni is on the classpath. On Dataproc, the Hadoop 3.3.x runtime puts zstd-jni 1.4.9-1 first, which has no Zstd.defaultCompressionLevel(). Every channel of the driver's NettyHttpService then fails with NoSuchMethodError, and batch pipeline runs stay in STARTING. netty-http 1.8.0 (cdapio/netty-http#137) builds the compressor with explicit gzip and deflate options, which is what CDAP offered before. Must merge together with the Netty 4.1.137 bump (#16306). Needs the netty-http 1.8.0 release first. Testing (JDK 8, netty-http 1.8.0-SNAPSHOT built from #137): - cdap-common: SpillableBodyConsumerTest, HttpsEnablerTest, RemoteTaskExecutorTest: 12 tests pass. - cdap-gateway: NettyRouter*Test, AuditLogTest, ConfigBasedRequestBlockingTest: 51 tests pass. - mvn dependency:tree: cdap-gateway resolves netty-http 1.8.0(-SNAPSHOT).
4c86655 to
73f445b
Compare
|



Repo: cdapio/cdap
Branch: vuln/cl-03-netty-bom → base develop (stacked; retarget to develop after the lower PRs merge)
Commit: 4c86655
Title
Bump netty 4.1.75.Final to 4.1.137.Final via netty-bom
Description
Updates
netty.versionfrom 4.1.75.Final to 4.1.137.Final. It also replacesthe three managed
io.nettyentries in the rootdependencyManagement(
netty-buffer,netty-codec-http,netty-all) with an import ofio.netty:netty-bom. The BOM puts every netty 4 artifact on the same version,including the transitive ones: codec, handler, transport, resolver, and the
epoll/kqueue native classifiers.
Why: image scanning of gcr.io/cdapio/cdap reports netty 4.1.75 CVEs in
master/lib. This PR is the Platform side of 64 netty bugs (7 Critical). Thebugs close only when these also land: CL-01 (cdap-etl-tools), CL-04
(events-writer Pub/Sub), and the Plugins and Dockerfile tracks.
Risk: low to medium. Netty 4.1.x keeps API compatibility. Newer releases parse
HTTP/1.1 more strictly, which affects the router and the netty-http services.
CDAP uses the JDK SSL provider, so the OpenSSL hostname change does not apply.
Testing:
SpillableBodyConsumerTest, CombineInputStreamTest, RemoteTaskExecutorTest:
31/31 pass.
NettyRouterPipelineTest, RouterPathLookupTest, RoutingToDataSetsTest: 78 run,
0 failures, 1 skipped (
@Ignore).test-compile). Checkstyle passes.mvn dependency:tree -Dincludes=io.nettyon cdap-common, cdap-gateway,cdap-app-fabric, cdap-watchdog, cdap-data-fabric, cdap-standalone, and
cdap-master
-Pdist,k8s: every netty 4 artifact is 4.1.137.Final.Note: Netty 3 (
io.netty:netty:3.10.6.Final, from Hadoop and ZooKeeper) is aseparate artifact. This PR does not change it.