Repository navigation
Remove Hadoop 2.6.0, Netty 3, log4j 1 and Jetty jars from adls-plugins - #169
Open
deepika-lalwani123 wants to merge 1 commit into
Open
deepika-lalwani123 wants to merge 1 commit into
deepika-lalwani123 wants to merge 1 commit into
Conversation
The image scan of gcr.io/cdapio/cdap reports these rows against /opt/cdap/master/artifacts/adls-plugins-*.jar/lib/: io.netty:netty 3.6.2.Final (2), log4j:log4j 1.2.17 (1), com.sun.jersey:jersey-core 1.9 (1), Jetty 9.4.54 jetty-server / jetty-http / jetty-security (5) and commons-io 2.8.0 inside velocity-engine-core-2.3 (1). None of them is used by plugin code. orc-mapreduce 1.1.0 only excluded hadoop-common and hadoop-hdfs, so it still packaged hadoop-mapreduce-client-core and hadoop-yarn-common 2.6.0, and with them Netty 3.6.2, log4j 1.2.17, slf4j-log4j12, jersey 1.9, guice 3, jaxb-impl and the codehaus jackson 1.9.13 jars. Hadoop is supplied by CDAP (hadoop-client 3.3.6 is already provided in the parent pom), so exclude org.apache.hadoop:* from orc-mapreduce. The plan marked the Jetty 9.4.54 source as "(verify)"; dependency:tree shows it is avro-mapred -> avro-ipc-jetty (jetty-server, -http, -io, -servlet, -security, -util, -util-ajax). avro-ipc / avro-ipc-jetty are only needed for Avro RPC (tether), not for AvroKeyOutputFormat, so they are excluded from avro-mapred (same as hydrator-plugins HP-1). Avro core used to arrive only through avro-ipc, so it is now declared directly (avro.version, 1.11.4) with its commons-compress / commons-io / commons-lang3 dependencies excluded, which keeps them out of the jar exactly as on develop (Hadoop supplies them). The snappy-java, xz and zstd-jni jars that avro-ipc brought also leave the jar; the ADLS sink has no compression option, so Avro output uses no codec. Versions: - hadoop-mapreduce-client-core / hadoop-yarn-common 2.6.0 -> removed (orc-mapreduce exclusion org.apache.hadoop:*, adls-plugins/pom.xml) - netty 3.6.2.Final, log4j 1.2.17, slf4j-log4j12 1.7.5, jersey-* 1.9, jackson-*-asl 1.9.13 -> removed (same exclusion) - jetty-* 9.4.54.v20240208, avro-ipc, avro-ipc-jetty, velocity-engine-core 2.3, snappy-java 1.1.10.5, xz 1.9, zstd-jni 1.5.5-5 -> removed (avro-mapred exclusions) - avro 1.11.4 -> unchanged, now a direct dependency Bugs: Netty 3 b/578597973, b/578686878; jersey-core b/578444945; log4j 1 b/578595367. Plugins side done for Jetty b/578497046, b/578498486, b/578499995, b/578565223, b/578580375. Risk: low-medium. Only jars no plugin code uses are dropped; ORC and Avro output formats now always run against CDAP's Hadoop 3.3.6 classes (hadoop-common was already excluded, so this was mostly the case). Needs an ADLS sink run with avro, orc and text output on a cluster. Testing: JDK 8. mvn -B clean verify (all 4 modules): BUILD SUCCESS, StructuredToTextTransformerTest: Tests run: 2, Failures: 0, Errors: 0 (the repo has no Checkstyle plugin configured). unzip -l adls-plugins/target/adls-plugins-1.19.0-SNAPSHOT.jar vs develop: 35 lib/ jars removed (the ones listed above plus aopalliance, asm, guice, guice-servlet, javax.inject, javax.annotation-api, javax.servlet-api, jaxb-impl), none added. azure-blob-store jar lib/ unchanged. mvn dependency:tree -pl adls-plugins -Dscope=runtime -Dincludes=org.apache.hadoop,org.eclipse.jetty,io.netty,log4j, com.sun.jersey,org.apache.avro: only avro 1.11.4, avro-mapred 1.11.4 and hadoop-azure-datalake 3.3.6 remain at runtime; Jetty, Netty, jersey and log4j appear only as provided (hadoop-client) or test.
This was referenced Nov 17, 2026
There was a problem hiding this comment.
Code Review
This pull request updates the adls-plugins/pom.xml file to optimize dependency packaging. It explicitly adds the org.apache.avro:avro dependency with exclusions for Jackson and Apache Commons, excludes avro-ipc and avro-ipc-jetty from avro-mapred to prevent bringing in Jetty, and simplifies exclusions in orc-mapreduce by using a wildcard exclusion for all org.apache.hadoop artifacts. There are no review comments, and I have no feedback to provide.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Remove Hadoop 2.6.0, Netty 3, log4j 1 and Jetty jars from adls-plugins
Repo: data-integrations/azure
Branch: vuln/az-1-remove-hadoop2-jetty-adls → base develop
Commit: 8c1c029
Part of the CVE fixes for the CDAP 6.11.2 patch release (container image scanning of gcr.io/cdapio/cdap).
The image scan of gcr.io/cdapio/cdap reports these rows against
/opt/cdap/master/artifacts/adls-plugins-*.jar/lib/: io.netty:netty
3.6.2.Final (2), log4j:log4j 1.2.17 (1), com.sun.jersey:jersey-core 1.9
(1), Jetty 9.4.54 jetty-server / jetty-http / jetty-security (5) and
commons-io 2.8.0 inside velocity-engine-core-2.3 (1). None of them is
used by plugin code.
orc-mapreduce 1.1.0 only excluded hadoop-common and hadoop-hdfs, so it
still packaged hadoop-mapreduce-client-core and hadoop-yarn-common
2.6.0, and with them Netty 3.6.2, log4j 1.2.17, slf4j-log4j12, jersey
1.9, guice 3, jaxb-impl and the codehaus jackson 1.9.13 jars. Hadoop is
supplied by CDAP (hadoop-client 3.3.6 is already provided in the
parent pom), so exclude org.apache.hadoop:* from orc-mapreduce.
The plan marked the Jetty 9.4.54 source as "(verify)"; dependency:tree
shows it is avro-mapred -> avro-ipc-jetty (jetty-server, -http, -io,
-servlet, -security, -util, -util-ajax). avro-ipc / avro-ipc-jetty are
only needed for Avro RPC (tether), not for AvroKeyOutputFormat, so they
are excluded from avro-mapred (same as cdapio/hydrator-plugins#2013). Avro core
used to arrive only through avro-ipc, so it is now declared directly
(avro.version, 1.11.4) with its commons-compress / commons-io /
commons-lang3 dependencies excluded, which keeps them out of the jar
exactly as on develop (Hadoop supplies them). The snappy-java, xz and
zstd-jni jars that avro-ipc brought also leave the jar; the ADLS sink
has no compression option, so Avro output uses no codec.
Versions:
(orc-mapreduce exclusion org.apache.hadoop:*, adls-plugins/pom.xml)
jackson-*-asl 1.9.13 -> removed (same exclusion)
2.3, snappy-java 1.1.10.5, xz 1.9, zstd-jni 1.5.5-5 -> removed
(avro-mapred exclusions)
Plugin-side changes for the Jetty findings are done.
Risk: low-medium. Only jars no plugin code uses are dropped; ORC and
Avro output formats now always run against CDAP's Hadoop 3.3.6 classes
(hadoop-common was already excluded, so this was mostly the case).
Needs an ADLS sink run with avro, orc and text output on a cluster.
Testing: JDK 8. mvn -B clean verify (all 4 modules): BUILD SUCCESS,
StructuredToTextTransformerTest: Tests run: 2, Failures: 0, Errors: 0
(the repo has no Checkstyle plugin configured). unzip -l
adls-plugins/target/adls-plugins-1.19.0-SNAPSHOT.jar vs develop: 35
lib/ jars removed (the ones listed above plus aopalliance, asm,
guice, guice-servlet, javax.inject, javax.annotation-api,
javax.servlet-api, jaxb-impl), none added. azure-blob-store jar lib/
unchanged. mvn dependency:tree -pl adls-plugins -Dscope=runtime
-Dincludes=org.apache.hadoop,org.eclipse.jetty,io.netty,log4j,
com.sun.jersey,org.apache.avro: only avro 1.11.4, avro-mapred 1.11.4
and hadoop-azure-datalake 3.3.6 remain at runtime; Jetty, Netty,
jersey and log4j appear only as provided (hadoop-client) or test.