Skip to content

Remove Hadoop 2.6.0, Netty 3, log4j 1 and Jetty jars from adls-plugins - #169

Open
deepika-lalwani123 wants to merge 1 commit into
developfrom
vuln/az-1-remove-hadoop2-jetty-adls
Open

deepika-lalwani123 wants to merge 1 commit into
developfrom
vuln/az-1-remove-hadoop2-jetty-adls

Conversation

@deepika-lalwani123

@deepika-lalwani123 deepika-lalwani123 commented Nov 17, 2026 •

Copy link
Copy Markdown

Remove Hadoop 2.6.0, Netty 3, log4j 1 and Jetty jars from adls-plugins
Repo: data-integrations/azure
Branch: vuln/az-1-remove-hadoop2-jetty-adls → base develop
Commit: 8c1c029
Part of the CVE fixes for the CDAP 6.11.2 patch release (container image scanning of gcr.io/cdapio/cdap).

The image scan of gcr.io/cdapio/cdap reports these rows against
/opt/cdap/master/artifacts/adls-plugins-*.jar/lib/: io.netty:netty
3.6.2.Final (2), log4j:log4j 1.2.17 (1), com.sun.jersey:jersey-core 1.9
(1), Jetty 9.4.54 jetty-server / jetty-http / jetty-security (5) and
commons-io 2.8.0 inside velocity-engine-core-2.3 (1). None of them is
used by plugin code.

orc-mapreduce 1.1.0 only excluded hadoop-common and hadoop-hdfs, so it
still packaged hadoop-mapreduce-client-core and hadoop-yarn-common
2.6.0, and with them Netty 3.6.2, log4j 1.2.17, slf4j-log4j12, jersey
1.9, guice 3, jaxb-impl and the codehaus jackson 1.9.13 jars. Hadoop is
supplied by CDAP (hadoop-client 3.3.6 is already provided in the
parent pom), so exclude org.apache.hadoop:* from orc-mapreduce.

The plan marked the Jetty 9.4.54 source as "(verify)"; dependency:tree
shows it is avro-mapred -> avro-ipc-jetty (jetty-server, -http, -io,
-servlet, -security, -util, -util-ajax). avro-ipc / avro-ipc-jetty are
only needed for Avro RPC (tether), not for AvroKeyOutputFormat, so they
are excluded from avro-mapred (same as cdapio/hydrator-plugins#2013). Avro core
used to arrive only through avro-ipc, so it is now declared directly
(avro.version, 1.11.4) with its commons-compress / commons-io /
commons-lang3 dependencies excluded, which keeps them out of the jar
exactly as on develop (Hadoop supplies them). The snappy-java, xz and
zstd-jni jars that avro-ipc brought also leave the jar; the ADLS sink
has no compression option, so Avro output uses no codec.

Versions:

  • hadoop-mapreduce-client-core / hadoop-yarn-common 2.6.0 -> removed
    (orc-mapreduce exclusion org.apache.hadoop:*, adls-plugins/pom.xml)
  • netty 3.6.2.Final, log4j 1.2.17, slf4j-log4j12 1.7.5, jersey-* 1.9,
    jackson-*-asl 1.9.13 -> removed (same exclusion)
  • jetty-* 9.4.54.v20240208, avro-ipc, avro-ipc-jetty, velocity-engine-core
    2.3, snappy-java 1.1.10.5, xz 1.9, zstd-jni 1.5.5-5 -> removed
    (avro-mapred exclusions)
  • avro 1.11.4 -> unchanged, now a direct dependency

Plugin-side changes for the Jetty findings are done.

Risk: low-medium. Only jars no plugin code uses are dropped; ORC and
Avro output formats now always run against CDAP's Hadoop 3.3.6 classes
(hadoop-common was already excluded, so this was mostly the case).
Needs an ADLS sink run with avro, orc and text output on a cluster.

Testing: JDK 8. mvn -B clean verify (all 4 modules): BUILD SUCCESS,
StructuredToTextTransformerTest: Tests run: 2, Failures: 0, Errors: 0
(the repo has no Checkstyle plugin configured). unzip -l
adls-plugins/target/adls-plugins-1.19.0-SNAPSHOT.jar vs develop: 35
lib/ jars removed (the ones listed above plus aopalliance, asm,
guice, guice-servlet, javax.inject, javax.annotation-api,
javax.servlet-api, jaxb-impl), none added. azure-blob-store jar lib/
unchanged. mvn dependency:tree -pl adls-plugins -Dscope=runtime
-Dincludes=org.apache.hadoop,org.eclipse.jetty,io.netty,log4j,
com.sun.jersey,org.apache.avro: only avro 1.11.4, avro-mapred 1.11.4
and hadoop-azure-datalake 3.3.6 remain at runtime; Jetty, Netty,
jersey and log4j appear only as provided (hadoop-client) or test.

The image scan of gcr.io/cdapio/cdap reports these rows against
/opt/cdap/master/artifacts/adls-plugins-*.jar/lib/: io.netty:netty
3.6.2.Final (2), log4j:log4j 1.2.17 (1), com.sun.jersey:jersey-core 1.9
(1), Jetty 9.4.54 jetty-server / jetty-http / jetty-security (5) and
commons-io 2.8.0 inside velocity-engine-core-2.3 (1). None of them is
used by plugin code.

orc-mapreduce 1.1.0 only excluded hadoop-common and hadoop-hdfs, so it
still packaged hadoop-mapreduce-client-core and hadoop-yarn-common
2.6.0, and with them Netty 3.6.2, log4j 1.2.17, slf4j-log4j12, jersey
1.9, guice 3, jaxb-impl and the codehaus jackson 1.9.13 jars. Hadoop is
supplied by CDAP (hadoop-client 3.3.6 is already provided in the
parent pom), so exclude org.apache.hadoop:* from orc-mapreduce.

The plan marked the Jetty 9.4.54 source as "(verify)"; dependency:tree
shows it is avro-mapred -> avro-ipc-jetty (jetty-server, -http, -io,
-servlet, -security, -util, -util-ajax). avro-ipc / avro-ipc-jetty are
only needed for Avro RPC (tether), not for AvroKeyOutputFormat, so they
are excluded from avro-mapred (same as hydrator-plugins HP-1). Avro core
used to arrive only through avro-ipc, so it is now declared directly
(avro.version, 1.11.4) with its commons-compress / commons-io /
commons-lang3 dependencies excluded, which keeps them out of the jar
exactly as on develop (Hadoop supplies them). The snappy-java, xz and
zstd-jni jars that avro-ipc brought also leave the jar; the ADLS sink
has no compression option, so Avro output uses no codec.

Versions:
- hadoop-mapreduce-client-core / hadoop-yarn-common 2.6.0 -> removed
  (orc-mapreduce exclusion org.apache.hadoop:*, adls-plugins/pom.xml)
- netty 3.6.2.Final, log4j 1.2.17, slf4j-log4j12 1.7.5, jersey-* 1.9,
  jackson-*-asl 1.9.13 -> removed (same exclusion)
- jetty-* 9.4.54.v20240208, avro-ipc, avro-ipc-jetty, velocity-engine-core
  2.3, snappy-java 1.1.10.5, xz 1.9, zstd-jni 1.5.5-5 -> removed
  (avro-mapred exclusions)
- avro 1.11.4 -> unchanged, now a direct dependency

Bugs: Netty 3 b/578597973, b/578686878; jersey-core b/578444945;
log4j 1 b/578595367. Plugins side done for Jetty b/578497046,
b/578498486, b/578499995, b/578565223, b/578580375.

Risk: low-medium. Only jars no plugin code uses are dropped; ORC and
Avro output formats now always run against CDAP's Hadoop 3.3.6 classes
(hadoop-common was already excluded, so this was mostly the case).
Needs an ADLS sink run with avro, orc and text output on a cluster.

Testing: JDK 8. mvn -B clean verify (all 4 modules): BUILD SUCCESS,
StructuredToTextTransformerTest: Tests run: 2, Failures: 0, Errors: 0
(the repo has no Checkstyle plugin configured). unzip -l
adls-plugins/target/adls-plugins-1.19.0-SNAPSHOT.jar vs develop: 35
lib/ jars removed (the ones listed above plus aopalliance, asm,
guice, guice-servlet, javax.inject, javax.annotation-api,
javax.servlet-api, jaxb-impl), none added. azure-blob-store jar lib/
unchanged. mvn dependency:tree -pl adls-plugins -Dscope=runtime
-Dincludes=org.apache.hadoop,org.eclipse.jetty,io.netty,log4j,
com.sun.jersey,org.apache.avro: only avro 1.11.4, avro-mapred 1.11.4
and hadoop-azure-datalake 3.3.6 remain at runtime; Jetty, Netty,
jersey and log4j appear only as provided (hadoop-client) or test.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the adls-plugins/pom.xml file to optimize dependency packaging. It explicitly adds the org.apache.avro:avro dependency with exclusions for Jackson and Apache Commons, excludes avro-ipc and avro-ipc-jetty from avro-mapred to prevent bringing in Jetty, and simplifies exclusions in orc-mapreduce by using a wildcard exclusion for all org.apache.hadoop artifacts. There are no review comments, and I have no feedback to provide.

@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant