From 8c1c02944c5a181468b1e2177538fbe43011d6f2 Mon Sep 17 00:00:00 2001 From: Deepika Lalwani Date: Tue, 17 Nov 2026 06:02:57 +0000 Subject: [PATCH 1/2] Remove Hadoop 2.6.0, Netty 3, log4j 1 and Jetty jars from adls-plugins The image scan of gcr.io/cdapio/cdap reports these rows against /opt/cdap/master/artifacts/adls-plugins-*.jar/lib/: io.netty:netty 3.6.2.Final (2), log4j:log4j 1.2.17 (1), com.sun.jersey:jersey-core 1.9 (1), Jetty 9.4.54 jetty-server / jetty-http / jetty-security (5) and commons-io 2.8.0 inside velocity-engine-core-2.3 (1). None of them is used by plugin code. orc-mapreduce 1.1.0 only excluded hadoop-common and hadoop-hdfs, so it still packaged hadoop-mapreduce-client-core and hadoop-yarn-common 2.6.0, and with them Netty 3.6.2, log4j 1.2.17, slf4j-log4j12, jersey 1.9, guice 3, jaxb-impl and the codehaus jackson 1.9.13 jars. Hadoop is supplied by CDAP (hadoop-client 3.3.6 is already provided in the parent pom), so exclude org.apache.hadoop:* from orc-mapreduce. The plan marked the Jetty 9.4.54 source as "(verify)"; dependency:tree shows it is avro-mapred -> avro-ipc-jetty (jetty-server, -http, -io, -servlet, -security, -util, -util-ajax). avro-ipc / avro-ipc-jetty are only needed for Avro RPC (tether), not for AvroKeyOutputFormat, so they are excluded from avro-mapred (same as hydrator-plugins HP-1). Avro core used to arrive only through avro-ipc, so it is now declared directly (avro.version, 1.11.4) with its commons-compress / commons-io / commons-lang3 dependencies excluded, which keeps them out of the jar exactly as on develop (Hadoop supplies them). The snappy-java, xz and zstd-jni jars that avro-ipc brought also leave the jar; the ADLS sink has no compression option, so Avro output uses no codec. Versions: - hadoop-mapreduce-client-core / hadoop-yarn-common 2.6.0 -> removed (orc-mapreduce exclusion org.apache.hadoop:*, adls-plugins/pom.xml) - netty 3.6.2.Final, log4j 1.2.17, slf4j-log4j12 1.7.5, jersey-* 1.9, jackson-*-asl 1.9.13 -> removed (same exclusion) - jetty-* 9.4.54.v20240208, avro-ipc, avro-ipc-jetty, velocity-engine-core 2.3, snappy-java 1.1.10.5, xz 1.9, zstd-jni 1.5.5-5 -> removed (avro-mapred exclusions) - avro 1.11.4 -> unchanged, now a direct dependency Bugs: Netty 3 b/578597973, b/578686878; jersey-core b/578444945; log4j 1 b/578595367. Plugins side done for Jetty b/578497046, b/578498486, b/578499995, b/578565223, b/578580375. Risk: low-medium. Only jars no plugin code uses are dropped; ORC and Avro output formats now always run against CDAP's Hadoop 3.3.6 classes (hadoop-common was already excluded, so this was mostly the case). Needs an ADLS sink run with avro, orc and text output on a cluster. Testing: JDK 8. mvn -B clean verify (all 4 modules): BUILD SUCCESS, StructuredToTextTransformerTest: Tests run: 2, Failures: 0, Errors: 0 (the repo has no Checkstyle plugin configured). unzip -l adls-plugins/target/adls-plugins-1.19.0-SNAPSHOT.jar vs develop: 35 lib/ jars removed (the ones listed above plus aopalliance, asm, guice, guice-servlet, javax.inject, javax.annotation-api, javax.servlet-api, jaxb-impl), none added. azure-blob-store jar lib/ unchanged. mvn dependency:tree -pl adls-plugins -Dscope=runtime -Dincludes=org.apache.hadoop,org.eclipse.jetty,io.netty,log4j, com.sun.jersey,org.apache.avro: only avro 1.11.4, avro-mapred 1.11.4 and hadoop-azure-datalake 3.3.6 remain at runtime; Jetty, Netty, jersey and log4j appear only as provided (hadoop-client) or test. --- adls-plugins/pom.xml | 48 +++++++++++++++++++++++++++++++++++++++----- 1 file changed, 43 insertions(+), 5 deletions(-) diff --git a/adls-plugins/pom.xml b/adls-plugins/pom.xml index 84ba8d2..61e867f 100644 --- a/adls-plugins/pom.xml +++ b/adls-plugins/pom.xml @@ -28,6 +28,35 @@ adls-plugins + + + org.apache.avro + avro + ${avro.version} + + + com.fasterxml.jackson.core + jackson-core + + + com.fasterxml.jackson.core + jackson-databind + + + org.apache.commons + commons-compress + + + commons-io + commons-io + + + org.apache.commons + commons-lang3 + + + org.apache.avro avro-mapred @@ -41,6 +70,16 @@ com.fasterxml.jackson.core jackson-databind + + + org.apache.avro + avro-ipc + + + org.apache.avro + avro-ipc-jetty + @@ -64,13 +103,12 @@ orc-mapreduce 1.1.0 + org.apache.hadoop - hadoop-hdfs - - - org.apache.hadoop - hadoop-common + * From db4cfb8f24bc2bb5a30b03e182285f10b5185ff6 Mon Sep 17 00:00:00 2001 From: Deepika Lalwani Date: Tue, 17 Nov 2026 06:03:56 +0000 Subject: [PATCH 2/2] Import jackson-bom 2.18.12 and manage jettison 1.5.4, iq80 snappy 0.5 The image scan of gcr.io/cdapio/cdap reports jackson-core (3) and jackson-databind (11) rows against both adls-plugins-*.jar and azure-blob-store-*.jar (2.13.4 / 2.13.4.2), jettison 1.1 (5 rows) and org.iq80.snappy 0.2 (1 row) against adls-plugins-*.jar. Only jackson-databind was pinned, so jackson-core and jackson-annotations stayed at 2.13.4. jettison and iq80 snappy are direct dependencies of orc-core 1.1.0 and cannot be excluded while adls stays on ORC 1.1.0. Import com.fasterxml.jackson:jackson-bom in the root dependencyManagement and manage jettison and iq80 snappy there. Versions: - jackson-databind 2.13.4.2 -> 2.18.12, jackson-core / -annotations 2.13.4 -> 2.18.12 (root jackson.databind.version replaced by jackson.version + jackson-bom import; adls-plugins and azure-blob-store drop their explicit version) - org.codehaus.jettison:jettison 1.1 -> 1.5.4 (root DM; adls-plugins); stax-api 1.0.1, a dependency of jettison 1.1 only, leaves the jar - org.iq80.snappy:snappy 0.2 -> 0.5 (root DM; adls-plugins) Bugs: none completed by this PR alone. Plugins side done for jettison b/578489064, b/578501183, b/578519621, b/578586713, b/578631934 (they also need Platform / Dockerfile). iq80 snappy is rows only (0 bugs). Risk: low. Jackson 2.15+ adds default stream read limits and 2.18 changes some property introspection; the plugins only use Jackson via Azure / Hadoop client code. jettison 1.5 keeps the org.codehaus.jettison.json API that orc-core 1.1.0 uses (JSON file dump only). Testing: JDK 8. mvn -B clean verify (all 4 modules): BUILD SUCCESS, StructuredToTextTransformerTest: Tests run: 2, Failures: 0, Errors: 0 (no Checkstyle plugin in this repo). mvn dependency:tree -Dincludes=com.fasterxml.jackson*,org.codehaus.jettison,org.iq80.snappy: every module resolves only Jackson 2.18.12; adls-plugins resolves jettison 1.5.4 and snappy 0.5. unzip -l vs the parent branch: adls lib/ now has jackson-* 2.18.12, jettison-1.5.4, snappy-0.5 (old 2.13.4*, jettison-1.1, snappy-0.2, stax-api-1.0.1 gone); azure-blob-store lib/ has jackson-* 2.18.12, nothing else changed. --- adls-plugins/pom.xml | 1 - azure-blob-store/pom.xml | 1 - pom.xml | 20 +++++++++++++++++++- 3 files changed, 19 insertions(+), 3 deletions(-) diff --git a/adls-plugins/pom.xml b/adls-plugins/pom.xml index 61e867f..5675acc 100644 --- a/adls-plugins/pom.xml +++ b/adls-plugins/pom.xml @@ -85,7 +85,6 @@ com.fasterxml.jackson.core jackson-databind - ${jackson.databind.version} junit diff --git a/azure-blob-store/pom.xml b/azure-blob-store/pom.xml index b35f2d1..053c5e3 100644 --- a/azure-blob-store/pom.xml +++ b/azure-blob-store/pom.xml @@ -42,7 +42,6 @@ com.fasterxml.jackson.core jackson-databind - ${jackson.databind.version} org.apache.commons diff --git a/pom.xml b/pom.xml index 1e5b084..a2768b2 100644 --- a/pom.xml +++ b/pom.xml @@ -80,7 +80,7 @@ 6.11.0 3.3.6 2.13.0 - 2.13.4.2 + 2.18.12 32.0.0-jre 1.11.4 widgets @@ -141,6 +141,24 @@ + + com.fasterxml.jackson + jackson-bom + ${jackson.version} + pom + import + + + + org.codehaus.jettison + jettison + 1.5.4 + + + org.iq80.snappy + snappy + 0.5 + io.cdap.cdap cdap-formats