Repository navigation
Import jackson-bom 2.18.12 and manage jettison 1.5.4, iq80 snappy 0.5 - #170
deepika-lalwani123 wants to merge 2 commits into
Conversation
The image scan of gcr.io/cdapio/cdap reports these rows against /opt/cdap/master/artifacts/adls-plugins-*.jar/lib/: io.netty:netty 3.6.2.Final (2), log4j:log4j 1.2.17 (1), com.sun.jersey:jersey-core 1.9 (1), Jetty 9.4.54 jetty-server / jetty-http / jetty-security (5) and commons-io 2.8.0 inside velocity-engine-core-2.3 (1). None of them is used by plugin code. orc-mapreduce 1.1.0 only excluded hadoop-common and hadoop-hdfs, so it still packaged hadoop-mapreduce-client-core and hadoop-yarn-common 2.6.0, and with them Netty 3.6.2, log4j 1.2.17, slf4j-log4j12, jersey 1.9, guice 3, jaxb-impl and the codehaus jackson 1.9.13 jars. Hadoop is supplied by CDAP (hadoop-client 3.3.6 is already provided in the parent pom), so exclude org.apache.hadoop:* from orc-mapreduce. The plan marked the Jetty 9.4.54 source as "(verify)"; dependency:tree shows it is avro-mapred -> avro-ipc-jetty (jetty-server, -http, -io, -servlet, -security, -util, -util-ajax). avro-ipc / avro-ipc-jetty are only needed for Avro RPC (tether), not for AvroKeyOutputFormat, so they are excluded from avro-mapred (same as hydrator-plugins HP-1). Avro core used to arrive only through avro-ipc, so it is now declared directly (avro.version, 1.11.4) with its commons-compress / commons-io / commons-lang3 dependencies excluded, which keeps them out of the jar exactly as on develop (Hadoop supplies them). The snappy-java, xz and zstd-jni jars that avro-ipc brought also leave the jar; the ADLS sink has no compression option, so Avro output uses no codec. Versions: - hadoop-mapreduce-client-core / hadoop-yarn-common 2.6.0 -> removed (orc-mapreduce exclusion org.apache.hadoop:*, adls-plugins/pom.xml) - netty 3.6.2.Final, log4j 1.2.17, slf4j-log4j12 1.7.5, jersey-* 1.9, jackson-*-asl 1.9.13 -> removed (same exclusion) - jetty-* 9.4.54.v20240208, avro-ipc, avro-ipc-jetty, velocity-engine-core 2.3, snappy-java 1.1.10.5, xz 1.9, zstd-jni 1.5.5-5 -> removed (avro-mapred exclusions) - avro 1.11.4 -> unchanged, now a direct dependency Bugs: Netty 3 b/578597973, b/578686878; jersey-core b/578444945; log4j 1 b/578595367. Plugins side done for Jetty b/578497046, b/578498486, b/578499995, b/578565223, b/578580375. Risk: low-medium. Only jars no plugin code uses are dropped; ORC and Avro output formats now always run against CDAP's Hadoop 3.3.6 classes (hadoop-common was already excluded, so this was mostly the case). Needs an ADLS sink run with avro, orc and text output on a cluster. Testing: JDK 8. mvn -B clean verify (all 4 modules): BUILD SUCCESS, StructuredToTextTransformerTest: Tests run: 2, Failures: 0, Errors: 0 (the repo has no Checkstyle plugin configured). unzip -l adls-plugins/target/adls-plugins-1.19.0-SNAPSHOT.jar vs develop: 35 lib/ jars removed (the ones listed above plus aopalliance, asm, guice, guice-servlet, javax.inject, javax.annotation-api, javax.servlet-api, jaxb-impl), none added. azure-blob-store jar lib/ unchanged. mvn dependency:tree -pl adls-plugins -Dscope=runtime -Dincludes=org.apache.hadoop,org.eclipse.jetty,io.netty,log4j, com.sun.jersey,org.apache.avro: only avro 1.11.4, avro-mapred 1.11.4 and hadoop-azure-datalake 3.3.6 remain at runtime; Jetty, Netty, jersey and log4j appear only as provided (hadoop-client) or test.
The image scan of gcr.io/cdapio/cdap reports jackson-core (3) and jackson-databind (11) rows against both adls-plugins-*.jar and azure-blob-store-*.jar (2.13.4 / 2.13.4.2), jettison 1.1 (5 rows) and org.iq80.snappy 0.2 (1 row) against adls-plugins-*.jar. Only jackson-databind was pinned, so jackson-core and jackson-annotations stayed at 2.13.4. jettison and iq80 snappy are direct dependencies of orc-core 1.1.0 and cannot be excluded while adls stays on ORC 1.1.0. Import com.fasterxml.jackson:jackson-bom in the root dependencyManagement and manage jettison and iq80 snappy there. Versions: - jackson-databind 2.13.4.2 -> 2.18.12, jackson-core / -annotations 2.13.4 -> 2.18.12 (root jackson.databind.version replaced by jackson.version + jackson-bom import; adls-plugins and azure-blob-store drop their explicit version) - org.codehaus.jettison:jettison 1.1 -> 1.5.4 (root DM; adls-plugins); stax-api 1.0.1, a dependency of jettison 1.1 only, leaves the jar - org.iq80.snappy:snappy 0.2 -> 0.5 (root DM; adls-plugins) Bugs: none completed by this PR alone. Plugins side done for jettison b/578489064, b/578501183, b/578519621, b/578586713, b/578631934 (they also need Platform / Dockerfile). iq80 snappy is rows only (0 bugs). Risk: low. Jackson 2.15+ adds default stream read limits and 2.18 changes some property introspection; the plugins only use Jackson via Azure / Hadoop client code. jettison 1.5 keeps the org.codehaus.jettison.json API that orc-core 1.1.0 uses (JSON file dump only). Testing: JDK 8. mvn -B clean verify (all 4 modules): BUILD SUCCESS, StructuredToTextTransformerTest: Tests run: 2, Failures: 0, Errors: 0 (no Checkstyle plugin in this repo). mvn dependency:tree -Dincludes=com.fasterxml.jackson*,org.codehaus.jettison,org.iq80.snappy: every module resolves only Jackson 2.18.12; adls-plugins resolves jettison 1.5.4 and snappy 0.5. unzip -l vs the parent branch: adls lib/ now has jackson-* 2.18.12, jettison-1.5.4, snappy-0.5 (old 2.13.4*, jettison-1.1, snappy-0.2, stax-api-1.0.1 gone); azure-blob-store lib/ has jackson-* 2.18.12, nothing else changed.
There was a problem hiding this comment.
Code Review
This pull request refactors dependency management across several modules. It introduces the Jackson BOM to manage Jackson dependencies consistently, upgrades Jackson to version 2.18.12, and adds explicit dependency management for Jettison and Snappy to address CVEs. Additionally, it refactors exclusions in adls-plugins to prevent unwanted transitive dependencies (such as Hadoop and Jetty) from being packaged. The review feedback suggests defining Maven properties for the newly added Jettison and Snappy versions in the properties section of the root pom.xml instead of hardcoding them in dependencyManagement.
| <hadoop.version>3.3.6</hadoop.version> | ||
| <cdap.plugin.version>2.13.0</cdap.plugin.version> | ||
| <jackson.databind.version>2.13.4.2</jackson.databind.version> | ||
| <jackson.version>2.18.12</jackson.version> |
There was a problem hiding this comment.
To maintain consistency with other dependency versions in this project, it is recommended to define the versions for jettison and snappy as properties in the <properties> section instead of hardcoding them in <dependencyManagement>.
| <jackson.version>2.18.12</jackson.version> | |
| <jackson.version>2.18.12</jackson.version> | |
| <jettison.version>1.5.4</jettison.version> | |
| <snappy.version>0.5</snappy.version> |
| <!-- orc-core 1.1.0 depends on jettison 1.1 and iq80 snappy 0.2 (CVE fixes in 1.5.4 / 0.5). --> | ||
| <dependency> | ||
| <groupId>org.codehaus.jettison</groupId> | ||
| <artifactId>jettison</artifactId> | ||
| <version>1.5.4</version> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>org.iq80.snappy</groupId> | ||
| <artifactId>snappy</artifactId> | ||
| <version>0.5</version> | ||
| </dependency> |
There was a problem hiding this comment.
Use the newly defined properties ${jettison.version} and ${snappy.version} here to avoid hardcoding the versions in <dependencyManagement>.
| <!-- orc-core 1.1.0 depends on jettison 1.1 and iq80 snappy 0.2 (CVE fixes in 1.5.4 / 0.5). --> | |
| <dependency> | |
| <groupId>org.codehaus.jettison</groupId> | |
| <artifactId>jettison</artifactId> | |
| <version>1.5.4</version> | |
| </dependency> | |
| <dependency> | |
| <groupId>org.iq80.snappy</groupId> | |
| <artifactId>snappy</artifactId> | |
| <version>0.5</version> | |
| </dependency> | |
| <!-- orc-core 1.1.0 depends on jettison 1.1 and iq80 snappy 0.2 (CVE fixes in 1.5.4 / 0.5). --> | |
| <dependency> | |
| <groupId>org.codehaus.jettison</groupId> | |
| <artifactId>jettison</artifactId> | |
| <version>${jettison.version}</version> | |
| </dependency> | |
| <dependency> | |
| <groupId>org.iq80.snappy</groupId> | |
| <artifactId>snappy</artifactId> | |
| <version>${snappy.version}</version> | |
| </dependency> |
|



Import jackson-bom 2.18.12 and manage jettison 1.5.4, iq80 snappy 0.5
Repo: data-integrations/azure
Branch: vuln/az-2-jackson-jettison-snappy → base develop (stacked)
Commit: db4cfb8
Part of the CVE fixes for the CDAP 6.11.2 patch release (container image scanning of gcr.io/cdapio/cdap).
The image scan of gcr.io/cdapio/cdap reports jackson-core (3) and
jackson-databind (11) rows against both adls-plugins-.jar and
azure-blob-store-.jar (2.13.4 / 2.13.4.2), jettison 1.1 (5 rows) and
org.iq80.snappy 0.2 (1 row) against adls-plugins-*.jar. Only
jackson-databind was pinned, so jackson-core and jackson-annotations
stayed at 2.13.4. jettison and iq80 snappy are direct dependencies of
orc-core 1.1.0 and cannot be excluded while adls stays on ORC 1.1.0.
Import com.fasterxml.jackson:jackson-bom in the root
dependencyManagement and manage jettison and iq80 snappy there.
Versions:
2.13.4 -> 2.18.12 (root jackson.databind.version replaced by
jackson.version + jackson-bom import; adls-plugins and
azure-blob-store drop their explicit version)
stax-api 1.0.1, a dependency of jettison 1.1 only, leaves the jar
None of the findings are fully resolved by this PR alone. The plugin
changes for the jettison findings are complete here (they also require
updates in the CDAP platform / Dockerfile).
Risk: low. Jackson 2.15+ adds default stream read limits and 2.18
changes some property introspection; the plugins only use Jackson via
Azure / Hadoop client code. jettison 1.5 keeps the
org.codehaus.jettison.json API that orc-core 1.1.0 uses (JSON file
dump only).
Testing: JDK 8. mvn -B clean verify (all 4 modules): BUILD SUCCESS,
StructuredToTextTransformerTest: Tests run: 2, Failures: 0, Errors: 0
(no Checkstyle plugin in this repo). mvn dependency:tree
-Dincludes=com.fasterxml.jackson*,org.codehaus.jettison,org.iq80.snappy:
every module resolves only Jackson 2.18.12; adls-plugins resolves
jettison 1.5.4 and snappy 0.5. unzip -l vs the parent branch: adls
lib/ now has jackson-* 2.18.12, jettison-1.5.4, snappy-0.5 (old
2.13.4*, jettison-1.1, snappy-0.2, stax-api-1.0.1 gone); azure-blob-store
lib/ has jackson-* 2.18.12, nothing else changed.