Repository navigation
Upgrade avro to 1.11.4 and commons-lang3 to 3.18.0 - #254
deepika-lalwani123 wants to merge 3 commits into
Conversation
kafka-plugins-client declared netty-all, netty-http and the kafka_2.12 broker at compile scope, so the plugin jar packaged about 30 Netty 4.1.75 / 4.1.48 modules, kafka_2.12 2.6.0 and ZooKeeper 3.5.8 under lib/. Image scanning of gcr.io/cdapio/cdap reports the Netty, kafka_2.12 and ZooKeeper CVEs against /opt/cdap/master/artifacts/kafka-plugins-client-*.jar/lib/. Only the embedded-broker unit tests use these jars: every kafka.*, scala.*, io.netty.*, io.cdap.http.* and org.apache.zookeeper.* import is under src/test. Main code uses only org.apache.kafka.clients/common from kafka-clients. The root POM already declares netty-all and netty-http as test scope; the client module overrode that. Versions: - io.netty:netty-all 4.1.75.Final: compile -> test (kafka-plugins-client) - io.cdap.http:netty-http 1.3.0: compile -> test (kafka-plugins-client) - org.apache.kafka:kafka_2.12 2.6.0: compile -> test (kafka-plugins-client), which also drops zookeeper 3.5.8, scala-library/-reflect 2.12.11, jackson-module-scala and metrics-core from the plugin jar Bugs: b/578427766, b/578434052, b/578497139 (kafka_2.12); Plugins side done for the kafka-plugins Netty bugs and zookeeper b/578633351. Risk: low. No main-code dependency on the removed jars. Scala comes from the Spark runtime that already provides spark-streaming at runtime. Testing: JDK 8, Checkstyle on (0 violations). mvn -B clean install (all modules): BUILD SUCCESS, 13 tests, 0 failures (KafkaBatchSourceTest, KafkaSinkAndAlertsPublisherTest, KafkaStreamingSource*Test, KafkaStreamingSourceUtilTest). unzip -l kafka-plugins-client-3.5.0-SNAPSHOT.jar: lib/ goes from 74 to 26 jars; no netty-*, kafka_2.12-*, zookeeper-* or scala-* left; nothing added. mvn dependency:tree -Dincludes=io.netty,org.apache.kafka, org.apache.zookeeper,io.cdap.http: kafka_2.12, netty-all and netty-http are test scope; the only compile-scope match is kafka-clients 2.6.0.
The kafka plugin jar packages jackson-databind 2.10.5.1 and
jackson-core 2.10.5. Image scanning of gcr.io/cdapio/cdap reports the
jackson-databind and jackson-core CVEs against
/opt/cdap/master/artifacts/kafka-plugins-client-*.jar/lib/.
Import com.fasterxml.jackson:jackson-bom in the root
dependencyManagement so every Jackson 2 module (including the ones that
come transitively through kafka_2.12 and the CDAP test jars) resolves to
one version.
Versions:
- jackson 2.10.5.1 -> 2.18.12 (root jackson.version, now also the
jackson-bom import version; kafka-plugins-client jackson-databind
keeps using ${jackson.version})
Bugs: no single-ID list for the kafka-plugins Jackson rows in the plan;
this is the Plugins side for the kafka-plugins jackson-databind and
jackson-core rows.
Risk: low. Jackson 2.x minor upgrade; plugin main code has no direct
Jackson imports. jackson-core-asl / jackson-mapper-asl 1.9.13 (Jackson
1, from avro 1.8.2) are untouched here and go away with the avro
upgrade.
Testing: JDK 8, Checkstyle on (0 violations).
mvn -B clean install (all modules): BUILD SUCCESS, 13 tests, 0 failures.
mvn dependency:tree -pl kafka-plugins-client -Dincludes=
com.fasterxml.jackson.*: every Jackson 2 artifact resolves to 2.18.12;
compile scope = jackson-databind/-core/-annotations 2.18.12. Plugin jar
lib/ has jackson-databind-2.18.12, jackson-core-2.18.12,
jackson-annotations-2.18.12 and no 2.10.x jar.
The kafka plugin jar packages avro 1.8.2 (which also shades Guava 11), commons-compress 1.8.1 and commons-lang3 3.9. Image scanning of gcr.io/cdapio/cdap reports the avro, commons-compress and commons-lang3 CVEs against /opt/cdap/master/artifacts/kafka-plugins-client-*.jar/lib/. avro 1.9 removed the Jackson 1 and Joda-Time APIs. Plugin main code has no direct avro imports (avro reaches it through cdap-formats), and the build compiles and the unit tests pass without code changes. commons-compress 1.26.x calls commons-io 2.12+ APIs (for example CloseShieldInputStream.wrap in the bzip2 codec that avro uses). Without a direct dependency Maven mediated commons-io to 2.4 from the provided hadoop-common, and it was not packaged, so the plugin would hit NoSuchMethodError / NoClassDefFoundError on the avro bzip2/xz/zstd codecs. kafka-plugins-client now declares commons-io 2.16.1. Versions: - avro 1.8.2 -> 1.11.4 (root avro.version) - commons-compress 1.8.1 -> 1.26.2 (root dependencyManagement) - commons-lang3 3.9 -> 3.18.0 (root dependency) - commons-io (not packaged, 2.4 at build time) -> 2.16.1 (kafka-plugins-client, new direct dependency) Bugs: no single-ID list for the kafka-plugins avro / commons-compress rows in the plan; this is the Plugins side for those rows. Also removes the Guava 11 shaded in avro 1.8.2 (Guava bugs b/578439199, b/578506400, b/578653882, b/578733862 stay out of scope; guava-13.0.1 is still packaged). Risk: medium. avro crosses the 1.9 API break (Jackson 1 / Joda removal). jackson-core-asl / jackson-mapper-asl 1.9.13 are no longer packaged. Needs a Kafka source/sink run with the avro format in a sandbox. Testing: JDK 8, Checkstyle on (0 violations). mvn -B clean install (all modules): BUILD SUCCESS, 13 tests, 0 failures. mvn dependency:tree -pl kafka-plugins-client: avro 1.11.4 -> commons-compress 1.26.2, commons-io 2.16.1, commons-lang3 3.18.0 (all compile); no avro 1.8.2 / commons-compress 1.8.1 at compile scope (avro 1.8.2 remains only under provided spark-core avro-mapred). Plugin jar lib/: avro-1.11.4, commons-compress-1.26.2, commons-io-2.16.1, commons-lang3-3.18.0; no jackson-*-asl, xz-1.5 or snappy-java-1.1.1.3. Not run: Kafka avro-format pipeline in a sandbox (P2).
There was a problem hiding this comment.
Code Review
This pull request updates several dependency versions, including upgrading Avro to 1.11.4, Jackson to 2.18.12, and commons-lang3 to 3.18.0, while introducing dependency management for Jackson modules and commons-compress to address security vulnerabilities. It also adds a direct dependency on commons-io in the Kafka client plugin and applies test scopes to Netty and Kafka dependencies. The review feedback suggests centralizing the commons-io version as a property in the parent POM to avoid hardcoding, and highlights that the Netty dependency declarations in the child POM are redundant and should be removed.
| <logback.version>1.0.9</logback.version> | ||
| <slf4j.version>1.7.25</slf4j.version> | ||
| <jackson.version>2.10.5.1</jackson.version> | ||
| <jackson.version>2.18.12</jackson.version> |
There was a problem hiding this comment.
| <dependency> | ||
| <groupId>commons-io</groupId> | ||
| <artifactId>commons-io</artifactId> | ||
| <version>2.16.1</version> |
| <groupId>io.netty</groupId> | ||
| <artifactId>netty-all</artifactId> | ||
| <version>${netty.version}</version> | ||
| <scope>test</scope> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>io.cdap.http</groupId> | ||
| <artifactId>netty-http</artifactId> | ||
| <version>${netty-http.version}</version> | ||
| <scope>test</scope> | ||
| </dependency> |
There was a problem hiding this comment.
The dependencies netty-all and netty-http are already declared in the parent POM's <dependencies> section (lines 292-303) with <scope>test</scope>. Since they are automatically inherited by all submodules, these declarations in kafka-plugins-client/pom.xml are completely redundant and can be safely removed.
Upgrade avro to 1.11.4 and commons-lang3 to 3.18.0
Repo: data-integrations/kafka-plugins
Branch: vuln/kp-3-avro-1.11.4 → base develop (stacked)
Commit: 75a2b64
Part of the CVE fixes for the CDAP 6.11.2 patch release (container image scanning of gcr.io/cdapio/cdap).
The kafka plugin jar packages avro 1.8.2 (which also shades Guava 11),
commons-compress 1.8.1 and commons-lang3 3.9. Image scanning of
gcr.io/cdapio/cdap reports the avro, commons-compress and commons-lang3
CVEs against /opt/cdap/master/artifacts/kafka-plugins-client-*.jar/lib/.
avro 1.9 removed the Jackson 1 and Joda-Time APIs. Plugin main code has
no direct avro imports (avro reaches it through cdap-formats), and the
build compiles and the unit tests pass without code changes.
commons-compress 1.26.x calls commons-io 2.12+ APIs (for example
CloseShieldInputStream.wrap in the bzip2 codec that avro uses). Without
a direct dependency Maven mediated commons-io to 2.4 from the provided
hadoop-common, and it was not packaged, so the plugin would hit
NoSuchMethodError / NoClassDefFoundError on the avro bzip2/xz/zstd
codecs. kafka-plugins-client now declares commons-io 2.16.1.
Versions:
(kafka-plugins-client, new direct dependency)
Addresses the avro and commons-compress CVEs in the plugins, and also
removes the Guava 11 shaded in avro 1.8.2 (the Guava findings stay out
of scope; guava-13.0.1 is still packaged).
Risk: medium. avro crosses the 1.9 API break (Jackson 1 / Joda removal).
jackson-core-asl / jackson-mapper-asl 1.9.13 are no longer packaged.
Needs a Kafka source/sink run with the avro format in a sandbox.
Testing: JDK 8, Checkstyle on (0 violations).
mvn -B clean install (all modules): BUILD SUCCESS, 13 tests, 0 failures.
mvn dependency:tree -pl kafka-plugins-client: avro 1.11.4 ->
commons-compress 1.26.2, commons-io 2.16.1, commons-lang3 3.18.0 (all
compile); no avro 1.8.2 / commons-compress 1.8.1 at compile scope (avro
1.8.2 remains only under provided spark-core avro-mapred). Plugin jar
lib/: avro-1.11.4, commons-compress-1.26.2, commons-io-2.16.1,
commons-lang3-3.18.0; no jackson-*-asl, xz-1.5 or snappy-java-1.1.1.3.
Not run: Kafka avro-format pipeline in a sandbox (P2).