Skip to content

Upgrade avro to 1.11.4 and commons-lang3 to 3.18.0 - #254

Open
deepika-lalwani123 wants to merge 3 commits into
developfrom
vuln/kp-3-avro-1.11.4
Open

deepika-lalwani123 wants to merge 3 commits into
developfrom
vuln/kp-3-avro-1.11.4

Conversation

@deepika-lalwani123

@deepika-lalwani123 deepika-lalwani123 commented Nov 17, 2026 •

Copy link
Copy Markdown

Upgrade avro to 1.11.4 and commons-lang3 to 3.18.0

Stacked PR: this branch is built on top of #252, #253. Against develop the diff includes those commits; please review only the top commit 75a2b64733. Merge after #253.

Repo: data-integrations/kafka-plugins
Branch: vuln/kp-3-avro-1.11.4 → base develop (stacked)
Commit: 75a2b64
Part of the CVE fixes for the CDAP 6.11.2 patch release (container image scanning of gcr.io/cdapio/cdap).

The kafka plugin jar packages avro 1.8.2 (which also shades Guava 11),
commons-compress 1.8.1 and commons-lang3 3.9. Image scanning of
gcr.io/cdapio/cdap reports the avro, commons-compress and commons-lang3
CVEs against /opt/cdap/master/artifacts/kafka-plugins-client-*.jar/lib/.

avro 1.9 removed the Jackson 1 and Joda-Time APIs. Plugin main code has
no direct avro imports (avro reaches it through cdap-formats), and the
build compiles and the unit tests pass without code changes.

commons-compress 1.26.x calls commons-io 2.12+ APIs (for example
CloseShieldInputStream.wrap in the bzip2 codec that avro uses). Without
a direct dependency Maven mediated commons-io to 2.4 from the provided
hadoop-common, and it was not packaged, so the plugin would hit
NoSuchMethodError / NoClassDefFoundError on the avro bzip2/xz/zstd
codecs. kafka-plugins-client now declares commons-io 2.16.1.

Versions:

  • avro 1.8.2 -> 1.11.4 (root avro.version)
  • commons-compress 1.8.1 -> 1.26.2 (root dependencyManagement)
  • commons-lang3 3.9 -> 3.18.0 (root dependency)
  • commons-io (not packaged, 2.4 at build time) -> 2.16.1
    (kafka-plugins-client, new direct dependency)

Addresses the avro and commons-compress CVEs in the plugins, and also
removes the Guava 11 shaded in avro 1.8.2 (the Guava findings stay out
of scope; guava-13.0.1 is still packaged).

Risk: medium. avro crosses the 1.9 API break (Jackson 1 / Joda removal).
jackson-core-asl / jackson-mapper-asl 1.9.13 are no longer packaged.
Needs a Kafka source/sink run with the avro format in a sandbox.

Testing: JDK 8, Checkstyle on (0 violations).
mvn -B clean install (all modules): BUILD SUCCESS, 13 tests, 0 failures.
mvn dependency:tree -pl kafka-plugins-client: avro 1.11.4 ->
commons-compress 1.26.2, commons-io 2.16.1, commons-lang3 3.18.0 (all
compile); no avro 1.8.2 / commons-compress 1.8.1 at compile scope (avro
1.8.2 remains only under provided spark-core avro-mapred). Plugin jar
lib/: avro-1.11.4, commons-compress-1.26.2, commons-io-2.16.1,
commons-lang3-3.18.0; no jackson-*-asl, xz-1.5 or snappy-java-1.1.1.3.
Not run: Kafka avro-format pipeline in a sandbox (P2).

kafka-plugins-client declared netty-all, netty-http and the kafka_2.12
broker at compile scope, so the plugin jar packaged about 30 Netty
4.1.75 / 4.1.48 modules, kafka_2.12 2.6.0 and ZooKeeper 3.5.8 under
lib/. Image scanning of gcr.io/cdapio/cdap reports the Netty, kafka_2.12
and ZooKeeper CVEs against
/opt/cdap/master/artifacts/kafka-plugins-client-*.jar/lib/.

Only the embedded-broker unit tests use these jars: every kafka.*,
scala.*, io.netty.*, io.cdap.http.* and org.apache.zookeeper.* import is
under src/test. Main code uses only org.apache.kafka.clients/common from
kafka-clients. The root POM already declares netty-all and netty-http as
test scope; the client module overrode that.

Versions:
- io.netty:netty-all 4.1.75.Final: compile -> test (kafka-plugins-client)
- io.cdap.http:netty-http 1.3.0: compile -> test (kafka-plugins-client)
- org.apache.kafka:kafka_2.12 2.6.0: compile -> test (kafka-plugins-client),
  which also drops zookeeper 3.5.8, scala-library/-reflect 2.12.11,
  jackson-module-scala and metrics-core from the plugin jar

Bugs: b/578427766, b/578434052, b/578497139 (kafka_2.12); Plugins side
done for the kafka-plugins Netty bugs and zookeeper b/578633351.

Risk: low. No main-code dependency on the removed jars. Scala comes from
the Spark runtime that already provides spark-streaming at runtime.

Testing: JDK 8, Checkstyle on (0 violations).
mvn -B clean install (all modules): BUILD SUCCESS, 13 tests, 0 failures
(KafkaBatchSourceTest, KafkaSinkAndAlertsPublisherTest,
KafkaStreamingSource*Test, KafkaStreamingSourceUtilTest).
unzip -l kafka-plugins-client-3.5.0-SNAPSHOT.jar: lib/ goes from 74 to
26 jars; no netty-*, kafka_2.12-*, zookeeper-* or scala-* left; nothing
added. mvn dependency:tree -Dincludes=io.netty,org.apache.kafka,
org.apache.zookeeper,io.cdap.http: kafka_2.12, netty-all and netty-http
are test scope; the only compile-scope match is kafka-clients 2.6.0.
The kafka plugin jar packages jackson-databind 2.10.5.1 and
jackson-core 2.10.5. Image scanning of gcr.io/cdapio/cdap reports the
jackson-databind and jackson-core CVEs against
/opt/cdap/master/artifacts/kafka-plugins-client-*.jar/lib/.

Import com.fasterxml.jackson:jackson-bom in the root
dependencyManagement so every Jackson 2 module (including the ones that
come transitively through kafka_2.12 and the CDAP test jars) resolves to
one version.

Versions:
- jackson 2.10.5.1 -> 2.18.12 (root jackson.version, now also the
  jackson-bom import version; kafka-plugins-client jackson-databind
  keeps using ${jackson.version})

Bugs: no single-ID list for the kafka-plugins Jackson rows in the plan;
this is the Plugins side for the kafka-plugins jackson-databind and
jackson-core rows.

Risk: low. Jackson 2.x minor upgrade; plugin main code has no direct
Jackson imports. jackson-core-asl / jackson-mapper-asl 1.9.13 (Jackson
1, from avro 1.8.2) are untouched here and go away with the avro
upgrade.

Testing: JDK 8, Checkstyle on (0 violations).
mvn -B clean install (all modules): BUILD SUCCESS, 13 tests, 0 failures.
mvn dependency:tree -pl kafka-plugins-client -Dincludes=
com.fasterxml.jackson.*: every Jackson 2 artifact resolves to 2.18.12;
compile scope = jackson-databind/-core/-annotations 2.18.12. Plugin jar
lib/ has jackson-databind-2.18.12, jackson-core-2.18.12,
jackson-annotations-2.18.12 and no 2.10.x jar.
The kafka plugin jar packages avro 1.8.2 (which also shades Guava 11),
commons-compress 1.8.1 and commons-lang3 3.9. Image scanning of
gcr.io/cdapio/cdap reports the avro, commons-compress and commons-lang3
CVEs against /opt/cdap/master/artifacts/kafka-plugins-client-*.jar/lib/.

avro 1.9 removed the Jackson 1 and Joda-Time APIs. Plugin main code has
no direct avro imports (avro reaches it through cdap-formats), and the
build compiles and the unit tests pass without code changes.

commons-compress 1.26.x calls commons-io 2.12+ APIs (for example
CloseShieldInputStream.wrap in the bzip2 codec that avro uses). Without
a direct dependency Maven mediated commons-io to 2.4 from the provided
hadoop-common, and it was not packaged, so the plugin would hit
NoSuchMethodError / NoClassDefFoundError on the avro bzip2/xz/zstd
codecs. kafka-plugins-client now declares commons-io 2.16.1.

Versions:
- avro 1.8.2 -> 1.11.4 (root avro.version)
- commons-compress 1.8.1 -> 1.26.2 (root dependencyManagement)
- commons-lang3 3.9 -> 3.18.0 (root dependency)
- commons-io (not packaged, 2.4 at build time) -> 2.16.1
  (kafka-plugins-client, new direct dependency)

Bugs: no single-ID list for the kafka-plugins avro / commons-compress
rows in the plan; this is the Plugins side for those rows. Also removes
the Guava 11 shaded in avro 1.8.2 (Guava bugs b/578439199, b/578506400,
b/578653882, b/578733862 stay out of scope; guava-13.0.1 is still
packaged).

Risk: medium. avro crosses the 1.9 API break (Jackson 1 / Joda removal).
jackson-core-asl / jackson-mapper-asl 1.9.13 are no longer packaged.
Needs a Kafka source/sink run with the avro format in a sandbox.

Testing: JDK 8, Checkstyle on (0 violations).
mvn -B clean install (all modules): BUILD SUCCESS, 13 tests, 0 failures.
mvn dependency:tree -pl kafka-plugins-client: avro 1.11.4 ->
commons-compress 1.26.2, commons-io 2.16.1, commons-lang3 3.18.0 (all
compile); no avro 1.8.2 / commons-compress 1.8.1 at compile scope (avro
1.8.2 remains only under provided spark-core avro-mapred). Plugin jar
lib/: avro-1.11.4, commons-compress-1.26.2, commons-io-2.16.1,
commons-lang3-3.18.0; no jackson-*-asl, xz-1.5 or snappy-java-1.1.1.3.
Not run: Kafka avro-format pipeline in a sandbox (P2).

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates several dependency versions, including upgrading Avro to 1.11.4, Jackson to 2.18.12, and commons-lang3 to 3.18.0, while introducing dependency management for Jackson modules and commons-compress to address security vulnerabilities. It also adds a direct dependency on commons-io in the Kafka client plugin and applies test scopes to Netty and Kafka dependencies. The review feedback suggests centralizing the commons-io version as a property in the parent POM to avoid hardcoding, and highlights that the Netty dependency declarations in the child POM are redundant and should be removed.

Comment thread pom.xml
<logback.version>1.0.9</logback.version>
<slf4j.version>1.7.25</slf4j.version>
<jackson.version>2.10.5.1</jackson.version>
<jackson.version>2.18.12</jackson.version>

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

To maintain consistency and centralize version management, define the commons-io version as a property in the parent POM's <properties> section rather than hardcoding it in the child module.

    <jackson.version>2.18.12</jackson.version>
    <commons-io.version>2.16.1</commons-io.version>

<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>2.16.1</version>

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Use the parent POM property ${commons-io.version} instead of hardcoding the version here to ensure consistent version management across modules.

Suggested change
<version>2.16.1</version>
<version>${commons-io.version}</version>

Comment on lines 215 to 225
<groupId>io.netty</groupId>
<artifactId>netty-all</artifactId>
<version>${netty.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>io.cdap.http</groupId>
<artifactId>netty-http</artifactId>
<version>${netty-http.version}</version>
<scope>test</scope>
</dependency>

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The dependencies netty-all and netty-http are already declared in the parent POM's <dependencies> section (lines 292-303) with <scope>test</scope>. Since they are automatically inherited by all submodules, these declarations in kafka-plugins-client/pom.xml are completely redundant and can be safely removed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build Trigger unit test build

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant