Skip to content

Upgrade kafka-clients to 3.9.2 - #255

Open
deepika-lalwani123 wants to merge 4 commits into
developfrom
vuln/kp-4-kafka-clients-3.9.2
Open

deepika-lalwani123 wants to merge 4 commits into
developfrom
vuln/kp-4-kafka-clients-3.9.2

Conversation

@deepika-lalwani123

@deepika-lalwani123 deepika-lalwani123 commented Nov 17, 2026 •

Copy link
Copy Markdown

Upgrade kafka-clients to 3.9.2

Stacked PR: this branch is built on top of #252, #253, #254. Against develop the diff includes those commits; please review only the top commit a55ca81e01. Merge after #254.

Repo: data-integrations/kafka-plugins
Branch: vuln/kafka-clients-3.9.2 → base develop (stacked)
Commit: a55ca81
Part of the CVE fixes for the CDAP 6.11.2 patch release (container image scanning of gcr.io/cdapio/cdap).

The kafka plugin jar packages kafka-clients 2.6.0. Image scanning of
gcr.io/cdapio/cdap reports the kafka-clients CVEs against
/opt/cdap/master/artifacts/kafka-plugins-client-*.jar/lib/. 3.9.2 is the
newest Kafka line that still runs on Java 8.

Code changes for the 3.x client API:

  • KafkaStreamingSourceUtil: ListOffsetRequest -> ListOffsetsRequest
    (class renamed in Kafka 2.7; same EARLIEST/LATEST_TIMESTAMP values).
  • Kafka10Reader: ConsumerRecord.checksum() was removed in 3.0. KafkaKey
    only uses it as a tie-breaker and Writable field, so compute the same
    partial checksum locally (CRC32 of timestamp + serialized key/value
    sizes).
  • KafkaBatchSink and KafkaAlertPublisher: 3.x producers default to
    enable.idempotence=true / acks=all, which needs IDEMPOTENT_WRITE on
    ACL-protected clusters. Set enable.idempotence=false by default to
    keep the 2.x behavior; user kafkaProperties / producerProperties still
    override it. (KafkaBatchSink passes it through the existing
    "additional." prefix that KafkaOutputFormat copies to the producer.)
  • EmbeddedKafkaServer (test): Kafka 3.0 removed the "port" broker
    property and KafkaConfig.port(); translate "port" to a PLAINTEXT
    listener, and use the 3.x KafkaServer constructor with Time.SYSTEM.

Packaging: kafka-clients 3.x declares snappy-java, lz4-java and zstd-jni
as runtime scope (2.6.0 had them at compile scope), so the bundle plugin
stopped embedding them and the snappy/lz4/zstd compression codecs would
fail at runtime. Embed-Dependency now embeds scope=compile|runtime; those
three are the only runtime-scope dependencies. kafka-clients 3.9.2 uses
at.yawk.lz4:lz4-java 1.10.1 (same net.jpountz packages); org.lz4:lz4-java
only remains under provided spark-core, so no exclusion is needed and only
one net.jpountz copy is packaged.

Versions:

  • kafka-clients 2.6.0 -> 3.9.2 (root kafka.version)
  • kafka_2.12 2.6.0 -> 3.9.2 (same property, test scope only; brings
    zookeeper 3.8.4 for the embedded broker)
  • lz4-java org.lz4 1.7.1 -> at.yawk.lz4 1.10.1 (from kafka-clients)
  • snappy-java 1.1.7.3 -> 1.10.12 (root dependencyManagement; 3.9.2
    brings 1.1.10.5)
  • zstd-jni 1.4.4-7 -> 1.5.6-4 (from kafka-clients)

Risk: medium-high. Major client upgrade. spark-streaming-kafka-0-10
stays at 3.1.2 (compiled against kafka 2.x) and runs on kafka-clients
3.9.2 in the unit tests. Producer idempotence is off by default as in
2.x. Needs produce/consume against Kafka 2.x and 3.x brokers, with and
without SASL, in a sandbox.

Testing: JDK 8, Checkstyle on (0 violations).
mvn -B clean install (all modules): BUILD SUCCESS, 13 tests, 0 failures
(embedded ZooKeeper + Kafka 3.9.2 broker: KafkaBatchSourceTest,
KafkaSinkAndAlertsPublisherTest, KafkaStreamingSource*Test,
KafkaStreamingSourceUtilTest). The FAILED program runs in the log
(RecoveringTransform, one testKafkaSink MapReduce run) are negative test
cases and appear identically on the parent branches.
mvn dependency:tree -pl kafka-plugins-client -Dincludes=org.apache.kafka,
org.lz4,at.yawk.lz4,org.xerial.snappy: kafka-clients 3.9.2 (compile),
at.yawk.lz4 1.10.1, snappy-java 1.1.10.12 (runtime), kafka_2.12 3.9.2
(test); org.lz4 1.7.1 only under provided spark-core. Plugin jar lib/:
kafka-clients-3.9.2, lz4-java-1.10.1, snappy-java-1.10.12,
zstd-jni-1.5.6-4; no kafka-clients-2.6.0.
Not run: Kafka 2.x / 3.x broker and SASL tests in a sandbox (P2).

kafka-plugins-client declared netty-all, netty-http and the kafka_2.12
broker at compile scope, so the plugin jar packaged about 30 Netty
4.1.75 / 4.1.48 modules, kafka_2.12 2.6.0 and ZooKeeper 3.5.8 under
lib/. Image scanning of gcr.io/cdapio/cdap reports the Netty, kafka_2.12
and ZooKeeper CVEs against
/opt/cdap/master/artifacts/kafka-plugins-client-*.jar/lib/.

Only the embedded-broker unit tests use these jars: every kafka.*,
scala.*, io.netty.*, io.cdap.http.* and org.apache.zookeeper.* import is
under src/test. Main code uses only org.apache.kafka.clients/common from
kafka-clients. The root POM already declares netty-all and netty-http as
test scope; the client module overrode that.

Versions:
- io.netty:netty-all 4.1.75.Final: compile -> test (kafka-plugins-client)
- io.cdap.http:netty-http 1.3.0: compile -> test (kafka-plugins-client)
- org.apache.kafka:kafka_2.12 2.6.0: compile -> test (kafka-plugins-client),
  which also drops zookeeper 3.5.8, scala-library/-reflect 2.12.11,
  jackson-module-scala and metrics-core from the plugin jar

Bugs: b/578427766, b/578434052, b/578497139 (kafka_2.12); Plugins side
done for the kafka-plugins Netty bugs and zookeeper b/578633351.

Risk: low. No main-code dependency on the removed jars. Scala comes from
the Spark runtime that already provides spark-streaming at runtime.

Testing: JDK 8, Checkstyle on (0 violations).
mvn -B clean install (all modules): BUILD SUCCESS, 13 tests, 0 failures
(KafkaBatchSourceTest, KafkaSinkAndAlertsPublisherTest,
KafkaStreamingSource*Test, KafkaStreamingSourceUtilTest).
unzip -l kafka-plugins-client-3.5.0-SNAPSHOT.jar: lib/ goes from 74 to
26 jars; no netty-*, kafka_2.12-*, zookeeper-* or scala-* left; nothing
added. mvn dependency:tree -Dincludes=io.netty,org.apache.kafka,
org.apache.zookeeper,io.cdap.http: kafka_2.12, netty-all and netty-http
are test scope; the only compile-scope match is kafka-clients 2.6.0.
The kafka plugin jar packages jackson-databind 2.10.5.1 and
jackson-core 2.10.5. Image scanning of gcr.io/cdapio/cdap reports the
jackson-databind and jackson-core CVEs against
/opt/cdap/master/artifacts/kafka-plugins-client-*.jar/lib/.

Import com.fasterxml.jackson:jackson-bom in the root
dependencyManagement so every Jackson 2 module (including the ones that
come transitively through kafka_2.12 and the CDAP test jars) resolves to
one version.

Versions:
- jackson 2.10.5.1 -> 2.18.12 (root jackson.version, now also the
  jackson-bom import version; kafka-plugins-client jackson-databind
  keeps using ${jackson.version})

Bugs: no single-ID list for the kafka-plugins Jackson rows in the plan;
this is the Plugins side for the kafka-plugins jackson-databind and
jackson-core rows.

Risk: low. Jackson 2.x minor upgrade; plugin main code has no direct
Jackson imports. jackson-core-asl / jackson-mapper-asl 1.9.13 (Jackson
1, from avro 1.8.2) are untouched here and go away with the avro
upgrade.

Testing: JDK 8, Checkstyle on (0 violations).
mvn -B clean install (all modules): BUILD SUCCESS, 13 tests, 0 failures.
mvn dependency:tree -pl kafka-plugins-client -Dincludes=
com.fasterxml.jackson.*: every Jackson 2 artifact resolves to 2.18.12;
compile scope = jackson-databind/-core/-annotations 2.18.12. Plugin jar
lib/ has jackson-databind-2.18.12, jackson-core-2.18.12,
jackson-annotations-2.18.12 and no 2.10.x jar.
The kafka plugin jar packages avro 1.8.2 (which also shades Guava 11),
commons-compress 1.8.1 and commons-lang3 3.9. Image scanning of
gcr.io/cdapio/cdap reports the avro, commons-compress and commons-lang3
CVEs against /opt/cdap/master/artifacts/kafka-plugins-client-*.jar/lib/.

avro 1.9 removed the Jackson 1 and Joda-Time APIs. Plugin main code has
no direct avro imports (avro reaches it through cdap-formats), and the
build compiles and the unit tests pass without code changes.

commons-compress 1.26.x calls commons-io 2.12+ APIs (for example
CloseShieldInputStream.wrap in the bzip2 codec that avro uses). Without
a direct dependency Maven mediated commons-io to 2.4 from the provided
hadoop-common, and it was not packaged, so the plugin would hit
NoSuchMethodError / NoClassDefFoundError on the avro bzip2/xz/zstd
codecs. kafka-plugins-client now declares commons-io 2.16.1.

Versions:
- avro 1.8.2 -> 1.11.4 (root avro.version)
- commons-compress 1.8.1 -> 1.26.2 (root dependencyManagement)
- commons-lang3 3.9 -> 3.18.0 (root dependency)
- commons-io (not packaged, 2.4 at build time) -> 2.16.1
  (kafka-plugins-client, new direct dependency)

Bugs: no single-ID list for the kafka-plugins avro / commons-compress
rows in the plan; this is the Plugins side for those rows. Also removes
the Guava 11 shaded in avro 1.8.2 (Guava bugs b/578439199, b/578506400,
b/578653882, b/578733862 stay out of scope; guava-13.0.1 is still
packaged).

Risk: medium. avro crosses the 1.9 API break (Jackson 1 / Joda removal).
jackson-core-asl / jackson-mapper-asl 1.9.13 are no longer packaged.
Needs a Kafka source/sink run with the avro format in a sandbox.

Testing: JDK 8, Checkstyle on (0 violations).
mvn -B clean install (all modules): BUILD SUCCESS, 13 tests, 0 failures.
mvn dependency:tree -pl kafka-plugins-client: avro 1.11.4 ->
commons-compress 1.26.2, commons-io 2.16.1, commons-lang3 3.18.0 (all
compile); no avro 1.8.2 / commons-compress 1.8.1 at compile scope (avro
1.8.2 remains only under provided spark-core avro-mapred). Plugin jar
lib/: avro-1.11.4, commons-compress-1.26.2, commons-io-2.16.1,
commons-lang3-3.18.0; no jackson-*-asl, xz-1.5 or snappy-java-1.1.1.3.
Not run: Kafka avro-format pipeline in a sandbox (P2).
The kafka plugin jar packages kafka-clients 2.6.0. Image scanning of
gcr.io/cdapio/cdap reports the kafka-clients CVEs against
/opt/cdap/master/artifacts/kafka-plugins-client-*.jar/lib/. 3.9.2 is the
newest Kafka line that still runs on Java 8.

Code changes for the 3.x client API:
- KafkaStreamingSourceUtil: ListOffsetRequest -> ListOffsetsRequest
  (class renamed in Kafka 2.7; same EARLIEST/LATEST_TIMESTAMP values).
- Kafka10Reader: ConsumerRecord.checksum() was removed in 3.0. KafkaKey
  only uses it as a tie-breaker and Writable field, so compute the same
  partial checksum locally (CRC32 of timestamp + serialized key/value
  sizes).
- KafkaBatchSink and KafkaAlertPublisher: 3.x producers default to
  enable.idempotence=true / acks=all, which needs IDEMPOTENT_WRITE on
  ACL-protected clusters. Set enable.idempotence=false by default to
  keep the 2.x behavior; user kafkaProperties / producerProperties still
  override it. (KafkaBatchSink passes it through the existing
  "additional." prefix that KafkaOutputFormat copies to the producer.)
- EmbeddedKafkaServer (test): Kafka 3.0 removed the "port" broker
  property and KafkaConfig.port(); translate "port" to a PLAINTEXT
  listener, and use the 3.x KafkaServer constructor with Time.SYSTEM.

Packaging: kafka-clients 3.x declares snappy-java, lz4-java and zstd-jni
as runtime scope (2.6.0 had them at compile scope), so the bundle plugin
stopped embedding them and the snappy/lz4/zstd compression codecs would
fail at runtime. Embed-Dependency now embeds scope=compile|runtime; those
three are the only runtime-scope dependencies. kafka-clients 3.9.2 uses
at.yawk.lz4:lz4-java 1.10.1 (same net.jpountz packages); org.lz4:lz4-java
only remains under provided spark-core, so no exclusion is needed and only
one net.jpountz copy is packaged.

Versions:
- kafka-clients 2.6.0 -> 3.9.2 (root kafka.version)
- kafka_2.12 2.6.0 -> 3.9.2 (same property, test scope only; brings
  zookeeper 3.8.4 for the embedded broker)
- lz4-java org.lz4 1.7.1 -> at.yawk.lz4 1.10.1 (from kafka-clients)
- snappy-java 1.1.7.3 -> 1.1.10.12 (root dependencyManagement; 3.9.2
  brings 1.1.10.5)
- zstd-jni 1.4.4-7 -> 1.5.6-4 (from kafka-clients)

Bugs: b/578590394; Plugins side done for b/578458133, b/578579893.

Risk: medium-high. Major client upgrade. spark-streaming-kafka-0-10
stays at 3.1.2 (compiled against kafka 2.x) and runs on kafka-clients
3.9.2 in the unit tests. Producer idempotence is off by default as in
2.x. Needs produce/consume against Kafka 2.x and 3.x brokers, with and
without SASL, in a sandbox.

Testing: JDK 8, Checkstyle on (0 violations).
mvn -B clean install (all modules): BUILD SUCCESS, 13 tests, 0 failures
(embedded ZooKeeper + Kafka 3.9.2 broker: KafkaBatchSourceTest,
KafkaSinkAndAlertsPublisherTest, KafkaStreamingSource*Test,
KafkaStreamingSourceUtilTest). The FAILED program runs in the log
(RecoveringTransform, one testKafkaSink MapReduce run) are negative test
cases and appear identically on the parent branches.
mvn dependency:tree -pl kafka-plugins-client -Dincludes=org.apache.kafka,
org.lz4,at.yawk.lz4,org.xerial.snappy: kafka-clients 3.9.2 (compile),
at.yawk.lz4 1.10.1, snappy-java 1.1.10.12 (runtime), kafka_2.12 3.9.2
(test); org.lz4 1.7.1 only under provided spark-core. Plugin jar lib/:
kafka-clients-3.9.2, lz4-java-1.10.1, snappy-java-1.1.10.12,
zstd-jni-1.5.6-4; no kafka-clients-2.6.0.
Not run: Kafka 2.x / 3.x broker and SASL tests in a sandbox (P2).

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request upgrades Kafka to version 3.9.2 and updates related dependencies like Avro and Jackson, adjusting the codebase for compatibility with Kafka 3.0+ (such as handling the removal of ConsumerRecord.checksum() and updating the embedded test server configuration). The review feedback highlights two key improvements: correcting the Jackson version in pom.xml from the non-existent 2.18.12 to 2.18.2 to avoid build failures, and optimizing the custom checksum method in Kafka10Reader.java to use bit-shifting instead of allocating a ByteBuffer per record, thereby reducing garbage collection pressure.

Comment thread pom.xml
<logback.version>1.0.9</logback.version>
<slf4j.version>1.7.25</slf4j.version>
<jackson.version>2.10.5.1</jackson.version>
<jackson.version>2.18.12</jackson.version>

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The Jackson version 2.18.12 does not exist on Maven Central. The latest stable version in the 2.18.x line is 2.18.2. Please correct this version to prevent dependency resolution and build failures.

Suggested change
<jackson.version>2.18.12</jackson.version>
<jackson.version>2.18.2</jackson.version>

Comment on lines +106 to +112
private static long checksum(ConsumerRecord<byte[], byte[]> record) {
ByteBuffer buffer = ByteBuffer.allocate(Long.BYTES + 2 * Integer.BYTES);
buffer.putLong(record.timestamp()).putInt(record.serializedKeySize()).putInt(record.serializedValueSize());
CRC32 crc = new CRC32();
crc.update(buffer.array());
return crc.getValue();
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Allocating a ByteBuffer and a CRC32 instance for every single record can create significant garbage collection pressure in high-throughput environments. We can optimize this by updating the CRC32 instance directly using bit-shifting, which avoids allocating the ByteBuffer entirely.

  private static long checksum(ConsumerRecord<byte[], byte[]> record) {
    CRC32 crc = new CRC32();
    long t = record.timestamp();
    crc.update((int) (t >>> 56));
    crc.update((int) (t >>> 48));
    crc.update((int) (t >>> 40));
    crc.update((int) (t >>> 32));
    crc.update((int) (t >>> 24));
    crc.update((int) (t >>> 16));
    crc.update((int) (t >>> 8));
    crc.update((int) t);

    int keySize = record.serializedKeySize();
    crc.update(keySize >>> 24);
    crc.update(keySize >>> 16);
    crc.update(keySize >>> 8);
    crc.update(keySize);

    int valSize = record.serializedValueSize();
    crc.update(valSize >>> 24);
    crc.update(valSize >>> 16);
    crc.update(valSize >>> 8);
    crc.update(valSize);

    return crc.getValue();
  }

@deepika-lalwani123 deepika-lalwani123 added the build Trigger unit test build label Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build Trigger unit test build

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant