Repository navigation
Upgrade kafka-clients to 3.9.2 - #255
deepika-lalwani123 wants to merge 4 commits into
Conversation
kafka-plugins-client declared netty-all, netty-http and the kafka_2.12 broker at compile scope, so the plugin jar packaged about 30 Netty 4.1.75 / 4.1.48 modules, kafka_2.12 2.6.0 and ZooKeeper 3.5.8 under lib/. Image scanning of gcr.io/cdapio/cdap reports the Netty, kafka_2.12 and ZooKeeper CVEs against /opt/cdap/master/artifacts/kafka-plugins-client-*.jar/lib/. Only the embedded-broker unit tests use these jars: every kafka.*, scala.*, io.netty.*, io.cdap.http.* and org.apache.zookeeper.* import is under src/test. Main code uses only org.apache.kafka.clients/common from kafka-clients. The root POM already declares netty-all and netty-http as test scope; the client module overrode that. Versions: - io.netty:netty-all 4.1.75.Final: compile -> test (kafka-plugins-client) - io.cdap.http:netty-http 1.3.0: compile -> test (kafka-plugins-client) - org.apache.kafka:kafka_2.12 2.6.0: compile -> test (kafka-plugins-client), which also drops zookeeper 3.5.8, scala-library/-reflect 2.12.11, jackson-module-scala and metrics-core from the plugin jar Bugs: b/578427766, b/578434052, b/578497139 (kafka_2.12); Plugins side done for the kafka-plugins Netty bugs and zookeeper b/578633351. Risk: low. No main-code dependency on the removed jars. Scala comes from the Spark runtime that already provides spark-streaming at runtime. Testing: JDK 8, Checkstyle on (0 violations). mvn -B clean install (all modules): BUILD SUCCESS, 13 tests, 0 failures (KafkaBatchSourceTest, KafkaSinkAndAlertsPublisherTest, KafkaStreamingSource*Test, KafkaStreamingSourceUtilTest). unzip -l kafka-plugins-client-3.5.0-SNAPSHOT.jar: lib/ goes from 74 to 26 jars; no netty-*, kafka_2.12-*, zookeeper-* or scala-* left; nothing added. mvn dependency:tree -Dincludes=io.netty,org.apache.kafka, org.apache.zookeeper,io.cdap.http: kafka_2.12, netty-all and netty-http are test scope; the only compile-scope match is kafka-clients 2.6.0.
The kafka plugin jar packages jackson-databind 2.10.5.1 and
jackson-core 2.10.5. Image scanning of gcr.io/cdapio/cdap reports the
jackson-databind and jackson-core CVEs against
/opt/cdap/master/artifacts/kafka-plugins-client-*.jar/lib/.
Import com.fasterxml.jackson:jackson-bom in the root
dependencyManagement so every Jackson 2 module (including the ones that
come transitively through kafka_2.12 and the CDAP test jars) resolves to
one version.
Versions:
- jackson 2.10.5.1 -> 2.18.12 (root jackson.version, now also the
jackson-bom import version; kafka-plugins-client jackson-databind
keeps using ${jackson.version})
Bugs: no single-ID list for the kafka-plugins Jackson rows in the plan;
this is the Plugins side for the kafka-plugins jackson-databind and
jackson-core rows.
Risk: low. Jackson 2.x minor upgrade; plugin main code has no direct
Jackson imports. jackson-core-asl / jackson-mapper-asl 1.9.13 (Jackson
1, from avro 1.8.2) are untouched here and go away with the avro
upgrade.
Testing: JDK 8, Checkstyle on (0 violations).
mvn -B clean install (all modules): BUILD SUCCESS, 13 tests, 0 failures.
mvn dependency:tree -pl kafka-plugins-client -Dincludes=
com.fasterxml.jackson.*: every Jackson 2 artifact resolves to 2.18.12;
compile scope = jackson-databind/-core/-annotations 2.18.12. Plugin jar
lib/ has jackson-databind-2.18.12, jackson-core-2.18.12,
jackson-annotations-2.18.12 and no 2.10.x jar.
The kafka plugin jar packages avro 1.8.2 (which also shades Guava 11), commons-compress 1.8.1 and commons-lang3 3.9. Image scanning of gcr.io/cdapio/cdap reports the avro, commons-compress and commons-lang3 CVEs against /opt/cdap/master/artifacts/kafka-plugins-client-*.jar/lib/. avro 1.9 removed the Jackson 1 and Joda-Time APIs. Plugin main code has no direct avro imports (avro reaches it through cdap-formats), and the build compiles and the unit tests pass without code changes. commons-compress 1.26.x calls commons-io 2.12+ APIs (for example CloseShieldInputStream.wrap in the bzip2 codec that avro uses). Without a direct dependency Maven mediated commons-io to 2.4 from the provided hadoop-common, and it was not packaged, so the plugin would hit NoSuchMethodError / NoClassDefFoundError on the avro bzip2/xz/zstd codecs. kafka-plugins-client now declares commons-io 2.16.1. Versions: - avro 1.8.2 -> 1.11.4 (root avro.version) - commons-compress 1.8.1 -> 1.26.2 (root dependencyManagement) - commons-lang3 3.9 -> 3.18.0 (root dependency) - commons-io (not packaged, 2.4 at build time) -> 2.16.1 (kafka-plugins-client, new direct dependency) Bugs: no single-ID list for the kafka-plugins avro / commons-compress rows in the plan; this is the Plugins side for those rows. Also removes the Guava 11 shaded in avro 1.8.2 (Guava bugs b/578439199, b/578506400, b/578653882, b/578733862 stay out of scope; guava-13.0.1 is still packaged). Risk: medium. avro crosses the 1.9 API break (Jackson 1 / Joda removal). jackson-core-asl / jackson-mapper-asl 1.9.13 are no longer packaged. Needs a Kafka source/sink run with the avro format in a sandbox. Testing: JDK 8, Checkstyle on (0 violations). mvn -B clean install (all modules): BUILD SUCCESS, 13 tests, 0 failures. mvn dependency:tree -pl kafka-plugins-client: avro 1.11.4 -> commons-compress 1.26.2, commons-io 2.16.1, commons-lang3 3.18.0 (all compile); no avro 1.8.2 / commons-compress 1.8.1 at compile scope (avro 1.8.2 remains only under provided spark-core avro-mapred). Plugin jar lib/: avro-1.11.4, commons-compress-1.26.2, commons-io-2.16.1, commons-lang3-3.18.0; no jackson-*-asl, xz-1.5 or snappy-java-1.1.1.3. Not run: Kafka avro-format pipeline in a sandbox (P2).
The kafka plugin jar packages kafka-clients 2.6.0. Image scanning of gcr.io/cdapio/cdap reports the kafka-clients CVEs against /opt/cdap/master/artifacts/kafka-plugins-client-*.jar/lib/. 3.9.2 is the newest Kafka line that still runs on Java 8. Code changes for the 3.x client API: - KafkaStreamingSourceUtil: ListOffsetRequest -> ListOffsetsRequest (class renamed in Kafka 2.7; same EARLIEST/LATEST_TIMESTAMP values). - Kafka10Reader: ConsumerRecord.checksum() was removed in 3.0. KafkaKey only uses it as a tie-breaker and Writable field, so compute the same partial checksum locally (CRC32 of timestamp + serialized key/value sizes). - KafkaBatchSink and KafkaAlertPublisher: 3.x producers default to enable.idempotence=true / acks=all, which needs IDEMPOTENT_WRITE on ACL-protected clusters. Set enable.idempotence=false by default to keep the 2.x behavior; user kafkaProperties / producerProperties still override it. (KafkaBatchSink passes it through the existing "additional." prefix that KafkaOutputFormat copies to the producer.) - EmbeddedKafkaServer (test): Kafka 3.0 removed the "port" broker property and KafkaConfig.port(); translate "port" to a PLAINTEXT listener, and use the 3.x KafkaServer constructor with Time.SYSTEM. Packaging: kafka-clients 3.x declares snappy-java, lz4-java and zstd-jni as runtime scope (2.6.0 had them at compile scope), so the bundle plugin stopped embedding them and the snappy/lz4/zstd compression codecs would fail at runtime. Embed-Dependency now embeds scope=compile|runtime; those three are the only runtime-scope dependencies. kafka-clients 3.9.2 uses at.yawk.lz4:lz4-java 1.10.1 (same net.jpountz packages); org.lz4:lz4-java only remains under provided spark-core, so no exclusion is needed and only one net.jpountz copy is packaged. Versions: - kafka-clients 2.6.0 -> 3.9.2 (root kafka.version) - kafka_2.12 2.6.0 -> 3.9.2 (same property, test scope only; brings zookeeper 3.8.4 for the embedded broker) - lz4-java org.lz4 1.7.1 -> at.yawk.lz4 1.10.1 (from kafka-clients) - snappy-java 1.1.7.3 -> 1.1.10.12 (root dependencyManagement; 3.9.2 brings 1.1.10.5) - zstd-jni 1.4.4-7 -> 1.5.6-4 (from kafka-clients) Bugs: b/578590394; Plugins side done for b/578458133, b/578579893. Risk: medium-high. Major client upgrade. spark-streaming-kafka-0-10 stays at 3.1.2 (compiled against kafka 2.x) and runs on kafka-clients 3.9.2 in the unit tests. Producer idempotence is off by default as in 2.x. Needs produce/consume against Kafka 2.x and 3.x brokers, with and without SASL, in a sandbox. Testing: JDK 8, Checkstyle on (0 violations). mvn -B clean install (all modules): BUILD SUCCESS, 13 tests, 0 failures (embedded ZooKeeper + Kafka 3.9.2 broker: KafkaBatchSourceTest, KafkaSinkAndAlertsPublisherTest, KafkaStreamingSource*Test, KafkaStreamingSourceUtilTest). The FAILED program runs in the log (RecoveringTransform, one testKafkaSink MapReduce run) are negative test cases and appear identically on the parent branches. mvn dependency:tree -pl kafka-plugins-client -Dincludes=org.apache.kafka, org.lz4,at.yawk.lz4,org.xerial.snappy: kafka-clients 3.9.2 (compile), at.yawk.lz4 1.10.1, snappy-java 1.1.10.12 (runtime), kafka_2.12 3.9.2 (test); org.lz4 1.7.1 only under provided spark-core. Plugin jar lib/: kafka-clients-3.9.2, lz4-java-1.10.1, snappy-java-1.1.10.12, zstd-jni-1.5.6-4; no kafka-clients-2.6.0. Not run: Kafka 2.x / 3.x broker and SASL tests in a sandbox (P2).
There was a problem hiding this comment.
Code Review
This pull request upgrades Kafka to version 3.9.2 and updates related dependencies like Avro and Jackson, adjusting the codebase for compatibility with Kafka 3.0+ (such as handling the removal of ConsumerRecord.checksum() and updating the embedded test server configuration). The review feedback highlights two key improvements: correcting the Jackson version in pom.xml from the non-existent 2.18.12 to 2.18.2 to avoid build failures, and optimizing the custom checksum method in Kafka10Reader.java to use bit-shifting instead of allocating a ByteBuffer per record, thereby reducing garbage collection pressure.
| <logback.version>1.0.9</logback.version> | ||
| <slf4j.version>1.7.25</slf4j.version> | ||
| <jackson.version>2.10.5.1</jackson.version> | ||
| <jackson.version>2.18.12</jackson.version> |
There was a problem hiding this comment.
| private static long checksum(ConsumerRecord<byte[], byte[]> record) { | ||
| ByteBuffer buffer = ByteBuffer.allocate(Long.BYTES + 2 * Integer.BYTES); | ||
| buffer.putLong(record.timestamp()).putInt(record.serializedKeySize()).putInt(record.serializedValueSize()); | ||
| CRC32 crc = new CRC32(); | ||
| crc.update(buffer.array()); | ||
| return crc.getValue(); | ||
| } |
There was a problem hiding this comment.
Allocating a ByteBuffer and a CRC32 instance for every single record can create significant garbage collection pressure in high-throughput environments. We can optimize this by updating the CRC32 instance directly using bit-shifting, which avoids allocating the ByteBuffer entirely.
private static long checksum(ConsumerRecord<byte[], byte[]> record) {
CRC32 crc = new CRC32();
long t = record.timestamp();
crc.update((int) (t >>> 56));
crc.update((int) (t >>> 48));
crc.update((int) (t >>> 40));
crc.update((int) (t >>> 32));
crc.update((int) (t >>> 24));
crc.update((int) (t >>> 16));
crc.update((int) (t >>> 8));
crc.update((int) t);
int keySize = record.serializedKeySize();
crc.update(keySize >>> 24);
crc.update(keySize >>> 16);
crc.update(keySize >>> 8);
crc.update(keySize);
int valSize = record.serializedValueSize();
crc.update(valSize >>> 24);
crc.update(valSize >>> 16);
crc.update(valSize >>> 8);
crc.update(valSize);
return crc.getValue();
}
Upgrade kafka-clients to 3.9.2
Repo: data-integrations/kafka-plugins
Branch: vuln/kafka-clients-3.9.2 → base develop (stacked)
Commit: a55ca81
Part of the CVE fixes for the CDAP 6.11.2 patch release (container image scanning of gcr.io/cdapio/cdap).
The kafka plugin jar packages kafka-clients 2.6.0. Image scanning of
gcr.io/cdapio/cdap reports the kafka-clients CVEs against
/opt/cdap/master/artifacts/kafka-plugins-client-*.jar/lib/. 3.9.2 is the
newest Kafka line that still runs on Java 8.
Code changes for the 3.x client API:
(class renamed in Kafka 2.7; same EARLIEST/LATEST_TIMESTAMP values).
only uses it as a tie-breaker and Writable field, so compute the same
partial checksum locally (CRC32 of timestamp + serialized key/value
sizes).
enable.idempotence=true / acks=all, which needs IDEMPOTENT_WRITE on
ACL-protected clusters. Set enable.idempotence=false by default to
keep the 2.x behavior; user kafkaProperties / producerProperties still
override it. (KafkaBatchSink passes it through the existing
"additional." prefix that KafkaOutputFormat copies to the producer.)
property and KafkaConfig.port(); translate "port" to a PLAINTEXT
listener, and use the 3.x KafkaServer constructor with Time.SYSTEM.
Packaging: kafka-clients 3.x declares snappy-java, lz4-java and zstd-jni
as runtime scope (2.6.0 had them at compile scope), so the bundle plugin
stopped embedding them and the snappy/lz4/zstd compression codecs would
fail at runtime. Embed-Dependency now embeds scope=compile|runtime; those
three are the only runtime-scope dependencies. kafka-clients 3.9.2 uses
at.yawk.lz4:lz4-java 1.10.1 (same net.jpountz packages); org.lz4:lz4-java
only remains under provided spark-core, so no exclusion is needed and only
one net.jpountz copy is packaged.
Versions:
zookeeper 3.8.4 for the embedded broker)
brings 1.1.10.5)
Risk: medium-high. Major client upgrade. spark-streaming-kafka-0-10
stays at 3.1.2 (compiled against kafka 2.x) and runs on kafka-clients
3.9.2 in the unit tests. Producer idempotence is off by default as in
2.x. Needs produce/consume against Kafka 2.x and 3.x brokers, with and
without SASL, in a sandbox.
Testing: JDK 8, Checkstyle on (0 violations).
mvn -B clean install (all modules): BUILD SUCCESS, 13 tests, 0 failures
(embedded ZooKeeper + Kafka 3.9.2 broker: KafkaBatchSourceTest,
KafkaSinkAndAlertsPublisherTest, KafkaStreamingSource*Test,
KafkaStreamingSourceUtilTest). The FAILED program runs in the log
(RecoveringTransform, one testKafkaSink MapReduce run) are negative test
cases and appear identically on the parent branches.
mvn dependency:tree -pl kafka-plugins-client -Dincludes=org.apache.kafka,
org.lz4,at.yawk.lz4,org.xerial.snappy: kafka-clients 3.9.2 (compile),
at.yawk.lz4 1.10.1, snappy-java 1.1.10.12 (runtime), kafka_2.12 3.9.2
(test); org.lz4 1.7.1 only under provided spark-core. Plugin jar lib/:
kafka-clients-3.9.2, lz4-java-1.10.1, snappy-java-1.10.12,
zstd-jni-1.5.6-4; no kafka-clients-2.6.0.
Not run: Kafka 2.x / 3.x broker and SASL tests in a sandbox (P2).