From 8c1c02944c5a181468b1e2177538fbe43011d6f2 Mon Sep 17 00:00:00 2001 From: Deepika Lalwani Date: Tue, 17 Nov 2026 06:02:57 +0000 Subject: [PATCH 1/3] Remove Hadoop 2.6.0, Netty 3, log4j 1 and Jetty jars from adls-plugins The image scan of gcr.io/cdapio/cdap reports these rows against /opt/cdap/master/artifacts/adls-plugins-*.jar/lib/: io.netty:netty 3.6.2.Final (2), log4j:log4j 1.2.17 (1), com.sun.jersey:jersey-core 1.9 (1), Jetty 9.4.54 jetty-server / jetty-http / jetty-security (5) and commons-io 2.8.0 inside velocity-engine-core-2.3 (1). None of them is used by plugin code. orc-mapreduce 1.1.0 only excluded hadoop-common and hadoop-hdfs, so it still packaged hadoop-mapreduce-client-core and hadoop-yarn-common 2.6.0, and with them Netty 3.6.2, log4j 1.2.17, slf4j-log4j12, jersey 1.9, guice 3, jaxb-impl and the codehaus jackson 1.9.13 jars. Hadoop is supplied by CDAP (hadoop-client 3.3.6 is already provided in the parent pom), so exclude org.apache.hadoop:* from orc-mapreduce. The plan marked the Jetty 9.4.54 source as "(verify)"; dependency:tree shows it is avro-mapred -> avro-ipc-jetty (jetty-server, -http, -io, -servlet, -security, -util, -util-ajax). avro-ipc / avro-ipc-jetty are only needed for Avro RPC (tether), not for AvroKeyOutputFormat, so they are excluded from avro-mapred (same as hydrator-plugins HP-1). Avro core used to arrive only through avro-ipc, so it is now declared directly (avro.version, 1.11.4) with its commons-compress / commons-io / commons-lang3 dependencies excluded, which keeps them out of the jar exactly as on develop (Hadoop supplies them). The snappy-java, xz and zstd-jni jars that avro-ipc brought also leave the jar; the ADLS sink has no compression option, so Avro output uses no codec. Versions: - hadoop-mapreduce-client-core / hadoop-yarn-common 2.6.0 -> removed (orc-mapreduce exclusion org.apache.hadoop:*, adls-plugins/pom.xml) - netty 3.6.2.Final, log4j 1.2.17, slf4j-log4j12 1.7.5, jersey-* 1.9, jackson-*-asl 1.9.13 -> removed (same exclusion) - jetty-* 9.4.54.v20240208, avro-ipc, avro-ipc-jetty, velocity-engine-core 2.3, snappy-java 1.1.10.5, xz 1.9, zstd-jni 1.5.5-5 -> removed (avro-mapred exclusions) - avro 1.11.4 -> unchanged, now a direct dependency Bugs: Netty 3 b/578597973, b/578686878; jersey-core b/578444945; log4j 1 b/578595367. Plugins side done for Jetty b/578497046, b/578498486, b/578499995, b/578565223, b/578580375. Risk: low-medium. Only jars no plugin code uses are dropped; ORC and Avro output formats now always run against CDAP's Hadoop 3.3.6 classes (hadoop-common was already excluded, so this was mostly the case). Needs an ADLS sink run with avro, orc and text output on a cluster. Testing: JDK 8. mvn -B clean verify (all 4 modules): BUILD SUCCESS, StructuredToTextTransformerTest: Tests run: 2, Failures: 0, Errors: 0 (the repo has no Checkstyle plugin configured). unzip -l adls-plugins/target/adls-plugins-1.19.0-SNAPSHOT.jar vs develop: 35 lib/ jars removed (the ones listed above plus aopalliance, asm, guice, guice-servlet, javax.inject, javax.annotation-api, javax.servlet-api, jaxb-impl), none added. azure-blob-store jar lib/ unchanged. mvn dependency:tree -pl adls-plugins -Dscope=runtime -Dincludes=org.apache.hadoop,org.eclipse.jetty,io.netty,log4j, com.sun.jersey,org.apache.avro: only avro 1.11.4, avro-mapred 1.11.4 and hadoop-azure-datalake 3.3.6 remain at runtime; Jetty, Netty, jersey and log4j appear only as provided (hadoop-client) or test. --- adls-plugins/pom.xml | 48 +++++++++++++++++++++++++++++++++++++++----- 1 file changed, 43 insertions(+), 5 deletions(-) diff --git a/adls-plugins/pom.xml b/adls-plugins/pom.xml index 84ba8d2..61e867f 100644 --- a/adls-plugins/pom.xml +++ b/adls-plugins/pom.xml @@ -28,6 +28,35 @@ adls-plugins + + + org.apache.avro + avro + ${avro.version} + + + com.fasterxml.jackson.core + jackson-core + + + com.fasterxml.jackson.core + jackson-databind + + + org.apache.commons + commons-compress + + + commons-io + commons-io + + + org.apache.commons + commons-lang3 + + + org.apache.avro avro-mapred @@ -41,6 +70,16 @@ com.fasterxml.jackson.core jackson-databind + + + org.apache.avro + avro-ipc + + + org.apache.avro + avro-ipc-jetty + @@ -64,13 +103,12 @@ orc-mapreduce 1.1.0 + org.apache.hadoop - hadoop-hdfs - - - org.apache.hadoop - hadoop-common + * From db4cfb8f24bc2bb5a30b03e182285f10b5185ff6 Mon Sep 17 00:00:00 2001 From: Deepika Lalwani Date: Tue, 17 Nov 2026 06:03:56 +0000 Subject: [PATCH 2/3] Import jackson-bom 2.18.12 and manage jettison 1.5.4, iq80 snappy 0.5 The image scan of gcr.io/cdapio/cdap reports jackson-core (3) and jackson-databind (11) rows against both adls-plugins-*.jar and azure-blob-store-*.jar (2.13.4 / 2.13.4.2), jettison 1.1 (5 rows) and org.iq80.snappy 0.2 (1 row) against adls-plugins-*.jar. Only jackson-databind was pinned, so jackson-core and jackson-annotations stayed at 2.13.4. jettison and iq80 snappy are direct dependencies of orc-core 1.1.0 and cannot be excluded while adls stays on ORC 1.1.0. Import com.fasterxml.jackson:jackson-bom in the root dependencyManagement and manage jettison and iq80 snappy there. Versions: - jackson-databind 2.13.4.2 -> 2.18.12, jackson-core / -annotations 2.13.4 -> 2.18.12 (root jackson.databind.version replaced by jackson.version + jackson-bom import; adls-plugins and azure-blob-store drop their explicit version) - org.codehaus.jettison:jettison 1.1 -> 1.5.4 (root DM; adls-plugins); stax-api 1.0.1, a dependency of jettison 1.1 only, leaves the jar - org.iq80.snappy:snappy 0.2 -> 0.5 (root DM; adls-plugins) Bugs: none completed by this PR alone. Plugins side done for jettison b/578489064, b/578501183, b/578519621, b/578586713, b/578631934 (they also need Platform / Dockerfile). iq80 snappy is rows only (0 bugs). Risk: low. Jackson 2.15+ adds default stream read limits and 2.18 changes some property introspection; the plugins only use Jackson via Azure / Hadoop client code. jettison 1.5 keeps the org.codehaus.jettison.json API that orc-core 1.1.0 uses (JSON file dump only). Testing: JDK 8. mvn -B clean verify (all 4 modules): BUILD SUCCESS, StructuredToTextTransformerTest: Tests run: 2, Failures: 0, Errors: 0 (no Checkstyle plugin in this repo). mvn dependency:tree -Dincludes=com.fasterxml.jackson*,org.codehaus.jettison,org.iq80.snappy: every module resolves only Jackson 2.18.12; adls-plugins resolves jettison 1.5.4 and snappy 0.5. unzip -l vs the parent branch: adls lib/ now has jackson-* 2.18.12, jettison-1.5.4, snappy-0.5 (old 2.13.4*, jettison-1.1, snappy-0.2, stax-api-1.0.1 gone); azure-blob-store lib/ has jackson-* 2.18.12, nothing else changed. --- adls-plugins/pom.xml | 1 - azure-blob-store/pom.xml | 1 - pom.xml | 20 +++++++++++++++++++- 3 files changed, 19 insertions(+), 3 deletions(-) diff --git a/adls-plugins/pom.xml b/adls-plugins/pom.xml index 61e867f..5675acc 100644 --- a/adls-plugins/pom.xml +++ b/adls-plugins/pom.xml @@ -85,7 +85,6 @@ com.fasterxml.jackson.core jackson-databind - ${jackson.databind.version} junit diff --git a/azure-blob-store/pom.xml b/azure-blob-store/pom.xml index b35f2d1..053c5e3 100644 --- a/azure-blob-store/pom.xml +++ b/azure-blob-store/pom.xml @@ -42,7 +42,6 @@ com.fasterxml.jackson.core jackson-databind - ${jackson.databind.version} org.apache.commons diff --git a/pom.xml b/pom.xml index 1e5b084..a2768b2 100644 --- a/pom.xml +++ b/pom.xml @@ -80,7 +80,7 @@ 6.11.0 3.3.6 2.13.0 - 2.13.4.2 + 2.18.12 32.0.0-jre 1.11.4 widgets @@ -141,6 +141,24 @@ + + com.fasterxml.jackson + jackson-bom + ${jackson.version} + pom + import + + + + org.codehaus.jettison + jettison + 1.5.4 + + + org.iq80.snappy + snappy + 0.5 + io.cdap.cdap cdap-formats From e55d87bcf37e34c8ebbcda1adb088b49f8f5a523 Mon Sep 17 00:00:00 2001 From: Deepika Lalwani Date: Tue, 17 Nov 2026 06:07:38 +0000 Subject: [PATCH 3/3] [deferred] Upgrade adls-plugins ORC 1.1.0 -> 1.9.8, drop protobuf 2.5.0 The image scan of gcr.io/cdapio/cdap reports 3 protobuf-java 2.5.0 rows against /opt/cdap/master/artifacts/adls-plugins-*.jar/lib/, which comes from orc-core 1.1.0. ORC 1.9.8 is the last ORC line that runs on Java 8 and uses protobuf 3. Deferred in the plan (AZ-3, with hydrator-plugins HP-5): the same three protobuf bugs stay open through hbase-shaded-mapreduce and the Hadoop 3.3.6 tarball, so this closes 0 bugs on its own. Kept as the last branch of the stack so it can land in the next patch. Correction to the plan: orc-core 1.9.8 does not drop the protobuf dependency. Its published (dependency-reduced) pom lists neither protobuf-java nor hive-storage-api, but the classes still reference com.google.protobuf.GeneratedMessageV3 and org.apache.hadoop.hive.ql.exec.vector.*, and neither is shaded into the jar. Both are therefore declared in adls-plugins/pom.xml; without protobuf-java there, Hadoop's provided protobuf 2.5.0 would win. Versions: - orc-mapreduce / orc-core 1.1.0 -> 1.9.8 (+ orc-shims 1.9.8, aircompressor 0.27, threeten-extra 1.7.1, kryo-shaded 3.0.3 -> 4.0.2; adls-plugins/pom.xml) - protobuf-java 2.5.0 -> 3.25.5 (direct dependency, adls-plugins) - hive-storage-api 2.1.0-pre-orc -> 2.8.1 (direct dependency, adls-plugins) - commons-lang3 3.12.0 (new, from ORC 1.9.8) -> 3.18.0 (root DM; azure-blob-store pins 3.14.0 itself and is unchanged) - jettison, iq80 snappy, objenesis, commons-cli / -codec / -lang 2.x (ORC 1.1.0 only) leave the jar; the jettison / snappy DM entries from the previous commit become no-ops for adls but stay harmless Bugs: none (protobuf b/578423011, b/578459565, b/578641178 stay open through the hbase-shaded / Hadoop exception). Removes the 3 adls protobuf-java 2.5.0 rows. Risk: high. ORC writer moves 8 minor versions (default file version 0.12, ZLIB compression unchanged). Needs an ADLS sink run with orc output on a cluster, and a check that the plugin class loader takes protobuf-java 3.25.5 from the plugin jar rather than Hadoop's 2.5.0. Testing: JDK 8. mvn -B clean verify (all 4 modules): BUILD SUCCESS, Tests run: 3, Failures: 0, Errors: 0 (new StructuredToOrcTransformerTest writes two records with OrcMapreduceRecordWriter and reads them back with OrcMapreduceRecordReader, which exercises protobuf and hive-storage-api; StructuredToTextTransformerTest). mvn dependency:tree -pl adls-plugins: orc-* 1.9.8, protobuf-java 3.25.5, hive-storage-api 2.8.1, commons-lang3 3.18.0 at compile; no protobuf 2.5.0 outside provided hadoop-client. unzip -l of the adls jar: protobuf-java-3.25.5, orc-*-1.9.8, hive-storage-api-2.8.1; no protobuf-java-2.5.0. --- adls-plugins/pom.xml | 20 +++- .../sink/StructuredToOrcTransformerTest.java | 97 +++++++++++++++++++ pom.xml | 6 ++ 3 files changed, 122 insertions(+), 1 deletion(-) create mode 100644 adls-plugins/src/test/java/io/cdap/plugin/sink/StructuredToOrcTransformerTest.java diff --git a/adls-plugins/pom.xml b/adls-plugins/pom.xml index 5675acc..a0a591b 100644 --- a/adls-plugins/pom.xml +++ b/adls-plugins/pom.xml @@ -100,7 +100,7 @@ org.apache.orc orc-mapreduce - 1.1.0 + 1.9.8 + + com.google.protobuf + protobuf-java + 3.25.5 + + + org.apache.hive + hive-storage-api + 2.8.1 + + + * + * + + + org.apache.hadoop hadoop-azure-datalake diff --git a/adls-plugins/src/test/java/io/cdap/plugin/sink/StructuredToOrcTransformerTest.java b/adls-plugins/src/test/java/io/cdap/plugin/sink/StructuredToOrcTransformerTest.java new file mode 100644 index 0000000..e89eb35 --- /dev/null +++ b/adls-plugins/src/test/java/io/cdap/plugin/sink/StructuredToOrcTransformerTest.java @@ -0,0 +1,97 @@ +/* + * Copyright © 2026 Cask Data, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); you may not + * use this file except in compliance with the License. You may obtain a copy of + * the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + * License for the specific language governing permissions and limitations under + * the License. + */ + +package io.cdap.plugin.sink; + +import io.cdap.cdap.api.data.format.StructuredRecord; +import io.cdap.cdap.api.data.schema.Schema; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.io.NullWritable; +import org.apache.orc.OrcFile; +import org.apache.orc.Reader; +import org.apache.orc.Writer; +import org.apache.orc.mapred.OrcStruct; +import org.apache.orc.mapreduce.OrcMapreduceRecordReader; +import org.apache.orc.mapreduce.OrcMapreduceRecordWriter; +import org.junit.Assert; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.TemporaryFolder; + +import java.io.File; + +/** + * Writes records produced by {@link StructuredToOrcTransformer} to an ORC file and reads them back. + */ +public class StructuredToOrcTransformerTest { + + private static final Schema SCHEMA = + Schema.recordOf("record", + Schema.Field.of("bool_field", Schema.nullableOf(Schema.of(Schema.Type.BOOLEAN))), + Schema.Field.of("int_field", Schema.of(Schema.Type.INT)), + Schema.Field.of("long_field", Schema.of(Schema.Type.LONG)), + Schema.Field.of("double_field", Schema.of(Schema.Type.DOUBLE)), + Schema.Field.of("bytes_field", Schema.of(Schema.Type.BYTES)), + Schema.Field.of("string_field", Schema.of(Schema.Type.STRING))); + + @Rule + public TemporaryFolder tmpFolder = new TemporaryFolder(); + + @Test + public void testWriteAndReadOrcFile() throws Exception { + StructuredToOrcTransformer transformer = new StructuredToOrcTransformer(SCHEMA); + Configuration conf = new Configuration(); + Path path = new Path(new File(tmpFolder.getRoot(), "test.orc").toURI()); + + OrcStruct first = transformer.transform(record(true, 1, "a")); + Writer writer = OrcFile.createWriter(path, OrcFile.writerOptions(conf).setSchema(first.getSchema())); + OrcMapreduceRecordWriter recordWriter = new OrcMapreduceRecordWriter<>(writer); + recordWriter.write(NullWritable.get(), first); + recordWriter.write(NullWritable.get(), transformer.transform(record(null, 2, "b"))); + recordWriter.close(null); + + Reader reader = OrcFile.createReader(path, OrcFile.readerOptions(conf)); + Assert.assertEquals(2, reader.getNumberOfRows()); + Assert.assertEquals("struct", reader.getSchema().toString()); + + OrcMapreduceRecordReader recordReader = new OrcMapreduceRecordReader<>(reader, reader.options()); + Assert.assertTrue(recordReader.nextKeyValue()); + OrcStruct row = recordReader.getCurrentValue(); + Assert.assertEquals("true", row.getFieldValue("bool_field").toString()); + Assert.assertEquals("1", row.getFieldValue("int_field").toString()); + Assert.assertEquals("a", row.getFieldValue("string_field").toString()); + Assert.assertTrue(recordReader.nextKeyValue()); + row = recordReader.getCurrentValue(); + Assert.assertNull(row.getFieldValue("bool_field")); + Assert.assertEquals("2", row.getFieldValue("int_field").toString()); + Assert.assertEquals("b", row.getFieldValue("string_field").toString()); + Assert.assertFalse(recordReader.nextKeyValue()); + recordReader.close(); + } + + private static StructuredRecord record(Boolean bool, int i, String s) { + return StructuredRecord.builder(SCHEMA) + .set("bool_field", bool) + .set("int_field", i) + .set("long_field", (long) i) + .set("double_field", (double) i) + .set("bytes_field", s.getBytes()) + .set("string_field", s) + .build(); + } +} diff --git a/pom.xml b/pom.xml index a2768b2..d15560d 100644 --- a/pom.xml +++ b/pom.xml @@ -159,6 +159,12 @@ snappy 0.5 + + + org.apache.commons + commons-lang3 + 3.18.0 + io.cdap.cdap cdap-formats