Repository navigation
[deferred] Upgrade adls-plugins ORC 1.1.0 -> 1.9.8, drop protobuf 2.5.0 - #171
deepika-lalwani123 wants to merge 3 commits into
Conversation
The image scan of gcr.io/cdapio/cdap reports these rows against /opt/cdap/master/artifacts/adls-plugins-*.jar/lib/: io.netty:netty 3.6.2.Final (2), log4j:log4j 1.2.17 (1), com.sun.jersey:jersey-core 1.9 (1), Jetty 9.4.54 jetty-server / jetty-http / jetty-security (5) and commons-io 2.8.0 inside velocity-engine-core-2.3 (1). None of them is used by plugin code. orc-mapreduce 1.1.0 only excluded hadoop-common and hadoop-hdfs, so it still packaged hadoop-mapreduce-client-core and hadoop-yarn-common 2.6.0, and with them Netty 3.6.2, log4j 1.2.17, slf4j-log4j12, jersey 1.9, guice 3, jaxb-impl and the codehaus jackson 1.9.13 jars. Hadoop is supplied by CDAP (hadoop-client 3.3.6 is already provided in the parent pom), so exclude org.apache.hadoop:* from orc-mapreduce. The plan marked the Jetty 9.4.54 source as "(verify)"; dependency:tree shows it is avro-mapred -> avro-ipc-jetty (jetty-server, -http, -io, -servlet, -security, -util, -util-ajax). avro-ipc / avro-ipc-jetty are only needed for Avro RPC (tether), not for AvroKeyOutputFormat, so they are excluded from avro-mapred (same as hydrator-plugins HP-1). Avro core used to arrive only through avro-ipc, so it is now declared directly (avro.version, 1.11.4) with its commons-compress / commons-io / commons-lang3 dependencies excluded, which keeps them out of the jar exactly as on develop (Hadoop supplies them). The snappy-java, xz and zstd-jni jars that avro-ipc brought also leave the jar; the ADLS sink has no compression option, so Avro output uses no codec. Versions: - hadoop-mapreduce-client-core / hadoop-yarn-common 2.6.0 -> removed (orc-mapreduce exclusion org.apache.hadoop:*, adls-plugins/pom.xml) - netty 3.6.2.Final, log4j 1.2.17, slf4j-log4j12 1.7.5, jersey-* 1.9, jackson-*-asl 1.9.13 -> removed (same exclusion) - jetty-* 9.4.54.v20240208, avro-ipc, avro-ipc-jetty, velocity-engine-core 2.3, snappy-java 1.1.10.5, xz 1.9, zstd-jni 1.5.5-5 -> removed (avro-mapred exclusions) - avro 1.11.4 -> unchanged, now a direct dependency Bugs: Netty 3 b/578597973, b/578686878; jersey-core b/578444945; log4j 1 b/578595367. Plugins side done for Jetty b/578497046, b/578498486, b/578499995, b/578565223, b/578580375. Risk: low-medium. Only jars no plugin code uses are dropped; ORC and Avro output formats now always run against CDAP's Hadoop 3.3.6 classes (hadoop-common was already excluded, so this was mostly the case). Needs an ADLS sink run with avro, orc and text output on a cluster. Testing: JDK 8. mvn -B clean verify (all 4 modules): BUILD SUCCESS, StructuredToTextTransformerTest: Tests run: 2, Failures: 0, Errors: 0 (the repo has no Checkstyle plugin configured). unzip -l adls-plugins/target/adls-plugins-1.19.0-SNAPSHOT.jar vs develop: 35 lib/ jars removed (the ones listed above plus aopalliance, asm, guice, guice-servlet, javax.inject, javax.annotation-api, javax.servlet-api, jaxb-impl), none added. azure-blob-store jar lib/ unchanged. mvn dependency:tree -pl adls-plugins -Dscope=runtime -Dincludes=org.apache.hadoop,org.eclipse.jetty,io.netty,log4j, com.sun.jersey,org.apache.avro: only avro 1.11.4, avro-mapred 1.11.4 and hadoop-azure-datalake 3.3.6 remain at runtime; Jetty, Netty, jersey and log4j appear only as provided (hadoop-client) or test.
The image scan of gcr.io/cdapio/cdap reports jackson-core (3) and jackson-databind (11) rows against both adls-plugins-*.jar and azure-blob-store-*.jar (2.13.4 / 2.13.4.2), jettison 1.1 (5 rows) and org.iq80.snappy 0.2 (1 row) against adls-plugins-*.jar. Only jackson-databind was pinned, so jackson-core and jackson-annotations stayed at 2.13.4. jettison and iq80 snappy are direct dependencies of orc-core 1.1.0 and cannot be excluded while adls stays on ORC 1.1.0. Import com.fasterxml.jackson:jackson-bom in the root dependencyManagement and manage jettison and iq80 snappy there. Versions: - jackson-databind 2.13.4.2 -> 2.18.12, jackson-core / -annotations 2.13.4 -> 2.18.12 (root jackson.databind.version replaced by jackson.version + jackson-bom import; adls-plugins and azure-blob-store drop their explicit version) - org.codehaus.jettison:jettison 1.1 -> 1.5.4 (root DM; adls-plugins); stax-api 1.0.1, a dependency of jettison 1.1 only, leaves the jar - org.iq80.snappy:snappy 0.2 -> 0.5 (root DM; adls-plugins) Bugs: none completed by this PR alone. Plugins side done for jettison b/578489064, b/578501183, b/578519621, b/578586713, b/578631934 (they also need Platform / Dockerfile). iq80 snappy is rows only (0 bugs). Risk: low. Jackson 2.15+ adds default stream read limits and 2.18 changes some property introspection; the plugins only use Jackson via Azure / Hadoop client code. jettison 1.5 keeps the org.codehaus.jettison.json API that orc-core 1.1.0 uses (JSON file dump only). Testing: JDK 8. mvn -B clean verify (all 4 modules): BUILD SUCCESS, StructuredToTextTransformerTest: Tests run: 2, Failures: 0, Errors: 0 (no Checkstyle plugin in this repo). mvn dependency:tree -Dincludes=com.fasterxml.jackson*,org.codehaus.jettison,org.iq80.snappy: every module resolves only Jackson 2.18.12; adls-plugins resolves jettison 1.5.4 and snappy 0.5. unzip -l vs the parent branch: adls lib/ now has jackson-* 2.18.12, jettison-1.5.4, snappy-0.5 (old 2.13.4*, jettison-1.1, snappy-0.2, stax-api-1.0.1 gone); azure-blob-store lib/ has jackson-* 2.18.12, nothing else changed.
The image scan of gcr.io/cdapio/cdap reports 3 protobuf-java 2.5.0 rows against /opt/cdap/master/artifacts/adls-plugins-*.jar/lib/, which comes from orc-core 1.1.0. ORC 1.9.8 is the last ORC line that runs on Java 8 and uses protobuf 3. Deferred in the plan (AZ-3, with hydrator-plugins HP-5): the same three protobuf bugs stay open through hbase-shaded-mapreduce and the Hadoop 3.3.6 tarball, so this closes 0 bugs on its own. Kept as the last branch of the stack so it can land in the next patch. Correction to the plan: orc-core 1.9.8 does not drop the protobuf dependency. Its published (dependency-reduced) pom lists neither protobuf-java nor hive-storage-api, but the classes still reference com.google.protobuf.GeneratedMessageV3 and org.apache.hadoop.hive.ql.exec.vector.*, and neither is shaded into the jar. Both are therefore declared in adls-plugins/pom.xml; without protobuf-java there, Hadoop's provided protobuf 2.5.0 would win. Versions: - orc-mapreduce / orc-core 1.1.0 -> 1.9.8 (+ orc-shims 1.9.8, aircompressor 0.27, threeten-extra 1.7.1, kryo-shaded 3.0.3 -> 4.0.2; adls-plugins/pom.xml) - protobuf-java 2.5.0 -> 3.25.5 (direct dependency, adls-plugins) - hive-storage-api 2.1.0-pre-orc -> 2.8.1 (direct dependency, adls-plugins) - commons-lang3 3.12.0 (new, from ORC 1.9.8) -> 3.18.0 (root DM; azure-blob-store pins 3.14.0 itself and is unchanged) - jettison, iq80 snappy, objenesis, commons-cli / -codec / -lang 2.x (ORC 1.1.0 only) leave the jar; the jettison / snappy DM entries from the previous commit become no-ops for adls but stay harmless Bugs: none (protobuf b/578423011, b/578459565, b/578641178 stay open through the hbase-shaded / Hadoop exception). Removes the 3 adls protobuf-java 2.5.0 rows. Risk: high. ORC writer moves 8 minor versions (default file version 0.12, ZLIB compression unchanged). Needs an ADLS sink run with orc output on a cluster, and a check that the plugin class loader takes protobuf-java 3.25.5 from the plugin jar rather than Hadoop's 2.5.0. Testing: JDK 8. mvn -B clean verify (all 4 modules): BUILD SUCCESS, Tests run: 3, Failures: 0, Errors: 0 (new StructuredToOrcTransformerTest writes two records with OrcMapreduceRecordWriter and reads them back with OrcMapreduceRecordReader, which exercises protobuf and hive-storage-api; StructuredToTextTransformerTest). mvn dependency:tree -pl adls-plugins: orc-* 1.9.8, protobuf-java 3.25.5, hive-storage-api 2.8.1, commons-lang3 3.18.0 at compile; no protobuf 2.5.0 outside provided hadoop-client. unzip -l of the adls jar: protobuf-java-3.25.5, orc-*-1.9.8, hive-storage-api-2.8.1; no protobuf-java-2.5.0.
There was a problem hiding this comment.
Code Review
This pull request updates several dependencies across the project, including upgrading Jackson to 2.18.12 using a BOM, upgrading orc-mapreduce to 1.9.8, and managing transitive dependencies to prevent version conflicts. It also introduces a new test class, StructuredToOrcTransformerTest, to verify ORC file writing and reading. Feedback on the new test class suggests using try-with-resources blocks for the Writer and OrcMapreduceRecordReader to prevent potential resource leaks, and explicitly specifying StandardCharsets.UTF_8 when converting strings to bytes to avoid platform-dependent encoding issues.
| OrcMapreduceRecordWriter<OrcStruct> recordWriter = new OrcMapreduceRecordWriter<>(writer); | ||
| recordWriter.write(NullWritable.get(), first); | ||
| recordWriter.write(NullWritable.get(), transformer.transform(record(null, 2, "b"))); | ||
| recordWriter.close(null); | ||
|
|
There was a problem hiding this comment.
The ORC Writer implements Closeable. If an exception is thrown during recordWriter.write(...) before recordWriter.close(null) is reached, the writer will not be closed, leading to a resource leak. Wrapping the Writer in a try-with-resources block ensures it is always closed properly.
| OrcMapreduceRecordWriter<OrcStruct> recordWriter = new OrcMapreduceRecordWriter<>(writer); | |
| recordWriter.write(NullWritable.get(), first); | |
| recordWriter.write(NullWritable.get(), transformer.transform(record(null, 2, "b"))); | |
| recordWriter.close(null); | |
| try (Writer writer = OrcFile.createWriter(path, OrcFile.writerOptions(conf).setSchema(first.getSchema()))) { | |
| OrcMapreduceRecordWriter<OrcStruct> recordWriter = new OrcMapreduceRecordWriter<>(writer); | |
| recordWriter.write(NullWritable.get(), first); | |
| recordWriter.write(NullWritable.get(), transformer.transform(record(null, 2, "b"))); | |
| recordWriter.close(null); | |
| } |
| OrcMapreduceRecordReader<OrcStruct> recordReader = new OrcMapreduceRecordReader<>(reader, reader.options()); | ||
| Assert.assertTrue(recordReader.nextKeyValue()); | ||
| OrcStruct row = recordReader.getCurrentValue(); | ||
| Assert.assertEquals("true", row.getFieldValue("bool_field").toString()); | ||
| Assert.assertEquals("1", row.getFieldValue("int_field").toString()); | ||
| Assert.assertEquals("a", row.getFieldValue("string_field").toString()); | ||
| Assert.assertTrue(recordReader.nextKeyValue()); | ||
| row = recordReader.getCurrentValue(); | ||
| Assert.assertNull(row.getFieldValue("bool_field")); | ||
| Assert.assertEquals("2", row.getFieldValue("int_field").toString()); | ||
| Assert.assertEquals("b", row.getFieldValue("string_field").toString()); | ||
| Assert.assertFalse(recordReader.nextKeyValue()); | ||
| recordReader.close(); |
There was a problem hiding this comment.
The OrcMapreduceRecordReader implements Closeable. If any of the assertions fail, recordReader.close() will not be called, resulting in a resource leak. Using a try-with-resources block ensures that the reader is closed under all circumstances.
| OrcMapreduceRecordReader<OrcStruct> recordReader = new OrcMapreduceRecordReader<>(reader, reader.options()); | |
| Assert.assertTrue(recordReader.nextKeyValue()); | |
| OrcStruct row = recordReader.getCurrentValue(); | |
| Assert.assertEquals("true", row.getFieldValue("bool_field").toString()); | |
| Assert.assertEquals("1", row.getFieldValue("int_field").toString()); | |
| Assert.assertEquals("a", row.getFieldValue("string_field").toString()); | |
| Assert.assertTrue(recordReader.nextKeyValue()); | |
| row = recordReader.getCurrentValue(); | |
| Assert.assertNull(row.getFieldValue("bool_field")); | |
| Assert.assertEquals("2", row.getFieldValue("int_field").toString()); | |
| Assert.assertEquals("b", row.getFieldValue("string_field").toString()); | |
| Assert.assertFalse(recordReader.nextKeyValue()); | |
| recordReader.close(); | |
| try (OrcMapreduceRecordReader<OrcStruct> recordReader = new OrcMapreduceRecordReader<>(reader, reader.options())) { | |
| Assert.assertTrue(recordReader.nextKeyValue()); | |
| OrcStruct row = recordReader.getCurrentValue(); | |
| Assert.assertEquals("true", row.getFieldValue("bool_field").toString()); | |
| Assert.assertEquals("1", row.getFieldValue("int_field").toString()); | |
| Assert.assertEquals("a", row.getFieldValue("string_field").toString()); | |
| Assert.assertTrue(recordReader.nextKeyValue()); | |
| row = recordReader.getCurrentValue(); | |
| Assert.assertNull(row.getFieldValue("bool_field")); | |
| Assert.assertEquals("2", row.getFieldValue("int_field").toString()); | |
| Assert.assertEquals("b", row.getFieldValue("string_field").toString()); | |
| Assert.assertFalse(recordReader.nextKeyValue()); | |
| } |
| .set("int_field", i) | ||
| .set("long_field", (long) i) | ||
| .set("double_field", (double) i) | ||
| .set("bytes_field", s.getBytes()) |
There was a problem hiding this comment.
Using String.getBytes() without specifying an explicit charset relies on the platform's default encoding, which can cause flaky tests or unexpected behavior across different environments. It is safer to specify StandardCharsets.UTF_8 explicitly.
| .set("bytes_field", s.getBytes()) | |
| .set("bytes_field", s.getBytes(java.nio.charset.StandardCharsets.UTF_8)) |
|



[deferred] Upgrade adls-plugins ORC 1.1.0 -> 1.9.8, drop protobuf 2.5.0
Repo: data-integrations/azure
Branch: vuln/az-3-orc-1.9.8 → base develop (stacked)
Commit: e55d87b
Part of the CVE fixes for the CDAP 6.11.2 patch release (container image scanning of gcr.io/cdapio/cdap).
The image scan of gcr.io/cdapio/cdap reports 3 protobuf-java 2.5.0 rows
against /opt/cdap/master/artifacts/adls-plugins-*.jar/lib/, which comes
from orc-core 1.1.0. ORC 1.9.8 is the last ORC line that runs on Java 8
and uses protobuf 3.
Deferred in the plan (#171, with cdapio/hydrator-plugins#2018): the same
three protobuf findings stay open through hbase-shaded-mapreduce and the
Hadoop 3.3.6 tarball, so this resolves 0 findings on its own. Kept as the
last branch of the stack so it can land in the next patch.
Correction to the plan: orc-core 1.9.8 does not drop the protobuf
dependency. Its published (dependency-reduced) pom lists neither
protobuf-java nor hive-storage-api, but the classes still reference
com.google.protobuf.GeneratedMessageV3 and
org.apache.hadoop.hive.ql.exec.vector.*, and neither is shaded into the
jar. Both are therefore declared in adls-plugins/pom.xml; without
protobuf-java there, Hadoop's provided protobuf 2.5.0 would win.
Versions:
aircompressor 0.27, threeten-extra 1.7.1, kryo-shaded 3.0.3 -> 4.0.2;
adls-plugins/pom.xml)
azure-blob-store pins 3.14.0 itself and is unchanged)
(ORC 1.1.0 only) leave the jar; the jettison / snappy DM entries
from the previous commit become no-ops for adls but stay harmless
The protobuf-java 2.5.0 findings stay open through the hbase-shaded /
Hadoop exception. Removes the 3 adls protobuf-java 2.5.0 rows.
Risk: high. ORC writer moves 8 minor versions (default file version
0.12, ZLIB compression unchanged). Needs an ADLS sink run with orc
output on a cluster, and a check that the plugin class loader takes
protobuf-java 3.25.5 from the plugin jar rather than Hadoop's 2.5.0.
Testing: JDK 8. mvn -B clean verify (all 4 modules): BUILD SUCCESS,
Tests run: 3, Failures: 0, Errors: 0 (new StructuredToOrcTransformerTest
writes two records with OrcMapreduceRecordWriter and reads them back
with OrcMapreduceRecordReader, which exercises protobuf and
hive-storage-api; StructuredToTextTransformerTest). mvn dependency:tree
-pl adls-plugins: orc-* 1.9.8, protobuf-java 3.25.5, hive-storage-api
2.8.1, commons-lang3 3.18.0 at compile; no protobuf 2.5.0 outside
provided hadoop-client. unzip -l of the adls jar: protobuf-java-3.25.5,
orc-*-1.9.8, hive-storage-api-2.8.1; no protobuf-java-2.5.0.