Skip to content

[deferred] Upgrade adls-plugins ORC 1.1.0 -> 1.9.8, drop protobuf 2.5.0 - #171

Draft
deepika-lalwani123 wants to merge 3 commits into
developfrom
vuln/az-3-orc-1.9.8
Draft

deepika-lalwani123 wants to merge 3 commits into
developfrom
vuln/az-3-orc-1.9.8

Conversation

@deepika-lalwani123

@deepika-lalwani123 deepika-lalwani123 commented Nov 17, 2026 •

Copy link
Copy Markdown

[deferred] Upgrade adls-plugins ORC 1.1.0 -> 1.9.8, drop protobuf 2.5.0

Draft: deferred in the plan (ORC upgrade); include in 6.11.2 or the next patch is pending.

Stacked PR: this branch is built on top of #169, #170. Against develop the diff includes those commits; please review only the top commit e55d87bcf3. Merge after #170.

Repo: data-integrations/azure
Branch: vuln/az-3-orc-1.9.8 → base develop (stacked)
Commit: e55d87b
Part of the CVE fixes for the CDAP 6.11.2 patch release (container image scanning of gcr.io/cdapio/cdap).

The image scan of gcr.io/cdapio/cdap reports 3 protobuf-java 2.5.0 rows
against /opt/cdap/master/artifacts/adls-plugins-*.jar/lib/, which comes
from orc-core 1.1.0. ORC 1.9.8 is the last ORC line that runs on Java 8
and uses protobuf 3.

Deferred in the plan (#171, with cdapio/hydrator-plugins#2018): the same
three protobuf findings stay open through hbase-shaded-mapreduce and the
Hadoop 3.3.6 tarball, so this resolves 0 findings on its own. Kept as the
last branch of the stack so it can land in the next patch.

Correction to the plan: orc-core 1.9.8 does not drop the protobuf
dependency. Its published (dependency-reduced) pom lists neither
protobuf-java nor hive-storage-api, but the classes still reference
com.google.protobuf.GeneratedMessageV3 and
org.apache.hadoop.hive.ql.exec.vector.*, and neither is shaded into the
jar. Both are therefore declared in adls-plugins/pom.xml; without
protobuf-java there, Hadoop's provided protobuf 2.5.0 would win.

Versions:

  • orc-mapreduce / orc-core 1.1.0 -> 1.9.8 (+ orc-shims 1.9.8,
    aircompressor 0.27, threeten-extra 1.7.1, kryo-shaded 3.0.3 -> 4.0.2;
    adls-plugins/pom.xml)
  • protobuf-java 2.5.0 -> 3.25.5 (direct dependency, adls-plugins)
  • hive-storage-api 2.1.0-pre-orc -> 2.8.1 (direct dependency, adls-plugins)
  • commons-lang3 3.12.0 (new, from ORC 1.9.8) -> 3.18.0 (root DM;
    azure-blob-store pins 3.14.0 itself and is unchanged)
  • jettison, iq80 snappy, objenesis, commons-cli / -codec / -lang 2.x
    (ORC 1.1.0 only) leave the jar; the jettison / snappy DM entries
    from the previous commit become no-ops for adls but stay harmless

The protobuf-java 2.5.0 findings stay open through the hbase-shaded /
Hadoop exception. Removes the 3 adls protobuf-java 2.5.0 rows.

Risk: high. ORC writer moves 8 minor versions (default file version
0.12, ZLIB compression unchanged). Needs an ADLS sink run with orc
output on a cluster, and a check that the plugin class loader takes
protobuf-java 3.25.5 from the plugin jar rather than Hadoop's 2.5.0.

Testing: JDK 8. mvn -B clean verify (all 4 modules): BUILD SUCCESS,
Tests run: 3, Failures: 0, Errors: 0 (new StructuredToOrcTransformerTest
writes two records with OrcMapreduceRecordWriter and reads them back
with OrcMapreduceRecordReader, which exercises protobuf and
hive-storage-api; StructuredToTextTransformerTest). mvn dependency:tree
-pl adls-plugins: orc-* 1.9.8, protobuf-java 3.25.5, hive-storage-api
2.8.1, commons-lang3 3.18.0 at compile; no protobuf 2.5.0 outside
provided hadoop-client. unzip -l of the adls jar: protobuf-java-3.25.5,
orc-*-1.9.8, hive-storage-api-2.8.1; no protobuf-java-2.5.0.

The image scan of gcr.io/cdapio/cdap reports these rows against
/opt/cdap/master/artifacts/adls-plugins-*.jar/lib/: io.netty:netty
3.6.2.Final (2), log4j:log4j 1.2.17 (1), com.sun.jersey:jersey-core 1.9
(1), Jetty 9.4.54 jetty-server / jetty-http / jetty-security (5) and
commons-io 2.8.0 inside velocity-engine-core-2.3 (1). None of them is
used by plugin code.

orc-mapreduce 1.1.0 only excluded hadoop-common and hadoop-hdfs, so it
still packaged hadoop-mapreduce-client-core and hadoop-yarn-common
2.6.0, and with them Netty 3.6.2, log4j 1.2.17, slf4j-log4j12, jersey
1.9, guice 3, jaxb-impl and the codehaus jackson 1.9.13 jars. Hadoop is
supplied by CDAP (hadoop-client 3.3.6 is already provided in the
parent pom), so exclude org.apache.hadoop:* from orc-mapreduce.

The plan marked the Jetty 9.4.54 source as "(verify)"; dependency:tree
shows it is avro-mapred -> avro-ipc-jetty (jetty-server, -http, -io,
-servlet, -security, -util, -util-ajax). avro-ipc / avro-ipc-jetty are
only needed for Avro RPC (tether), not for AvroKeyOutputFormat, so they
are excluded from avro-mapred (same as hydrator-plugins HP-1). Avro core
used to arrive only through avro-ipc, so it is now declared directly
(avro.version, 1.11.4) with its commons-compress / commons-io /
commons-lang3 dependencies excluded, which keeps them out of the jar
exactly as on develop (Hadoop supplies them). The snappy-java, xz and
zstd-jni jars that avro-ipc brought also leave the jar; the ADLS sink
has no compression option, so Avro output uses no codec.

Versions:
- hadoop-mapreduce-client-core / hadoop-yarn-common 2.6.0 -> removed
  (orc-mapreduce exclusion org.apache.hadoop:*, adls-plugins/pom.xml)
- netty 3.6.2.Final, log4j 1.2.17, slf4j-log4j12 1.7.5, jersey-* 1.9,
  jackson-*-asl 1.9.13 -> removed (same exclusion)
- jetty-* 9.4.54.v20240208, avro-ipc, avro-ipc-jetty, velocity-engine-core
  2.3, snappy-java 1.1.10.5, xz 1.9, zstd-jni 1.5.5-5 -> removed
  (avro-mapred exclusions)
- avro 1.11.4 -> unchanged, now a direct dependency

Bugs: Netty 3 b/578597973, b/578686878; jersey-core b/578444945;
log4j 1 b/578595367. Plugins side done for Jetty b/578497046,
b/578498486, b/578499995, b/578565223, b/578580375.

Risk: low-medium. Only jars no plugin code uses are dropped; ORC and
Avro output formats now always run against CDAP's Hadoop 3.3.6 classes
(hadoop-common was already excluded, so this was mostly the case).
Needs an ADLS sink run with avro, orc and text output on a cluster.

Testing: JDK 8. mvn -B clean verify (all 4 modules): BUILD SUCCESS,
StructuredToTextTransformerTest: Tests run: 2, Failures: 0, Errors: 0
(the repo has no Checkstyle plugin configured). unzip -l
adls-plugins/target/adls-plugins-1.19.0-SNAPSHOT.jar vs develop: 35
lib/ jars removed (the ones listed above plus aopalliance, asm,
guice, guice-servlet, javax.inject, javax.annotation-api,
javax.servlet-api, jaxb-impl), none added. azure-blob-store jar lib/
unchanged. mvn dependency:tree -pl adls-plugins -Dscope=runtime
-Dincludes=org.apache.hadoop,org.eclipse.jetty,io.netty,log4j,
com.sun.jersey,org.apache.avro: only avro 1.11.4, avro-mapred 1.11.4
and hadoop-azure-datalake 3.3.6 remain at runtime; Jetty, Netty,
jersey and log4j appear only as provided (hadoop-client) or test.
The image scan of gcr.io/cdapio/cdap reports jackson-core (3) and
jackson-databind (11) rows against both adls-plugins-*.jar and
azure-blob-store-*.jar (2.13.4 / 2.13.4.2), jettison 1.1 (5 rows) and
org.iq80.snappy 0.2 (1 row) against adls-plugins-*.jar. Only
jackson-databind was pinned, so jackson-core and jackson-annotations
stayed at 2.13.4. jettison and iq80 snappy are direct dependencies of
orc-core 1.1.0 and cannot be excluded while adls stays on ORC 1.1.0.

Import com.fasterxml.jackson:jackson-bom in the root
dependencyManagement and manage jettison and iq80 snappy there.

Versions:
- jackson-databind 2.13.4.2 -> 2.18.12, jackson-core / -annotations
  2.13.4 -> 2.18.12 (root jackson.databind.version replaced by
  jackson.version + jackson-bom import; adls-plugins and
  azure-blob-store drop their explicit version)
- org.codehaus.jettison:jettison 1.1 -> 1.5.4 (root DM; adls-plugins);
  stax-api 1.0.1, a dependency of jettison 1.1 only, leaves the jar
- org.iq80.snappy:snappy 0.2 -> 0.5 (root DM; adls-plugins)

Bugs: none completed by this PR alone. Plugins side done for jettison
b/578489064, b/578501183, b/578519621, b/578586713, b/578631934 (they
also need Platform / Dockerfile). iq80 snappy is rows only (0 bugs).

Risk: low. Jackson 2.15+ adds default stream read limits and 2.18
changes some property introspection; the plugins only use Jackson via
Azure / Hadoop client code. jettison 1.5 keeps the
org.codehaus.jettison.json API that orc-core 1.1.0 uses (JSON file
dump only).

Testing: JDK 8. mvn -B clean verify (all 4 modules): BUILD SUCCESS,
StructuredToTextTransformerTest: Tests run: 2, Failures: 0, Errors: 0
(no Checkstyle plugin in this repo). mvn dependency:tree
-Dincludes=com.fasterxml.jackson*,org.codehaus.jettison,org.iq80.snappy:
every module resolves only Jackson 2.18.12; adls-plugins resolves
jettison 1.5.4 and snappy 0.5. unzip -l vs the parent branch: adls
lib/ now has jackson-* 2.18.12, jettison-1.5.4, snappy-0.5 (old
2.13.4*, jettison-1.1, snappy-0.2, stax-api-1.0.1 gone); azure-blob-store
lib/ has jackson-* 2.18.12, nothing else changed.
The image scan of gcr.io/cdapio/cdap reports 3 protobuf-java 2.5.0 rows
against /opt/cdap/master/artifacts/adls-plugins-*.jar/lib/, which comes
from orc-core 1.1.0. ORC 1.9.8 is the last ORC line that runs on Java 8
and uses protobuf 3.

Deferred in the plan (AZ-3, with hydrator-plugins HP-5): the same three
protobuf bugs stay open through hbase-shaded-mapreduce and the Hadoop
3.3.6 tarball, so this closes 0 bugs on its own. Kept as the last
branch of the stack so it can land in the next patch.

Correction to the plan: orc-core 1.9.8 does not drop the protobuf
dependency. Its published (dependency-reduced) pom lists neither
protobuf-java nor hive-storage-api, but the classes still reference
com.google.protobuf.GeneratedMessageV3 and
org.apache.hadoop.hive.ql.exec.vector.*, and neither is shaded into the
jar. Both are therefore declared in adls-plugins/pom.xml; without
protobuf-java there, Hadoop's provided protobuf 2.5.0 would win.

Versions:
- orc-mapreduce / orc-core 1.1.0 -> 1.9.8 (+ orc-shims 1.9.8,
  aircompressor 0.27, threeten-extra 1.7.1, kryo-shaded 3.0.3 -> 4.0.2;
  adls-plugins/pom.xml)
- protobuf-java 2.5.0 -> 3.25.5 (direct dependency, adls-plugins)
- hive-storage-api 2.1.0-pre-orc -> 2.8.1 (direct dependency, adls-plugins)
- commons-lang3 3.12.0 (new, from ORC 1.9.8) -> 3.18.0 (root DM;
  azure-blob-store pins 3.14.0 itself and is unchanged)
- jettison, iq80 snappy, objenesis, commons-cli / -codec / -lang 2.x
  (ORC 1.1.0 only) leave the jar; the jettison / snappy DM entries
  from the previous commit become no-ops for adls but stay harmless

Bugs: none (protobuf b/578423011, b/578459565, b/578641178 stay open
through the hbase-shaded / Hadoop exception). Removes the 3 adls
protobuf-java 2.5.0 rows.

Risk: high. ORC writer moves 8 minor versions (default file version
0.12, ZLIB compression unchanged). Needs an ADLS sink run with orc
output on a cluster, and a check that the plugin class loader takes
protobuf-java 3.25.5 from the plugin jar rather than Hadoop's 2.5.0.

Testing: JDK 8. mvn -B clean verify (all 4 modules): BUILD SUCCESS,
Tests run: 3, Failures: 0, Errors: 0 (new StructuredToOrcTransformerTest
writes two records with OrcMapreduceRecordWriter and reads them back
with OrcMapreduceRecordReader, which exercises protobuf and
hive-storage-api; StructuredToTextTransformerTest). mvn dependency:tree
-pl adls-plugins: orc-* 1.9.8, protobuf-java 3.25.5, hive-storage-api
2.8.1, commons-lang3 3.18.0 at compile; no protobuf 2.5.0 outside
provided hadoop-client. unzip -l of the adls jar: protobuf-java-3.25.5,
orc-*-1.9.8, hive-storage-api-2.8.1; no protobuf-java-2.5.0.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates several dependencies across the project, including upgrading Jackson to 2.18.12 using a BOM, upgrading orc-mapreduce to 1.9.8, and managing transitive dependencies to prevent version conflicts. It also introduces a new test class, StructuredToOrcTransformerTest, to verify ORC file writing and reading. Feedback on the new test class suggests using try-with-resources blocks for the Writer and OrcMapreduceRecordReader to prevent potential resource leaks, and explicitly specifying StandardCharsets.UTF_8 when converting strings to bytes to avoid platform-dependent encoding issues.

Comment on lines +62 to +66
OrcMapreduceRecordWriter<OrcStruct> recordWriter = new OrcMapreduceRecordWriter<>(writer);
recordWriter.write(NullWritable.get(), first);
recordWriter.write(NullWritable.get(), transformer.transform(record(null, 2, "b")));
recordWriter.close(null);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The ORC Writer implements Closeable. If an exception is thrown during recordWriter.write(...) before recordWriter.close(null) is reached, the writer will not be closed, leading to a resource leak. Wrapping the Writer in a try-with-resources block ensures it is always closed properly.

Suggested change
OrcMapreduceRecordWriter<OrcStruct> recordWriter = new OrcMapreduceRecordWriter<>(writer);
recordWriter.write(NullWritable.get(), first);
recordWriter.write(NullWritable.get(), transformer.transform(record(null, 2, "b")));
recordWriter.close(null);
try (Writer writer = OrcFile.createWriter(path, OrcFile.writerOptions(conf).setSchema(first.getSchema()))) {
OrcMapreduceRecordWriter<OrcStruct> recordWriter = new OrcMapreduceRecordWriter<>(writer);
recordWriter.write(NullWritable.get(), first);
recordWriter.write(NullWritable.get(), transformer.transform(record(null, 2, "b")));
recordWriter.close(null);
}

Comment on lines +72 to +84
OrcMapreduceRecordReader<OrcStruct> recordReader = new OrcMapreduceRecordReader<>(reader, reader.options());
Assert.assertTrue(recordReader.nextKeyValue());
OrcStruct row = recordReader.getCurrentValue();
Assert.assertEquals("true", row.getFieldValue("bool_field").toString());
Assert.assertEquals("1", row.getFieldValue("int_field").toString());
Assert.assertEquals("a", row.getFieldValue("string_field").toString());
Assert.assertTrue(recordReader.nextKeyValue());
row = recordReader.getCurrentValue();
Assert.assertNull(row.getFieldValue("bool_field"));
Assert.assertEquals("2", row.getFieldValue("int_field").toString());
Assert.assertEquals("b", row.getFieldValue("string_field").toString());
Assert.assertFalse(recordReader.nextKeyValue());
recordReader.close();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The OrcMapreduceRecordReader implements Closeable. If any of the assertions fail, recordReader.close() will not be called, resulting in a resource leak. Using a try-with-resources block ensures that the reader is closed under all circumstances.

Suggested change
OrcMapreduceRecordReader<OrcStruct> recordReader = new OrcMapreduceRecordReader<>(reader, reader.options());
Assert.assertTrue(recordReader.nextKeyValue());
OrcStruct row = recordReader.getCurrentValue();
Assert.assertEquals("true", row.getFieldValue("bool_field").toString());
Assert.assertEquals("1", row.getFieldValue("int_field").toString());
Assert.assertEquals("a", row.getFieldValue("string_field").toString());
Assert.assertTrue(recordReader.nextKeyValue());
row = recordReader.getCurrentValue();
Assert.assertNull(row.getFieldValue("bool_field"));
Assert.assertEquals("2", row.getFieldValue("int_field").toString());
Assert.assertEquals("b", row.getFieldValue("string_field").toString());
Assert.assertFalse(recordReader.nextKeyValue());
recordReader.close();
try (OrcMapreduceRecordReader<OrcStruct> recordReader = new OrcMapreduceRecordReader<>(reader, reader.options())) {
Assert.assertTrue(recordReader.nextKeyValue());
OrcStruct row = recordReader.getCurrentValue();
Assert.assertEquals("true", row.getFieldValue("bool_field").toString());
Assert.assertEquals("1", row.getFieldValue("int_field").toString());
Assert.assertEquals("a", row.getFieldValue("string_field").toString());
Assert.assertTrue(recordReader.nextKeyValue());
row = recordReader.getCurrentValue();
Assert.assertNull(row.getFieldValue("bool_field"));
Assert.assertEquals("2", row.getFieldValue("int_field").toString());
Assert.assertEquals("b", row.getFieldValue("string_field").toString());
Assert.assertFalse(recordReader.nextKeyValue());
}

.set("int_field", i)
.set("long_field", (long) i)
.set("double_field", (double) i)
.set("bytes_field", s.getBytes())

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Using String.getBytes() without specifying an explicit charset relies on the platform's default encoding, which can cause flaky tests or unexpected behavior across different environments. It is safer to specify StandardCharsets.UTF_8 explicitly.

Suggested change
.set("bytes_field", s.getBytes())
.set("bytes_field", s.getBytes(java.nio.charset.StandardCharsets.UTF_8))

@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant