Skip to content

[deferred] Upgrade format-orc to ORC 1.9.8 and drop protobuf 2.5.0 - #2018

Draft
deepika-lalwani123 wants to merge 6 commits into
developfrom
vuln/hp-5-orc-1.9.8
Draft

deepika-lalwani123 wants to merge 6 commits into
developfrom
vuln/hp-5-orc-1.9.8

Conversation

@deepika-lalwani123

@deepika-lalwani123 deepika-lalwani123 commented Nov 17, 2026 •

Copy link
Copy Markdown

[deferred] Upgrade format-orc to ORC 1.9.8 and drop protobuf 2.5.0

Draft: deferred in the plan (ORC upgrade); include in 6.11.2 or the next patch is pending.

Stacked PR: this branch is built on top of #2013, #2014, #2015, #2016, #2017. Against develop the diff includes those commits; please review only the top commit 1973c07ac2. Merge after #2017.

Repo: cdapio/hydrator-plugins
Branch: vuln/hp-5-orc-1.9.8 → base develop (stacked)
Commit: 1973c07
Part of the CVE fixes for the CDAP 6.11.2 patch release (container image scanning of gcr.io/cdapio/cdap).

Image scanning of gcr.io/cdapio/cdap reports protobuf-java 2.5.0 in
format-orc-*.jar (3 CVEs), packaged as a direct dependency for
orc-mapreduce 1.1.0. ORC 1.9.8 (the last Java 8 line; ORC 2.x needs
Java 17) is generated with protobuf 3, so protobuf-java moves to
3.25.5. Deferred in the plan: it closes no bug on its own, because
hbase-shaded-mapreduce and the Hadoop 3.3.6 tarball keep protobuf
2.5.0 (those findings stay under the protobuf 2.5.0 exception).

The published orc-core 1.9.8 pom does not declare hive-storage-api
(orc-core needs org.apache.hadoop.hive.ql.exec.vector.*), and its
commons-lang3 3.12.0 loses to 3.5 from hydrator-common, so format-orc
declares hive-storage-api 2.8.1 (the version ORC 1.9.8 builds with)
and commons-lang3 3.18.0 directly so they are packaged. No Java code
change was needed: the plugin only uses TypeDescription,
CompressionKind, OrcStruct and OrcOutputFormat, which are unchanged.

Versions:

Bugs: none closed alone (see above). Removes the protobuf 2.5.0 copy
from format-orc; data-integrations/azure#171 (azure adls) is the matching
change in the azure repo.

Risk: high. ORC 1.1 -> 1.9 changes the writer (file version 0.12,
new defaults such as zstd support, dictionary and stripe settings);
files are still readable by older readers that support ORC 0.12. ORC
1.9.8 was built against aircompressor 0.27; here it runs with the
managed 2.0.3.

Testing: JDK 8, Checkstyle on (0 violations).
mvn -DcloudBuild clean verify -pl format-orc,core-plugins -am: BUILD
SUCCESS; core-plugins 328 tests (2 skipped), including
ETLTPFSTestRun.testOrc (writes ORC through the TPFSOrc sink and reads
it back) and the parquet / avro TPFS tests. format-orc has no unit
tests of its own. dependency:tree -pl format-orc shows orc 1.9.8,
protobuf-java 3.25.5, hive-storage-api 2.8.1, commons-lang3 3.18.0,
aircompressor 2.0.3 at compile scope; no protobuf 2.5.0 or iq80 snappy.
unzip -l format-orc jar: lib/ has orc-core/-mapreduce/-shims-1.9.8,
protobuf-java-3.25.5, hive-storage-api-2.8.1, commons-lang3-3.18.0,
aircompressor-2.0.3 (no protobuf-java-2.5.0, no snappy-0.2/0.5).
ORC read/write in a 6.11 sandbox (P2) not run.

Image scanning of gcr.io/cdapio/cdap reports jline 3.9.0 and
jetty-http 9.4.51.v20230217 inside database-plugins-*.jar. dependency:tree
shows both come from hadoop-client 3.3.6 (compile scope) ->
hadoop-yarn-client 3.3.6 (jline 3.9.0, websocket-client -> jetty-client
-> jetty-http/-util/-io 9.4.51). The main code only uses
org.apache.hadoop.{conf,io,util,mapreduce,mapred.lib.db} classes, which
come from hadoop-common / hadoop-mapreduce-client-core that CDAP provides
at runtime, so hadoop-client is now provided. This also stops packaging
the rest of the Hadoop client tree (httpclient, commons-io 2.8.0,
commons-compress 1.21, reload4j, kerby, curator, jersey-client, jackson
jaxrs 2.12.7, ...) in the database-plugins jar.

The root-managed avro-mapred 1.11.4 pulled avro-ipc and avro-ipc-jetty,
which brought velocity-engine-core 2.3 (with commons-io 2.8.0) into the
format-avro jar (and through it format-parquet / google-cloud). No plugin
uses Avro IPC, so both are excluded.

Versions:
- hadoop-client 3.3.6: compile -> provided (database-plugins/pom.xml)
- avro-ipc, avro-ipc-jetty 1.11.4: excluded from avro-mapred (root
  dependencyManagement)

Bugs: Plugins side done for jline b/578444907, b/578489244, b/578492108,
b/578644778, b/578663138 and jetty-http b/578515513.

Risk: low. No main-code change; Hadoop classes are supplied by CDAP.
Downstream projects that relied on database-plugins pulling hadoop-client
transitively at compile scope must declare it themselves (none in this
repo).

Testing: JDK 8, Checkstyle on (0 violations).
mvn -DcloudBuild clean verify -pl database-plugins,format-avro,format-parquet -am:
BUILD SUCCESS; hydrator-common, format-common, format-avro,
format-parquet and database-plugins unit tests pass (database-plugins: 35
run, 1 skipped: DBConnectorTest assumes live-DB system properties). dependency:tree
-Dincludes=org.jline,org.eclipse.jetty,org.apache.avro:avro-ipc*,org.apache.velocity
shows jline/jetty only at provided/test scope and no avro-ipc or velocity.
unzip -l database-plugins jar: lib/ now only has hydrator-common, guava,
opentracing-* and common-0.1.4 (no jline, jetty, hadoop-*); format-avro
jar has no avro-ipc / velocity. DB source/sink against real MySQL /
Postgres not run (needs a database).
Image scanning of gcr.io/cdapio/cdap reports CVEs in jars packaged by
hydrator-plugins: log4j-core 2.20.0 / log4j-api 2.21.1 (format-xls,
core-plugins via poi), bcprov-jdk15to18 1.84 (core-plugins), commons-io
2.5 (format-delimited), json-smart 2.2.1 (transform-plugins via
json-path 2.2.0) and iq80 snappy 0.2 (format-orc via orc-core 1.1.0).

The root POM now imports log4j-bom and manages commons-io, json-smart
and iq80 snappy, so modules no longer pin their own versions. json-path
stays on 2.2.0 (2.9 would pull json-smart 2.5.x).

Versions:
- log4j2 (log4j-core, log4j-api) 2.20.0 / 2.21.1 -> 2.25.5 (root
  log4j2.version, log4j-bom import; format-xls log4j-core.version removed)
- bcprov-jdk15to18 1.84 -> 1.85 (core-plugins)
- commons-io 2.5 / 2.15.0 -> 2.16.1 (root commons-io.version + managed
  entry; format-delimited property and core-plugins / format-xls pins
  removed)
- json-smart 2.2.1 -> 2.4.10 (root json-smart.version, managed)
- iq80 snappy 0.2 -> 0.5 (root iq80-snappy.version, managed)

Bugs: Plugins side done for log4j-core b/578431300, b/578491647,
b/578514361, b/578514875. iq80 snappy is rows only (b/578567520 stays
open through the Platform leveldb exception).

Risk: low-medium. Same-major bumps. log4j 2.25.x and commons-io 2.16.1
still run on Java 8. commons-compress (1.26.0 in core-plugins /
format-xls) is not touched in this PR.

Testing: JDK 8, Checkstyle on (0 violations).
mvn -DcloudBuild clean verify -pl core-plugins,format-xls,format-delimited,transform-plugins,format-orc -am:
BUILD SUCCESS; core-plugins 328 tests (2 skipped), transform-plugins 100
(1 skipped), format-xls / format-delimited / format-orc and
dependencies all pass. dependency:tree
-Dincludes=org.apache.logging.log4j,org.bouncycastle,commons-io,net.minidev:json-smart,org.iq80.snappy
shows only log4j 2.25.5, bcprov-jdk15to18 1.85, commons-io 2.16.1,
json-smart 2.4.10 and snappy 0.5 at compile scope (smbj's
bcprov-jdk15on 1.70 is runtime scope and is not embedded). unzip -l of
the built jars: core-plugins lib/ has bcprov-jdk15to18-1.85,
commons-io-2.16.1, log4j-api-2.25.5; format-xls has log4j-core/-api
2.25.5, commons-io-2.16.1; format-delimited commons-io-2.16.1;
transform-plugins json-smart-2.4.10; format-orc snappy-0.5. Excel / ORC
/ SFTP pipelines in a sandbox not run.
Image scanning of gcr.io/cdapio/cdap reports org.json:json 20160212 in
transform-plugins-*.jar (3 CVEs). transform-plugins is the only module
that uses ${json.version}; only XMLToJSON uses org.json (JSONObject,
XML.toJSONObject). Kept separate from the other minor bumps because
newer org.json changes behavior: numbers are parsed as BigDecimal /
BigInteger where needed, and XML.toJSONObject keeps strings with leading
zeros and some other values that the old version coerced.

Versions:
- org.json:json 20160212 -> 20260719 (root json.version)

Bugs: none closed alone in hydrator-plugins (the 3 org.json bugs close
once WR-3b and GC-3b also land).

Risk: medium. The XMLToJSON output can differ for numeric-looking
values (for example leading zeros and very large / high-precision
numbers).

Testing: JDK 8, Checkstyle on (0 violations).
mvn -DcloudBuild clean verify -pl transform-plugins -am: BUILD SUCCESS;
transform-plugins 100 tests pass (1 skipped, ValueMapperTest, as on
the HP-2a run), including XMLToJSONConverterTest (4), XMLParserTest (8),
XMLMultiParserTest (2), JSONParserTest (8), JSONFormatterTest (2).
unzip -l transform-plugins jar: lib/json-20260719.jar (no
json-20160212). A sandbox pipeline with XMLToJSON was not run.
Image scanning of gcr.io/cdapio/cdap reports jackson-databind 2.13.4.2
and jackson-core 2.13.4 / 2.14.3 in format-avro, format-parquet (also
nested in google-cloud through format-avro) and spark-plugins jars. The
root POM now imports com.fasterxml.jackson:jackson-bom so every jackson
module (core, databind, annotations, module-scala for the Spark tests)
resolves to one version; jackson.databind.version now points at the new
jackson.version property, so existing module references keep working.

Versions:
- jackson (core, databind, annotations, modules) 2.13.4.2 / 2.14.3 ->
  2.18.12 (root jackson.version + jackson-bom import)

Bugs: none closed alone; the jackson rows still have a copy shaded in
parquet-jackson 1.12.0 (HP-4) and in Platform / other plugins.

Risk: medium. 2.15 added stream read limits (max string length 20M,
nesting depth 1000, number length 1000) and 2.18 changed some
property introspection. 2.18.x still runs on Java 8. The repo has no
maven-shade-plugin usage, so the shade >= 3.5 rule does not apply.

Testing: JDK 8, Checkstyle on (0 violations).
mvn -DcloudBuild clean verify (whole reactor: 17 modules): BUILD SUCCESS;
all unit tests pass (core-plugins 328 / 2 skipped, transform-plugins
100 / 1 skipped, database-plugins 35 / 1 skipped, spark-plugins,
http-plugins, solrsearch-plugins, format-* modules). dependency:tree
-Dincludes=com.fasterxml.jackson.core,com.fasterxml.jackson.module,com.fasterxml.jackson.dataformat
shows only 2.18.12 at compile scope. unzip -l: format-avro,
format-parquet and spark-plugins lib/ contain jackson-*-2.18.12 only
(format-parquet still has parquet-jackson-1.12.0, fixed in HP-4).
The submodules (google-cloud, kafka, azure, ...) are separate repos and
are not built here (-DcloudBuild).
Image scanning of gcr.io/cdapio/cdap reports parquet-avro 1.12.0 (1
Critical, 1 High) and the jackson-core 2.11.4 / jackson-databind 2.11.4
copy shaded inside parquet-jackson 1.12.0 in format-parquet-*.jar.

parquet-avro and the other parquet modules move to 1.15.2 and
parquet-jackson to 1.17.1 (it only holds the shaded jackson; 1.17.1
has a fixed copy). snappy-java (transform-plugins, root
snappy.version) and aircompressor (parquet-hadoop) are aligned to the
cross-track targets.

Code: format-parquet ships its own copy of
org.apache.parquet.avro.AvroSchemaConverter (to read INT96 as bytes).
It was based on an old parquet release and shadows the 1.15.2 class,
so AvroParquetWriter failed with NoSuchMethodError
(AvroSchemaConverter(ParquetConfiguration)). The copy is re-based on
the parquet-avro 1.15.2 source with the same INT96 change (INT96 ->
Avro bytes unless parquet.avro.readInt96AsFixed is set).
AvroIndexedRecordConverter / NoOpConverter are unchanged (1.12 -> 1.15
upstream changes to that class are formatting only).

parquet-hadoop 1.15.2 declares parquet-jackson at runtime scope, which
the bundle plugin does not embed (Embed-Dependency scope=compile), so
format-parquet declares parquet-jackson explicitly to keep it in the
plugin jar.

Versions:
- parquet-avro, -hadoop, -column, -common, -encoding,
  -format-structures 1.12.0 -> 1.15.2 (root parquet.version, managed)
- parquet-jackson 1.12.0 -> 1.17.1 (root parquet-jackson.version,
  managed; explicit dependency in format-parquet)
- snappy-java 1.1.2 -> 1.1.10.12 (root snappy.version)
- aircompressor: new compile dependency of parquet-hadoop 1.15.2
  (2.0.2) -> pinned to 2.0.3 (root aircompressor.version, managed; also
  replaces 0.21 from Spark's orc-core on the test classpath)

Bugs: closes b/578421211 (Critical) and b/578599613 (High) alone;
Plugins side done for the jackson bugs whose last hydrator copy was
parquet-jackson 1.12.0.

Risk: medium. New aircompressor 2.0.3, zstd-jni 1.5.6-6 and commons-pool 1.6 are now
packaged (parquet-hadoop 1.15 compile deps). The managed parquet
version also applies to the Spark 3.3.2 test classpath (parquet
1.12.x in Spark); all Spark-based unit tests pass. The plugin
classloader loading parquet from the plugin jar on Spark 3.5.8 still
needs the P2 sandbox check. All new jars are Java 8 bytecode (52).

Testing: JDK 8, Checkstyle on (0 violations).
mvn -DcloudBuild clean verify (whole reactor) passed for every module
except format-parquet (NoSuchMethodError above, fixed in this commit);
after the fix: mvn clean verify -pl format-parquet,core-plugins -am
BUILD SUCCESS (format-parquet ParquetInputFormatProviderTest 3/3,
core-plugins 328 / 2 skipped) and mvn clean verify -pl format-parquet
-am BUILD SUCCESS with the explicit parquet-jackson. transform-plugins
(100 / 1 skipped, CompressorTest / DecompressorTest with snappy),
database, http, solr, spark plugins passed in the reactor run.
dependency:tree -Dincludes=org.apache.parquet,io.airlift,org.xerial.snappy
shows only parquet 1.15.2, parquet-jackson 1.17.1, aircompressor 2.0.3
and snappy-java 1.1.10.12 at compile scope. unzip -l format-parquet jar:
lib/ has parquet-*-1.15.2, parquet-jackson-1.17.1, aircompressor-2.0.3,
no parquet-*-1.12.0.
Image scanning of gcr.io/cdapio/cdap reports protobuf-java 2.5.0 in
format-orc-*.jar (3 CVEs), packaged as a direct dependency for
orc-mapreduce 1.1.0. ORC 1.9.8 (the last Java 8 line; ORC 2.x needs
Java 17) is generated with protobuf 3, so protobuf-java moves to
3.25.5. Deferred in the plan: it closes no bug on its own, because
hbase-shaded-mapreduce and the Hadoop 3.3.6 tarball keep protobuf
2.5.0 (b/578423011, b/578459565, b/578641178 stay under the protobuf
2.5.0 exception).

The published orc-core 1.9.8 pom does not declare hive-storage-api
(orc-core needs org.apache.hadoop.hive.ql.exec.vector.*), and its
commons-lang3 3.12.0 loses to 3.5 from hydrator-common, so format-orc
declares hive-storage-api 2.8.1 (the version ORC 1.9.8 builds with)
and commons-lang3 3.18.0 directly so they are packaged. No Java code
change was needed: the plugin only uses TypeDescription,
CompressionKind, OrcStruct and OrcOutputFormat, which are unchanged.

Versions:
- orc-mapreduce / orc-core 1.1.0 -> 1.9.8 (format-orc orc.version);
  adds orc-shims 1.9.8, threeten-extra 1.7.1, annotations 17.0.0
- protobuf-java 2.5.0 -> 3.25.5 (format-orc protobuf.version)
- hive-storage-api: new direct dependency 2.8.1 (format-orc)
- commons-lang3 3.5 -> 3.18.0 in the format-orc jar
- iq80 snappy 0.5 (HP-2a) is no longer used: orc-core 1.9 uses
  aircompressor (2.0.3, managed in HP-4) instead

Bugs: none closed alone (see above). Removes the protobuf 2.5.0 copy
from format-orc; AZ-3 (azure adls) is the matching change in the azure
repo.

Risk: high. ORC 1.1 -> 1.9 changes the writer (file version 0.12,
new defaults such as zstd support, dictionary and stripe settings);
files are still readable by older readers that support ORC 0.12. ORC
1.9.8 was built against aircompressor 0.27; here it runs with the
managed 2.0.3.

Testing: JDK 8, Checkstyle on (0 violations).
mvn -DcloudBuild clean verify -pl format-orc,core-plugins -am: BUILD
SUCCESS; core-plugins 328 tests (2 skipped), including
ETLTPFSTestRun.testOrc (writes ORC through the TPFSOrc sink and reads
it back) and the parquet / avro TPFS tests. format-orc has no unit
tests of its own. dependency:tree -pl format-orc shows orc 1.9.8,
protobuf-java 3.25.5, hive-storage-api 2.8.1, commons-lang3 3.18.0,
aircompressor 2.0.3 at compile scope; no protobuf 2.5.0 or iq80 snappy.
unzip -l format-orc jar: lib/ has orc-core/-mapreduce/-shims-1.9.8,
protobuf-java-3.25.5, hive-storage-api-2.8.1, commons-lang3-3.18.0,
aircompressor-2.0.3 (no protobuf-java-2.5.0, no snappy-0.2/0.5).
ORC read/write in a 6.11 sandbox (P2) not run.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates various project dependencies and refactors AvroSchemaConverter in the format-parquet module to align with parquet-avro 1.15.2, adding support for INT96, UUID, and local timestamp logical types. Feedback on the changes highlights two critical dependency version typos in the root pom.xml: json.version is set to a future release (20260719) and jackson.version is set to a non-existent version (2.18.12). Both should be corrected to valid, stable releases to avoid build and dependency resolution failures.

Comment thread pom.xml
<zookeeper.version>3.4.5</zookeeper.version>
<json-path.version>2.2.0</json-path.version>
<json.version>20160212</json.version>
<json.version>20260719</json.version>

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The version 20260719 for json.version appears to be a typo. org.json:json versions are typically based on actual release dates, and 2026 is in the future. Please verify and correct this to a valid version (e.g., 20231013 or 20240303) to prevent build failures.

Suggested change
<json.version>20260719</json.version>
<json.version>20240303</json.version>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

False positive due to the bot's knowledge cutoff — org.json:json:20260719 is a valid published release on Maven Central (https://repo1.maven.org/maven2/org/json/json/20260719/). Keeping 20260719 (also from stacked PR #2015).

Comment thread pom.xml
<parquet-jackson.version>1.17.1</parquet-jackson.version>
<spark.version>3.3.2</spark.version>
<jackson.databind.version>2.13.4.2</jackson.databind.version>
<jackson.version>2.18.12</jackson.version>

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The version 2.18.12 for jackson.version appears to be a typo. The Jackson 2.18 release line currently only goes up to 2.18.2 (or similar). This will cause dependency resolution to fail. Please correct this to a valid Jackson version.

Suggested change
<jackson.version>2.18.12</jackson.version>
<jackson.version>2.18.2</jackson.version>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

False positive due to the bot's knowledge cutoff — com.fasterxml.jackson:jackson-bom:2.18.12 is a valid published release on Maven Central (https://repo1.maven.org/maven2/com/fasterxml/jackson/jackson-bom/2.18.12/). Keeping 2.18.12 (also from stacked PR #2016).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build Trigger unit test build

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants