Skip to content

Exclude avro-ipc and avro-ipc-jetty from avro-mapred - #1632

Open
deepika-lalwani123 wants to merge 1 commit into
developfrom
vuln/gc-1-exclude-avro-ipc
Open

deepika-lalwani123 wants to merge 1 commit into
developfrom
vuln/gc-1-exclude-avro-ipc

Conversation

@deepika-lalwani123

@deepika-lalwani123 deepika-lalwani123 commented Nov 17, 2026 •

Copy link
Copy Markdown

Exclude avro-ipc and avro-ipc-jetty from avro-mapred
Repo: data-integrations/google-cloud
Branch: vuln/gc-1-exclude-avro-ipc → base develop
Commit: 20c6176
Part of the CVE fixes for the CDAP 6.11.2 patch release (container image scanning of gcr.io/cdapio/cdap).

avro-mapred 1.11.4 pulls in avro-ipc and avro-ipc-jetty, which bring
velocity-engine-core 2.3 (shaded commons-io 2.8.0, CVE-2024-47554) and
org.tukaani:xz into lib/ of google-cloud-0.25.0-SNAPSHOT.jar, as seen in
the image scan of gcr.io/cdapio/cdap
(/opt/cdap/master/artifacts/google-cloud-*.jar/lib/velocity-engine-core-2.3.jar).

The plugins only use org.apache.avro.mapred.AvroKey and
org.apache.avro.mapreduce.* (BigQuery sink AvroOutputFormat /
AvroRecordWriter, Dataplex tests). Avro RPC and tether jobs, the only
users of avro-ipc, are not used. The Avro output codecs the plugins
allow are snappy and deflate (FileSetUtil), so dropping xz is safe.

Versions:

  • org.apache.avro:avro-ipc 1.11.4 -> removed (exclusion on avro-mapred)
  • org.apache.avro:avro-ipc-jetty 1.11.4 -> removed (exclusion on avro-mapred)
  • velocity-engine-core 2.3 and xz 1.9 no longer packaged

The other commons-io 2.8.0 copy (inside format-avro-2.13.0.jar) is
removed by cdapio/hydrator-plugins#2013.

Risk: very low. No plugin code path uses avro-ipc.

Testing: JDK 8, Checkstyle on (0 violations).
mvn clean verify: 647 tests, 0 failures, 5 errors, 8 skipped. The 5
errors are cloud ITs that pick up the Linux workstation's ambient GCP
project and fail with PERMISSION_DENIED (GCSConnectorTest x2,
SpannerConnectorTest x2, BigQueryConnectorTest); they fail the same way on
develop. All other unit tests pass, including AvroRecordWriterTest and
DataPlexOutputFormatProviderTest.
mvn dependency:tree: avro-mapred:1.11.4 has no children; avro-ipc,
avro-ipc-jetty, velocity-engine-core and xz are gone.
unzip -l target/google-cloud-0.25.0-SNAPSHOT.jar: lib/ has no avro-ipc*,
velocity-* or xz-* jar.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the pom.xml file to exclude the avro-ipc and avro-ipc-jetty dependencies from avro-mapred, as these are not required for the plugins and introduce unnecessary transitive dependencies. I have no feedback to provide as there were no review comments.

@vishwasvaidya-cloudsufi vishwasvaidya-cloudsufi added the build Trigger unit test build label Nov 17, 2026
avro-mapred 1.11.4 pulls in avro-ipc and avro-ipc-jetty, which bring
velocity-engine-core 2.3 (shaded commons-io 2.8.0, CVE-2024-47554) and
org.tukaani:xz into lib/ of google-cloud-0.25.0-SNAPSHOT.jar, as seen in
the image scan of gcr.io/cdapio/cdap
(/opt/cdap/master/artifacts/google-cloud-*.jar/lib/velocity-engine-core-2.3.jar).

The plugins only use org.apache.avro.mapred.AvroKey and
org.apache.avro.mapreduce.* (BigQuery sink AvroOutputFormat /
AvroRecordWriter, Dataplex tests). Avro RPC and tether jobs, the only
users of avro-ipc, are not used. The Avro output codecs the plugins
allow are snappy and deflate (FileSetUtil), so dropping xz is safe.

Versions:
- org.apache.avro:avro-ipc 1.11.4 -> removed (exclusion on avro-mapred)
- org.apache.avro:avro-ipc-jetty 1.11.4 -> removed (exclusion on avro-mapred)
- velocity-engine-core 2.3 and xz 1.9 no longer packaged

Bugs: none closed alone. The other commons-io 2.8.0 copy (inside
format-avro-2.13.0.jar) is removed by hydrator-plugins HP-1.

Risk: very low. No plugin code path uses avro-ipc.

Testing: JDK 8, Checkstyle on (0 violations).
mvn clean verify: 647 tests, 0 failures, 5 errors, 8 skipped. The 5
errors are cloud ITs that pick up the Cloudtop's ambient GCP project and
fail with PERMISSION_DENIED (GCSConnectorTest x2, SpannerConnectorTest
x2, BigQueryConnectorTest); they fail the same way on develop. All other
unit tests pass, including AvroRecordWriterTest and
DataPlexOutputFormatProviderTest.
mvn dependency:tree: avro-mapred:1.11.4 has no children; avro-ipc,
avro-ipc-jetty, velocity-engine-core and xz are gone.
unzip -l target/google-cloud-0.25.0-SNAPSHOT.jar: lib/ has no avro-ipc*,
velocity-* or xz-* jar.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build Trigger unit test build

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants