Skip to content

Bump avro from 1.8.2 to 1.11.4 - #157

Open
deepika-lalwani123 wants to merge 3 commits into
developfrom
vuln/ds-3-avro-1.11.4
Open

deepika-lalwani123 wants to merge 3 commits into
developfrom
vuln/ds-3-avro-1.11.4

Conversation

@deepika-lalwani123

@deepika-lalwani123 deepika-lalwani123 commented Nov 17, 2026 •

Copy link
Copy Markdown

Bump avro from 1.8.2 to 1.11.4

Stacked PR: this branch is built on top of #155, #156. Against develop the diff includes those commits; please review only the top commit 7c0ca28318. Merge after #156.

Repo: data-integrations/datastream-delta-plugins
Branch: vuln/ds-3-avro-1.11.4 → base develop (stacked)
Commit: 7c0ca28
Part of the CVE fixes for the CDAP 6.11.2 patch release (container image scanning of gcr.io/cdapio/cdap).

avro 1.8.2 and its commons-compress 1.8.1 are reported by image
scanning of gcr.io/cdapio/cdap in
/opt/cdap/master/artifacts/datastream-delta-plugins-*.jar/lib/
(avro 2 rows, commons-compress 6 rows; avro-1.8.2.jar also carries a
shaded Guava 11 that is reported as Guava rows). avro 1.11.4 is the
fixed release that still runs on Java 8 (1.12 needs Java 11).

avro 1.9 dropped Jackson 1 (codehaus) and Joda from its API; the
plugin only uses the generic reader API (DataFileReader,
GenericDatumReader, GenericRecord, LogicalType) and compiles unchanged.
Two packaging follow-ups:

  • avro 1.11.4 brings commons-compress 1.26.2, which pulls
    commons-lang3 3.14.0 (CVE-2025-48924); commons-lang3 is managed to
    3.18.0.
  • snappy-java and xz are optional dependencies since avro 1.9, but
    1.8.2 packaged them. They are declared directly (snappy-java at the
    cross-track 1.1.10.12, xz 1.9) so Avro files written with the snappy
    or xz codec stay readable.

Versions:

  • avro 1.8.2 -> 1.11.4 (avro.version)
  • commons-compress 1.8.1 -> 1.26.2 (via avro), new commons-io 2.16.1
  • commons-lang3 3.18.0 (managed; was not packaged)
  • snappy-java 1.1.1.3 -> 1.1.10.12, xz 1.5 -> 1.9 (direct)
  • jackson-core-asl / jackson-mapper-asl 1.9.13 and paranamer 2.7 ->
    no longer packaged; avro now uses jackson-databind 2.18.12 (Import jackson-bom 2.18.12 and a protobuf 3.25.5 floor #156 BOM)

Removes the Guava 11 copy shaded in avro-1.8.2.jar (Guava findings
stay open as an exception, out of scope).

Risk: low-medium (treated as MAJOR for testing). Crosses the avro 1.9
API break; no code change was needed.

Testing: JDK 8. mvn clean package: BUILD SUCCESS, Checkstyle 0
violations, 16 unit tests pass, including the 8
DatastreamEventConsumerTest cases that decode Avro records and logical
types. mvn dependency:tree
-Dincludes=org.apache.avro,org.apache.commons,commons-io,org.codehaus.jackson,org.xerial.snappy,org.tukaani:
avro 1.11.4 -> commons-compress 1.26.2 -> commons-io 2.16.1,
commons-lang3 3.18.0; snappy-java 1.1.10.12; xz 1.9; no codehaus
jackson. unzip -l of the plugin jar vs #156 matches the list above.
Not run: Datastream -> BigQuery replication against a real Datastream
stream (needs GCP credentials).

grpc-all 1.49.0 was a compile-scope dependency, so the datastream
plugin jar packaged grpc-netty with unshaded Netty 4.1.77.Final
(netty-buffer, -codec, -codec-http, -codec-http2, -common, -handler,
-resolver, -transport), grpc-okhttp, grpc-rls, grpc-testing and, through
grpc-testing, junit 4.12 and hamcrest-core 1.3. Image scanning of
gcr.io/cdapio/cdap reports these under
/opt/cdap/master/artifacts/datastream-delta-plugins-*.jar/lib/
(40 rows). The exclusion meant to drop grpc-netty had groupId and
artifactId swapped, so it never matched.

Only DatastreamTableRegistryTest uses io.grpc directly (io.grpc.Status,
which is in grpc-api), so grpc-all moves to test scope and the
exclusion is fixed. The runtime transport stays grpc-netty-shaded.

The plugin jar embeds only compile-scope dependencies, and gax-grpc
declares grpc-auth, grpc-grpclb, grpc-services and grpc-xds at runtime
scope; grpc-all used to lift them to compile scope. They are now
declared directly at compile scope (versions from libraries-bom 26.1.1;
grpc-xds, which that BOM does not manage, pinned to the same 1.48.0)
so they stay in the plugin jar. As a side effect grpc-xds's own
compile deps grpc-alts 1.48.0, opencensus-proto 0.2.0 and re2j 1.6 are
now packaged too (they were runtime-only, so missing, before).

delta-api's junit was already excluded in this repo.

Versions:
- grpc-all 1.49.0 compile -> test scope (exclusion fixed to
  io.grpc:grpc-netty)
- io.netty 4.1.77.Final (8 jars), grpc-netty 1.48.0, grpc-okhttp
  1.48.0, grpc-rls 1.49.0, grpc-testing 1.48.0, junit 4.12,
  hamcrest-core 1.3 -> no longer packaged
- grpc-auth, grpc-grpclb, grpc-services, grpc-xds 1.48.0: now direct
  compile deps (unchanged versions)

Bugs: plugin side for the unshaded Netty 4.1.77 rows (plan DS-1:
38 bugs touched, all also need other copies removed); junit
b/578455300 closes once delta DL-1 and database-delta DBD-1 also land.

Risk: low. No code change. The packaged gRPC set is the same except
the removed test/alternate-transport jars and the three added xds
deps.

Testing: JDK 8. mvn clean package: BUILD SUCCESS, Checkstyle 0
violations, 16 unit tests pass (DatastreamTableRegistryTest,
DatastreamEventConsumerTest, DatastreamConfigTest,
DatastreamAssessmentTest, UtilsTest; the credential-gated tests are
skipped by their assumptions as on develop). mvn dependency:tree
-Dincludes=io.netty,junit,org.hamcrest,io.grpc: no io.netty artifact at
any scope; junit/hamcrest and grpc-okhttp/rls/testing only under
grpc-all at test scope. unzip -l of the plugin jar vs develop: removed
lib/netty-*-4.1.77.Final.jar (8), grpc-all, grpc-netty, grpc-okhttp,
grpc-rls, grpc-testing, junit-4.12, hamcrest-core-1.3; added grpc-alts,
opencensus-proto, re2j; lib/grpc-netty-shaded-1.48.0.jar kept.
libraries-bom 26.1.1 brings jackson-core 2.13.3 and protobuf-java /
protobuf-java-util 3.21.4, which image scanning of gcr.io/cdapio/cdap
reports in /opt/cdap/master/artifacts/datastream-delta-plugins-*.jar/lib/
(jackson-core 3 rows, protobuf-java 4 rows).

Import com.fasterxml.jackson:jackson-bom 2.18.12 and
com.google.protobuf:protobuf-bom 3.25.5 before libraries-bom so they
take precedence. Nothing that libraries-bom manages at a higher
version is lowered: its protobuf (3.21.4) and jackson (2.13.3) are
both older than the new floor. Upgrading libraries-bom itself is out of
scope (WR-5-style change).

Versions:
- jackson-core 2.13.3 -> 2.18.12 (jackson-bom import, jackson.version)
- protobuf-java, protobuf-java-util 3.21.4 -> 3.25.5 (protobuf-bom
  import, protobuf.version)

Bugs: Plugins side done for the datastream jackson-core and protobuf
rows (plan DS-2: 7 bugs touched, none closed by this PR alone).

Risk: low-medium. protobuf 3.25 runtime runs code generated by 3.21
(the datastream v1 protos); jackson 2.15+ adds stream read limits,
which only matter for very large JSON documents from GCS.

Testing: JDK 8. mvn clean package: BUILD SUCCESS, Checkstyle 0
violations, 16 unit tests pass. mvn dependency:tree
-Dincludes=com.fasterxml.jackson.core,com.google.protobuf: only
jackson-core 2.18.12 and protobuf-java / protobuf-java-util 3.25.5.
unzip -l of the plugin jar vs DS-1: only those three jars changed
version.
avro 1.8.2 and its commons-compress 1.8.1 are reported by image
scanning of gcr.io/cdapio/cdap in
/opt/cdap/master/artifacts/datastream-delta-plugins-*.jar/lib/
(avro 2 rows, commons-compress 6 rows; avro-1.8.2.jar also carries a
shaded Guava 11 that is reported as Guava rows). avro 1.11.4 is the
fixed release that still runs on Java 8 (1.12 needs Java 11).

avro 1.9 dropped Jackson 1 (codehaus) and Joda from its API; the
plugin only uses the generic reader API (DataFileReader,
GenericDatumReader, GenericRecord, LogicalType) and compiles unchanged.
Two packaging follow-ups:
- avro 1.11.4 brings commons-compress 1.26.2, which pulls
  commons-lang3 3.14.0 (CVE-2025-48924); commons-lang3 is managed to
  3.18.0.
- snappy-java and xz are optional dependencies since avro 1.9, but
  1.8.2 packaged them. They are declared directly (snappy-java at the
  cross-track 1.1.10.12, xz 1.9) so Avro files written with the snappy
  or xz codec stay readable.

Versions:
- avro 1.8.2 -> 1.11.4 (avro.version)
- commons-compress 1.8.1 -> 1.26.2 (via avro), new commons-io 2.16.1
- commons-lang3 3.18.0 (managed; was not packaged)
- snappy-java 1.1.1.3 -> 1.1.10.12, xz 1.5 -> 1.9 (direct)
- jackson-core-asl / jackson-mapper-asl 1.9.13 and paranamer 2.7 ->
  no longer packaged; avro now uses jackson-databind 2.18.12 (DS-2 BOM)

Bugs: b/578435773. Also removes the Guava 11 copy shaded in
avro-1.8.2.jar (Guava bugs stay open as an exception, out of scope).

Risk: low-medium (treated as MAJOR for testing). Crosses the avro 1.9
API break; no code change was needed.

Testing: JDK 8. mvn clean package: BUILD SUCCESS, Checkstyle 0
violations, 16 unit tests pass, including the 8
DatastreamEventConsumerTest cases that decode Avro records and logical
types. mvn dependency:tree
-Dincludes=org.apache.avro,org.apache.commons,commons-io,org.codehaus.jackson,org.xerial.snappy,org.tukaani:
avro 1.11.4 -> commons-compress 1.26.2 -> commons-io 2.16.1,
commons-lang3 3.18.0; snappy-java 1.1.10.12; xz 1.9; no codehaus
jackson. unzip -l of the plugin jar vs DS-2 matches the list above.
Not run: Datastream -> BigQuery replication against a real Datastream
stream (needs GCP credentials; covered by wave smoke P4).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant