Repository navigation
Bump avro from 1.8.2 to 1.11.4 - #157
Open
deepika-lalwani123 wants to merge 3 commits into
Open
deepika-lalwani123 wants to merge 3 commits into
deepika-lalwani123 wants to merge 3 commits into
Conversation
grpc-all 1.49.0 was a compile-scope dependency, so the datastream plugin jar packaged grpc-netty with unshaded Netty 4.1.77.Final (netty-buffer, -codec, -codec-http, -codec-http2, -common, -handler, -resolver, -transport), grpc-okhttp, grpc-rls, grpc-testing and, through grpc-testing, junit 4.12 and hamcrest-core 1.3. Image scanning of gcr.io/cdapio/cdap reports these under /opt/cdap/master/artifacts/datastream-delta-plugins-*.jar/lib/ (40 rows). The exclusion meant to drop grpc-netty had groupId and artifactId swapped, so it never matched. Only DatastreamTableRegistryTest uses io.grpc directly (io.grpc.Status, which is in grpc-api), so grpc-all moves to test scope and the exclusion is fixed. The runtime transport stays grpc-netty-shaded. The plugin jar embeds only compile-scope dependencies, and gax-grpc declares grpc-auth, grpc-grpclb, grpc-services and grpc-xds at runtime scope; grpc-all used to lift them to compile scope. They are now declared directly at compile scope (versions from libraries-bom 26.1.1; grpc-xds, which that BOM does not manage, pinned to the same 1.48.0) so they stay in the plugin jar. As a side effect grpc-xds's own compile deps grpc-alts 1.48.0, opencensus-proto 0.2.0 and re2j 1.6 are now packaged too (they were runtime-only, so missing, before). delta-api's junit was already excluded in this repo. Versions: - grpc-all 1.49.0 compile -> test scope (exclusion fixed to io.grpc:grpc-netty) - io.netty 4.1.77.Final (8 jars), grpc-netty 1.48.0, grpc-okhttp 1.48.0, grpc-rls 1.49.0, grpc-testing 1.48.0, junit 4.12, hamcrest-core 1.3 -> no longer packaged - grpc-auth, grpc-grpclb, grpc-services, grpc-xds 1.48.0: now direct compile deps (unchanged versions) Bugs: plugin side for the unshaded Netty 4.1.77 rows (plan DS-1: 38 bugs touched, all also need other copies removed); junit b/578455300 closes once delta DL-1 and database-delta DBD-1 also land. Risk: low. No code change. The packaged gRPC set is the same except the removed test/alternate-transport jars and the three added xds deps. Testing: JDK 8. mvn clean package: BUILD SUCCESS, Checkstyle 0 violations, 16 unit tests pass (DatastreamTableRegistryTest, DatastreamEventConsumerTest, DatastreamConfigTest, DatastreamAssessmentTest, UtilsTest; the credential-gated tests are skipped by their assumptions as on develop). mvn dependency:tree -Dincludes=io.netty,junit,org.hamcrest,io.grpc: no io.netty artifact at any scope; junit/hamcrest and grpc-okhttp/rls/testing only under grpc-all at test scope. unzip -l of the plugin jar vs develop: removed lib/netty-*-4.1.77.Final.jar (8), grpc-all, grpc-netty, grpc-okhttp, grpc-rls, grpc-testing, junit-4.12, hamcrest-core-1.3; added grpc-alts, opencensus-proto, re2j; lib/grpc-netty-shaded-1.48.0.jar kept.
libraries-bom 26.1.1 brings jackson-core 2.13.3 and protobuf-java / protobuf-java-util 3.21.4, which image scanning of gcr.io/cdapio/cdap reports in /opt/cdap/master/artifacts/datastream-delta-plugins-*.jar/lib/ (jackson-core 3 rows, protobuf-java 4 rows). Import com.fasterxml.jackson:jackson-bom 2.18.12 and com.google.protobuf:protobuf-bom 3.25.5 before libraries-bom so they take precedence. Nothing that libraries-bom manages at a higher version is lowered: its protobuf (3.21.4) and jackson (2.13.3) are both older than the new floor. Upgrading libraries-bom itself is out of scope (WR-5-style change). Versions: - jackson-core 2.13.3 -> 2.18.12 (jackson-bom import, jackson.version) - protobuf-java, protobuf-java-util 3.21.4 -> 3.25.5 (protobuf-bom import, protobuf.version) Bugs: Plugins side done for the datastream jackson-core and protobuf rows (plan DS-2: 7 bugs touched, none closed by this PR alone). Risk: low-medium. protobuf 3.25 runtime runs code generated by 3.21 (the datastream v1 protos); jackson 2.15+ adds stream read limits, which only matter for very large JSON documents from GCS. Testing: JDK 8. mvn clean package: BUILD SUCCESS, Checkstyle 0 violations, 16 unit tests pass. mvn dependency:tree -Dincludes=com.fasterxml.jackson.core,com.google.protobuf: only jackson-core 2.18.12 and protobuf-java / protobuf-java-util 3.25.5. unzip -l of the plugin jar vs DS-1: only those three jars changed version.
avro 1.8.2 and its commons-compress 1.8.1 are reported by image scanning of gcr.io/cdapio/cdap in /opt/cdap/master/artifacts/datastream-delta-plugins-*.jar/lib/ (avro 2 rows, commons-compress 6 rows; avro-1.8.2.jar also carries a shaded Guava 11 that is reported as Guava rows). avro 1.11.4 is the fixed release that still runs on Java 8 (1.12 needs Java 11). avro 1.9 dropped Jackson 1 (codehaus) and Joda from its API; the plugin only uses the generic reader API (DataFileReader, GenericDatumReader, GenericRecord, LogicalType) and compiles unchanged. Two packaging follow-ups: - avro 1.11.4 brings commons-compress 1.26.2, which pulls commons-lang3 3.14.0 (CVE-2025-48924); commons-lang3 is managed to 3.18.0. - snappy-java and xz are optional dependencies since avro 1.9, but 1.8.2 packaged them. They are declared directly (snappy-java at the cross-track 1.1.10.12, xz 1.9) so Avro files written with the snappy or xz codec stay readable. Versions: - avro 1.8.2 -> 1.11.4 (avro.version) - commons-compress 1.8.1 -> 1.26.2 (via avro), new commons-io 2.16.1 - commons-lang3 3.18.0 (managed; was not packaged) - snappy-java 1.1.1.3 -> 1.1.10.12, xz 1.5 -> 1.9 (direct) - jackson-core-asl / jackson-mapper-asl 1.9.13 and paranamer 2.7 -> no longer packaged; avro now uses jackson-databind 2.18.12 (DS-2 BOM) Bugs: b/578435773. Also removes the Guava 11 copy shaded in avro-1.8.2.jar (Guava bugs stay open as an exception, out of scope). Risk: low-medium (treated as MAJOR for testing). Crosses the avro 1.9 API break; no code change was needed. Testing: JDK 8. mvn clean package: BUILD SUCCESS, Checkstyle 0 violations, 16 unit tests pass, including the 8 DatastreamEventConsumerTest cases that decode Avro records and logical types. mvn dependency:tree -Dincludes=org.apache.avro,org.apache.commons,commons-io,org.codehaus.jackson,org.xerial.snappy,org.tukaani: avro 1.11.4 -> commons-compress 1.26.2 -> commons-io 2.16.1, commons-lang3 3.18.0; snappy-java 1.1.10.12; xz 1.9; no codehaus jackson. unzip -l of the plugin jar vs DS-2 matches the list above. Not run: Datastream -> BigQuery replication against a real Datastream stream (needs GCP credentials; covered by wave smoke P4).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bump avro from 1.8.2 to 1.11.4
Repo: data-integrations/datastream-delta-plugins
Branch: vuln/ds-3-avro-1.11.4 → base develop (stacked)
Commit: 7c0ca28
Part of the CVE fixes for the CDAP 6.11.2 patch release (container image scanning of gcr.io/cdapio/cdap).
avro 1.8.2 and its commons-compress 1.8.1 are reported by image
scanning of gcr.io/cdapio/cdap in
/opt/cdap/master/artifacts/datastream-delta-plugins-*.jar/lib/
(avro 2 rows, commons-compress 6 rows; avro-1.8.2.jar also carries a
shaded Guava 11 that is reported as Guava rows). avro 1.11.4 is the
fixed release that still runs on Java 8 (1.12 needs Java 11).
avro 1.9 dropped Jackson 1 (codehaus) and Joda from its API; the
plugin only uses the generic reader API (DataFileReader,
GenericDatumReader, GenericRecord, LogicalType) and compiles unchanged.
Two packaging follow-ups:
commons-lang3 3.14.0 (CVE-2025-48924); commons-lang3 is managed to
3.18.0.
1.8.2 packaged them. They are declared directly (snappy-java at the
cross-track 1.1.10.12, xz 1.9) so Avro files written with the snappy
or xz codec stay readable.
Versions:
no longer packaged; avro now uses jackson-databind 2.18.12 (Import jackson-bom 2.18.12 and a protobuf 3.25.5 floor #156 BOM)
Removes the Guava 11 copy shaded in avro-1.8.2.jar (Guava findings
stay open as an exception, out of scope).
Risk: low-medium (treated as MAJOR for testing). Crosses the avro 1.9
API break; no code change was needed.
Testing: JDK 8. mvn clean package: BUILD SUCCESS, Checkstyle 0
violations, 16 unit tests pass, including the 8
DatastreamEventConsumerTest cases that decode Avro records and logical
types. mvn dependency:tree
-Dincludes=org.apache.avro,org.apache.commons,commons-io,org.codehaus.jackson,org.xerial.snappy,org.tukaani:
avro 1.11.4 -> commons-compress 1.26.2 -> commons-io 2.16.1,
commons-lang3 3.18.0; snappy-java 1.1.10.12; xz 1.9; no codehaus
jackson. unzip -l of the plugin jar vs #156 matches the list above.
Not run: Datastream -> BigQuery replication against a real Datastream
stream (needs GCP credentials).