Repository navigation
Move Google Cloud clients to libraries-bom with grpc-bom 1.85.1 - #1074
Open
deepika-lalwani123 wants to merge 6 commits into
Open
deepika-lalwani123 wants to merge 6 commits into
deepika-lalwani123 wants to merge 6 commits into
Conversation
Image scanning of gcr.io/cdapio/cdap reports logback-classic and logback-core 1.2.11 inside the wrangler-service (DataPrep system app) jar. Wrangler does not declare logback and no wrangler code imports ch.qos.logback; it came in only as a compile dependency of io.cdap.cdap:cdap-etl-core (confirmed with dependency:tree, the only path). CDAP provides the logging backend at runtime, so the jar does not need its own copy. Versions: - logback-classic / logback-core 1.2.11 -> not packaged (exclusion of ch.qos.logback:* on cdap-etl-core in wrangler-service) - logback-classic 1.2.13 added at test scope only (logback.version in wrangler-service), declared before hadoop-mapreduce-client-core so it stays the SLF4J binding for unit tests; without it slf4j-reload4j binds and FileTypeDetectorTest fails with ExceptionInInitializerError Bugs: Plugins side done for b/578430422, b/578585447, b/578633383, b/578635398, b/578643563, b/578657694, b/578688736 (the logback CVEs fixed only in 1.3.x/1.5.x stay blocked by the Platform logback exception). Risk: low. Only the packaged jar list changes; logging in the app goes through slf4j to the CDAP-provided backend as for other plugins. Testing: JDK 8, Checkstyle on (0 violations). mvn clean verify -pl wrangler-service: Tests run: 17, Failures: 0, Errors: 0, Skipped: 6 (same as develop). dependency:tree -Dincludes=ch.qos.logback on wrangler-service shows only logback-classic/core 1.2.13 at test scope. unzip -l of wrangler-service-4.12.0-SNAPSHOT.jar: lib/ no longer has logback-classic-1.2.11.jar or logback-core-1.2.11.jar; no other lib/ change. DataPrep sandbox logging check (P3) not run here.
Patch-level bumps for CVEs that image scanning of gcr.io/cdapio/cdap reports in the wrangler-service and wrangler-transform jars (httpclient 4.5.3, avro 1.11.3). httpcore was not pinned: it resolved to 4.4.13 (via google-cloud-spanner) in wrangler-service and 4.4.6 in wrangler-core / wrangler-transform; it is now managed at 4.4.16 to match httpclient 4.5.14. netty is used only at test scope in wrangler (netty-all, netty-http); the bump keeps the netty-bom import aligned with the cross-repo target and changes no packaged jar. Versions: - httpclient 4.5.3 -> 4.5.14 (httpclient.version, now also in root dependencyManagement) - httpcore 4.4.13 / 4.4.6 -> 4.4.16 (new httpcore.version, root dependencyManagement) - avro 1.11.3 -> 1.11.4 (avro.version); avro 1.11.4 also moves its jackson-core/databind/annotations from 2.14.2 to 2.14.3 - netty-bom 4.1.118.Final -> 4.1.140.Final (netty.version) Bugs: none close alone. Plugins (wrangler) side done for httpclient b/578423875 (also needs google-cloud GC-4 and Platform); the avro bugs also need KP-3 / DS-3. Risk: low. Same-line patch releases. avro 1.11.4 rejects untrusted java-class schema properties (CVE-2024-47561 fix); wrangler does not use avro reflect/specific classes from schema props. Testing: JDK 8, Checkstyle on (0 violations). mvn clean install (all modules): BUILD SUCCESS. Tests: API 31/0 fail, Proto 3/0, Core 456/0 (2 skipped), Storage 37/0, Service 17/0 (6 skipped, same as develop), Transform 2/0. dependency:tree -Dincludes=org.apache.httpcomponents,org.apache.avro on wrangler-core/-service/-transform: only httpclient 4.5.14, httpcore 4.4.16, avro 1.11.4 at compile scope. unzip -l: wrangler-service and wrangler-transform lib/ now have httpclient-4.5.14, httpcore-4.4.16, avro-1.11.4 (old jars gone).
Minor-version bumps for CVEs that image scanning of gcr.io/cdapio/cdap reports in the wrangler-service and wrangler-transform jars. Where the library is transitive it is pinned in the root dependencyManagement. Real sources (checked with dependency:tree; the plan guessed some): - commons-beanutils 1.9.3 comes from org.everit.json.schema 1.11.1 (wrangler-core), not from commons-validator - commons-compress 1.25.0 comes from poi-ooxml 5.2.5 - json-smart 2.2.1 comes from json-path 2.2.0 - log4j-api 2.17.2 comes from poi 5.2.5 (log4j-bom import) Versions: - commons-compress 1.25.0 -> 1.26.2 (new commons-compress.version, DM) - commons-lang3 3.5 -> 3.18.0 (commons-lang3.version; needed by compress 1.26 and itself a CVE fix) - commons-beanutils 1.9.3 -> 1.11.0 (new commons-beanutils.version, DM) - bcprov-jdk18on 1.78.1 -> 1.85 (bouncycastle.version) - log4j 2.17.2 -> 2.25.5 (log4j.version, log4j-bom import) - json-smart 2.2.1 -> 2.4.10 (new json-smart.version, DM); this also moves accessors-smart 1.1 -> 2.4.9 - protobuf-java 3.11.3 -> 3.25.5 (protobuf.version); protobuf-java-util is now managed at the same version (was 3.11.4 via the Google Cloud clients in wrangler-service) Bugs: closes b/578429610, b/578444267, b/578489829, b/578574083, b/583757186 (bcprov) and b/578581625 (json-smart). Plugins side done for b/578439719, b/578443850, b/578572706, b/578584043, b/578584206, b/578645797. Risk: medium. commons-lang3 jumps 3.5 -> 3.18 (source compatible; deprecations only). protobuf 3.25 runtime runs the older 3.x gencode of the Google Cloud 1.x clients. accessors-smart 2.4.9 declares asm 9.3 but org.ow2.asm:asm 5.0.4 (via kryo/reflectasm, nearer) still wins; checked at runtime that json-smart bean mapping (JSONValue.parse(json, Bean.class)), JSONValue.toJSONString and JsonPath.read work on the wrangler-core runtime classpath with asm 5.0.4. Testing: JDK 8, Checkstyle on (0 violations). mvn clean install (all modules): BUILD SUCCESS. Tests: API 31/0 fail, Proto 3/0, Core 456/0 (2 skipped), Storage 37/0, Service 17/0 (6 skipped, same as develop), Transform 2/0. dependency:tree on wrangler-core/-service/-transform (compile scope): commons-compress 1.26.2, commons-lang3 3.18.0, commons-beanutils 1.11.0, bcprov-jdk18on 1.85, log4j-api 2.25.5, json-smart 2.4.10, protobuf-java(-util) 3.25.5; no old version left. unzip -l lib/ of wrangler-service and wrangler-transform shows the same new jars and none of the old ones.
Image scanning of gcr.io/cdapio/cdap reports org.json:json 20180813 in the wrangler-service and wrangler-transform jars. It is a direct dependency of wrangler-core (directives use JSONObject, JSONTokener, JSONException and XML: parse-xml-to-json, JsParser, json conformer, regex filter, fill-null-or-empty). Kept separate from the other minor bumps because newer org.json changes parsing behavior. Versions: - org.json:json 20180813 -> 20260719 (new org.json.version; wrangler-core and root dependencyManagement, which also aligns the copy that org.everit.json.schema asks for) Behavior change: XML.toJSONObject now returns a BigDecimal for numbers with an exponent, so parse-xml-to-json renders 303246306303E8 as 3.03246306303E+19 instead of 3.03246306303E19 (same value). The XmlToJsonTest expectation is updated for that; with keepStrings=true the output is unchanged. Bugs: wrangler copy done for b/578504385, b/578575239, b/584531579 (they close when hydrator-plugins HP-2b and google-cloud GC-3b land too). Risk: medium. Newer org.json parses numbers more exactly (BigDecimal / BigInteger) and is stricter about malformed input; all wrangler-core directive tests pass. Testing: JDK 8, Checkstyle on (0 violations). mvn clean install (all modules): BUILD SUCCESS. Tests: API 31/0 fail, Proto 3/0, Core 456/0 (2 skipped), Storage 37/0, Service 17/0 (6 skipped), Transform 2/0. JSON/XML directive tests: XmlToJsonTest 1, JsonConformerTest 5, RowRegexFilterTest 4, FillNullOrEmptyTest 5, WriteAsJsonMapTest 1, WriteAsJsonObjectTest 1, JsonFunctionsTest 2, SchemaConverterTest 5, JsParserTest, JsPathTest: all pass. XmlToJsonTest failed before the expectation update (E19 vs E+19 only). dependency:tree -Dincludes=org.json: only json 20260719 at compile scope in wrangler-core/-service/-transform; unzip -l lib/ of both plugin jars shows json-20260719.jar and no json-20180813.jar.
Image scanning of gcr.io/cdapio/cdap reports jackson-core 2.7.9 and jackson-core/databind 2.14.2 in the wrangler-service and wrangler-transform jars. Wrangler declares no Jackson dependency and has no com.fasterxml imports; every copy is transitive (checked with dependency:tree): - jackson-core 2.7.9: com.microsoft.azure:azure-data-lake-store-sdk 2.3.3 (wrangler-service), not from aws-java-sdk-s3 as assumed; it was packaged next to databind 2.14.x, i.e. a mismatched pair - jackson-core/databind/annotations 2.14.x: org.apache.avro:avro (wrangler-core) - jackson-dataformat-cbor 2.12.6: aws-java-sdk-core (wrangler-service) Importing the BOM in the root dependencyManagement aligns all of them. The jackson-databind exclusion on aws-java-sdk-s3 in wrangler-service stays. Versions: - jackson-core 2.7.9 / 2.14.3 -> 2.18.12 - jackson-databind, jackson-annotations 2.14.3 -> 2.18.12 - jackson-dataformat-cbor 2.12.6 -> 2.18.12 (new jackson.version, com.fasterxml.jackson:jackson-bom import) Bugs: none close here alone; this finishes the wrangler copies of the Jackson rows (they also need hydrator-plugins HP-4, google-cloud GC-4 and Platform). Risk: medium. Jackson 2.18 still targets Java 8 and is API compatible for the users here (avro, google-http-client-jackson2, AWS SDK, ADLS SDK); 2.18 adds default stream read constraints (max string/number length) that only affect very large documents. Testing: JDK 8, Checkstyle on (0 violations). mvn clean install (all modules): BUILD SUCCESS. Tests: API 31/0 fail, Proto 3/0, Core 456/0 (2 skipped), Storage 37/0, Service 17/0 (6 skipped), Transform 2/0. dependency:tree -Dincludes=com.fasterxml.jackson.*: only 2.18.12 at compile scope in wrangler-core/-service/-transform. unzip -l lib/: wrangler-service has jackson-annotations/core/databind/dataformat-cbor 2.18.12 (jackson-core-2.7.9, databind 2.14.x and cbor 2.12.6 gone); wrangler-transform has annotations/core/databind 2.18.12 only. S3/ADLS/GCS connection checks in a sandbox (P3) not run here.
Image scanning of gcr.io/cdapio/cdap reports 42 rows in grpc-netty-shaded-1.31.1.jar (shaded Netty 4.1.48: codec-http, -http2, handler, ...) inside the wrangler-service (DataPrep) jar. It came from google-cloud-spanner 1.61.0 and the other pinned 1.x Google Cloud clients (bigquery 1.110.1, storage 1.106.0, core 1.93.4). They are now managed by com.google.cloud:libraries-bom, with io.grpc:grpc-bom 1.85.1 imported first so grpc (and its shaded Netty 4.2.17) is newer than what the BOM ships (grpc 1.83.0). Both BOMs are imported in wrangler-service only, the one module that uses Google Cloud clients. In the root pom libraries-bom also manages Guava, which moved the Guava 13 that the CDAP unit-test framework needs to 33.x and broke 12 wrangler-storage tests (IncompatibleClassChangeError). Versions (wrangler-service): - google-cloud-spanner 1.61.0 -> 6.127.0, google-cloud-bigquery 1.110.1 -> 2.76.0, google-cloud-storage 1.106.0 -> 2.78.0, google-cloud-core 1.93.4 -> 2.80.0, google-oauth-client 1.34.1 -> 1.39.0 (libraries-bom 26.93.0; google.cloud.*.version, google.storage.version, google.oauth.client.version removed) - grpc 1.31.1 -> 1.85.1 (new grpc.version, grpc-bom), incl. grpc-netty-shaded (shaded Netty 4.1.48 -> 4.2.17.Final) - protobuf-java(-util) 3.25.5 -> 4.33.6 in wrangler-service only (new protobuf4.version; the clients are generated for protobuf 4; the other modules stay on 3.25.5) - guava 31.0.1-jre -> 33.6.0-jre in wrangler-service only (from the BOM; gax 2.90 / grpc 1.85 need Guava 33) - gax 1.54 -> 2.90, google-auth-library 0.20 -> 1.57, google-http-client 1.34 -> 2.2.0, gson 2.10.1 -> 2.14.0 (BOM) - New in the jar: arrow 17 (+ netty-buffer/netty-common 4.1.140.Final, via google-cloud-bigquery), bigquerystorage, monitoring, opentelemetry, grpc-xds and others; lib/ grows from 172 to 215 jars No source change was needed: wrangler-service main and test code compile against Spanner 6.x, BigQuery 2.x and Storage 2.x as is. Bugs: Plugins side done for the Netty bugs b/578422530, b/578425552, b/578426660, b/578427732, b/578435805, b/578435812, b/578442798, b/578452844, b/578498484, b/578501458, b/578511634, b/578512623, b/578520387, b/578523920, b/578524138, b/578525371, b/578559603, b/578566449, b/578568144, b/578571769, b/578631548, b/578636943, b/578637410, b/578646421, b/578646479, b/578655606, b/578663185. Risk: high. Major-version jumps of the GCS, BigQuery and Spanner clients (new transports, retries, defaults) in the DataPrep connections; Guava 33 and protobuf 4 in the app classloader. All jars in lib/ are Java 8 bytecode (checked: no class file above major 52). Needs the GCS / BigQuery / Spanner connection checks (browse, sampling, proxy) in a 6.11 sandbox (P3) before release. Testing: JDK 8, Checkstyle on (0 violations). mvn clean install (all modules): BUILD SUCCESS. Tests: API 31/0 fail, Proto 3/0, Core 456/0 (2 skipped), Storage 37/0, Service 17/0 (6 skipped, same as develop), Transform 2/0. dependency:tree -Dincludes=io.grpc on wrangler-service: every io.grpc artifact is 1.85.1 (grpc-netty-shaded 1.85.1); no 1.31.1 left. unzip -l lib/: grpc-netty-shaded-1.85.1.jar (its io.netty.versions.properties says 4.2.17.Final), no grpc 1.31.1 or Netty 4.1.48 jar. Offline smoke run on the wrangler-service runtime classpath: StorageOptions/BigQueryOptions/SpannerOptions with NoCredentials build their services, Spanner admin clients, Statement binding, Type.Code, Date and Timestamp work (no linkage errors). Live GCS / BigQuery / Spanner calls not run (no credentials here).
There was a problem hiding this comment.
Code Review
This pull request updates and aligns multiple dependency versions across the project's POM files (including pom.xml, wrangler-core/pom.xml, and wrangler-service/pom.xml) to address CVEs and ensure compatibility. Key changes include upgrading Bouncy Castle, Netty, Log4j, Protobuf, and Jackson, as well as introducing BOMs for Google Cloud libraries and gRPC. A unit test in XmlToJsonTest.java was also updated to match the exponent formatting of the newer org.json library. All review comments were filtered out as they pertained to version numbers, so there is no additional feedback to provide.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Move Google Cloud clients to libraries-bom with grpc-bom 1.85.1
Repo: data-integrations/wrangler
Branch: vuln/wr-5-libraries-bom-grpc → base develop (stacked)
Commit: 818bbf6
Part of the CVE fixes for the CDAP 6.11.2 patch release (container image scanning of gcr.io/cdapio/cdap).
Image scanning of gcr.io/cdapio/cdap reports 42 rows in
grpc-netty-shaded-1.31.1.jar (shaded Netty 4.1.48: codec-http, -http2,
handler, ...) inside the wrangler-service (DataPrep) jar. It came from
google-cloud-spanner 1.61.0 and the other pinned 1.x Google Cloud
clients (bigquery 1.110.1, storage 1.106.0, core 1.93.4). They are now
managed by com.google.cloud:libraries-bom, with io.grpc:grpc-bom
1.85.1 imported first so grpc (and its shaded Netty 4.2.17) is newer
than what the BOM ships (grpc 1.83.0).
Both BOMs are imported in wrangler-service only, the one module that
uses Google Cloud clients. In the root pom libraries-bom also manages
Guava, which moved the Guava 13 that the CDAP unit-test framework
needs to 33.x and broke 12 wrangler-storage tests
(IncompatibleClassChangeError).
Versions (wrangler-service):
1.110.1 -> 2.76.0, google-cloud-storage 1.106.0 -> 2.78.0,
google-cloud-core 1.93.4 -> 2.80.0, google-oauth-client 1.34.1 ->
1.39.0 (libraries-bom 26.93.0; google.cloud.*.version,
google.storage.version, google.oauth.client.version removed)
grpc-netty-shaded (shaded Netty 4.1.48 -> 4.2.17.Final)
protobuf4.version; the clients are generated for protobuf 4; the
other modules stay on 3.25.5)
BOM; gax 2.90 / grpc 1.85 need Guava 33)
google-http-client 1.34 -> 2.2.0, gson 2.10.1 -> 2.14.0 (BOM)
4.1.140.Final, via google-cloud-bigquery), bigquerystorage,
monitoring, opentelemetry, grpc-xds and others; lib/ grows from 172
to 215 jars
No source change was needed: wrangler-service main and test code
compile against Spanner 6.x, BigQuery 2.x and Storage 2.x as is.
Plugin changes are done for the Netty findings.
Risk: high. Major-version jumps of the GCS, BigQuery and Spanner
clients (new transports, retries, defaults) in the DataPrep
connections; Guava 33 and protobuf 4 in the app classloader. All jars
in lib/ are Java 8 bytecode (checked: no class file above major 52).
Needs the GCS / BigQuery / Spanner connection checks (browse,
sampling, proxy) in a 6.11 sandbox (P3) before release.
Testing: JDK 8, Checkstyle on (0 violations).
mvn clean install (all modules): BUILD SUCCESS. Tests: API 31/0 fail,
Proto 3/0, Core 456/0 (2 skipped), Storage 37/0, Service 17/0
(6 skipped, same as develop), Transform 2/0.
dependency:tree -Dincludes=io.grpc on wrangler-service: every io.grpc
artifact is 1.85.1 (grpc-netty-shaded 1.85.1); no 1.31.1 left.
unzip -l lib/: grpc-netty-shaded-1.85.1.jar (its
io.netty.versions.properties says 4.2.17.Final), no grpc 1.31.1 or
Netty 4.1.48 jar. Offline smoke run on the wrangler-service runtime
classpath: StorageOptions/BigQueryOptions/SpannerOptions with
NoCredentials build their services, Spanner admin clients,
Statement binding, Type.Code, Date and Timestamp work (no linkage
errors). Live GCS / BigQuery / Spanner calls not run (no credentials
here).