From 20c6176f487ec78c23fa5dcd1694f972c6f07361 Mon Sep 17 00:00:00 2001 From: Deepika Lalwani Date: Tue, 17 Nov 2026 06:05:48 +0000 Subject: [PATCH 1/5] Exclude avro-ipc and avro-ipc-jetty from avro-mapred avro-mapred 1.11.4 pulls in avro-ipc and avro-ipc-jetty, which bring velocity-engine-core 2.3 (shaded commons-io 2.8.0, CVE-2024-47554) and org.tukaani:xz into lib/ of google-cloud-0.25.0-SNAPSHOT.jar, as seen in the image scan of gcr.io/cdapio/cdap (/opt/cdap/master/artifacts/google-cloud-*.jar/lib/velocity-engine-core-2.3.jar). The plugins only use org.apache.avro.mapred.AvroKey and org.apache.avro.mapreduce.* (BigQuery sink AvroOutputFormat / AvroRecordWriter, Dataplex tests). Avro RPC and tether jobs, the only users of avro-ipc, are not used. The Avro output codecs the plugins allow are snappy and deflate (FileSetUtil), so dropping xz is safe. Versions: - org.apache.avro:avro-ipc 1.11.4 -> removed (exclusion on avro-mapred) - org.apache.avro:avro-ipc-jetty 1.11.4 -> removed (exclusion on avro-mapred) - velocity-engine-core 2.3 and xz 1.9 no longer packaged Bugs: none closed alone. The other commons-io 2.8.0 copy (inside format-avro-2.13.0.jar) is removed by hydrator-plugins HP-1. Risk: very low. No plugin code path uses avro-ipc. Testing: JDK 8, Checkstyle on (0 violations). mvn clean verify: 647 tests, 0 failures, 5 errors, 8 skipped. The 5 errors are cloud ITs that pick up the Cloudtop's ambient GCP project and fail with PERMISSION_DENIED (GCSConnectorTest x2, SpannerConnectorTest x2, BigQueryConnectorTest); they fail the same way on develop. All other unit tests pass, including AvroRecordWriterTest and DataPlexOutputFormatProviderTest. mvn dependency:tree: avro-mapred:1.11.4 has no children; avro-ipc, avro-ipc-jetty, velocity-engine-core and xz are gone. unzip -l target/google-cloud-0.25.0-SNAPSHOT.jar: lib/ has no avro-ipc*, velocity-* or xz-* jar. --- pom.xml | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/pom.xml b/pom.xml index e09f83e83..8c695a643 100644 --- a/pom.xml +++ b/pom.xml @@ -653,6 +653,16 @@ avro-mapred ${avro.version} + + + org.apache.avro + avro-ipc + + + org.apache.avro + avro-ipc-jetty + com.fasterxml.jackson.core jackson-core From e40376708a4943043e9c4388e48723be1057533a Mon Sep 17 00:00:00 2001 From: Deepika Lalwani Date: Tue, 17 Nov 2026 06:11:56 +0000 Subject: [PATCH 2/5] Bump snappy-java, httpclient and httpcore patch versions Patch-level alignment with the cross-track targets used by the Platform CLs. No row of the current gcr.io/cdapio/cdap image scan points at these top-level jars in google-cloud, but they are below the versions used everywhere else and would re-open on the next scan: snappy-java 1.1.10.5 is below the 1.1.10.12 target, and httpcore 4.4.14 (transitive) did not match httpclient. 4.5.14 / 4.4.16 are what google-http-client-apache-v2 already uses, so nothing is downgraded. Versions: - org.xerial.snappy:snappy-java 1.1.10.5 -> 1.1.10.12 (direct dependency) - org.apache.httpcomponents:httpclient 4.5.13 -> 4.5.14 (httpclient.version, now also in dependencyManagement) - org.apache.httpcomponents:httpcore 4.4.14 -> 4.4.16 (new httpcore.version property, managed in dependencyManagement) Bugs: none (no scanned rows; prevents a re-open). Risk: very low. Same-line patch releases. Testing: JDK 8, Checkstyle on (0 violations). mvn clean verify: 647 tests, 0 failures, 5 errors, 8 skipped; the 5 errors are the same cloud ITs as on develop (GCSConnectorTest x2, SpannerConnectorTest x2, BigQueryConnectorTest: PERMISSION_DENIED on the Cloudtop's ambient GCP project, no test credentials). mvn dependency:tree: snappy-java 1.1.10.12, httpclient 4.5.14 and httpcore 4.4.16 at compile scope; no other version of them on the runtime classpath (httpcore-nio 4.4.5 stays test-only via cdap-unit-test). unzip -l target/google-cloud-0.25.0-SNAPSHOT.jar: lib/httpclient-4.5.14.jar, lib/httpcore-4.4.16.jar, lib/snappy-java-1.1.10.12.jar. --- pom.xml | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/pom.xml b/pom.xml index 8c695a643..d22d08b81 100644 --- a/pom.xml +++ b/pom.xml @@ -80,7 +80,8 @@ 3.3.6 1.4.13 1.4.13 - 4.5.13 + 4.5.14 + 4.4.16 2.13.4.2 4.13.1 2.0.2 @@ -103,6 +104,16 @@ google-api-services-storage v1-rev20211201-1.32.1 + + org.apache.httpcomponents + httpclient + ${httpclient.version} + + + org.apache.httpcomponents + httpcore + ${httpcore.version} + @@ -816,7 +827,7 @@ org.xerial.snappy snappy-java - 1.1.10.5 + 1.1.10.12 From 574800d5abe75061bbe59662a9dc43268f1c5313 Mon Sep 17 00:00:00 2001 From: Deepika Lalwani Date: Tue, 17 Nov 2026 06:18:16 +0000 Subject: [PATCH 3/5] Bump Jackson to 2.18.12, protobuf to 3.25.5, oauth-client to 1.33.3 The image scan of gcr.io/cdapio/cdap reports these top-level jars in lib/ of google-cloud-0.25.0-SNAPSHOT.jar: - jackson-core 2.12.3 (CVE-2025-49128, CVE-2025-52999, CVE-2026-68494, CVE-2026-89425), jackson-databind 2.13.4.2 (11 CVEs) and jackson-dataformat-cbor 2.9.9 (CVE-2020-28491) - protobuf-java 3.19.4 (CVE-2022-3171, CVE-2022-3509, CVE-2022-3510, CVE-2024-7254) - google-oauth-client 1.31.2 (CVE-2021-22573) Jackson is now aligned with an imported jackson-bom, so the cbor module that comes in through google-cloud-bigquery moves with core/databind. protobuf-bom aligns protobuf-java-util (3.17.3) with protobuf-java. No libraries-bom is imported here, so nothing brings a protobuf 4.x that could be downgraded. google-oauth-client 1.33.3 is built against google-http-client 1.41.x, so google-http-client-bom 1.41.8 is imported to keep the whole google-http-client family (core, jackson2, gson, apache-v2, appengine, protobuf) on one version instead of mixing 1.38.0 / 1.39.2 with 1.41. Versions: - jackson-core 2.12.3, jackson-databind / jackson-annotations 2.13.4.x, jackson-dataformat-cbor 2.9.9 -> 2.18.12 (jackson.core.version, now also used for the jackson-bom import) - protobuf-java 3.19.4 -> 3.25.5 (google.protobuf.java.version); protobuf-java-util 3.17.3 -> 3.25.5 (protobuf-bom import) - google-oauth-client 1.31.2 -> 1.33.3 (managed, new google.oauth.client.version) - google-http-client* 1.38.0 / 1.39.2 -> 1.41.8 (google-http-client-bom import, new google.http.client.version) The copies bundled inside bigtable-hbase-1.x-hadoop-1.17.1.jar (jackson-core 2.11.1, protobuf 3.12.2, oauth-client 1.31.0) are handled by GC-4; the Jackson copy inside format-avro-2.13.0.jar is handled in hydrator-plugins. Bugs: jackson-dataformat-cbor b/578686984 (closed by this change alone). Risk: medium. Minor-version jumps. Jackson 2.18 keeps the 2.x API but has stricter defaults (e.g. StreamReadConstraints limits on very large numbers / strings). protobuf 3.25 runtime is backward compatible with code generated by older 3.x protoc (grpc 1.39 / gax). Testing: JDK 8, Checkstyle on (0 violations). mvn clean verify: 647 tests, 0 failures, 5 errors, 8 skipped; the 5 errors are the same cloud ITs as on develop (GCSConnectorTest x2, SpannerConnectorTest x2, BigQueryConnectorTest: PERMISSION_DENIED on the Cloudtop's ambient GCP project, no test credentials). BigQuery, GCS, Spanner, Datastore, Pub/Sub, Dataplex unit tests pass. mvn dependency:tree: jackson-* 2.18.12, protobuf-java / protobuf-java-util 3.25.5, google-oauth-client 1.33.3, google-http-client* 1.41.8 at compile scope; 2.12.3 / 2.13.4.2 / 2.9.9 / 3.19.4 / 1.31.2 are gone. unzip -l target/google-cloud-0.25.0-SNAPSHOT.jar: lib/ has only the new versions (jackson-core-2.18.12, jackson-databind-2.18.12, jackson-dataformat-cbor-2.18.12, protobuf-java-3.25.5, google-oauth-client-1.33.3, google-http-client-1.41.8). Not run: e2e tests (need GCP credentials). --- pom.xml | 35 +++++++++++++++++++++++++++++++++-- 1 file changed, 33 insertions(+), 2 deletions(-) diff --git a/pom.xml b/pom.xml index d22d08b81..353d4026b 100644 --- a/pom.xml +++ b/pom.xml @@ -64,6 +64,8 @@ 6.11.0 2.13.0 3.2.6 + 1.41.8 + 1.33.3 0.7.1 hadoop3-2.2.21 1.17.1 @@ -74,7 +76,7 @@ 1.24.7 2.3.0 1.105.1 - 3.19.4 + 3.25.5 1.3.0-rc3 27.0.1-jre 3.3.6 @@ -82,7 +84,7 @@ 1.4.13 4.5.14 4.4.16 - 2.13.4.2 + 2.18.12 4.13.1 2.0.2 1.7.5 @@ -94,6 +96,35 @@ + + + com.fasterxml.jackson + jackson-bom + ${jackson.core.version} + pom + import + + + + com.google.protobuf + protobuf-bom + ${google.protobuf.java.version} + pom + import + + + + com.google.http-client + google-http-client-bom + ${google.http.client.version} + pom + import + + + com.google.oauth-client + google-oauth-client + ${google.oauth.client.version} + com.google.apis google-api-services-bigquery From a6fc36e9b32350b7efd75dd8e224898c9116f482 Mon Sep 17 00:00:00 2001 From: Deepika Lalwani Date: Tue, 17 Nov 2026 06:25:28 +0000 Subject: [PATCH 4/5] Bump org.json to 20260719 The image scan of gcr.io/cdapio/cdap reports org.json:json 20180813 (CVE-2022-45688, CVE-2023-5072, CVE-2026-59171) in lib/ of google-cloud-0.25.0-SNAPSHOT.jar. It comes in through tink 1.3.0-rc3 (and tink-gcpkms) and is used directly by GCSTinkTool and TinkDecryptor (JSONObject) to write and read the Tink encryption metadata of encrypted GCS files, and by Tink's JsonKeysetReader / JsonKeysetWriter. Kept separate from the other MINOR bumps because newer org.json releases change parsing behavior (stricter handling of malformed input, number conversion). A new unit test, TinkKeysetJsonTest, writes the metadata the same way GCSTinkTool does (JsonKeysetWriter -> JSONObject.toString(4)) and reads it back the same way TinkDecryptor does (new JSONObject -> getJSONObject("keyset") -> JsonKeysetReader), then decrypts a stream with the result, with a local AEAD key in place of Cloud KMS. It passes with both 20180813 and 20260719. Versions: - org.json:json 20180813 -> 20260719 (new org.json.version property, managed in dependencyManagement) Bugs: b/578504385, b/578575239, b/584531579 (closed once the hydrator-plugins HP-2b and wrangler WR-3b copies are also bumped). Risk: medium. org.json behavior changes; the Tink metadata round trip and Tink-encrypted read path are covered by the new test. Testing: JDK 8, Checkstyle on (0 violations). mvn clean verify: 648 tests, 0 failures, 5 errors, 8 skipped; the 5 errors are the same cloud ITs as on develop (GCSConnectorTest x2, SpannerConnectorTest x2, BigQueryConnectorTest: PERMISSION_DENIED on the Cloudtop's ambient GCP project, no test credentials). TinkKeysetJsonTest passes on this branch and with the old 20180813. mvn dependency:tree -Dincludes=org.json: tink:1.3.0-rc3 -> org.json:json:20260719 only. unzip -l target/google-cloud-0.25.0-SNAPSHOT.jar: lib/json-20260719.jar. Not run: GCS e2e with a real Cloud KMS key (needs GCP credentials). --- pom.xml | 7 ++ .../gcp/gcs/source/TinkKeysetJsonTest.java | 93 +++++++++++++++++++ 2 files changed, 100 insertions(+) create mode 100644 src/test/java/io/cdap/plugin/gcp/gcs/source/TinkKeysetJsonTest.java diff --git a/pom.xml b/pom.xml index 353d4026b..595154893 100644 --- a/pom.xml +++ b/pom.xml @@ -78,6 +78,7 @@ 1.105.1 3.25.5 1.3.0-rc3 + 20260719 27.0.1-jre 3.3.6 1.4.13 @@ -125,6 +126,12 @@ google-oauth-client ${google.oauth.client.version} + + + org.json + json + ${org.json.version} + com.google.apis google-api-services-bigquery diff --git a/src/test/java/io/cdap/plugin/gcp/gcs/source/TinkKeysetJsonTest.java b/src/test/java/io/cdap/plugin/gcp/gcs/source/TinkKeysetJsonTest.java new file mode 100644 index 000000000..8231b745e --- /dev/null +++ b/src/test/java/io/cdap/plugin/gcp/gcs/source/TinkKeysetJsonTest.java @@ -0,0 +1,93 @@ +/* + * Copyright © 2026 Cask Data, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); you may not + * use this file except in compliance with the License. You may obtain a copy of + * the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + * License for the specific language governing permissions and limitations under + * the License. + */ + +package io.cdap.plugin.gcp.gcs.source; + +import com.google.common.io.ByteStreams; +import com.google.crypto.tink.Aead; +import com.google.crypto.tink.JsonKeysetReader; +import com.google.crypto.tink.JsonKeysetWriter; +import com.google.crypto.tink.KeysetHandle; +import com.google.crypto.tink.StreamingAead; +import com.google.crypto.tink.aead.AeadConfig; +import com.google.crypto.tink.aead.AeadKeyTemplates; +import com.google.crypto.tink.streamingaead.StreamingAeadConfig; +import com.google.crypto.tink.streamingaead.StreamingAeadKeyTemplates; +import org.json.JSONObject; +import org.junit.Assert; +import org.junit.BeforeClass; +import org.junit.Test; + +import java.io.ByteArrayInputStream; +import java.io.ByteArrayOutputStream; +import java.io.InputStream; +import java.io.OutputStream; +import java.util.Base64; + +import static java.nio.charset.StandardCharsets.UTF_8; + +/** + * Checks that the encryption metadata written by {@link GCSTinkTool} can be parsed with org.json and read back by + * Tink the same way {@link TinkDecryptor} does. A local AEAD key stands in for the Cloud KMS key. + */ +public class TinkKeysetJsonTest { + + private static final int ITERATIONS = 10; + + @BeforeClass + public static void setupClass() throws Exception { + AeadConfig.register(); + StreamingAeadConfig.register(); + } + + @Test + public void testMetadataRoundTrip() throws Exception { + Aead keysetAead = KeysetHandle.generateNew(AeadKeyTemplates.AES128_GCM).getPrimitive(Aead.class); + byte[] plaintext = "some plaintext to encrypt".getBytes(UTF_8); + byte[] associatedData = "gs://bucket/path/file.csv".getBytes(UTF_8); + + // Each iteration uses a new keyset, so different random key ids go through the JSON round trip. + for (int i = 0; i < ITERATIONS; i++) { + KeysetHandle handle = KeysetHandle.generateNew(StreamingAeadKeyTemplates.AES256_GCM_HKDF_4KB); + + // Encrypt and write the metadata the same way as GCSTinkTool. + ByteArrayOutputStream ciphertext = new ByteArrayOutputStream(); + try (OutputStream os = handle.getPrimitive(StreamingAead.class).newEncryptingStream(ciphertext, + associatedData)) { + os.write(plaintext); + } + ByteArrayOutputStream keyset = new ByteArrayOutputStream(); + handle.write(JsonKeysetWriter.withOutputStream(keyset), keysetAead); + JSONObject metadataObj = new JSONObject(); + metadataObj.put("kms", "gcp-kms://projects/p/locations/global/keyRings/r/cryptoKeys/k"); + metadataObj.put("aad", Base64.getEncoder().encodeToString(associatedData)); + metadataObj.put("keyset", new JSONObject(new String(keyset.toByteArray(), UTF_8))); + String metadataJson = metadataObj.toString(4); + + // Parse and read it back the same way as TinkDecryptor. + JSONObject metadata = new JSONObject(metadataJson); + KeysetHandle readHandle = KeysetHandle.read(JsonKeysetReader.withJsonObject(metadata.getJSONObject("keyset")), + keysetAead); + byte[] aad = Base64.getDecoder().decode(metadata.getString("aad")); + Assert.assertEquals(handle.getKeysetInfo().getPrimaryKeyId(), readHandle.getKeysetInfo().getPrimaryKeyId()); + + try (InputStream is = readHandle.getPrimitive(StreamingAead.class) + .newDecryptingStream(new ByteArrayInputStream(ciphertext.toByteArray()), aad)) { + Assert.assertArrayEquals(plaintext, ByteStreams.toByteArray(is)); + } + } + } +} From 6812dd161b76f7cf40618a9abc532b0daf97b797 Mon Sep 17 00:00:00 2001 From: Deepika Lalwani Date: Tue, 17 Nov 2026 06:40:18 +0000 Subject: [PATCH 5/5] Move Bigtable source/sink to bigtable-hbase 2.x and HBase 2.5 shaded The image scan of gcr.io/cdapio/cdap reports 64 CVEs in jars bundled in lib/ of google-cloud-0.25.0-SNAPSHOT.jar by the Bigtable plugins: - bigtable-hbase-1.x-hadoop-1.17.1.jar: Netty 4.1.48 (codec, codec-http, codec-http2, common, handler, handler-proxy, native-epoll), jackson-core 2.11.1, protobuf 3.12.2, guava 29.0, gson 2.8.6, oauth-client 1.31.0, httpclient 4.5.12 - hbase-shaded-client-1.4.13.jar / hbase-shaded-server-1.4.13.jar: Hadoop 2.7.4 (hadoop-common, yarn-server-common), avro 1.7.7, commons-compress 1.4.1, commons-io 2.4, commons-net 3.1, snappy-java 1.0.5, protobuf 2.5.0, guava 12.0.1, gson 2.8.5, httpclient 4.5.2, java-xmlbuilder 0.4 None of these can be fixed inside the 1.x line. bigtable-hbase-1.x-mapreduce 1.17.1 is replaced by the shaded bigtable-hbase-2.x-hadoop 2.21.0 (its unshaded hbase-client 2.6.4 is excluded), and hbase-shaded-client / -server 1.4.13 by hbase-shaded-mapreduce 2.5.16-hadoop3, which provides the HBase 2 client API and TableInputFormat / TableOutputFormat with HBase's own third-party jars relocated. hbase-shaded-mapreduce declares Hadoop 3.4.3 jars (hadoop-distcp, -hdfs-client, -annotations), slf4j and log4j2 as compile dependencies; they are excluded because CDAP provides Hadoop and logging. javax.annotation-api 1.2 from it is excluded so the 1.3.2 used by the Google Cloud clients stays. google-cloud-bigtable 1.17.1 (admin / data client) is only used by the e2e test setup, so it moves from the plugin's compile dependencies to the e2e-tests profile at test scope. Code changes (HBase 2 API): - BigtableSink: HTableDescriptor / HColumnDescriptor -> TableDescriptorBuilder / ColumnFamilyDescriptorBuilder; Table.getTableDescriptor().getFamiliesKeys() -> Table.getDescriptor().getColumnFamilyNames() - BigtableSource: same getDescriptor() change - HBaseResultToRecordTransformer: Cell.getFamilyArray() / getQualifierArray() / getValueArray() -> CellUtil.cloneFamily / cloneQualifier / cloneValue (the *Array() getters return the backing array, which is only correct when offset = 0 and length = array length) - BigtableInputFormat / BigtableOutputFormat: no change needed; TableInputFormat.createRecordReader and TableOutputFormat getRecordWriter / checkOutputSpecs / setConf keep the same signatures. BigtableConfiguration / BigtableOptionsFactory keys used by the plugins are unchanged in 2.x. Versions: - com.google.cloud.bigtable:bigtable-hbase-1.x-mapreduce 1.17.1 -> bigtable-hbase-2.x-hadoop 2.21.0 (google.cloud.bigtable.version) - org.apache.hbase:hbase-shaded-client 1.4.13 and hbase-shaded-server 1.4.13 -> hbase-shaded-mapreduce 2.5.16-hadoop3 (new hbase-shaded-mapreduce.version; the two old properties are removed) - com.google.cloud:google-cloud-bigtable 1.17.1: compile -> test scope in the e2e-tests profile - New small runtime jars: opentelemetry-api / -context 1.49.0, opentelemetry-semconv 1.29.0-alpha, audience-annotations 0.13.0, jakarta.annotation-api 1.3.5, jakarta.validation-api 2.0.2, jakarta.inject 2.6.1, javassist 3.30.2-GA Bugs: b/578424402, b/578429556, b/578451083, b/578453650, b/578458777, b/578489641, b/578495801, b/578496916, b/578501872, b/578521895, b/578579901, b/578591566, b/578635525, b/578636017, b/578637267, b/578639096, b/578662249, b/578746169, b/578756088, b/578782461 (closed by this change alone). Plugins side done for b/571156599, b/578423875, b/578436101, b/578436419, b/578437560, b/578496596, b/578559913, b/578580043, b/578580079, b/578597409, b/578600761, b/578643165, b/578659719. Not fixed / net-new copies (checked by unzipping the built plugin jar): - bigtable-hbase-2.x-hadoop-2.21.0.jar bundles grpc-netty-shaded with Netty 4.2.15.Final and jackson-core 2.18.3, so the 13 Netty CVEs fixed only in 4.2.16+ and jackson-core CVE-2026-89425 stay open (exception). - hbase-shaded-mapreduce-2.5.16-hadoop3.jar (hbase-thirdparty 4.1.14) bundles relocated Netty 4.1.131.Final (31 modules incl. netty-all), Jetty 9.4.58.v20250814 (10 modules), protobuf-java 2.5.0, httpclient 4.5.13 / httpcore 4.4.13, commons-codec 1.15, zookeeper 3.8.6, jackson 2.21.1, guava 33.4.8, gson 2.13.2, jersey 2.47. The Netty / Jetty / protobuf 2.5.0 copies are expected to show up as new rows on the next scan; run a Pantheon / Container Analysis scan of the built jar before merging. Risk: high. Major-version client change for the Bigtable source and sink (HBase 1.4 API -> HBase 2.5 API, Bigtable HBase client 1.x -> 2.x on the veneer data client). bigtable-hbase-2.x 2.21.0 is built against hbase-client 2.6.4 but runs here on the 2.5.16 client classes; the connection constructor used by HBase 2.5's ConnectionFactory exists. Testing: JDK 8, Checkstyle on (0 violations). mvn clean verify: 648 tests, 0 failures, 5 errors, 8 skipped; the 5 errors are the same cloud ITs as on develop (GCSConnectorTest x2, SpannerConnectorTest x2, BigQueryConnectorTest: PERMISSION_DENIED on the Cloudtop's ambient GCP project, no test credentials). Bigtable unit tests pass: HBaseResultToRecordTransformerTest, RecordToHBaseMutationTransformerTest, BigtableSinkConfigTest, BigtableSourceConfigTest. Runtime smoke check (scratch class on the test classpath, not committed): BigtableConfiguration.configure, Scan -> TableMapReduceUtil.convertScanToString, TableDescriptorBuilder, HBase ConnectionFactory.createConnection -> hbase2_x.BigtableConnection, Connection.getTable, BigtableInputFormat.setConf and BigtableOutputFormat.setConf all work; the channel reached the Bigtable API and got NOT_FOUND for the fake instance, as expected. mvn dependency:tree: no bigtable-hbase-1.x-*, hbase-shaded-client / -server or google-cloud-bigtable at compile scope; no Hadoop jar at compile scope. unzip -l target/google-cloud-0.25.0-SNAPSHOT.jar: lib/ has bigtable-hbase-2.x-hadoop-2.21.0.jar and hbase-shaded-mapreduce-2.5.16-hadoop3.jar and no bigtable-hbase-1.x-*, hbase-shaded-client-*, hbase-shaded-server-*, google-cloud-bigtable-*, grpc-google-cloud-bigtable-* or hadoop-* jar. Not run: Bigtable e2e tests and Bigtable source/sink on MapReduce and Spark in a CDAP 6.11 sandbox (need a Bigtable instance and credentials). --- pom.xml | 91 ++++++------------- .../gcp/bigtable/sink/BigtableSink.java | 16 ++-- .../gcp/bigtable/source/BigtableSource.java | 4 +- .../HBaseResultToRecordTransformer.java | 6 +- 4 files changed, 40 insertions(+), 77 deletions(-) diff --git a/pom.xml b/pom.xml index 595154893..afa1f6843 100644 --- a/pom.xml +++ b/pom.xml @@ -68,7 +68,7 @@ 1.33.3 0.7.1 hadoop3-2.2.21 - 1.17.1 + 2.21.0 1.137.1 2.0.2 1.108.1 @@ -81,8 +81,7 @@ 20260719 27.0.1-jre 3.3.6 - 1.4.13 - 1.4.13 + 2.5.16-hadoop3 4.5.14 4.4.16 2.18.12 @@ -399,33 +398,18 @@ + com.google.cloud.bigtable - bigtable-hbase-1.x-mapreduce + bigtable-hbase-2.x-hadoop ${google.cloud.bigtable.version} org.apache.hbase hbase-client - - org.apache.hbase - hbase-server - - - org.apache.hbase - hbase-annotations - - - - org.apache.hbase - hbase-common - - - org.apache.hadoop - hadoop-common - @@ -510,53 +494,29 @@ org.apache.hbase - hbase-shaded-client - ${hbase-shaded-client.version} + hbase-shaded-mapreduce + ${hbase-shaded-mapreduce.version} - org.slf4j - slf4j-log4j12 - - - log4j - log4j - - - org.apache.htrace - htrace-core + org.apache.hadoop + * - - - - - org.apache.hbase - hbase-shaded-server - ${hbase-shaded-server.version} - org.slf4j - slf4j-log4j12 + * - log4j - log4j + org.apache.logging.log4j + * + - org.apache.htrace - htrace-core - - - org.apache.hadoop - hadoop-common + javax.annotation + javax.annotation-api @@ -949,12 +909,6 @@ 0.2.0 - - - com.google.cloud - google-cloud-bigtable - 1.17.1 - com.google.api.grpc proto-google-common-protos @@ -1341,6 +1295,13 @@ 0.5.0-SNAPSHOT test + + + com.google.cloud + google-cloud-bigtable + 1.17.1 + test + ch.qos.logback logback-classic diff --git a/src/main/java/io/cdap/plugin/gcp/bigtable/sink/BigtableSink.java b/src/main/java/io/cdap/plugin/gcp/bigtable/sink/BigtableSink.java index 1301f0cc3..e6196c26f 100644 --- a/src/main/java/io/cdap/plugin/gcp/bigtable/sink/BigtableSink.java +++ b/src/main/java/io/cdap/plugin/gcp/bigtable/sink/BigtableSink.java @@ -41,14 +41,14 @@ import io.cdap.plugin.gcp.bigtable.common.HBaseColumn; import io.cdap.plugin.gcp.common.SourceOutputFormatProvider; import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.hbase.HColumnDescriptor; -import org.apache.hadoop.hbase.HTableDescriptor; import org.apache.hadoop.hbase.TableName; import org.apache.hadoop.hbase.client.Admin; +import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder; import org.apache.hadoop.hbase.client.Connection; import org.apache.hadoop.hbase.client.Mutation; import org.apache.hadoop.hbase.client.Put; import org.apache.hadoop.hbase.client.Table; +import org.apache.hadoop.hbase.client.TableDescriptorBuilder; import org.apache.hadoop.hbase.io.ImmutableBytesWritable; import org.apache.hadoop.hbase.mapreduce.TableOutputFormat; import org.apache.hadoop.hbase.util.Bytes; @@ -222,15 +222,15 @@ private void validateInputSchema(Schema inputSchema, FailureCollector collector) private void createTable(Connection connection, TableName tableName, FailureCollector collector) { try (Admin admin = connection.getAdmin()) { - HTableDescriptor tableDescriptor = new HTableDescriptor(tableName); + TableDescriptorBuilder tableDescriptor = TableDescriptorBuilder.newBuilder(tableName); config.getColumnMappings(collector) .values() .stream() .map(HBaseColumn::getFamily) .distinct() - .map(HColumnDescriptor::new) - .forEach(tableDescriptor::addFamily); - admin.createTable(tableDescriptor); + .map(family -> ColumnFamilyDescriptorBuilder.of(family)) + .forEach(tableDescriptor::setColumnFamily); + admin.createTable(tableDescriptor.build()); } catch (IOException e) { collector.addFailure( String.format("Failed to create table '%s' in Bigtable : %s", tableName, e.getMessage()), null) @@ -242,8 +242,8 @@ private void createTable(Connection connection, TableName tableName, FailureColl private void validateExistingTable(Connection connection, TableName tableName, FailureCollector collector) throws IOException { try (Table table = connection.getTable(tableName)) { - Set existingFamilies = table.getTableDescriptor() - .getFamiliesKeys() + Set existingFamilies = table.getDescriptor() + .getColumnFamilyNames() .stream() .map(Bytes::toString) .collect(Collectors.toSet()); diff --git a/src/main/java/io/cdap/plugin/gcp/bigtable/source/BigtableSource.java b/src/main/java/io/cdap/plugin/gcp/bigtable/source/BigtableSource.java index d1d05ee82..294081310 100644 --- a/src/main/java/io/cdap/plugin/gcp/bigtable/source/BigtableSource.java +++ b/src/main/java/io/cdap/plugin/gcp/bigtable/source/BigtableSource.java @@ -192,8 +192,8 @@ private void validateOutputSchema(Configuration configuration, FailureCollector TableName tableName = TableName.valueOf(config.table); try (Connection connection = BigtableConfiguration.connect(configuration); Table table = connection.getTable(tableName)) { - Set existingFamilies = table.getTableDescriptor() - .getFamiliesKeys() + Set existingFamilies = table.getDescriptor() + .getColumnFamilyNames() .stream() .map(Bytes::toString) .collect(Collectors.toSet()); diff --git a/src/main/java/io/cdap/plugin/gcp/bigtable/source/HBaseResultToRecordTransformer.java b/src/main/java/io/cdap/plugin/gcp/bigtable/source/HBaseResultToRecordTransformer.java index 7caba7bee..045c33e94 100644 --- a/src/main/java/io/cdap/plugin/gcp/bigtable/source/HBaseResultToRecordTransformer.java +++ b/src/main/java/io/cdap/plugin/gcp/bigtable/source/HBaseResultToRecordTransformer.java @@ -19,6 +19,7 @@ import io.cdap.cdap.api.data.format.UnexpectedFormatException; import io.cdap.cdap.api.data.schema.Schema; import org.apache.hadoop.hbase.Cell; +import org.apache.hadoop.hbase.CellUtil; import org.apache.hadoop.hbase.client.Result; import org.apache.hadoop.hbase.util.Bytes; @@ -49,10 +50,11 @@ public StructuredRecord transform(Result result) { recordBuilder.set(keyAlias, value); } for (Cell cell : result.rawCells()) { - String columnName = Bytes.toString(cell.getFamilyArray()) + ':' + Bytes.toString(cell.getQualifierArray()); + String columnName = Bytes.toString(CellUtil.cloneFamily(cell)) + ':' + + Bytes.toString(CellUtil.cloneQualifier(cell)); String fieldName = columnMappings.get(columnName); if (fieldName != null) { - Object value = convertBytesToFieldValue(cell.getValueArray(), fieldName); + Object value = convertBytesToFieldValue(CellUtil.cloneValue(cell), fieldName); recordBuilder.set(fieldName, value); } }