Repository navigation
[cherrypick][6.11] Upgrade Bigtable HBase client to 2.x to remove shaded Hadoop 2.7.4 - #1631
Open
sahusanket wants to merge 2 commits into
Open
sahusanket wants to merge 2 commits into
sahusanket wants to merge 2 commits into
Conversation
There was a problem hiding this comment.
Code Review
This pull request upgrades Google Cloud Bigtable and HBase dependencies in pom.xml to support HBase 2.x and Hadoop 3. Specifically, it updates google.cloud.bigtable.version to 2.18.3, replaces bigtable-hbase-1.x-mapreduce with bigtable-hbase-2.x-hadoop, and replaces the shaded HBase client/server dependencies with hbase-shaded-mapreduce. Feedback indicates a potential version mismatch where a hardcoded dependency on google-cloud-bigtable version 1.17.1 remains elsewhere in the POM, which should be updated to use the new property to prevent runtime classpath conflicts.
…doop 2.7.4. hbase-shaded-client and hbase-shaded-server 1.4.13 bundle hadoop-common 2.7.4 and hadoop-yarn-server-common 2.7.4 inside their shaded JARs, triggering 6 S1 Hadoop 2.x CVEs even when hadoop.version is 3.3.6. Move to bigtable-hbase-2.x-hadoop 2.18.3 and hbase-shaded-mapreduce 2.6.6-hadoop3 (with org.apache.hadoop transitive dependencies excluded). hbase-shaded-mapreduce 2.6.6-hadoop3 is compiled for Java 8 (class version 52.0), keeps HBase's internal protobuf 2.5.0 shaded away from the Datastore plugin, provides all HBase client, mapreduce, and token classes, and does not bundle Hadoop classes so the CDAP platform Hadoop 3.3.6 is used at runtime.
sahusanket
force-pushed
the
fix-bigtable-hbase-hadoop-cve-0.24
branch
from
November 17, 2026 11:09
4076488 to
354a331
Compare
shruthi713
approved these changes
Nov 17, 2026
vsethi09
reviewed
Nov 17, 2026
| <groupId>io.cdap.plugin</groupId> | ||
| <artifactId>google-cloud</artifactId> | ||
| <version>0.24.7</version> | ||
| <version>0.24.8-SNAPSHOT</version> |
Contributor
There was a problem hiding this comment.
Please split this into separate PR to decouple for rollback.
vsethi09
approved these changes
Nov 17, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
bigtable-hbase-1.x-mapreduce(1.17.1) pulls inhbase-shaded-clientandhbase-shaded-server(1.4.13), which bundle unrelocated Hadoop2.7.4classes (hadoop-commonandhadoop-yarn-server-common2.7.4) inside the shaded JARs. Even though the plugin and CDAP runtime use Hadoop3.3.6, the bundled Hadoop2.7.4classes remain inside the packaged artifact (lib/hbase-shaded-*-1.4.13.jar), shadowingorg.apache.hadoop.util.VersionInfo(2.7.4) on the plugin classpath and triggering Hadoop2.7.4CVEs (CVE-2017-7669,CVE-2020-9492,CVE-2021-37404,CVE-2022-25168,CVE-2022-26612,CVE-2021-33036).Change
com.google.cloud.bigtable:bigtable-hbase-1.x-mapreduce:1.17.1tocom.google.cloud.bigtable:bigtable-hbase-2.x-hadoop:2.18.3(excluding unshadedorg.apache.hbase:*).hbase-shaded-client:1.4.13andhbase-shaded-server:1.4.13withorg.apache.hbase:hbase-shaded-mapreduce:2.6.6-hadoop3(excluding transitiveorg.apache.hadoop:*dependencies so they are provided by the CDAP runtime's Hadoop3.3.6).hbase-shaded-mapreduce:2.6.6-hadoop3is compiled for Java 8 (class version52.0), keeps HBase's internalprotobuf-java 2.5.0shaded underorg.apache.hbase.thirdparty.*so it does not conflict with the Datastore plugin'sprotobuf-java 3.x, provides all required HBase client,TableInputFormat,TableOutputFormat,TableMapReduceUtil, andTokenUtilclasses, and bundles zeroorg.apache.hadoop.*classes.Tests
mvn clean testandmvn package -DskipTestson Java 8 (OpenJDK 1.8.0_432), verifyingmaven-enforcer-plugin,BigtableSinkConfigTest,RecordToHBaseMutationTransformerTest,BigtableSourceConfigTest, andHBaseResultToRecordTransformerTest.lib/insidegoogle-cloud-*.jar): confirmedhbase-shaded-client-1.4.13.jarandhbase-shaded-server-1.4.13.jarare removed, no transitive Hadoop JARs are embedded inlib/, and runtimeVersionInfo.getVersion()resolves to3.3.6instead of2.7.4.Bigtablebatchsinkandbatchsourceplugin descriptors, validated both stages via the Pipeline Studio validation API (HTTP 200,failures: []), and deployed batch read/write pipelines (HTTP 200).BigtableSink.prepareRunto auto-create a target Cloud Bigtable table and column family viaHTableDescriptor/Admin.createTable, transformedStructuredRecordrows (string,int,long,double,boolean) into HBasePutmutations, and wrote them to Cloud Bigtable viaBigtableOutputFormat(TableOutputFormat) on Hadoop3.3.6.BigtableSource.prepareRun(validateOutputSchemaandTableMapReduceUtil.convertScanToString) and read back the written rows from Cloud Bigtable viaBigtableInputFormat(TableInputFormat) andHBaseResultToRecordTransformerfor both a full table scan and a bounded[startRow, stopRow)scan, confirming 100% identical output records between the baseline (1.x) and upgraded (2.x) plugin builds.ValidationException(Column family 'nonexistent_cf' does not exist in target table ...) as expected.Build with unit tests,Build e2e tests / build (bigtable), andBuild e2e tests / build (datastore)passed on CI.