Skip to content

[cherrypick][6.11] Upgrade Bigtable HBase client to 2.x to remove shaded Hadoop 2.7.4 - #1631

Open
sahusanket wants to merge 2 commits into
release/0.24from
fix-bigtable-hbase-hadoop-cve-0.24
Open

sahusanket wants to merge 2 commits into
release/0.24from
fix-bigtable-hbase-hadoop-cve-0.24

Conversation

@sahusanket

@sahusanket sahusanket commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Problem

bigtable-hbase-1.x-mapreduce (1.17.1) pulls in hbase-shaded-client and hbase-shaded-server (1.4.13), which bundle unrelocated Hadoop 2.7.4 classes (hadoop-common and hadoop-yarn-server-common 2.7.4) inside the shaded JARs. Even though the plugin and CDAP runtime use Hadoop 3.3.6, the bundled Hadoop 2.7.4 classes remain inside the packaged artifact (lib/hbase-shaded-*-1.4.13.jar), shadowing org.apache.hadoop.util.VersionInfo (2.7.4) on the plugin classpath and triggering Hadoop 2.7.4 CVEs (CVE-2017-7669, CVE-2020-9492, CVE-2021-37404, CVE-2022-25168, CVE-2022-26612, CVE-2021-33036).

Change

  • Upgrade com.google.cloud.bigtable:bigtable-hbase-1.x-mapreduce:1.17.1 to com.google.cloud.bigtable:bigtable-hbase-2.x-hadoop:2.18.3 (excluding unshaded org.apache.hbase:*).
  • Replace hbase-shaded-client:1.4.13 and hbase-shaded-server:1.4.13 with org.apache.hbase:hbase-shaded-mapreduce:2.6.6-hadoop3 (excluding transitive org.apache.hadoop:* dependencies so they are provided by the CDAP runtime's Hadoop 3.3.6).
  • hbase-shaded-mapreduce:2.6.6-hadoop3 is compiled for Java 8 (class version 52.0), keeps HBase's internal protobuf-java 2.5.0 shaded under org.apache.hbase.thirdparty.* so it does not conflict with the Datastore plugin's protobuf-java 3.x, provides all required HBase client, TableInputFormat, TableOutputFormat, TableMapReduceUtil, and TokenUtil classes, and bundles zero org.apache.hadoop.* classes.

Tests

  1. Unit Tests & Build Checks:
    • Ran mvn clean test and mvn package -DskipTests on Java 8 (OpenJDK 1.8.0_432), verifying maven-enforcer-plugin, BigtableSinkConfigTest, RecordToHBaseMutationTransformerTest, BigtableSourceConfigTest, and HBaseResultToRecordTransformerTest.
    • Verified the packaged OSGi bundle (lib/ inside google-cloud-*.jar): confirmed hbase-shaded-client-1.4.13.jar and hbase-shaded-server-1.4.13.jar are removed, no transitive Hadoop JARs are embedded in lib/, and runtime VersionInfo.getVersion() resolves to 3.3.6 instead of 2.7.4.
  2. End-to-End Cloud Bigtable Verification (Baseline vs. Upgraded Plugin on CDAP 6.11 / Hadoop 3.3.6):
    • Stage Validation & Pipeline Deployment: Uploaded the plugin artifact to a CDAP 6.11 instance, verified Bigtable batchsink and batchsource plugin descriptors, validated both stages via the Pipeline Studio validation API (HTTP 200, failures: []), and deployed batch read/write pipelines (HTTP 200).
    • Bigtable BatchSink (Write & Table Auto-Creation): Executed BigtableSink.prepareRun to auto-create a target Cloud Bigtable table and column family via HTableDescriptor / Admin.createTable, transformed StructuredRecord rows (string, int, long, double, boolean) into HBase Put mutations, and wrote them to Cloud Bigtable via BigtableOutputFormat (TableOutputFormat) on Hadoop 3.3.6.
    • Bigtable BatchSource (Full Scan & Row-Range Scan): Executed BigtableSource.prepareRun (validateOutputSchema and TableMapReduceUtil.convertScanToString) and read back the written rows from Cloud Bigtable via BigtableInputFormat (TableInputFormat) and HBaseResultToRecordTransformer for both a full table scan and a bounded [startRow, stopRow) scan, confirming 100% identical output records between the baseline (1.x) and upgraded (2.x) plugin builds.
    • Negative Column-Family Validation: Verified that configuring a non-existent column family against an existing Cloud Bigtable table throws ValidationException (Column family 'nonexistent_cf' does not exist in target table ...) as expected.
    • GitHub Actions E2E Suite: Build with unit tests, Build e2e tests / build (bigtable), and Build e2e tests / build (datastore) passed on CI.

@sahusanket sahusanket added the build Trigger unit test build label Nov 16, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request upgrades Google Cloud Bigtable and HBase dependencies in pom.xml to support HBase 2.x and Hadoop 3. Specifically, it updates google.cloud.bigtable.version to 2.18.3, replaces bigtable-hbase-1.x-mapreduce with bigtable-hbase-2.x-hadoop, and replaces the shaded HBase client/server dependencies with hbase-shaded-mapreduce. Feedback indicates a potential version mismatch where a hardcoded dependency on google-cloud-bigtable version 1.17.1 remains elsewhere in the POM, which should be updated to use the new property to prevent runtime classpath conflicts.

Comment thread pom.xml Resolved
@sahusanket sahusanket self-assigned this Nov 17, 2026
@sahusanket
sahusanket requested a review from vsethi09 November 17, 2026 11:06
…doop 2.7.4.

hbase-shaded-client and hbase-shaded-server 1.4.13 bundle hadoop-common 2.7.4 and hadoop-yarn-server-common 2.7.4 inside their shaded JARs, triggering 6 S1 Hadoop 2.x CVEs even when hadoop.version is 3.3.6.

Move to bigtable-hbase-2.x-hadoop 2.18.3 and hbase-shaded-mapreduce 2.6.6-hadoop3 (with org.apache.hadoop transitive dependencies excluded). hbase-shaded-mapreduce 2.6.6-hadoop3 is compiled for Java 8 (class version 52.0), keeps HBase's internal protobuf 2.5.0 shaded away from the Datastore plugin, provides all HBase client, mapreduce, and token classes, and does not bundle Hadoop classes so the CDAP platform Hadoop 3.3.6 is used at runtime.
@sahusanket
sahusanket force-pushed the fix-bigtable-hbase-hadoop-cve-0.24 branch from 4076488 to 354a331 Compare November 17, 2026 11:09
@sahusanket sahusanket changed the title [cherrypick] Upgrade Bigtable HBase client to 2.x to remove shaded Hadoop 2.7.4 [cherrypick][6.11] Upgrade Bigtable HBase client to 2.x to remove shaded Hadoop 2.7.4 Nov 17, 2026
Comment thread pom.xml
<groupId>io.cdap.plugin</groupId>
<artifactId>google-cloud</artifactId>
<version>0.24.7</version>
<version>0.24.8-SNAPSHOT</version>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please split this into separate PR to decouple for rollback.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build Trigger unit test build

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants