Skip to content

Bundle snappy-java in format-parquet plugin jar - #2021

Open
sahusanket wants to merge 1 commit into
developfrom
fix/format-parquet-bundle-snappy
Open

sahusanket wants to merge 1 commit into
developfrom
fix/format-parquet-bundle-snappy

Conversation

@sahusanket

Copy link
Copy Markdown
Contributor

Batch pipelines writing or reading Snappy-compressed Parquet files fail at runtime with ClassNotFoundException: org.xerial.snappy.Snappy.

format-parquet inherits hadoop-common at provided scope (depth 2 to snappy-java), which wins Maven's nearest-dependency resolution over parquet-avro -> parquet-hadoop -> snappy-java (depth 3, compile scope). Because maven-bundle-plugin only embeds compile-scope dependencies into lib/ and CDAP's FilterClassLoader blocks org.xerial.snappy.* from the parent Hadoop/Spark classloader, snappy-java is missing at runtime unless declared directly in format-parquet/pom.xml (this remains needed alongside #2017).

Declare org.xerial.snappy:snappy-java directly at compile scope in format-parquet/pom.xml and bump <snappy.version> from 1.1.2 to 1.1.10.5 so lib/snappy-java-1.1.10.5.jar is packaged in the plugin bundle.

Tested with ParquetInputFormatProviderTest#testSnappyParquetWriteAndRead, jar tf bundle verification, and an end-to-end GCS CSV-to-Snappy-Parquet Spark batch pipeline on Dataproc 2.3.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request upgrades the snappy.version from 1.1.2 to 1.1.10.5 in the root pom.xml and adds the snappy-java dependency to the format-parquet module. Additionally, it introduces a new unit test testSnappyParquetWriteAndRead to verify the writing and reading of Parquet files using Snappy compression. There are no review comments, and I have no feedback to provide.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant