Repository navigation
Only offer gzip and deflate in the HTTP response compressor - #137
Open
deepika-lalwani123 wants to merge 1 commit into
Open
deepika-lalwani123 wants to merge 1 commit into
deepika-lalwani123 wants to merge 1 commit into
Conversation
NettyHttpService added HttpContentCompressor with the no-arg constructor. In newer Netty 4.1.x releases that constructor also turns on zstd (and brotli) whenever those libraries are on the classpath. When an older zstd-jni comes first on the classpath (for example zstd-jni 1.4.9-1 from a Hadoop 3.3.x runtime), creating the compressor fails with NoSuchMethodError: Zstd.defaultCompressionLevel(), and every channel initialization in the service fails. CDAP hits this on Dataproc after moving to Netty 4.1.137.Final. Build the compressor with explicit gzip and deflate options instead, so the available encodings no longer depend on what else is on the classpath. gzip and deflate are what the service offered before. Testing: HttpServerTest and HttpsServerTest (102 tests) pass, including the new testCompressResponseOnlyGzipOrDeflate. A standalone check with Netty 4.1.137.Final plus zstd-jni 1.4.9-1 on the classpath reproduces the NoSuchMethodError with the old constructor and passes with the new one.
There was a problem hiding this comment.
Code Review
This pull request configures the NettyHttpService to explicitly support only gzip and deflate compression. This avoids potential NoSuchMethodError runtime failures in environments like Hadoop or Spark where older versions of zstd-jni might be loaded on the classpath. A new test case has been added to HttpServerTest to verify that unsupported encodings are ignored while gzip is correctly handled. There are no review comments, and the changes are well-implemented.
This was referenced Nov 17, 2026
deepika-lalwani123
added a commit
to cdapio/cdap
that referenced
this pull request
Nov 17, 2026
netty-http 1.7.0 creates HttpContentCompressor with the no-arg constructor. With Netty 4.1.137.Final (netty-bom bump), that constructor turns on zstd whenever zstd-jni is on the classpath. On Dataproc, the Hadoop 3.3.x runtime puts zstd-jni 1.4.9-1 first, which has no Zstd.defaultCompressionLevel(). Every channel of the driver's NettyHttpService then fails with NoSuchMethodError, and batch pipeline runs stay in STARTING. netty-http 1.8.0 (cdapio/netty-http#137) builds the compressor with explicit gzip and deflate options, which is what CDAP offered before. Must merge together with the Netty 4.1.137 bump (#16306). Needs the netty-http 1.8.0 release first. Testing (JDK 8, netty-http 1.8.0-SNAPSHOT built from #137): - cdap-common: SpillableBodyConsumerTest, HttpsEnablerTest, RemoteTaskExecutorTest: 12 tests pass. - cdap-gateway: NettyRouter*Test, AuditLogTest, ConfigBasedRequestBlockingTest: 51 tests pass. - mvn dependency:tree: cdap-gateway resolves netty-http 1.8.0(-SNAPSHOT).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
NettyHttpServiceaddsnew HttpContentCompressor()to every channel pipeline. In newer Netty 4.1.x releases, the no-arg constructor also turns on zstd (and brotli) whenever those libraries are on the classpath. It then callsZstd.defaultCompressionLevel().When an older zstd-jni is loaded first, creating the compressor throws:
That older zstd-jni is 1.4.9-1, which ships with Hadoop 3.3.x. As a result, every channel initialization in the service fails.
CDAP hits this after moving to Netty 4.1.137.Final (cdapio/cdap#16306, a CVE fix for 6.11.2): batch pipeline runs on Dataproc stay in STARTING, because the driver's messaging service can't accept any connection.
Change
Build the compressor with explicit options:
Testing
HttpServerTestandHttpsServerTestpass: 102 tests, JDK 8, Netty 4.1.75. This includes the newtestCompressResponseOnlyGzipOrDeflate:Accept-Encoding: zstd, br, snappyreturns an uncompressed response.zstd, br, gzipreturns gzip.new HttpContentCompressor()new HttpContentCompressor(gzip(), deflate())MutualAuthServerTestfails with Netty 4.1.137 (bad_certificate) with or without this change. That failure is unrelated to this PR; this repo still builds with Netty 4.1.75.Release
CDAP needs a netty-http 1.8.0 release with this change. CDAP then bumps
netty.http.versionfrom 1.7.0 to 1.8.0, together with the Netty 4.1.137 upgrade.