Skip to content

Only offer gzip and deflate in the HTTP response compressor - #137

Open
deepika-lalwani123 wants to merge 1 commit into
developfrom
fix/compressor-gzip-deflate
Open

deepika-lalwani123 wants to merge 1 commit into
developfrom
fix/compressor-gzip-deflate

Conversation

@deepika-lalwani123

Copy link
Copy Markdown

Problem

NettyHttpService adds new HttpContentCompressor() to every channel pipeline. In newer Netty 4.1.x releases, the no-arg constructor also turns on zstd (and brotli) whenever those libraries are on the classpath. It then calls Zstd.defaultCompressionLevel().

When an older zstd-jni is loaded first, creating the compressor throws:

java.lang.NoSuchMethodError: 'int com.github.luben.zstd.Zstd.defaultCompressionLevel()'
  at io.netty.handler.codec.compression.ZstdConstants.<clinit>
  at io.netty.handler.codec.http.HttpContentCompressor.<init>
  at io.cdap.http.NettyHttpService$4.initChannel

That older zstd-jni is 1.4.9-1, which ships with Hadoop 3.3.x. As a result, every channel initialization in the service fails.

CDAP hits this after moving to Netty 4.1.137.Final (cdapio/cdap#16306, a CVE fix for 6.11.2): batch pipeline runs on Dataproc stay in STARTING, because the driver's messaging service can't accept any connection.

Change

Build the compressor with explicit options:

new HttpContentCompressor(StandardCompressionOptions.gzip(), StandardCompressionOptions.deflate())
  • gzip and deflate are what the service offered before, so normal clients see no change.
  • The constructor exists in Netty 4.1.75 (this repo's version) and in 4.1.137.
  • Encodings that aren't offered (zstd, br, snappy) fall back to an uncompressed response.

Testing

  • HttpServerTest and HttpsServerTest pass: 102 tests, JDK 8, Netty 4.1.75. This includes the new testCompressResponseOnlyGzipOrDeflate:
    • Accept-Encoding: zstd, br, snappy returns an uncompressed response.
    • zstd, br, gzip returns gzip.
  • Standalone check with Netty 4.1.137.Final on the classpath:
zstd-jni on classpath new HttpContentCompressor() new HttpContentCompressor(gzip(), deflate())
1.4.9-1 NoSuchMethodError OK
1.5.2-1 OK OK
none OK OK
  • MutualAuthServerTest fails with Netty 4.1.137 (bad_certificate) with or without this change. That failure is unrelated to this PR; this repo still builds with Netty 4.1.75.

Release

CDAP needs a netty-http 1.8.0 release with this change. CDAP then bumps netty.http.version from 1.7.0 to 1.8.0, together with the Netty 4.1.137 upgrade.

NettyHttpService added HttpContentCompressor with the no-arg
constructor. In newer Netty 4.1.x releases that constructor also turns on
zstd (and brotli) whenever those libraries are on the classpath. When an
older zstd-jni comes first on the classpath (for example zstd-jni 1.4.9-1
from a Hadoop 3.3.x runtime), creating the compressor fails with
NoSuchMethodError: Zstd.defaultCompressionLevel(), and every channel
initialization in the service fails. CDAP hits this on Dataproc after
moving to Netty 4.1.137.Final.

Build the compressor with explicit gzip and deflate options instead, so
the available encodings no longer depend on what else is on the
classpath. gzip and deflate are what the service offered before.

Testing: HttpServerTest and HttpsServerTest (102 tests) pass, including
the new testCompressResponseOnlyGzipOrDeflate. A standalone check with
Netty 4.1.137.Final plus zstd-jni 1.4.9-1 on the classpath reproduces
the NoSuchMethodError with the old constructor and passes with the new
one.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request configures the NettyHttpService to explicitly support only gzip and deflate compression. This avoids potential NoSuchMethodError runtime failures in environments like Hadoop or Spark where older versions of zstd-jni might be loaded on the classpath. A new test case has been added to HttpServerTest to verify that unsupported encodings are ignored while gzip is correctly handled. There are no review comments, and the changes are well-implemented.

deepika-lalwani123 added a commit to cdapio/cdap that referenced this pull request Nov 17, 2026
netty-http 1.7.0 creates HttpContentCompressor with the no-arg
constructor. With Netty 4.1.137.Final (netty-bom bump), that constructor
turns on zstd whenever zstd-jni is on the classpath. On Dataproc, the
Hadoop 3.3.x runtime puts zstd-jni 1.4.9-1 first, which has no
Zstd.defaultCompressionLevel(). Every channel of the driver's
NettyHttpService then fails with NoSuchMethodError, and batch pipeline
runs stay in STARTING.

netty-http 1.8.0 (cdapio/netty-http#137) builds the compressor with
explicit gzip and deflate options, which is what CDAP offered before.

Must merge together with the Netty 4.1.137 bump (#16306). Needs the
netty-http 1.8.0 release first.

Testing (JDK 8, netty-http 1.8.0-SNAPSHOT built from #137):
- cdap-common: SpillableBodyConsumerTest, HttpsEnablerTest,
  RemoteTaskExecutorTest: 12 tests pass.
- cdap-gateway: NettyRouter*Test, AuditLogTest,
  ConfigBasedRequestBlockingTest: 51 tests pass.
- mvn dependency:tree: cdap-gateway resolves netty-http 1.8.0(-SNAPSHOT).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant