Skip to content

websocket: connection attempts on the input connector now honor context cancellation - #483

Open
Leward wants to merge 3 commits into
mainfrom
VM-75-f-03-websocket-dial-context
Open

websocket: connection attempts on the input connector now honor context cancellation#483
Leward wants to merge 3 commits into
mainfrom
VM-75-f-03-websocket-dial-context

Conversation

@Leward

@Leward Leward commented Aug 27, 2026

Copy link
Copy Markdown

What
Ensures connection attempts on the websocket input connector properly honor context cancellation and deadlines during the HTTP upgrade/handshake phase.

Why
Previously, Gorilla WebSocket only honored context cancellation during TCP/TLS handshakes, causing hangs against unresponsive endpoints for up to the default 45-second handshake timeout and delaying graceful shutdown.

Key Changes:

  • Added a dialContext helper using context.AfterFunc on net.Conn to interrupt hanging HTTP upgrade exchanges upon context cancellation.
  • Concealed ctx.Deadline() via hiddenDeadlineContext to avoid competing socket timers and ensure deterministic context error returns.
  • Updated websocket input connector and added test cases covering pre-dial cancellation, mid-handshake cancellation, and deadline timeouts.

Out of Scope:

  • ReadMessage() will be handled separately
  • similar fix forwebsocket output will be handled in a follow-up PR

Refs:

  • CON-545
  • VM-75

@CLAassistant

CLAassistant commented Aug 27, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

if client, res, err = dialer.Dial(w.urlStr, headers); err != nil {
return nil, err
}
} else if client, res, err = dialer.Dial(w.urlStr, headers); err != nil {

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The branching was redundant and has been simplified

Comment thread internal/impl/io/websocket.go Outdated
@@ -0,0 +1,72 @@
// Copyright 2025 Redpanda Data, Inc.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file has been introduced because in a follow-up change the websocket output will need to use those functions.

Comment thread internal/impl/io/websocket.go Outdated
@claude

claude Bot commented Aug 27, 2026

Copy link
Copy Markdown

Commits

  1. refactor: update/add comments and simplify cleanup logic — invalid prefix. refactor is not a known system/area, and the message is not chore: nor a sentence-case repo-wide message. It also has an empty body, is vague ("update/add comments and simplify cleanup logic"), and mixes a CHANGELOG.md edit with behaviour-affecting changes to internal/impl/io/input_websocket.go. This is a fixup of 244695e and should be squashed into it.
  2. refactor(websocket): move the shared dial and test helpers to their own files — same prefix problem: refactor is not a system name, so refactor(websocket): does not match system(subsystem):. Should be websocket: move the shared dial and test helpers to their own files.
  3. websocket: add a failing test for a dial that ignores its context — the body states "This commit is expected to fail CI." A knowingly-red commit breaks git bisect and green-history guarantees; the test and the fix should land together (or the test should be skipped/marked until the fix commit).
  4. CHANGELOG.md (documentation) is edited inside the code commits (244695e, f52c31b) rather than in its own commit, which the multi-commit granularity rule asks for.

Review

The approach is sound and well-reasoned: honouring the context through the HTTP upgrade phase genuinely requires poking the raw net.Conn deadline, and hiding ctx.Deadline() to avoid a competing socket timer is the right call. Cleanup on the success path (stop(), then close the conn if ctx.Err() != nil) is correct, and res.Body is still closed by the existing defer in getConn. Test structure follows the project patterns (table-driven, t.Context(), require.ErrorIs, helpers with t.Helper()/t.Cleanup).

One issue found:

  1. internal/impl/io/websocket.go:51 — the context.AfterFunc watcher is registered inside NetDialContext, but gorilla applies c.SetDeadline(now + HandshakeTimeout) to the same conn immediately after that closure returns (DefaultDialer.HandshakeTimeout is 45s, and gorilla adds it to the context before its ctx.Deadline() wrap). If the context is done in that window, the watcher fires once, its past deadline is overwritten by gorilla's 45s deadline, and the upgrade read blocks for the full handshake timeout — the exact hang this PR fixes. See the inline comment for the suggested fix.

Comment thread internal/impl/io/websocket.go Outdated
@Leward
Leward force-pushed the VM-75-f-03-websocket-dial-context branch from bf682a6 to 60ae0e1 Compare August 28, 2026 04:23
@claude

claude Bot commented Aug 28, 2026

Copy link
Copy Markdown

Commits
LGTM

Review
Reviewed the new dialContext helper, the getConn call site, and the added tests. The cancellation logic holds up: NetDialContext is invoked synchronously by gorilla (including through the proxy/TLS wrappers), so the stop capture is race-free; the stop()-before-ctx.Err() ordering correctly closes a connection that raced to success against cancellation; and hiddenDeadlineContext still propagates Done() into gorilla’s HandshakeTimeout-derived context, so the TCP/TLS dial keeps cancelling and the 45s socket deadline is still installed. The error path returns res so the existing defer res.Body.Close() in getConn still applies. License headers, file placement, and the table-driven tests with require match project conventions, and the new behavior is covered by TestWebsocketConnectContextDone.

LGTM

@redpanda-data redpanda-data deleted a comment from claude Bot Aug 28, 2026
Comment thread internal/impl/io/websocket.go Outdated
Comment thread internal/impl/io/websocket_test.go Outdated
Leward and others added 3 commits September 1, 2026 14:01
getConn received a context but discarded it: dialer.Dial always used
context.Background, so a peer that accepted the TCP connection and never
completed the HTTP upgrade handshake held the input for gorilla's 45s
default handshake timeout. AsyncReader has no other way to stop the reader
than the context it passes to Connect, so this could outlast the
Kubernetes default terminationGracePeriodSeconds (30s) and turn a graceful
shutdown into a kill.

Add a dialContext helper that keeps a handle on the raw connection via
NetDialContext and uses context.AfterFunc to close it once the outer
context is done, waking a handshake read that gorilla's own timeout
wouldn't catch in time. The resulting error is mapped back to ctx.Err so
callers can tell a shutdown from a flaky peer. The helper hides the
context's deadline from gorilla so gorilla's own connection deadline
doesn't race the watcher.

dialContext and its test helpers are free functions with no reader state,
so they live in shared websocket.go/websocket_test.go files rather than
the input-only ones, ready for the output side to reuse.
Co-authored-by: Joseph Woodward <joseph.woodward@redpanda.com>
Co-authored-by: Joseph Woodward <joseph.woodward@redpanda.com>
@Leward
Leward force-pushed the VM-75-f-03-websocket-dial-context branch from c742850 to f47750c Compare September 1, 2026 18:01
@claude

claude Bot commented Sep 1, 2026

Copy link
Copy Markdown

Commits

  1. Unsquashed fixups. Commits e55e513 and f47750c share the identical message Set copyright header to current year and each only flip Copyright 2025Copyright 2026 on one line of internal/impl/io/websocket.go / internal/impl/io/websocket_test.go. Both files are introduced by this PR in 4ce00ea, so these are fixups on the PR's own first commit — they should be squashed into websocket: make the input dial honor context cancellation rather than shipped as two separate commits with the same message.
  2. Message format. Set copyright header to current year uses the sentence-case plain form, which is reserved for repo-wide changes not scoped to one system. These commits touch only internal/impl/io/websocket*.go, so if they are kept as standalone commits they should be scoped (websocket: set copyright header to current year) or classified as housekeeping (chore: ...).

4ce00ea (websocket: make the input dial honor context cancellation) is correctly scoped, imperative, self-contained, and its body explains the why. No issue there.

Review

The change adds a dialContext helper that attaches a context.AfterFunc watcher to the raw net.Conn via NetDialContext, so a peer that completes the TCP handshake and then goes silent no longer pins the input for gorilla's 45s HandshakeTimeout. I traced the lifecycle of the watcher (stop() is always called, and gorilla invokes NetDialContext synchronously so there is no race on the stop variable), the cancellation propagation through hiddenDeadlineContext (Done() is delegated, so context.WithTimeout still registers as a child and cancellation propagates), the already-cancelled-context path, and the cancel-between-dial-return-and-stop() window — in every case the connection is closed and ctx.Err() is returned. Response bodies are still drained by the existing defer in getConn. The table-driven test covers pre-dial cancellation, mid-handshake cancellation, and deadline expiry, with requireAccepted guarding against the deadline case passing via an earlier failure; t.Cleanup ordering (LIFO) cancels the context before the listener's connections are closed, so the helpers are sound.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants