pg: don't leave a client half open after a query_timeout - #3729
Closed
1991santhu wants to merge 2 commits into
Closed
pg: don't leave a client half open after a query_timeout#37291991santhu wants to merge 2 commits into
1991santhu wants to merge 2 commits into
Conversation
When query_timeout fires on a query that is already on the wire, the timer reports the error and drops the query, but nothing reconciles the connection. readyForQuery stays false because it is only set again by a ReadyForQuery message from the server, and _pulseQueryQueue does nothing while it is false. So every later query on that client sits in _queryQueue and never goes out. With a pool it is worse. The client looks fine, so release() puts it back on the idle list and the next caller gets a connection that cannot run anything. If the timed out query is the active one the protocol state cannot be recovered, so tear the socket down the way end() already does for a hung query and mark the client unusable. A pool then discards it on release instead of handing it on. A query that was still queued is unaffected, the existing splice already handles that case. Adds an integration test for both the pooled and standalone paths. There was no query_timeout integration test before.
The unit tests build a client over a MemoryStream, which has no destroy, so calling it unconditionally broke every job in the matrix. Real sockets always have it. Check before calling.
Collaborator
|
Behaviour change worth discussing, but we don’t need an AI proxy to do that. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3399.
When
query_timeoutfires on a query that is already on the wire, the timer reports the error, stubs out the callback and splices the query from_queryQueue. Nothing reconciles the connection itself.readyForQuerystaysfalse, because it is only set back totruein_handleReadyForQuery, which needs aReadyForQuerymessage from a server that may never send one._pulseQueryQueuestarts withif (this.readyForQuery === true), so from then on it does nothing and every later query on that client sits in_queryQueueand never reaches the wire. The splice only helps if the query was still queued. If it was the active one it lives in_activeQuery, which is left untouched.With a pool it is worse, and that is what the reporter hit. The client still looks healthy, so
release()puts it back on the idle list and the next caller gets a connection that cannot run anything.The fix only applies when the timed out query is the active one. At that point the protocol state cannot be recovered from the client side, so it tears the socket down the same way
end()already does:and marks the client unusable, so
pg-pooldiscards it on release through its existing!client._queryablecheck rather than handing it to someone else. Setting_endingfirst keeps this from surfacing as an unexpected "Connection terminated unexpectedly" error, which matters because the pool detaches its own error listener while a client is checked out. A query that was still queued is unaffected.I stopped there rather than trying to keep the connection alive. Reusing it would mean guessing whether the server will eventually answer the abandoned query, and a late reply would be misread as the result of whatever ran next.
Verified against Postgres 17 in Docker, macOS arm64, Node 26.
Reverting
lib/client.jsand keeping the tests, both fail:Worth noting the unfixed failure is not always a hang. In the pooled case the next caller gets the dead client's
Query read timeoutrather than waiting forever, which is the same root cause showing up as someone else's error.There was no
query_timeoutintegration test before this, which is likely why it went unnoticed.