Skip to content

pg: don't leave a client half open after a query_timeout - #3729

Closed
1991santhu wants to merge 2 commits into
brianc:masterfrom
1991santhu:fix/query-timeout-leaves-client-unusable
Closed

pg: don't leave a client half open after a query_timeout#3729
1991santhu wants to merge 2 commits into
brianc:masterfrom
1991santhu:fix/query-timeout-leaves-client-unusable

Conversation

@1991santhu

Copy link
Copy Markdown

Fixes #3399.

When query_timeout fires on a query that is already on the wire, the timer reports the error, stubs out the callback and splices the query from _queryQueue. Nothing reconciles the connection itself. readyForQuery stays false, because it is only set back to true in _handleReadyForQuery, which needs a ReadyForQuery message from a server that may never send one. _pulseQueryQueue starts with if (this.readyForQuery === true), so from then on it does nothing and every later query on that client sits in _queryQueue and never reaches the wire. The splice only helps if the query was still queued. If it was the active one it lives in _activeQuery, which is left untouched.

With a pool it is worse, and that is what the reporter hit. The client still looks healthy, so release() puts it back on the idle list and the next caller gets a connection that cannot run anything.

The fix only applies when the timed out query is the active one. At that point the protocol state cannot be recovered from the client side, so it tears the socket down the same way end() already does:

if (this._getActiveQuery() || !this._queryable) {
  // if we have an active query we need to force a disconnect
  // on the socket - otherwise a hung query could block end forever
  this.connection.stream.destroy()
}

and marks the client unusable, so pg-pool discards it on release through its existing !client._queryable check rather than handing it to someone else. Setting _ending first keeps this from surfacing as an unexpected "Connection terminated unexpectedly" error, which matters because the pool detaches its own error listener while a client is checked out. A query that was still queued is unaffected.

I stopped there rather than trying to keep the connection alive. Reusing it would mean guessing whether the server will eventually answer the abandoned query, and a late reply would be misread as the result of whatever ran next.

Verified against Postgres 17 in Docker, macOS arm64, Node 26.

$ PGHOST=127.0.0.1 PGPORT=55433 PGUSER=postgres PGPASSWORD=secret PGDATABASE=postgres \
    node test/integration/client/query-timeout-tests.js
query-timeout-tests.js
  query_timeout does not leave a pooled client unusable ✔
  query_timeout marks the client unusable rather than queueing forever ✔

Reverting lib/client.js and keeping the tests, both fail:

query-timeout-tests.js
  query_timeout does not leave a pooled client unusable FAILED!

Error: Query read timeout
    at /private/tmp/npg/packages/pg-pool/index.js:45:11
    at async Test.action (test/integration/client/query-timeout-tests.js:20:18)

Worth noting the unfixed failure is not always a hang. In the pooled case the next caller gets the dead client's Query read timeout rather than waiting forever, which is the same root cause showing up as someone else's error.

There was no query_timeout integration test before this, which is likely why it went unnoticed.

When query_timeout fires on a query that is already on the wire, the timer
reports the error and drops the query, but nothing reconciles the connection.
readyForQuery stays false because it is only set again by a ReadyForQuery
message from the server, and _pulseQueryQueue does nothing while it is false.
So every later query on that client sits in _queryQueue and never goes out.

With a pool it is worse. The client looks fine, so release() puts it back on
the idle list and the next caller gets a connection that cannot run anything.

If the timed out query is the active one the protocol state cannot be
recovered, so tear the socket down the way end() already does for a hung query
and mark the client unusable. A pool then discards it on release instead of
handing it on. A query that was still queued is unaffected, the existing splice
already handles that case.

Adds an integration test for both the pooled and standalone paths. There was no
query_timeout integration test before.
The unit tests build a client over a MemoryStream, which has no destroy, so
calling it unconditionally broke every job in the matrix. Real sockets always
have it. Check before calling.
@charmander

Copy link
Copy Markdown
Collaborator

Behaviour change worth discussing, but we don’t need an AI proxy to do that.

@charmander charmander closed this Aug 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PG queries are not timing out on server going down

2 participants