Repository navigation
Conversation
…nzema#148) There was a leftover line from the MPL license appendix still at the top of the license file. Related to jeroenrinzema#94
This is a breaking change that changes ParseFn to take a Query struct instead of a query string. The reason is that the query string alone is not always enough to implement a handler that works correctly: 1. Clients might send parameter OIDs to the server and then encode the value as binary. psycopg does this for instance with integers and chooses the smallest integer type tat fits the integer. A handler cannot parse that binary correctly without knowing the type that the client sent to the server. 2. Using the Simple Query protocol it's valid to separate multiple queries from eachother with semicolons. With the extended protocol this is invalid, but you do have to parse the parameter placeholders. Supporting both semicolons and parameters at the same-time can be non-trivial to implement and the protocol does not require it, but without the handler knowing if the query was sent using the simple or extended protocol it's impossible to know which of the two the handler should be supporting. Note: instead of adding parameters for each of these to the ParseFn there's now a new Query struct that encapsulates them all. This will allow any future additions to not require a breaking change.
PostgreSQL 18 introduced protocol version 3.2, and the 19beta1 libpq defaults to announcing a "grease" protocol version of 3.9999 together with a `_pq_.test_protocol_negotiation` option to verify that servers correctly implement negotiation. Such clients abort the connection with `server incorrectly accepted "grease" protocol version 3.9999 without negotiation` unless the server responds appropriately. When a client requests a newer minor version than the 3.0 implemented here, or sends `_pq_.`-prefixed protocol extension options we don't recognize, the server now replies with a `NegotiateProtocolVersion` message reporting the version it downgraded to (`3.0`) and echoing back the unrecognized options. Parameters prefixed with `_pq_.` are protocol-level options rather than regular server parameters, so they are peeled off the StartupMessage instead of being stored as GUCs. Because we always negotiate down to 3.0 the rest of the connection keeps using the 3.0 wire format, so nothing else needs to change. The minor version is the only negotiable part: a differing major version cannot be downgraded, so it is rejected with a `FATAL` error before the rest of the StartupMessage is consumed.
…zema#144) * Report error when an unknown portal is executed * Support reporting transaction state in ReadyForQuery This adds a new TxStatus handler that can be used to send different transaction statuses than ServerIdle in the ReadyForQuery responses. I needed this functionality because without it the `commit()` methods of pgJDBC and psycopg are broken. Their `commit()` method is clever and does not send a `COMMIT` query when the transaction status reported by the server is Idle. * fix: reconcile merge conflict with updated ParseFn and mock client APIs The merge with main combined main's new handshake_test.go (written against the pre-Query ParseFn signature and the no-arg mock client helpers) with this branch's API changes, leaving the test files uncompilable: - Update ParseFn test handlers to the Query-based signature (jeroenrinzema#145). - Pass the expected status to mock Client.ReadyForQuery and a message regex to Client.Error, matching the helper signatures introduced here. --------- Co-authored-by: Jeroen Rinzema <jeroen@rinzema.dev>
* Report error when an unknown portal is executed * Support reporting transaction state in ReadyForQuery This adds a new TxStatus handler that can be used to send different transaction statuses than ServerIdle in the ReadyForQuery responses. I needed this functionality because without it the `commit()` methods of pgJDBC and psycopg are broken. Their `commit()` method is clever and does not send a `COMMIT` query when the transaction status reported by the server is Idle. * Add a handler for the Sync message --------- Co-authored-by: Jeroen Rinzema <jeroen@rinzema.dev>
Merge tag v0.20.0 (eef30e8) so release ancestry stays visible, and accept its wire.Query ParseFn, client parameter OIDs, protocol negotiation, TxStatus/Sync hooks, idle-ReadyForQuery portal cleanup, unknown-portal 34000 on synchronous Execute, RowWriter/WriteRows, 32 KiB frame batching and bounded encode scratch. Semantic resolutions: - Portal.execute decides batching from the client's MaxRows, captured before PortalSuspension(false) rewrites the internal limit to NoLimit. A finite legacy Execute is therefore never batchable, and Rows also falls back to Row whenever a legacy limit is set, so ErrRowLimitExceeded (54000) still fires at limit+1. - The per-column encode accumulator threads through v0.20's typemap and scratch parameters. Row and the batched Rows path both feed it, so the aggregated observer keeps its Execute-boundary publishing and v0.20's single context lookup per Rows call. - Keep DataWriter.Formats, the resumed parallel portal writer rebind, per-caller Shutdown deadlines and the WriteError-based cache error recovery (now passing ctx for TxStatus). - The suspended-portal regression resumes after a Flush, or after a Sync inside a transaction block, because v0.20 closes all portals when an idle ReadyForQuery is sent; it still fails without the parallel rebind. - Shopify tests adopt the wire.Query ParseFn signature.
PortalSuspension(false) converts a finite Execute to an internal NoLimit so the handler never parks. Cover that the client's MaxRows still decides batching: WriteRows under a finite legacy Execute writes one frame per row, completes at the exact limit and fails with 54000 at limit+1 (sync and parallel), while an unlimited Execute still batches. Also cover the Rows guard on its own, observer totals and single lookup through batched rows, and suspended-portal teardown by explicit Close and by idle Sync.
Reject repeated GSSENC requests and reuse the startup reader so pipelined follow-up packets remain available.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Upstream v0.20.0 (
eef30e8) addswire.Queryparse metadata, protocol version negotiation, TxStatus/Sync hooks, a 34000 error for unknown portals, and batched materialized rows with bounded encode scratch (upstream jeroenrinzema#151, which supersedes Shopify #16). This PR merges the tag as a real second parent on top of the batched-observer port (#18), keeping every Shopify contract from #15. #16 is intentionally not included.What runs (Shopify delta on top of v0.20)
Behavior and risks
ParseFntakeswire.Query{Query, ParameterOIDs, SimpleQuery}.PortalSuspension(false): a finite Execute is never batchable, andRowsalso falls back toRowwhenever a legacy limit is set. A naive merge (batchable: limit == NoLimitafter the internal rewrite) would letWriteRowsstream past MaxRows;rows_compatibility_test.gofails on that variant.PortalSuspension(false), so this does not affect it.unknown portalerror._pq_.options get NegotiateProtocolVersion; a different major version is rejected.WriteRowsbatches (≤32 KiB chunks);Rowstill writes one frame per row.EncodeObserver/DataWriter.Formats, resumed parallel-portal writer rebind, per-caller Shutdown deadlines, cache error recovery (WriteErrornow takes ctx), close behavior.Where to look
git show 446f849 --remerge-diff— the hand resolutions:cache.go(clientLimit → batchable),writer.go(legacy limit, Rows guard, observer stats plus scratch),row.go(tm/scratch plus observer arguments),options.go(both option sets).git diff v0.20.0 HEAD -- . ':!*_test.go'— the whole Shopify delta on v0.20 (~200 lines).rows_compatibility_test.go— MaxRows/batching and observer-through-Rows regressions.pipeline_compatibility_test.go— resumed-portal regression, now resuming after Flush or a transaction-block Sync; it still fails without the parallel rebind.handshake.go/handshake_gss_test.go— bounded GSSENC negotiation, pipelined startup preservation, and repeated-request rejection.Later merges from #18 apply cleanly.
885ed04brings in the publication-order fix.b35afd8brings in #18's cherry-pick of upstreameef30e8(golangci-lint 2.13.2); this branch already haseef30e8through the v0.20.0 merge, so the tree is identical to885ed04and the CI files do not appear in this PR's diff.Validation
go test -race ./...passes with default pgx 5.4.3, with QE's pgx 5.7.6 (alternate modfile), and with Go 1.26. Focused-count=10 -racepasses.golangci-lint2.13.2 (upstream's pinned version) reports 0 issues. The merge commit alone builds and passes.batchablewithout the Rows guard fails the WriteRows 54000/exact-limit tests on both sync and parallel paths; the naivebatchablealone fails the not-batched test; the guard alone is covered byTestRowsKeepsLegacyLimitWhenBatchable; removing the rebind fails both parallel suspension cases.testandlintpass onb35afd8. On2637206, the GSS tests pass under-race -count=10; the vulnerable recursive implementation fails the pipelined and repeated-GSS regressions. Fullgo test -race ./...and golangci-lint 2.13.2 pass. Nogo.modchanges. No merge or tag.Stack: #15 → #18 → this. QE requires v0.20.0 and will pin this head as
v0.15.3-shopify.0.20260929200001-2637206111f2(2637206) in shop/world #2109171.