Skip to content

Reconcile upstream v0.20.0 on the batched-observer fork - #19

Draft
seifghazi wants to merge 13 commits into
sg/qe-wire-observer-batching-v019-20260928from
sg/qe-wire-v020-20260928
Draft

seifghazi wants to merge 13 commits into
sg/qe-wire-observer-batching-v019-20260928from
sg/qe-wire-v020-20260928

Conversation

@seifghazi

@seifghazi seifghazi commented Sep 28, 2026 •

Copy link
Copy Markdown

Why

Upstream v0.20.0 (eef30e8) adds wire.Query parse metadata, protocol version negotiation, TxStatus/Sync hooks, a 34000 error for unknown portals, and batched materialized rows with bounded encode scratch (upstream jeroenrinzema#151, which supersedes Shopify #16). This PR merges the tag as a real second parent on top of the batched-observer port (#18), keeping every Shopify contract from #15. #16 is intentionally not included.

What runs (Shopify delta on top of v0.20)

Execute(portal, MaxRows=N)
 └─ Portal.execute                                              cache.go
     clientLimit := N
     PortalSuspension(false)?  dw.limit = N; internal limit = NoLimit   (handler never parks)
     dw.batchable = clientLimit == NoLimit          ◀ client MaxRows, not the rewritten limit
     handler ─▶ WriteRows(w, rows)
                 └─ dw.Rows                                     writer.go
                     !batchable || dw.limit != NoLimit ─▶ dw.Row per row   (54000 at N+1)
                     else StartBatch(32 KiB) ─▶ columns.write(tm, scratch, observer=nil, encodeStats)
     encode stats flushed at PortalSuspended / Complete / error / teardown   (unchanged from #18)

Sync ─▶ handleSync ─▶ SyncConn? ─▶ readyForQuery: TxStatus idle ⇒ Portals.Close()

Behavior and risks

  • Breaking API: ParseFn takes wire.Query{Query, ParameterOIDs, SimpleQuery}.
  • MaxRows with PortalSuspension(false): a finite Execute is never batchable, and Rows also falls back to Row whenever a legacy limit is set. A naive merge (batchable: limit == NoLimit after the internal rewrite) would let WriteRows stream past MaxRows; rows_compatibility_test.go fails on that variant.
  • Portal lifetime: an idle ReadyForQuery now closes every portal, so a suspended portal can only be resumed after Flush or inside a TxStatus transaction block. QE uses PortalSuspension(false), so this does not affect it.
  • Execute on a missing portal returns 34000 on the synchronous path. The parallel pipeline path still returns upstream's plain unknown portal error.
  • Protocol negotiation: unsupported minor versions or _pq_. options get NegotiateProtocolVersion; a different major version is rejected.
  • GSS encryption negotiation is bounded to one request and reuses the existing reader. Repeated GSSENC is rejected before authentication instead of recursively growing the connection goroutine stack; pipelined GSSENC→Startup and GSSENC→SSLRequest→Startup remain valid.
  • Only unlimited Execute + WriteRows batches (≤32 KiB chunks); Row still writes one frame per row.
  • Kept: EncodeObserver/DataWriter.Formats, resumed parallel-portal writer rebind, per-caller Shutdown deadlines, cache error recovery (WriteError now takes ctx), close behavior.

Where to look

  1. git show 446f849 --remerge-diff — the hand resolutions: cache.go (clientLimit → batchable), writer.go (legacy limit, Rows guard, observer stats plus scratch), row.go (tm/scratch plus observer arguments), options.go (both option sets).
  2. git diff v0.20.0 HEAD -- . ':!*_test.go' — the whole Shopify delta on v0.20 (~200 lines).
  3. rows_compatibility_test.go — MaxRows/batching and observer-through-Rows regressions.
  4. pipeline_compatibility_test.go — resumed-portal regression, now resuming after Flush or a transaction-block Sync; it still fails without the parallel rebind.
  5. handshake.go / handshake_gss_test.go — bounded GSSENC negotiation, pipelined startup preservation, and repeated-request rejection.

Later merges from #18 apply cleanly. 885ed04 brings in the publication-order fix. b35afd8 brings in #18's cherry-pick of upstream eef30e8 (golangci-lint 2.13.2); this branch already has eef30e8 through the v0.20.0 merge, so the tree is identical to 885ed04 and the CI files do not appear in this PR's diff.

Validation

  • go test -race ./... passes with default pgx 5.4.3, with QE's pgx 5.7.6 (alternate modfile), and with Go 1.26. Focused -count=10 -race passes. golangci-lint 2.13.2 (upstream's pinned version) reports 0 issues. The merge commit alone builds and passes.
  • Negative controls: a naive batchable without the Rows guard fails the WriteRows 54000/exact-limit tests on both sync and parallel paths; the naive batchable alone fails the not-batched test; the guard alone is covered by TestRowsKeepsLegacyLimitWhenBatchable; removing the rebind fails both parallel suspension cases.
  • GitHub Actions test and lint pass on b35afd8. On 2637206, the GSS tests pass under -race -count=10; the vulnerable recursive implementation fails the pipelined and repeated-GSS regressions. Full go test -race ./... and golangci-lint 2.13.2 pass. No go.mod changes. No merge or tag.

Stack: #15 → #18 → this. QE requires v0.20.0 and will pin this head as v0.15.3-shopify.0.20260929200001-2637206111f2 (2637206) in shop/world #2109171.

JelteF and others added 11 commits July 19, 2026 20:49
…nzema#148)

There was a leftover line from the MPL license appendix still at the top
of the license file.

Related to jeroenrinzema#94
This is a breaking change that changes ParseFn to take a Query struct
instead of a query string. The reason is that the query string alone is
not always enough to implement a handler that works correctly:

1. Clients might send parameter OIDs to the server and then encode the
   value as binary. psycopg does this for instance with integers and
   chooses the smallest integer type tat fits the integer. A handler
   cannot parse that binary correctly without knowing the type that the
   client sent to the server.
2. Using the Simple Query protocol it's valid to separate multiple
   queries from eachother with semicolons. With the extended protocol
   this is invalid, but you do have to parse the parameter
   placeholders. Supporting both semicolons and parameters at the
   same-time can be non-trivial to implement and the protocol does not
   require it, but without the handler knowing if the query was sent
   using the simple or extended protocol it's impossible to know which
   of the two the handler should be supporting.

Note: instead of adding parameters for each of these to the ParseFn
there's now a new Query struct that encapsulates them all. This will
allow any future additions to not require a breaking change.
PostgreSQL 18 introduced protocol version 3.2, and the 19beta1 libpq
defaults to announcing a "grease" protocol version of 3.9999 together
with a `_pq_.test_protocol_negotiation` option to verify that servers
correctly implement negotiation. Such clients abort the connection with
`server incorrectly accepted "grease" protocol version 3.9999 without
negotiation` unless the server responds appropriately.

When a client requests a newer minor version than the 3.0 implemented
here, or sends `_pq_.`-prefixed protocol extension options we don't
recognize, the server now replies with a `NegotiateProtocolVersion`
message reporting the version it downgraded to (`3.0`) and echoing back
the unrecognized options. Parameters prefixed with `_pq_.` are
protocol-level options rather than regular server parameters, so they
are peeled off the StartupMessage instead of being stored as GUCs.
Because we always negotiate down to 3.0 the rest of the connection keeps
using the 3.0 wire format, so nothing else needs to change.

The minor version is the only negotiable part: a differing major version
cannot be downgraded, so it is rejected with a `FATAL` error before the
rest of the StartupMessage is consumed.
…zema#144)

* Report error when an unknown portal is executed

* Support reporting transaction state in ReadyForQuery

This adds a new TxStatus handler that can be used to send different
transaction statuses than ServerIdle in the ReadyForQuery responses.

I needed this functionality because without it the `commit()` methods
of pgJDBC and psycopg are broken. Their `commit()` method is clever and
does not send a `COMMIT` query when the transaction status reported by
the server is Idle.

* fix: reconcile merge conflict with updated ParseFn and mock client APIs

The merge with main combined main's new handshake_test.go (written against
the pre-Query ParseFn signature and the no-arg mock client helpers) with this
branch's API changes, leaving the test files uncompilable:

- Update ParseFn test handlers to the Query-based signature (jeroenrinzema#145).
- Pass the expected status to mock Client.ReadyForQuery and a message regex
  to Client.Error, matching the helper signatures introduced here.

---------

Co-authored-by: Jeroen Rinzema <jeroen@rinzema.dev>
* Report error when an unknown portal is executed

* Support reporting transaction state in ReadyForQuery

This adds a new TxStatus handler that can be used to send different
transaction statuses than ServerIdle in the ReadyForQuery responses.

I needed this functionality because without it the `commit()` methods
of pgJDBC and psycopg are broken. Their `commit()` method is clever and
does not send a `COMMIT` query when the transaction status reported by
the server is Idle.

* Add a handler for the Sync message

---------

Co-authored-by: Jeroen Rinzema <jeroen@rinzema.dev>
Merge tag v0.20.0 (eef30e8) so release ancestry stays visible, and accept
its wire.Query ParseFn, client parameter OIDs, protocol negotiation,
TxStatus/Sync hooks, idle-ReadyForQuery portal cleanup, unknown-portal
34000 on synchronous Execute, RowWriter/WriteRows, 32 KiB frame batching
and bounded encode scratch.

Semantic resolutions:
- Portal.execute decides batching from the client's MaxRows, captured
  before PortalSuspension(false) rewrites the internal limit to NoLimit.
  A finite legacy Execute is therefore never batchable, and Rows also
  falls back to Row whenever a legacy limit is set, so ErrRowLimitExceeded
  (54000) still fires at limit+1.
- The per-column encode accumulator threads through v0.20's typemap and
  scratch parameters. Row and the batched Rows path both feed it, so the
  aggregated observer keeps its Execute-boundary publishing and v0.20's
  single context lookup per Rows call.
- Keep DataWriter.Formats, the resumed parallel portal writer rebind,
  per-caller Shutdown deadlines and the WriteError-based cache error
  recovery (now passing ctx for TxStatus).
- The suspended-portal regression resumes after a Flush, or after a Sync
  inside a transaction block, because v0.20 closes all portals when an
  idle ReadyForQuery is sent; it still fails without the parallel rebind.
- Shopify tests adopt the wire.Query ParseFn signature.
PortalSuspension(false) converts a finite Execute to an internal NoLimit
so the handler never parks. Cover that the client's MaxRows still decides
batching: WriteRows under a finite legacy Execute writes one frame per row,
completes at the exact limit and fails with 54000 at limit+1 (sync and
parallel), while an unlimited Execute still batches. Also cover the Rows
guard on its own, observer totals and single lookup through batched rows,
and suspended-portal teardown by explicit Close and by idle Sync.
Brings #18's backport of upstream eef30e8 (golangci-lint v2.13.2 and the
Go-version/Makefile cache key) into the v0.20 branch. This branch already
contains eef30e8 itself through the v0.20.0 merge, so both sides made the
same change and the resulting tree is identical to the previous head.
Reject repeated GSSENC requests and reuse the startup reader so pipelined follow-up packets remain available.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants