feat(rpsql): add pkg/rpsql go-jet query layer for Redpanda SQL [AI-1675] - #2
Draft
c-julin wants to merge 4 commits into
Draft
feat(rpsql): add pkg/rpsql go-jet query layer for Redpanda SQL [AI-1675]#2c-julin wants to merge 4 commits into
c-julin wants to merge 4 commits into
Conversation
Adds pkg/rpsql: a type-safe query layer for Redpanda SQL (rpsql) built on go-jet, focused on reading external catalog (Iceberg/Kafka) tables via the catalog => table operator. - CatalogTable: wraps a catalog => table read in a CTE so only the raw seed is untyped; the outer query (projections, WHERE, GROUP BY, aggregates, composite access) stays fully type-safe. - Field/FieldString/FieldInt/...: composite (col).field member access. - ExecArgs/Query: run go-jet statements over a pgx Querier under QueryExecModeExec (required by rpsql's text-defaulted bind params). - gen + rpsql-jet-gen CLI: generate go-jet table/model code by introspecting a live rpsql via information_schema (go-jet's own generator relies on regclass, which rpsql lacks). Reuses the repo's existing go-jet/pgx deps; no go.mod changes. Unit + build-tagged integration tests validated against a live rpsql instance.
…ad) [AI-1675]
Adds a proto-first Redpanda SQL read-model generator to the plugin: annotate
a message with (gojet.v1.rpsql_read){catalog, table, cte_name} and the plugin
emits a type-safe go-jet read model bound to catalog => table (pkg/rpsql.
CatalogTable) in the normal buf generate pass — no DDL, migrations, or Docker.
- options.proto: RpsqlRead message + MessageOptions extension (field 77232).
- resolve_rpsql.go: maps scalar/enum/timestamp/duration -> typed columns and
recurses single nested messages into composite (col).field accessors
(incl. nested composites). Repeated/map fields are skipped: rpsql cannot yet
read array / composite-array columns (variant planner error, no UNNEST).
- emit_rpsql.go: hand-written emitter -> <Message>Read with CTE() + accessors.
- main.go: resolve/emit read models alongside the write path, bypassing FK
resolution and the DDL/Docker pass.
- e2e: RpsqlOrder fixture (scalars, enum, Timestamp, single + nested composite,
skipped repeated/map) + generated read model; SQL-shape test and a live-rpsql
integration test (validated against a neutral orders_seed_test fixture).
The read model now assembles proto messages directly instead of requiring a hand-defined scan struct: <Message>Read.Select(ctx, db, where...) returns []*<pb.Message>. It projects leaf columns (composite members via Field), scans into an unexported row, and builds the proto — enum text->value, timestamptz-> timestamppb, nested composite->nested message, honoring proto presence. Repeated fields stay zero-valued (rpsql array limitation). Validated against live rpsql (orders_seed_test) via the e2e integration test.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
pkg/rpsql— a type-safe query layer for Redpanda SQL (rpsql) built on go-jet. Primary driver: reading external catalog (Iceberg/Kafka) tables via the non-standardcatalog => tableoperator, which go-jet's typed table API cannot emit.Jira: AI-1675.
What it adds
CatalogTable— reads an external catalog table by wrappingSELECT * FROM catalog => tablein a CTE. Only that one-line seed is raw; the outer query (projections,WHERE,GROUP BY, aggregates, composite access) is fully type-safe.Field/FieldString/FieldInt/ … — composite (struct) member access(col).field, including nested composites. go-jet emitscol.field, which rpsql parses astable.column, so this is required.ExecArgs/Query— run a go-jet statement over a pgxQuerierunderQueryExecModeExec(required: rpsql describes untyped bind params astext, so pgx's default mode can't encode non-text params).gen+rpsql-jet-genCLI — generate go-jet table/model code by introspecting a live rpsql viainformation_schema(the stock go-jet generator relies onregclass, which rpsql lacks). Reusespostgres.Dialecttemplates so output shape matches the rest of the repo.Why go-jet + a CTE
postgresdialect builds correct read SQL.=>, which go-jet cannot emit (identifier quoting + an unexported serialize interface). The CTE confines=>to a raw seed while its plain-identifier name gives the outer query a typed, addressable table.rpsql read-side constraints (handled / documented)
No
ILIKE,DISTINCT ON,RETURNING,ON CONFLICT, or#> / #>>path operators. Arrow operators-> / ->>work on json/jsonb; composite access uses(col).field.Testing
go test ./pkg/rpsql/): SQL-serialization assertions forFieldandCatalogTable, no DB.-tags integration, gated on a reachable rpsql +RPSQL_TEST_*env): managed-table aggregate, composite field access, generator round-trip, and an external catalog read against a neutralorders_seed_testfixture. All pass against a live localdev rpsql.Notes
CatalogTable/Field/ generator split) before wiring into a consumer.go.modchanges — uses the repo's existing go-jet / pgx / lib-pq deps.