Add a converter for Alibaba Cloud Hologres Semantic View - #320
Open
TimothyHologres wants to merge 13 commits into
Open
Add a converter for Alibaba Cloud Hologres Semantic View#320TimothyHologres wants to merge 13 commits into
TimothyHologres wants to merge 13 commits into
Conversation
Hologres V5.0.0+ supports CREATE SEMANTIC VIEW, so an Ossie converter for it needs a way to label Hologres expressions. Without a dedicated token, PostgreSQL-specific syntax such as `col::text` would have to be mislabelled as ANSI_SQL, which misleads consumers about portability. Hologres is PostgreSQL wire- and dialect-compatible, so the validator maps the new token to sqlglot's postgres dialect and keeps SQL parsing enabled rather than skipping validation.
Sets up the converter sub-project and the cross-cutting helpers both directions need: the custom_extensions stash protocol, YAML 1.2 loading, SQL quoting, and a sqlglot expression layer. Two details are worth calling out because they are easy to get wrong: - sqlglot does not quote reserved words on output (it emits a bare `select` for the identifier `select`, a syntax error), so quote_identifier carries its own reserved-word set. It also quotes anything not already lower case, since PostgreSQL folds unquoted identifiers. - Metric aggregates are matched by exact sqlglot node type rather than isinstance(node, AggFunc): `stddev` and `array_agg` are AggFuncs too, but Hologres only accepts count/sum/avg/min/max. The live dependency group holds the database driver and is excluded from default-groups so CI never installs a driver it cannot use.
Emits SQL DDL rather than YAML because Hologres has no YAML import function -- the DDL is the only way to create a Semantic View. The conversion is fail-closed: Hologres enforces single-table definitions, a five-function aggregate whitelist, and REFERENCES targets that must be the referenced table's primary key. Each of those is checked offline with the offending field named, rather than emitting DDL the server will reject. Two mappings do real work. Ossie field expressions are unqualified bare columns while Hologres needs alias.column, so columns are qualified through the sqlglot AST -- a "prefix only bare identifiers" shortcut would leave `first || last` half-qualified and unresolvable. And Ossie metrics are model-level while Hologres namespaces them under an owning alias, so the owner is inferred from the expression's column references; `count(*)` has none, so it needs an explicit owner and fails closed without one, since guessing would silently change the number under a fan-out join. Also fixes reserved-word quoting inside expressions: sqlglot renders the identifier `user` bare, so a column named `user` on an alias named `order` produced the invalid `order.user`. Both golden .sql fixtures were executed against a real Hologres 5.0.0 instance and the resulting views queried, including the fan-out case where order revenue must not be inflated by joined order_items rows.
Consumes the model_yaml that Hologres publishes in hologres.hg_semantic_view_properties, rather than the ddl_text beside it, because it is already structured and needs no SQL statement parser. The Hologres table alias becomes the Ossie dataset name. That keeps every dimension, metric and relationship reference valid without rewriting, and makes Hologres' own `sum(o.amount)` already correct as a dataset-qualified Ossie metric expression. Expressions are labelled ANSI_SQL unless they actually need PostgreSQL syntax, decided by checking whether sqlglot renders the AST identically outside the postgres dialect. Labelling an ordinary SUM(x) as HOLOGRES would hide it from every other Ossie converter looking for a portable expression; `j -> 'k'` and the 1-based `arr[1]` genuinely do differ and are labelled HOLOGRES. Both model_yaml fixtures were read back from a real Hologres 5.0.0 instance after executing the exported DDL, so the full loop is verified end to end: importing them reproduces the original Ossie fixtures exactly, and the output passes the repository's own validation/validate.py.
Wires both directions up to a command line, following the export/import subcommand shape the other converters use. Two options exist because of Hologres constraints rather than preference: --drop-if-exists, since there is no CREATE OR REPLACE or ALTER SEMANTIC VIEW and a definition can only be changed by recreating it; and --metric-owner, to supply the owning table for a COUNT(*) metric whose expression names no column.
Also fixes a real grammar bug found by running the TPC-DS DDL against the instance: Hologres rejects a bare top-level operator in a definition expression. `a || b`, `a + 1` and `a::text` are all syntax errors in a DIMENSIONS clause, while `(a || b)`, `(a + 1)` and `cast(a as text)` are accepted -- even though the same operators are fine inside a function call's argument list. The manual's "operators are supported" does not mention this, so the exporter now adds the required parentheses and the importer strips them back off, keeping round trips from accumulating wrapping. The canonical examples/tpcds_semantic_model.yaml is referenced directly rather than copied into a fixture, so these tests cannot drift from it. Two of its five metrics are structurally inexpressible -- customer_lifetime_value and store_productivity are ratios spanning two datasets -- so the default is to fail closed naming the metric, with --skip-unsupported-metrics to convert the rest. The resulting 5-table snowflake was created on a real Hologres 5.0.0 instance and queried across all four dimension tables, including the computed customer_full_name dimension that needs both full qualification and the parentheses above.
A CREATE SEMANTIC VIEW statement can only really be validated by a Hologres server, so these tests create the view, query it, and read the model back. They prove three things the offline tests cannot: the generated DDL is accepted, the resulting view answers queries with the right numbers, and importing Hologres' own readback reproduces the model we started from. The fan-out case is checked with real rows rather than just parsed: each order has several order_items rows, and order revenue must stay 300/200 rather than being multiplied by them. Beijing's credit of 2500 likewise includes a customer with no orders at all, which is what aggregating before joining buys. Skipped unless HOLOGRES_HOST/USER/PASSWORD/DB are set, and the driver lives in a non-default dependency group, so CI stays hermetic. Credentials are read only from the environment and none are stored in the repository. Everything is created inside an ossie_hologres_it schema that is dropped on teardown. Also renames the fixtures' placeholder database from a real instance's name to `retail`, so no internal identifier ships upstream.
Mirrors the sibling converter workflows, reusing their pinned action SHAs and the 3.11-3.14 matrix. The path filter additionally watches examples/tpcds_semantic_model.yaml, validation/validate.py and core-spec/osi-schema.json. This converter's tests read those three files directly instead of copying them, so without listing them a change to the canonical example could break the golden fixture without CI noticing on the PR that caused it. `uv sync` deliberately omits the live group: those tests need a Hologres instance and skip themselves without one.
Records the mapping, and separates limitations that come from Hologres from choices the converter makes. The limitations section is the part worth reading before using this: single-table definitions, the five-function aggregate whitelist, no derived or ratio metrics, and REFERENCES targets that must be a primary key. Also notes what the fixtures actually are -- DDL executed against a real instance and that instance's own readback -- so a future maintainer knows they are verified artifacts rather than hand-written guesses, and that a live test guards them against Hologres changing its output shape.
Follow-up to adding the HOLOGRES token to the Dialect enum. Two places described the available dialects but not this one. expression_language.md is where someone writing a dialect-specific expression looks, and its Common Dialect Variations table already carries a PostgreSQL column, so the note points HOLOGRES at that column and names the cases that actually justify the tag rather than ANSI_SQL. docs/index.md listed the dialects as if exhaustively but was already missing BIGQUERY and MAQL, so the list is now completed against the schema enum instead of having HOLOGRES appended to a wrong list.
Adds it to the Existing Artifacts list under Tooling & Ecosystem Support, noting the asymmetry that distinguishes it from the other spokes: export emits CREATE SEMANTIC VIEW DDL because Hologres has no YAML import function, and import consumes the model_yaml Hologres publishes for every Semantic View.
Drops the per-expression dialect decision. Every expression is now labelled ANSI_SQL, which removes this converter's need for a vendor dialect token. The original motivation was to label `x::text` honestly, but that does not hold up: the portable spelling is always available and sqlglot already normalizes to `CAST(x AS TEXT)` in both directions, so `::text` never reached the output. The per-expression test that remained was also unsound. It compared sqlglot's postgres rendering against its default rendering, but sqlglot's default dialect is not ANSI SQL, so it passed `ILIKE` as portable while flagging the standard `SUBSTRING`, `EXTRACT` and `DATE_TRUNC` as vendor-specific. And over-labelling turned out to be the more damaging error: a converter that looks for an ANSI_SQL expression and finds none drops the field (see converters/databricks/ossie_to_metric_view.py), so tagging a `DATE_TRUNC` dimension as vendor-specific loses it silently, whereas an optimistic label at worst surfaces as a SQL error on the target engine. PostgreSQL-only syntax such as `j -> 'k'` is therefore labelled slightly optimistically, which the README states plainly. This matches the NVIDIA GSF converter's reasoning that anything else stays ANSI_SQL rather than being labelled inaccurately.
This reverts commit cdfa891 and its follow-up documentation in b76db78, restoring core-spec/, python/, validation/ and docs/ to their previous state. The converter no longer needs the token. Its stated motivation -- labelling `x::text` honestly -- does not survive scrutiny: Hologres is PostgreSQL-compatible but the portable `CAST(x AS TEXT)` spelling is always available, and sqlglot normalizes to it, so the shorthand never reached the output in the first place. Removing it keeps the contribution out of the specification-change process (7-day discussion, three binding +1) for a token that would have been used only for occasional PostgreSQL-only operators, and where over-labelling costs more than it saves: a converter that finds no ANSI_SQL expression drops the field. The Vendor name HOLOGRES used by the custom_extensions stash is unaffected -- Ossie's Vendor field is free-form and needs no spec change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
converters/hologres/, a bidirectional converter between Apache Ossie semantic models and Alibaba Cloud Hologres Semantic View (available since Hologres V5.0.0).A Hologres Semantic View is an in-database object declaring physical tables, their relationships, business dimensions and metrics. Queries reference dimensions and
AGG(metric)rather than repeating joins and aggregations, and the engine aggregates each metric within its own minimal join subtree so a one-to-many join cannot inflate a total.The two directions are asymmetric, because of how Hologres exposes definitions:
CREATE SEMANTIC VIEWSQL DDLmodel_yamlfromhologres.hg_semantic_view_propertiesExport emits DDL because Hologres has no YAML import function (
hg_create_semantic_view_from_yaml()does not exist); the documentation directs cross-environment rebuilds to the originalddl_text. Emitting themodel_yamlshape instead would be symmetrical with the other spokes but produce a file Hologres cannot consume.This PR contains no specification changes. An earlier revision proposed adding a
HOLOGREStoken to theDialectenum; that was dropped after the motivation failed to hold up — see #319 for the reasoning. Everything is read and written asANSI_SQL. TheHOLOGRESvendor name used for thecustom_extensionsstash needs no spec change, sinceVendoris free-form.Scope:
converters/hologres/**, a path-filtered CI workflow, and one row each inconverters/README.mdandROADMAP.md.Mapping
dataset+source+primary_keyTABLES (alias AS schema.table PRIMARY KEY (...))relationship(frommany →toone)RELATIONSHIPS (name AS from(cols) REFERENCES to(cols))dataset.fields[]DIMENSIONS (alias.name AS expr)metrics[]METRICS (owner.name AS agg_expr)description/ stringai_contextCOMMENTThree mappings do real work:
sum(o.amount)already correct as a dataset-qualified Ossie metric expression.alias.column, so columns are qualified through the sqlglot AST. A "prefix only bare identifiers" shortcut would leavec_first_name || ' ' || c_last_name— which is in this repo's own TPC-DS example — half-qualified and unresolvable.COUNT(*)has none, so it requires an explicit owner and fails closed without one: guessing would silently change the number under a fan-out join.One open question for reviewers
Whether a converter's export direction may emit SQL DDL rather than a data document. This would be the first such converter in the repo. I checked whether the CLI plugin contract constrains it and it appears not to —
convert.from_ossiedeclares onlyinvoke, with no output-format field, whileAcceptsis documented as "populated on ToOssie only" (cli/internal/plugin/plugin.go:26,44-46). Caveats and alternatives are discussed in #319. Since Snowflake Semantic Views are also created through DDL, this likely sets a precedent, so I am happy to follow the community's preference.Limitations, all imposed by Hologres
The converter fails closed and names the offending field rather than emitting DDL the server will reject:
count/sum/avg/min/maxREFERENCEStarget must be the target table's primary keyCREATE OR REPLACE/ALTER SEMANTIC VIEW;--drop-if-existsrecreatesUsing this repository's
examples/tpcds_semantic_model.yaml, 2 of its 5 metrics (customer_lifetime_value,store_productivity) are cross-dataset ratios with no Semantic View form. The default is to fail naming them;--skip-unsupported-metricsconverts the rest.One limitation is worth flagging because it is not in the Hologres documentation: a bare top-level operator in a definition is a syntax error.
a || b,a + 1anda::textare all rejected in aDIMENSIONSclause, while(a || b),(a + 1)andcast(a as text)are accepted — even though the same operators are fine inside a function call's argument list. This was found by executing the generated TPC-DS DDL against a live instance. The exporter adds the required parentheses; the importer strips them back off.Verification
Beyond the unit tests, the behaviour was verified end to end against a real Hologres 5.0.0 instance:
order_itemsrows per order, revenue stays 300/200 rather than being multiplied, and one city's credit total includes a customer with no orders at allThe checked-in fixtures are therefore not hand-written: each
*_semantic_view.sqlwas executed on that instance and each*_model_yaml.yamlis the instance's readback of it. A live test re-checks the readback against the committed fixture, so if Hologres changes its output shape the suite says so rather than quietly testing a stale format.Those live tests are skipped unless
HOLOGRES_*environment variables are set, and the database driver sits in a non-default dependency group, so CI stays hermetic. No credentials are in the repository.Disclosure
This implementation was produced with AI assistance. In line with the ASF Generative Tooling Guidance, I have reviewed the code and take full responsibility for it; the behaviour described above was verified by me against a live instance.
Related Issues
Relates to #319
Checklist
Specification
core-spec/and follow the existing structure — N/A: this PR makes no spec changes (an earlier revision did; it was dropped, see Add a converter for Alibaba Cloud Hologres Semantic View #319)Ontology
ontology/are consistent with spec changes — N/A, noneConverters
converters/is updated to reflect spec or ontology changesValidation
validation/are updated if the spec changed — N/A, spec unchangedvalidation/validate.pyin a subprocessDocumentation
docs/is updated to reflect any user-facing changes — N/A: no spec or CLI change outside this converter;converters/hologres/README.md,converters/README.mdandROADMAP.mdare updatedCONTRIBUTING.mdis updated if the contribution process changed — N/AExamples
examples/are added or updated for any new spec constructs or converter support — N/A: no new spec constructs. The tests read the existingexamples/tpcds_semantic_model.yamldirectly rather than copying it, so they cannot drift from it; the CI path filter watches that file for the same reason.Tests
pytest/ CI green) — 256 offline tests pass; the 9 live tests pass against a Hologres 5.0.0 instance and skip without credentialsCompliance
sqlglotandjsonschemaare already used byvalidation/andconverters/gsf.psycopg[binary]is new to the repo; it is used only by the live tests and lives in alivedependency group excluded fromdefault-groups, so CI and a normaluv syncnever install it. Happy to drop the live tests, or gate them differently, if adding it is not acceptable.