perf: speed up SQL auto-categorization

[nc-dirknilius]

Description

Motivation

SqlModel.is_breaking_change currently calls SQLGlot's general-purpose diff implementation to answer a much narrower question: whether a rendered query changed exclusively through added SELECT projections.

SQLGlot's tree diff computes candidate matchings across both ASTs, introducing quadratic work in its matching phases.
SQLMesh performs this work synchronously for each model during auto-categorization. We encountered the resulting
amplification in a project with more than 600 models: a broadly used macro change can require hundreds of models to
be categorized, repeatedly paying the superlinear cost for large rendered queries and causing sqlmesh plan to spend
more than an hour in categorization without completing; the plan was cancelled at that point.

The existing _additive_projection_change fallback handles cases where SQLGlot emits spurious Move or Update edits, such as repeated cast types. However, the fallback runs only after the expensive tree diff has completed, so it does not address the performance problem.

What changed

This replaces the general tree diff and its fallback with a specialized AST comparison:

• Walk both query ASTs in lockstep.
• Require identical node types, scalar arguments, and non-projection child lists.
• Treat each previous SELECT projection list as an ordered subsequence of the current list.
• Allow new projections before, between, or after existing projections.
• Preserve duplicate projections and structurally similar expressions such as repeated CAST types.
• Apply the same comparison recursively to CTEs, EXISTS queries, UNIONs, and other nested query structures.
• Return the conservative, undetermined result for every change that cannot be proven to be projection-only.

The comparison is linear in the traversed AST and projection lists rather than performing general candidate matching across the trees.

Safety and behavior

The specialized comparison preserves SQLMesh's conservative categorization rules:

• Removed, replaced, or reordered projections remain undetermined.
• Changes to FROM, WHERE, GROUP BY, ORDER BY, DISTINCT, or other query structure remain undetermined.
• Directly projected UDTFs remain undetermined because they can change row cardinality.
• A UDTF is accepted only when its nearest Subquery ancestor is contained within the added projection.
• Mid-list additions remain undetermined when GROUP BY or ORDER BY uses ordinal references, because inserting a projection can shift the referenced output.
• Changes inside an existing scalar-subquery projection remain undetermined.
• Any failed structural comparison returns None, retaining SQLMesh's conservative fallback behavior.

There are no public API or configuration changes.

Performance

Measured on main at e075200 using Python 3.13.13 and SQLGlot 30.8.0 on macOS ARM64.

Both ASTs were parsed before timing. Each benchmark copied the previous query and added one scalar projection. The measurements time only the existing sqlglot.diff call and the specialized comparison.

Input	AST nodes	sqlglot.diff	Specialized comparison	Speedup
Synthetic, 100 projections	1,104	87.9 ms	1.4 ms	62×
Synthetic, 500 projections	5,504	2.590 s	7.4 ms	351×
Synthetic, 1,000 projections	11,004	11.126 s	14.0 ms	797×
Real-world Redshift model	16,157	12.442 s	20.6 ms	605×

These are single-process wall-clock measurements intended to show the order-of-magnitude difference; parsing and rendering are excluded from both sides.

A representative breaking change on the same large model retained the conservative result while dropping comparison time from approximately 12.1 seconds to 20 milliseconds.

Test Plan

Added coverage for:

• Existing projection additions, removals, replacements, and reordering.
• Repeated CAST types.
• A new projection identical to an existing projection.
• Direct and aliased UDTFs.
• UDTFs inside projection subqueries.
• UDTFs inside derived-table projections.
• CTE projection additions.
• Projection additions inside EXISTS.
• Changes inside existing scalar-subquery projections.
• Projection additions across UNION branches.
• Nested ordinal ORDER BY safeguards.

Validation performed:

• make style
  • Passed: ruff, ruff-format, mypy, and migration validation.
• python -m pytest tests/core/test_snapshot.py -k 'categorize_change_sql' -q
  • 8 passed.
• Differential comparison of 500 generated projection and non-projection mutations against the current implementation.
  • No categorization mismatches.
• make fast-test
  • The exact target was run twice. Its parallel phase consistently completed 2,576 tests successfully, skipped 4,
    and reported one order-dependent error in
    tests/dbt/cli/test_selectors.py::test_select_by_dbt_names[dbt_select6-expected6].
  • The error occurs when another DBT test's temporary namespace package named macros remains in the same xdist
    worker and dbt-common attempts to inspect it. No changed code is present in the traceback.
  • Running all 14 test_select_by_dbt_names cases serially passes.
  • Running the parallel fast suite without that parametrized test passes: 2,563 passed and 4 skipped.
  • The phases that follow the parallel phase in make fast-test also pass when invoked separately:
    3 isolated, 1 registry_isolation, and 159 dialect_isolated tests.

Checklist

• I have run make style and fixed any issues.
• I have added tests for my changes.
• All existing tests pass (make fast-test).
  • All tests pass when the order-dependent DBT selector case is separated from the parallel run, as detailed above.
• My commits are signed off (git commit -s) per the DCO.

[mday-io]

Nice find and fix. Confirmed significant speedups myself (160x on a 400 projection model). Looks like all of the categorizations are safe and / or safer (in the case of ORDER BY 2 being NON_BREAKING and now classified as undetermined).

Only one additional easy win - the ORDER BY 2 fix won't kick in when ORDER BY sits after a UNION. Since you've already modified that area, worth a patch.

Some other nits listed in-line

[nc-dirknilius]

Nice find and fix. Confirmed significant speedups myself (160x on a 400 projection model). Looks like all of the categorizations are safe and / or safer (in the case of ORDER BY 2 being NON_BREAKING and now classified as undetermined).

Only one additional easy win - the ORDER BY 2 fix won't kick in when ORDER BY sits after a UNION. Since you've already modified that area, worth a patch.

Some other nits listed in-line

Hey @mday-io,

Thanks for looking into this and your comments. They are all valid. I made appropriate changes. Especially the blind spot when UNION is used is now covered.

Please take a second look.

[mday-io]

@nc-dirknilius hey the style test failed. May you run Make Style again please?

[nc-dirknilius]

@nc-dirknilius hey the style test failed. May you run Make Style again please?

Hey @mday-io,

It failed on files not touched in this PR. This error was merged in from main.

[mday-io]

@nc-dirknilius ah, thanks. Will fix.