fix(sof): deterministic SQL view ordering with one final preview LIMIT

[Gordex2014]

Closes #1623.

Outcome

SQL-backed ViewDefinition runs (PostgreSQL and SQLite in-DB runners) now have a total, deterministic order. A preview's _limit is applied as one final SQL LIMIT after every filter, expansion, union and recursion, and returns exactly the first N rows of the unlimited result — for flat, expanded (forEach/forEachOrNull, nested, chained, cartesian), unionAll, repeat and indexed shapes.

Visual explanation

https://vanish.gordex.dev/s/brave-atlas-pt/ — before/after preview query path, ordering contract, filter placement, benchmark results and trade-off. Temporary link, expires 2026-10-07T22:32:06Z.

Ordering contract (all keys ascending)

Shape	Order
Ordinary / expanded	resource last_updated, resource id, every expansion occurrence ordinal (a forEachOrNull miss sorts as -1)
unionAll	first visible column (unchanged primary key, explicit NULL placement: PostgreSQL NULLS LAST, SQLite NULLS FIRST), resource keys, shared enclosing ordinals, branch number, branch-local identity
repeat	first visible column (same as above), resource keys, collision-free pre-order traversal identity, post-repeat ordinals

Tie-breakers travel as hidden, positionally aliased columns behind an outer projection of the visible columns only — no helper columns reach results and no user column name is reserved. The contract is documented for users in the book ("Result Order and _limit in HFS") and READMEs, and for developers in rustdoc (SofRunner and the sof::emit module).

What changed

• Structural runtime filters: _since, Patient and Group filters are placed in every union branch and every repeat seed (previously spliced as text before the last ORDER BY, which filtered only the last union branch and produced invalid SQL for some repeat views). SQL text heuristics (inject_before_order_by, strip_trailing_order_by) are removed.
• PostgreSQL LIMIT pushdown for every shape (OutputLimitStrategy removed). Limits above i64::MAX keep the client-side cap; intrinsic LIMIT 1 OFFSET N for indexed selections is unchanged.
• repeat: collision-free traversal identity (PostgreSQL bigint[], SQLite fixed-width binary-sorted text). %rowIndex under repeat is ranked over the full identity before post-repeat joins, fixing collisions (multi-answer children, repeated paths, multiple seeds) and PostgreSQL multi-path / post-repeat %rowIndex, which previously failed to prepare.
• Indexed forEach (x[N]): deterministic pick, membership filter honoured in every scope, trailing where() applied to the selected occurrence, %rowIndex = 0 as in the evaluator; x.where()[N] is rejected (422) instead of returning wrong rows.
• %rowIndex inside where() criteria reads the enclosing iteration's index, matching the evaluator (SQL targets only).
• Declared columns for SQL-runner output: buffered JSON/CSV/Parquet/Arrow/fhir and CSV/Parquet export shards use the ViewDefinition's declared layout, so a PostgreSQL column that is NULL in the first row (or first row of a shard) is no longer dropped. Raw NDJSON objects are unchanged.
• unionAll views as SQLQuery/SQLView dependencies now materialize (deduplicated dependency-table columns) instead of failing with a duplicate-column 422.

Compatibility

• Rows that previously tied may come back in a different, now fixed, order in unlimited runs, $sql-export shards and SQLQuery dependency insertion order. Rows with different primary keys keep their relative order.
• Intentional content corrections: %rowIndex values listed above; indexed forEach no longer emits rows for absent/rejected selections; runtime filters now apply to every union branch/seed; SQL-runner tabular formats carry all declared columns.
• PostgreSQL and SQLite are each deterministic but not guaranteed to agree with each other (collation, NULL placement).
• Unchanged: MongoDB (pipelines verified identical to the previous behaviour by golden tests) and in-process runners; dependency row cap (sof(sql-query): a depends-on ViewDefinition is materialized in full before the query's WHERE, so any query over a view past HFS_SOF_SQLQUERY_MAX_SOURCE_ROWS_PER_VD fails with row limit exceeded #1473) behaviour; export and dependency runs remain unlimited.

Performance (PostgreSQL 16, 20,000 Patients / 2,000 QuestionnaireResponses; base vs this branch, same data)

Preview _limit=50, server rows produced and execution (EXPLAIN ANALYZE of the executed statement):

View	Rows produced by PG	Execution
nullable forEachOrNull	152 → 50	0.3 → 0.4 ms
nested expansion	120,000 → 50 (10 resources read instead of 20,000)	224.7 → 0.3 ms
unionAll	100,000 → 50 (top-N heapsort)	141.7 → 50.0 ms
repeat (multi-path)	140,000 → 50 (top-N; previously external merge sort on disk)	345.0 → 437.3 ms

End-to-end request, preview _limit=50 (median):

View	Warm	Cold (PG restarted, OS page cache dropped, 0 KiB of the DB resident)
nullable	1.8 → 1.4 ms	6.8 → 6.9 ms
expansion	88.2 → 1.6 ms	179.9 → 6.9 ms
unionAll	111.6 → 34.3 ms	187.9 → 102.7 ms
repeat	240.3 → 313.7 ms	343.6 → 404.3 ms
flat (control)	1.7 → 1.4 ms	—

In the web UI ViewDefinitions playground (50-row preview) the displayed rows are identical before/after; displayed time drops from 84 → 3 ms (expansion) and 122 → 36 ms (union).

Known trade-off (follow-up issue): unlimited runs pay for the total order — warm, expansion 211 → 296 ms, union 188 → 222 ms, repeat 354 → 556 ms on this fixture (wider sort keys; repeat sort spills more). unionAll/repeat previews still evaluate their full input because the first visible column remains the primary key. Unlimited contents are identical (multiset-verified).

Tests

• Hand-written expected-order oracles for every shape (including the issue's exact nullable fixture and tied last_updated resources inserted in reverse id order), limits 0/1/50/large/oversized against the unlimited prefix, executed-SQL assertions (one final LIMIT, server rows = returned rows via pg_stat_statements).
• Planner/statistics stability: PostgreSQL 14 conditions (before/after ANALYZE, individual join toggles, sort/incremental sort off, work_mem extremes, parallel workers, custom/generic/auto plans — plans verified to change) and SQLite 9 conditions; order identical across all.
• %rowIndex and indexed results compared with the in-process evaluator; row_index.json unchanged.
• REST: complex export shards concatenated vs unlimited run (full rows, multiplicity, key order), dependency full contents, declared columns on PostgreSQL and SQLite, exact visible keys.
• MongoDB golden pipeline tests; SQL-on-FHIR conformance unchanged on SQLite, PostgreSQL and MongoDB.

Checks run locally

Focused persistence, SOF and REST suites on SQLite, PostgreSQL and MongoDB; clippy (CI allow-list) for persistence, rest, sof and hfs feature sets; rustfmt; git diff --check. CI pending.

Review

Two independent adversarial reviews (lint/CI and issue alignment). Findings fixed: indexed where() filter, %rowIndex scope in criteria, MongoDB isolation, rustdoc accuracy, test feature gates; cold measurements added. Remaining accepted trade-off: unlimited-run cost (follow-up issue).

Out of scope (pre-existing, follow-ups)

• Unlimited-run performance of the total order (new issue).
• MongoDB ignores a trailing where() on indexed forEach.
• Nested select under an indexed forEach is not lowered.
• UI guided-form validation reports forEach is unknown inside unionAll/nested selects.
• Known evaluator vs SQL differences (JSON-null array elements, character indexing of single strings, $this on empty forEachOrNull context).

[Gordex2014]

Follow-up issue to be created: unlimited-run performance

This PR makes every SQL-backed ViewDefinition run deterministically ordered so previews push one final LIMIT. Previews get much faster, but unlimited runs ($sql-export, SQL Query/SQL View dependency materialization, and $sql-run without _limit) now sort by wider keys. On the benchmark fixture (PostgreSQL 16, 20,000 Patients / 2,000 QuestionnaireResponses, warm median): nested expansion 211 → 296 ms, unionAll 188 → 222 ms, multi-path repeat 354 → 556 ms. Contents are unchanged.

We accept this trade-off here and will open a separate issue to optimize unlimited runs without changing the limited-preview contract. Candidate directions: a more compact traversal identity for repeat, skipping ordering keys where the consumer re-sorts (dependency materialization), an ascending (tenant_id, resource_type, last_updated, id) index on PostgreSQL, and — as a separate compatibility decision — a resource-first primary order for unionAll/repeat.

[codecov]

Codecov Report

❌ Patch coverage is 98.82759% with 17 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
crates/persistence/src/sof/compiler.rs	97.77%	6 Missing ⚠️
crates/sof/src/sqlquery/engine.rs	97.26%	6 Missing ⚠️
crates/rest/src/handlers/sof/run.rs	98.47%	3 Missing ⚠️
crates/persistence/src/sof/compile_path.rs	95.45%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!