feat(load_expr): column_stat_sources computes a column's stats over the expression it came from (#1092)

[paddymul]

/load_expr takes two new fields, stat_sources and column_stat_sources, so a column's summary stats can be computed over the expression the column came from instead of over the loaded expression. A compare grid that loads an outer join of V1 and V2 can then show V2's own length, null_count, distinct_per and histogram for price_v2, not the join's.

Change

stat_sources names expressions by the fields that would load each as a session of its own (build_dir, and optionally data_id, cache_storage_path, project_root). column_stat_sources maps a column of the session's expression to a list of {source, column, prefix?}.

• The hook is XorqDataflow._get_full_sd. Mapped columns join the skip set, so no stat query touches them in the raw, cleaned or filtered scope. Each source then gets its own XorqStatPipeline.process_table(<source expr>, stat_columns=…, scope_id=make_scope_id(<source data_id>)) against the source's cache_storage_path. Cells already there are hits, missing ones are computed over the source column and written to the source's cache, where the source's own grid finds them later. There is no fallback to the session's expression.
• The stat set run over a source is XORQ_STATS_V2 plus load_project_stat_klasses(<source project_root>). A stat from the session's project_root does not run on a mapped column.
• The unprefixed source's summary replaces the column's, dtype and length included, and is part of the raw summary, so it wins over init_sd. orig_col_name and rewritten_col_name stay the session's. A prefixed source's keys land as <prefix><key>.
• Under stats_delivery: deferred the sources run when the dataflow goes to the full tier, so they arrive in the same stats_update. The schema tier issues no source query.
• Both fields are stored in the session's dataflow_kwargs, so /reload_expr replays them, and they count as config for the warm-session exit, so a re-POST rebuilds the session.
• Malformed requests are a 400 with an error_code: invalid_stat_sources, invalid_column_stat_sources (including a mapped column without exactly one unprefixed source), unknown_stat_source, column_not_found (a target missing from the session's expression) and stat_source_column_not_found. A source that fails to load, or a stat that errors over it, is a stat error on the mapped column ((<column>, stat_source) or (<column>, <stat>) in errs), and the session still loads.
• Sources load lazily and are released with the session (SessionState.release_xorq_state, and after /reload_expr swaps the dataflow), like the main expression's memtables (load_expr creates a new xorq-datafusion backend per call; contexts accumulate and dominate server memory #896).

The summaries of a mapped column don't depend on the session's scope, so each source run is computed once per dataflow and shared by the three scopes.

Tests

TestColumnStatSources and TestLoadExprColumnStatSources in tests/unit/server/test_load_expr.py, over a V1/V2 pair where V1 has keys V2 lacks and the join is an outer join on the key:

• price_v2's summary equals the V2 session's for price, values and types, with length 6 and null_count 0 where the join has 8 and 2.
• No stat query reads price_v2 in the raw, clean and filtered scopes, with the unmapped join as a control that does.
• A cold source writes its cells under its cache path and make_scope_id(<data_id>). A second load issues no stat query, and opening the source as a session of its own issues none for price. Cells the source's own session wrote are hits.
• A project stat in the source's project_root appears on the mapped column, and one in the session's project_root runs on the other columns only.
• Prefixed keys, a source that fails to load, the deferred stage, each validation error as a 400 with nothing loaded, /reload_expr replay, and a re-POST that changes the mapping.

The tests were pushed first and failed on CI (run 37656666811, Python / Test (3.13): 14 failed, 1896 passed; all 14 are the new tests). The fix commit also corrects one assertion in the cold-source test: V2's own session computes key and name too, so the check that it issues no query is limited to price. Locally the unit suite passes except artifact_test.py::TestToHtml (two tests, no built static assets) and test_mcp_uvx_install.py::test_view_data_call, which fail the same way on main.

Not in this PR

As the issue lists: add_analysis on a live session (#1063) does not run over sources, post-processing views of a session with mapped columns, and pandas, polars and the widget.

Closes #1092.

🤖 Generated with Claude Code

[github-actions]

📦 TestPyPI package published

pip install --index-strategy unsafe-best-match --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ buckaroo==0.15.10.dev37800307589

or with uv:

uv pip install --index-strategy unsafe-best-match --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ buckaroo==0.15.10.dev37800307589

MCP server for Claude Code

claude mcp add buckaroo-table -- uvx --from "buckaroo[mcp]==0.15.10.dev37800307589" --index-strategy unsafe-best-match --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ buckaroo-table

📖 Docs preview

🎨 Storybook preview

[paddymul]

Review pass

A /code-review pass over this PR found 10 issues. Tests for #1–#4 went in first (7d39174, failing on CI in all Python test jobs), then the fixes (97d6e64, CI green).

Fixed in 97d6e64

1. Stat sources ignored cache_dir. A source build loaded with load_expr_build_dir(build_dir) and nothing else, so its cache nodes resolved under ~/.cache/xorq and its stat queries recomputed the embedder's snapshots there (the server: /load_expr loads builds without a cache_dir — cache nodes resolve to ~/.cache/xorq, so an embedder's baked snapshots are never read #972 problem, for sources). Sources now load and heal in their spec's cache_dir (new optional field), else the session's, on both /load_expr and /reload_expr. Test: test_stat_sources_read_the_embedder_snapshots.
2. No warm exit for a host that resends the sources. stat_sources/column_stat_sources were in the truthiness has_config tuple, so an identical re-POST always rebuilt. They are now compared with the session's stored values, like stats_tier. Test: test_an_identical_repost_takes_the_warm_exit.
3. A rebuild dropped the mapping. A re-POST that changed data_id or stats_tier but omitted the sources rebuilt without them. They now carry over like data_id (same build only), and an explicit null clears them. Tests: test_a_repost_that_omits_them_keeps_them, test_a_repost_with_null_clears_them.
4. Failed loads leaked memtables. The 500 path released neither the session expression nor the sources; the 400 path released only the sources. Both now release both, and a failed /reload_expr releases its new set. Test: test_a_failed_load_releases_the_sessions_and_sources_memtables.

StatSourceSet.from_fields also replaces the three copies of the "either field is set" check (part of #10).

Open

5. A source's load error, or a failed run, is cached for the life of the set and never retried.
6. Prefix collisions (prefix: "null_" turning count into null_count, duplicate prefixes) silently overwrite stats. Proposed: reject duplicate prefixes at parse time, and in validate() reject any overlap between the unprefixed source's stat keys and prefix + key for the prefixed ones (keys are known from each source's stat set, no query needed).
7. /reload_expr releases the replaced set while its deferred stats stage may still be querying it.
8. _get_schema_sd ignores the set, so a mapped column's length changes between the schema and full tiers.
9. One set-wide lock serializes every source run across scopes.
10. _StatSource.scope_id/_compute duplicate XorqDataflow._stat_scope_id and the XorqDfStatsV2 pipeline wiring.

🤖 Generated with Claude Code