feat(xorq): stats tier machinery and the xorq schema tier (rows-first s1)

[paddymul]

Problem

On /load_expr, XorqServerDataflow computes summary stats inside its constructor. Over 115 tallyman loads stats are a median 93% of firstpull.load_expr (p50 1.29 s, max 215.8 s), and session.xorq_dataflow is only assigned after construction, so no row is served until they finish. A schema-only construction takes about 1 ms against 393 ms with stats on a 27-column entry, but nothing builds a dataflow at that level today, and three details of the dataflow get in the way:

• merged_sd and column_config need more than an empty sd. An empty merged_sd yields zero columns, and skip_stat_columns drops _type, so every column renders as obj.
• _scope_cache_key ignores what the sd contains. After a schema-only construction, assigning full stats to summary_sd hits the three cached scope entries and never reaches merged_sd. _populate_sd_cache also stores the current summary_sd under a new filt key without checking it belongs there.
• add_analysis builds DFStatsClass itself (and processes the frame twice), so it bypasses any gate placed in _get_summary_sd.

Phase and plan references

Rows-first s1, from buckaroo2-reports/plans/: plan 2 (02-rows-first-xorq-and-lazy-polars.md) section 7 "Phase 1" and sections 3, 4.1 and 4.2; plan 1 (01-rows-first-stats-separate-plumbing.md) sections 3 (constraints 1 to 4 and 11), 4.0 and 9 "Phase 1", limited to the parts plan 2 Phase 1 owns (the _populate_sd_cache guard and the _handle_widget_change split). Plan 3 is background only.

Approach

• CustomizableDataflow(..., stats_tier="full" | "schema"). stats_tier is a traitlets Enum on DataFlow that _summary_sd observes: assigning it refuses an unknown value and computes the new tier's stats. set_stats_tier(tier, summary=None) switches without that compute when the stats are already in hand: a (sd, errs) computed elsewhere, or the entry an earlier visit left in summary_stats_cache. XorqDataflow implements the schema tier; pandas and polars raise NotImplementedError for it until the next phase.
• The tier is checked once, in CustomizableDataflow._get_summary_sd, which calls _get_full_sd or _get_schema_sd(processed_df, scope). XorqDataflow overrides _get_full_sd and keeps only its pandas-frame branch in _get_summary_sd.
• The schema tier's sd has, for every column, orig_col_name, rewritten_col_name, dtype, the is_* flags, _type and length. Types come from the same typing_stats and _type stats the pipeline runs (new schema_stats(dtype) in xorq_stats_v2.py), length from the cached _expr_count. No stat query is issued. The clean scope's sd has no length: its frame is rebuilt on every cache miss, so counting it would be a query of its own. The filtered frame's count is the one df_meta takes anyway. A column in skip_stat_columns gets only orig_col_name, rewritten_col_name, dtype and length, as it does at the full tier, so its _type comes from init_sd at both tiers.
• The tier is part of _scope_cache_key (new optional tier= argument, defaulting to the dataflow's) and of the _summary_sd dedupe key, which _summary_sd now records only after the stats are computed. _populate_sd_cache stores summary_sd under the filt key only if it was computed for the current frame, klass list and tier. Otherwise it leaves the entry out, and the next _summary_sd run for that state fills it.
• add_analysis builds the klass list with with_stat, the replace-or-append rule both pipelines' add_stat now share, validates it, then reruns _summary_sd, at either tier. It no longer builds DFStatsClass itself.
• The merged_sd observer body is now the pure assemble_merged_sd(init_sd, cleaned_sd, raw_sd, processed_sd, processed_df, chains, clean_sd=None, filt_sd=None).
• _handle_widget_change calls _build_df_data_dict (rows, all_stats, placeholder) and then _build_df_display_args (column_config, pinned_rows, overrides), in the order it set the traits before. BuckarooInfiniteWidget._handle_widget_change calls the same two builders instead of repeating them.
• /load_expr accepts stats_tier (full | schema) and stats_delivery (inline | deferred). They are stored on the session as stats_tier and stats_delivery, beside dataflow_kwargs, and replayed by /reload_expr, which also accepts them in an optional body. The dataflow is built at dataflow_stats_tier(stats_tier, stats_delivery): schema when delivery is deferred, else stats_tier. A field the body omits keeps the session's value, as cache_dir does. /load resets the pair to DEFAULT_STATS_TIER and DEFAULT_STATS_DELIVERY (session.py), as it clears the rest of the xorq state. Unknown values return 400 (invalid_stats_tier, invalid_stats_delivery).
• The fields are not in has_config. The warm short-circuit compares the requested pair with the stored one and rebuilds only when they differ.
• Under deferred this phase only builds the schema-tier dataflow and publishes it. Stats never arrive; that is the next phases.

What changes

• buckaroo/dataflow/dataflow.py: STATS_TIERS, assemble_merged_sd, the stats_tier trait and set_stats_tier, the tier dispatch in _get_summary_sd with the _get_full_sd and _get_schema_sd hooks, the tier in _scope_cache_key and in the _summary_sd dedupe key, the _populate_sd_cache guard, add_analysis, _build_df_data_dict and _build_df_display_args.
• buckaroo/xorq_buckaroo.py, buckaroo/customizations/xorq_stats_v2.py: XorqDataflow._get_full_sd and _get_schema_sd, schema_stats.
• buckaroo/pluggable_analysis_framework/stat_pipeline.py, xorq_stat_pipeline.py: with_stat, used by both add_stats.
• buckaroo/buckaroo_widget.py: BuckarooInfiniteWidget._handle_widget_change.
• buckaroo/server/session.py: SessionState.stats_tier and stats_delivery, STATS_DELIVERIES, DEFAULT_STATS_TIER, DEFAULT_STATS_DELIVERY, dataflow_stats_tier.
• buckaroo/server/handlers.py: _stats_policy_from_body, LoadHandler, LoadExprHandler and ReloadExprHandler.

Tests

Three test and fix commits, in this order (the third is a test-only fix, below):

1. A characterization test (passes on main): a default-tier XorqBuckarooWidget and BuckarooWidget keep the order in which a search change reaches the dataflow's and widget's traits, and their merged_sd keys and values (tests/unit/test_xorq_buckaroo_widget.py, tests/unit/basic_widget_test.py).
2. Failing tests, in the existing files:
   • tests/unit/server/test_load_expr.py, TestStatsTierSchema: a schema-tier XorqServerDataflow matches full stats on pinned_rows, data_key, summary_stats_key and column_config without minWidth, and minWidth is the one difference for a float with a 1e9 maximum; its sd is the schema keys of the full sd; a spy on Expr.execute, Expr.to_pyarrow and XorqStatPipeline._execute sees only CountStar through construction, a search and add_analysis (a control test shows the spy does see stat queries at the full tier); init_sd hints and column_config_overrides apply; a sorted infinite_request works; a schema-tier state with a user op and a search active writes nothing under a full-tier key; switching the tier gives a merged_sd equal to a full-tier dataflow's for raw, clean and filt; full-tier entries written to the cache first are used without a stat query (both reworked in the second follow-up below); an sd from another tier is not cached under the new tier's key.
   • TestLoadExprStatsPolicy: defaults, schema tier, deferred publishing a schema dataflow that serves rows with no stat query, 400s, a warm re-POST with an unchanged pair short-circuiting (also when the fields are omitted, and when the default pair is sent on every POST) and with a changed pair rebuilding, /reload_expr replaying and replacing the pair.
   • tests/unit/dataflow/scoped_summary_stats_test.py: assemble_merged_sd equals merged_sd with init_sd overrides, cleaning and a search; its startup branch; it does not mutate its inputs.
   • tests/unit/dataflow/customizable_dataflow_test.py: the two builders are callable apart, and building display args serializes no stats.

On the failing-tests commit (9b96220) all nine Python / Test matrix jobs failed (3.11 to 3.14, Max Versions 3.11 to 3.14, Windows). CI's annotation only says the step exited 1, so the reasons come from running that commit locally: 23 of the new tests fail: the dataflow tests on the missing stats_tier argument, the handler tests on the missing session attributes and validation, the assemble and split tests on the missing functions, and the deferred-load test on the stat queries it still issues. The characterization tests pass. test_the_spy_sees_stat_queries_at_the_full_tier and test_the_default_pair_sent_on_every_post_does_not_defeat_the_warm_exit pass on main by design: the first is a control for the spy, the second guards the has_config requirement, which only a wrong fix can break.

The implementation commit (7f56c20) passed every Python / Test job except the four Max Versions ones. Those resolve pandas 3, which reports a string column's dtype as str, and my pandas characterization test asserted object. I reproduced it in a Max Versions environment (pandas 3.0.6, polars 1.44.2, xorq 0.4.5), loosened that one assertion in 21cd73e, and the unit suite then passed there (1213 passed). On 21cd73e every check completed successfully, including all nine Python / Test jobs, Python / Lint, the JS job and the six Playwright jobs.

Review follow-up: the schema tier first ignored skip_stat_columns, so for a skipped column its schema-derived _type and is_* keys overrode the _type in init_sd (an int64 column that init_sd types as float merged as integer and rendered with zero fraction digits). test_skipped_column_keeps_init_sd_typing in TestStatsTierSchema covers it. It was committed alone (1c89cfd) and Python / Test (3.11) failed on that commit; the fix is 10d9d01. The 3.14 job passes on the test-only commit because xorq is gated to python<3.14 and these tests skip there.

Second review follow-up, after the rebase onto main with #1040. The failing tests are in 739f2c1: assigning stats_tier changed nothing and accepted any string; set_stats_tier did not exist (the two tests that assigned summary_sd after flipping the tier now go through it or the trait); add_analysis never reran the summary stats, so an added stat never reached merged_sd; at the schema tier an op under a search counted the rebuilt clean-scope frame as well as the filtered one; and /load left the stats policy for the next /load_expr to inherit. All seven failed locally. The fixes are 55d70bb (dataflow) and 9609772 (server), and c61b221 has the infinite widget reuse the builders and moves test imports to module level.

The rebase rewrote the earlier commits: 9b96220, 7f56c20, 21cd73e, 1c89cfd and 10d9d01 above are now 689aae8, 5db0cfc, f304948, 0ef794e and 91e1ff5. The CI results quoted for them ran before the rebase.

Construction on a 200,000-row, 20-column parquet (10 float and 10 string columns) took about 395 ms at the full tier and 4 ms at the schema tier, measured in-process with XorqServerDataflow.

Why default behaviour is unchanged

• stats_tier defaults to full everywhere; dataflow_stats_tier("full", "inline") is full, which is what /load_expr and /reload_expr send when the fields are absent.
• The full-tier cascade is the same code path. The cache keys gain a |full component (they are opaque, in-process strings). The _populate_sd_cache guard compares (id(processed_df), id(analysis_klasses), tier) with the key _summary_sd records after computing. That matches at the full tier unless the last run raised, and then the filt entry is left out rather than filled with the previous frame's sd.
• assemble_merged_sd is the old observer body moved; the observer passes it the same values. _handle_widget_change builds the same two values in the same order, which the characterization commit pins.
• /reload_expr still works with no body or a body it cannot parse (it reads the stats fields leniently).
• Two full-tier changes are deliberate. add_analysis now reruns the summary stats once, so an added stat reaches merged_sd; before, it ran the pipeline twice and dropped the result. A klass that fails DAG validation now raises (DAGConfigError, or TypeError for a v1 class) and leaves the list as it was, at both tiers and on both backends; before, the full tier printed "Unit tests failed" (pandas) or "DAG validation failed" (xorq) and carried on.
• The existing tests pass unchanged.

Deviations from the plan

• The plan says the schema branch lives in _get_summary_sd of both xorq_buckaroo.py and the base dataflow.py. The base branch calls a new _get_schema_sd hook that raises NotImplementedError; pandas and polars implement it in the next phase.
• Plan 1 section 4.0 describes stats_tier as dataflow-level, and plan 2 says deferred "builds a schema-tier dataflow". I read stats_tier on the session as the tier the session is headed for and stats_delivery="deferred" as forcing the dataflow to be built at schema until later phases deliver the rest. So stats_tier="full" with deferred is a schema dataflow whose session still targets full.
• Plan 2 constraint 1 says to replace the summary_stats_cache entry before assigning full stats. With the tier in the key that is unnecessary: the full-tier entry is a different key. Assigning stats_tier computes the new tier's stats, and set_stats_tier installs stats computed elsewhere, or full-tier entries already cached, without recomputing the filtered scope (both are tested).
• The plan words the _populate_sd_cache change as a guard against a deferral returning the previous sd. The deferral path here builds a fresh schema sd for each frame, so what the guard covers is a summary_sd left from an earlier frame because the last run raised. It leaves that entry out rather than computing it. A hook that deliberately returns a stale sd would still be stored; none exists.
• stats_tier is a trait on DataFlow (the base), since _summary_sd and its dedupe key live there.
• Omitting stats_tier or stats_delivery (or sending null) keeps the session's value rather than resetting to the default. The plan leaves this open; it follows cache_dir (server: /load_expr loads builds without a cache_dir — cache nodes resolve to ~/.cache/xorq, so an embedder's baked snapshots are never read #972).
• /reload_expr takes an optional body for the pair; the plan only says it replays the stored one.
• Only ag_grid_specs.minWidth differs between tiers in the default styling, so the comparison drops that one key. Other klasses that read min, max or histogram will differ the same way.
• For a skipped column the full tier's pipeline pre-fills min, max and distinct_count with None, and those override init_sd's min and max in merged_sd. The schema tier leaves them to init_sd, so a klass that reads min or max of a skipped column sees init_sd's values at the schema tier and None at the full tier. The default styling's column_config is the same at both.

Not in this PR

Pandas and polars schema tiers; df_meta.stats, stats_request, stats_update, ?caps= and stats_gen; resumable stat units and StatRun; the filtered-count memoization; any default flip. Under deferred, stats do not arrive yet; the phase that delivers them should install them with set_stats_tier("full", summary).

🤖 Generated with Claude Code

[github-actions]

📦 TestPyPI package published

pip install --index-strategy unsafe-best-match --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ buckaroo==0.15.9.dev37502725670

or with uv:

uv pip install --index-strategy unsafe-best-match --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ buckaroo==0.15.9.dev37502725670

MCP server for Claude Code

claude mcp add buckaroo-table -- uvx --from "buckaroo[mcp]==0.15.9.dev37502725670" --index-strategy unsafe-best-match --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ buckaroo-table

📖 Docs preview

🎨 Storybook preview

[paddymul]

Review of this PR found one issue, a medium. I reproduced it before changing anything and addressed it.

Finding: the schema tier ignored skip_stat_columns (buckaroo/xorq_buckaroo.py, _get_schema_sd).
For a column in skip_stat_columns the full tier gives only orig_col_name, rewritten_col_name, dtype and length, so its _type comes from init_sd. _get_schema_sd never read skip_stat_columns, and in assemble_merged_sd the raw sd overrides init_sd, so the schema-derived _type and is_* keys won. With an int64 qty column, init_sd={'qty': {'_type': 'float', ...}} and skip_stat_columns=['qty'], the full tier merged _type float with a float displayer at 3 fraction digits, and the schema tier merged integer with a float displayer at 0 fraction digits. The column_config differed beyond minWidth, and the PR body did not say so.

How it was addressed:

• 1c89cfd adds test_skipped_column_keeps_init_sd_typing to TestStatsTierSchema, committed alone. Locally it was the only failure in the unit suite, and Python / Test (3.11) failed on that commit. The 3.14 job passes there because xorq is gated to python<3.14 and these tests skip.
• 10d9d01 makes _get_schema_sd give a skipped column only name, dtype and length, as the full tier does. The reviewer's probe now shows the same column_config at both tiers, and the new test compares them for all columns. The change is limited to the schema tier, so default behaviour is untouched and no existing test changed.

The PR body now describes the skipped-column behaviour. It also records one remaining difference: for a skipped column the full tier pre-fills min and max with None, and those override init_sd's values in merged_sd. The schema tier keeps init_sd's min and max. The default styling's column_config is the same either way, but a klass that reads min or max of a skipped column would see a different value at each tier.

Declined: nothing.

CI on 10d9d01: all checks completed successfully, including every Python / Test job, Python / Lint, the JS job and the Playwright jobs.

[paddymul]

A second review of this PR (at 10d9d01) found 14 issues. I rebased the branch onto main after #1040 merged, so the earlier fixes have new SHAs (old ones in parentheses), and the rest of the work sits on top of that.

Fixed

1. Schema-tier add_analysis kept a klass that failed DAG validation, so the list grew from 16 to 17 and every later call raised. Test 7ea4626 (72c6f59), fix ba9e081 (2ae5b06): the list is validated before it is assigned.
2. Switching to the full tier reran full stats for every scope, because the _populate_sd_cache guard compared against a dedupe key still recorded at schema. Test 739f2c1, fix 55d70bb: stats_tier is now a traitlets Enum that _summary_sd observes, so assigning it computes the new tier's stats once. set_stats_tier(tier, summary=None) installs stats already in hand (computed elsewhere, or cached by an earlier visit) without recomputing the filtered scope.
3. The guard's recompute dropped the errs it got back. Fixed in 55d70bb: the guard no longer computes anything. A summary_sd that isn't the current frame's is left out of the cache, and the next run fills it.
4. /reload_expr returned 500 on a body that isn't valid UTF-8. Test 227a664 (c297899), fix a03bfa0 (1e040e4): _optional_body catches ValueError.
5. /load left the session's stats policy behind, so the next /load_expr inherited a deferred delivery. Test 739f2c1, fix 9609772: /load resets the pair.
6. Full-tier add_analysis ran two stat pipelines and discarded both, so an added stat never reached merged_sd. Test 739f2c1, fix 55d70bb: it builds the list without running stats, then reruns _summary_sd once.
7. _analysis_klasses_with was a third copy of add_stat's replace-or-append rule. Fixed in 55d70bb: with_stat in stat_pipeline.py, used by both pipelines and by add_analysis.
8. BuckarooInfiniteWidget._handle_widget_change repeated _build_df_data_dict and _build_df_display_args. Fixed in c61b221: it calls them.
9. The tier was checked in three places with disagreeing predicates. Fixed in 55d70bb: one check in CustomizableDataflow._get_summary_sd, dispatching to _get_full_sd or _get_schema_sd.
10. The schema tier issued a COUNT for the rebuilt clean-scope frame. Test 739f2c1, fix 55d70bb: the clean scope's schema sd has no length, so an op under a search issues one COUNT, the filtered frame's, which df_meta needs anyway.
11. The default policy was hard-coded in the handler as well as on SessionState. Fixed in 9609772: DEFAULT_STATS_TIER and DEFAULT_STATS_DELIVERY in session.py.
12. Imports inside test functions. Fixed in c61b221.

• Root cause of 2 and 4: stats_tier was a plain attribute that nothing validated. It is now the trait above, and stats_tier = "Full" raises TraitError (test 739f2c1).

Changed in the process. At the full tier, a klass that fails DAG validation now makes add_analysis raise (DAGConfigError, or TypeError for a v1 class), and the list is left as it was. Before, it printed "Unit tests failed" (pandas) or "DAG validation failed" (xorq) and carried on. Both tiers and both backends now behave the same. The PR body lists this under default behaviour.

Corrected: 3. The finding was that one failed stats run leaves wrong filtered stats. _summary_sd did record its dedupe key before computing, and 55d70bb now records it afterwards. But the review's reproduction misread its own output. After the failed run operations never receives the search op, so the "filtered length 5" it printed came from the old unfiltered entry.

Tracing again turned up a separate bug, already present on main. When _operation_result applies a user op while a search is active, the summary cascade runs with self.operations still holding the user's unmerged list. The op+search frame's stats are therefore cached under the key for the op-only chain. Once the search is cleared, cleaned_length shows 2 where a freshly built dataflow shows 5. No failure is needed to trigger it, and it reproduces identically on main and on this PR's base, so it is not addressed here. Filed as #1067.

Declined: nothing.

For #1024 and #1035: assign_full_stats flips stats_tier and then assigns summary_sd. Now that the tier is a trait, the flip itself computes the stats, so those PRs should call dataflow.set_stats_tier("full", computed) instead.

Local: 1642 passed, 17 skipped, 16 xfailed; ruff and paddy-format pass. CI on c61b221: all 24 Checks jobs passed, including every Python / Test job, Python / Lint, the JS job and the Playwright jobs, and Build and Deploy and Storybook passed.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.