feat(stats): xorq scalar tier units, served as fragments and never assigned (rows-first p34) by paddymul · Pull Request #1031 · buckaroo-data/buckaroo

paddymul · 2026-10-04T10:20:56Z

Stacked on #1029 (feat/rowsfirst-p33-policy-wire-contract), which carries #1021, #1022, #1024, #1026, #1028 and the stats policy of #1019 and #1023. This PR is based on main so the repo's Checks workflow runs on it (its pull_request trigger uses branches: "*", which does not match a base branch containing a slash). The diff therefore includes the commits of those PRs until they merge. The commits of this phase are 3bc2c09b and e214db09 (failing tests) and 60abb8f3 (implementation); read only those (git diff origin/feat/rowsfirst-p33-policy-wire-contract...HEAD).

Problem

A session whose policy target is scalar (the size rule, a host that named scalar, or the ceiling lowering full) has no units to run: the dataflow is held at the schema tier, the status is not_computed, and a stats_request from a client that advertised stats_ondemand is refused with not_requestable. The xorq pipeline plans one batch aggregate that includes approx_median and an approx_nunique per column, and one histogram query per column, so there is no cheaper run to serve. Nothing in the code states that only a full run may write summary_stats_cache, which matters once a second kind of run exists: _populate_sd_cache skips a key it finds, so a scalar or column-scoped sd stored under the full key would later be served as the complete one.

Phase and plan references

Rows-first p34, from buckaroo2-reports/plans/: plan 3 (03-no-summary-stats-for-large-files.md) section 6 "Phase 4" with section 3.4 (xorq scalar is the scalar class of the batch without approx_median and distinct_count) and the cache rule in section 3.3 (only full runs write summary_stats_cache; scalar and scoped runs behave like plan 2's filt run). Plan 1 sections 3 and 4.0, plan 2 sections 3 and 4.1.

Approach

The tier. StatState gains tier (full, the default, or scalar). The xorq pipeline plans a scalar run as the batch alone (batch, or batch:<i> per column chunk when the host opted into chunking), with no histogram unit. The batch leaves out the aggregates that provide median and distinct_count; the other stats run as before. histogram_bins is a pure function of min and max (and distinct_count, which is unknown at this tier), so it comes out of the batch and color_map needs no histogram query. The tier reports none of median, distinct_count, distinct_per and histogram (SCALAR_OMITTED_KEYS): the accumulator withholds them, so a fragment and the assembled sd carry no key the tier did not compute. A stat that reads an omitted key is left out with it, except that distinct_count is external to the DAG (as it always was), so a stat that reads it runs on the unknown None. The pandas and polars pipelines have only the full tier and refuse a scalar state.
Runs. StatRun takes an optional state, so a run can be planned at another tier or over a column group without changing the stats class. It exposes tier and assigns (true only for a full run over every column). A run's key stays (stats_gen, scope) for the full run, which is what the existing tests and complete_stats use, and is (stats_gen, scope, tier, columns) for any other (stat_run_key). start_stat_run(session, scope, tier, columns) builds either. A column-scoped run is not reachable from the wire yet (phase 6a's demand scan will call it).
The cache rule. Only the full run over every column reaches the final assignment (complete_stats, assign_full_stats). A scalar or scoped run lives under its own key, runs no assignment, and writes nothing to summary_stats_cache, summary_sd, the session snapshot or the status. _run_summary refuses a run that does not assign, so a scalar run stored under the full key fails the generation instead of being cached as complete. The schema-tier entries the dataflow cascade writes (feat(xorq): stats tier machinery and the xorq schema tier (rows-first s1) #1021) are keyed by tier and are unchanged.
Serving it. stats_wire.serves_scalar_tier(session, client) is true for a client that advertised stats_update and stats_ondemand, on a session whose status is not_computed and whose policy target is scalar. A stats_request from such a client runs the scalar units (incremental: true for the time budget, otherwise the whole run) and answers stats_update with tier: "scalar", final and remaining. The payload is the assembled slice of the columns the client has not seen (partial_payload), read through the client's own cursor, so a client that connects later gets the run's fragments with no unit run again. The session stays not_computed with the schema-tier snapshot: nothing is assigned. A unit that raises aborts that request with stats_aborted reason error and fails the run, not the session, so the full tier stays open.
Other clients. A client without both bits is served as before. A stats_update-only or no-caps client of a target the server chose below full is owed the full stats and gets them (pending, then stats_request, or synchronously at connect), and a scalar run that an ondemand client already ran is never used for that. A tier the host named (scalar) still reaches a stats_update-only client as not_computed with not_requestable, as feat(server): stats policy wire contract and handler plumbing (rows-first p33) #1029 pinned. A schema target is still refused.

What changes

buckaroo/pluggable_analysis_framework/stat_units.py: StatState.tier, UNIT_TIERS.
buckaroo/pluggable_analysis_framework/xorq_stat_pipeline.py: the scalar plan, the scalar batch, SCALAR_OMITTED_KEYS, withheld keys on the accumulator, a refusal of a histogram unit at the scalar tier.
buckaroo/pluggable_analysis_framework/stat_pipeline.py: the pandas and polars pipelines refuse a non-full state.
buckaroo/server/stat_run.py: stat_run_key, StatRun(state=), tier, assigns.
buckaroo/server/stats_wire.py: start_stat_run(tier, columns), serves_scalar_tier, the scalar branch of the stats_request handler (_serve_scalar), the guard in _run_summary.
buckaroo/server/session.py: comments on dataflow_stats_tier and STATS_STATUSES.
No change to handlers.py, websocket_handler.py, dataflow.py, the policy module or any client.

Tests

Three commits: 3bc2c09b and e214db09 (failing tests, each pushed alone) and 60abb8f3 (implementation). The second tests commit covers four behaviours that a deliberate regression of the implementation showed the first set did not catch. Tests are in the existing files of their kind.

test_xorq_stats_v2.py, TestScalarTier: the plan is the batch alone; a run issues one batch query and no histogram query; the batch holds no ApproxMedian and no CountDistinct (the same search finds them in the full batch); the scalar stats are the full stats minus median, distinct_count, distinct_per and histogram, equal where they overlap; the accumulator reports what the fragments do; the bins equal the full tier's and need no histogram query; a low-cardinality integer gets bins the full tier does not; a skipped column; a chunked batch equals the single batch; a histogram unit is refused; an unknown tier is refused; a stat that reads an omitted key is left out; a stat that reads distinct_count runs on the unknown.
test_paf_v2.py and test_data_loading_polars.py: the pandas and polars pipelines refuse the scalar tier; the keys of a run; a column-scoped run does not assign.
test_stats_policy.py, TestScalarTierServing (pure, every Python and Windows): which client of which session is served the scalar tier.
test_load_expr.py, TestScalarTierWire and TestScalarTierRequests (xorq, over HTTP and WebSocket): an ondemand client pulls the tier of a target reached by size, by the host and by the ceiling, whole and incremental, with one aggregate query and none of the approximate aggregates; the payload rows equal the full run's for the keys they cover; a run assigns nothing (dataflow, snapshot, cache keys, status); a later client reads the run's fragments with no query; a scalar run is never served as the complete stats to a no-caps client or after the full stats exist; runs of each tier and column group are stored apart; runs that assign nothing leave the cache alone while the one full assignment is unchanged; a scalar run stored under the full key is not assigned; one chunk per request over a chunked batch with the tier on every reply; a failing unit aborts the request and not the session; the request and unit spans carry the scalar tier; a dataflow-field change drops the run and starts one over the new state.
Added with the implementation because they pass on the earlier code: a stats_update-only client of a host-named scalar target is still refused and one of an auto target still pulls full units, and a schema target is still refused (test_the_clients_that_cannot_take_a_scalar_target_are_served_as_before, test_a_schema_target_still_refuses_the_request).
One comparison in a test of the first commit was corrected in the implementation commit: it compared the merged payloads of a chunked run with a whole run, and a chunk whose columns have no mean sends no row for it, where the whole run's wide payload has an empty cell. It now compares the cells that hold a value. The test failed on 3bc2c09b for the intended reason (the request was refused), not on this comparison.

On 3bc2c09b all nine Python / Test jobs failed (3.11 to 3.14, Max Versions 3.11 to 3.14, Windows), with all 28 checks completed; the Docs / Build + Check Links job failed there too (see below). On e214db09 all nine failed again with all 28 completed, and the docs job passed. CI logs were not read (the REST log API is rate-limited for this account), so the reasons rest on the local run of the same commits: on 3bc2c09b 47 of the new tests fail (each on the missing tier argument or on a not_requestable refusal where a scalar stats_update is expected) and the other 1887 unit tests pass; e214db09 adds four failing tests.

CI on 60abb8f3: all 28 checks completed, 27 succeeded (one of them the Read the Docs status) and deploy was skipped. That includes all nine Python / Test jobs, Python / Lint, Python / Typecheck, Docs / Build + Check Links, the JS job, the wheel build and the Playwright jobs.

Local checks on 60abb8f3:

The full unit suite (pytest ./tests/unit -m "not slow") gives 1940 passed and 5 skipped (the 1887 of feat(server): stats policy wire contract and handler plumbing (rows-first p33) #1029, the 51 failing tests and the 2 added with the implementation), and the same 1940 and 5 in an environment resolved with the newest versions the Max Versions job uses (pandas 3.0.6, polars 1.44.2, xorq 0.4.5, numpy 2.5.3).
Thirty-two deliberate regressions of the implementation (an omitted key not omitted, the scalar plan with histogram units, the accumulator or the emission not withholding, a histogram unit allowed at the scalar tier, the unknown-tier and pandas-tier checks removed, the run key ignoring the tier or the columns, assigns always or too often true, the _run_summary guard removed, serves_scalar_tier ignoring the ondemand bits, the status or the target, a reply with the wrong tier or always or never final, the budget or the cursor ignored, start_stat_run ignoring its tier or columns, a failing unit failing the session or being retried, and a scalar request that assigns) each make at least one test fail. One of them (the scalar tier built from the full tier's per-column funcs) survived the first set of tests and is what the second commit's test_a_stat_that_reads_an_omitted_key_is_left_out_with_it kills.
A real server process (python -m buckaroo.server --no-browser on port 8934, BUCKAROO_STATS_FULL_AUTO_ROWS=1000, a 50,000-row xorq build, stats_tier: "auto", stats_delivery: "deferred"), started and stopped by me. A client with both bits got not_computed, reason size, tier_target scalar and the dtype row only. Its incremental stats_request was answered with stats_update tier: "scalar", final: true, remaining: 0 in 29.7 ms and the rows dtype, histogram_bins, max, mean, min, non_null_count, null_count, std. A second client with both bits was told not_computed and, on asking, got the same rows with no new query. A stats_update-only client was told pending. A client with no caps got complete with the full rows (distinct_count, histogram, median among them), and after that the first client's request was answered with tier: "full". A schema target was refused with not_requestable.
Cost on a 2,000,000-row, 27-column parquet file (nine float, nine integer and nine string columns): the scalar run is one batch query and took 0.36 to 0.39 s over three repeats, the full run is 28 units and took 1.76 to 1.78 s (one machine, load not controlled).

Why default behaviour is unchanged

StatState.tier defaults to full; the full plan, run and accumulator are untouched, and the existing tests pass unchanged.
The scalar branch of the request handler is reached only by a client that sends both capability bits on a session whose policy target is scalar. That session is only reachable with stats_tier: "scalar" or auto on a deferred session, or full over the ceiling, all opt-in since feat(server): stats policy wire contract and handler plumbing (rows-first p33) #1029. No client sends stats_ondemand yet.
Clients without the bits, stats_update-only clients and no-caps clients see the messages they saw before; df_meta.stats is unchanged.

Deviations from the plan

The scalar fragments are the full fragments minus four keys, and histogram_bins reads min, max and an unknown distinct_count, not mean, std and null_count. A low-cardinality integer column therefore gets bins the full tier will not (plan 3 question 12), and distinct_per is not reported either, since it is a ratio of the unknown.
The scalar tier is not a dataflow tier: dataflow_stats_tier and STATS_TIERS are unchanged, a scalar target is built at the schema tier, and its stats are served only as fragments from a StatRun. An inline session with stats_tier: "scalar" carries no scalar stats in its first message; the client asks for them.
The status of a scalar target stays not_computed (with the policy fields feat(server): stats policy wire contract and handler plumbing (rows-first p33) #1029 reports), and a run that finishes changes nothing on the session, so a client that connects later is told the same and replays the run. The plan's "tier reached so far" is reported by the tier of each stats_update, not by df_meta.stats.
Which tier a request gets is the session's target, not a request field. tier, force, requestable and the ceiling on a request are phase 6a's.
Under a search filter a scalar run describes the filtered table. Those are the filt scope's stats, which the wire does not carry: it sends the bare keys (wire_stat_keys), which come from the unfiltered scope, so the stats_update of a filtered state holds the schema rows only. The partial updates of a full run under a filter are the same. Carrying filtered stats is the later design of plan 1 section 6.
final on a scalar reply carries no df_display_args: with nothing assigned there is no rebuilt config to send, and styling the merged partial sd directly is the config upgrade of phases 5 and 6b.
omitted_keys is not filled for the scalar tier. The tier's key set is fixed, which a client can read from the tier, and omitted_keys is the per-key refinement of @stat(max_rows=).
No firstpull.stats_total span for a scalar run, which has no completion step; stats.unit and stats.request spans carry it (tier is scalar).
The Docs / Build + Check Links job failed on 3bc2c09b and passed on e214db09. This PR changes no docs file, and both link checks it runs (pytest --check-links over docs/source/*.rst and the example notebooks) pass locally (10 and 42 tests). The log was not readable, so which link failed is not known; the governing variable is the response of an external link target at the time of that run, and the job's own log or a re-run would decide it.

Not in this PR

A lazy polars scalar unit, client rendering of the scalar tier, footer statistics.
tier, force and columns as a scope on a request, the cost guard, the demand scan (phase 6a). start_stat_run accepts columns so that phase has a place to start.
The config upgrade on a scalar reply (phases 5 and 6b), and filtered stats on the wire.

🤖 Generated with Claude Code

…the _handle_widget_change split (rows-first s1) A default-tier XorqBuckarooWidget and BuckarooWidget publish df_data_dict, then df_display_args, then the rest of the widget_args_tuple observers on a search change, and merged_sd carries the full stat set. These pass on main and pin the behaviour the split must keep. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…(rows-first p31) New tests/unit/server/test_stats_policy.py for buckaroo/server/stats_policy.py, which does not exist yet: - resolve_stats_policy: a table of (backend, source_kind, rows, cols, host_tier) against (tier_target, auto_request, requestable, reason), including the boundaries of each threshold and the three tallyman entries over 70 s. - The ceiling holds for host_tier="full", for a reload that re-resolves against a larger row count, and for a force request, and every tier a result lists as requestable is one a force would be granted. - Probes add no data query: the parquet footer probe reads a small share of a counting file object and decodes no row group, the dtype probe never collects a LazyFrame or executes a xorq expression, and a known xorq count is an input. - route_polars_entry returns "xorq" above R and "eager" at or below it. - Threshold overrides from BUCKAROO_* environment variables. Nothing imports the module yet. The tests error at fixture setup on the missing module until the implementation lands. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

… p31) New buckaroo/server/stats_policy.py, pure and imported by nothing yet: - resolve_stats_policy(backend, source_kind, rows, cols, bytes, host_tier, limits) returns tier_target, auto_request, requestable, reason and estimate over the tiers schema < scalar < full. A ceiling is computed inside the function, so every caller gets the lower of the requested and ceiling tiers with reason "ceiling". A host tier lowers freely and raises only to the ceiling. Eager pandas and polars resolve to full. - route_polars_entry(rows, cols) returns "xorq" above R rows and "eager" at or below it. - probe_dtypes reads a schema without collecting or executing anything, and probe_parquet_rows reads a parquet footer's num_rows without decoding data. A xorq count is an input to the policy, never computed by it. - The thresholds are provisional constants gathered in StatsLimits, each with a BUCKAROO_* environment override read on every call. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…-handler fields (rows-first s1) A schema-tier XorqServerDataflow should match full stats on pinned_rows, data_key, summary_stats_key and (except the stats-derived minWidth) column_config, issue no data query besides the cached count, and keep init_sd hints and sorted windows working. A pending state must write nothing under a full-tier cache key, and a later full assignment must reach merged_sd for the raw, clean and filt scopes. assemble_merged_sd must equal merged_sd, and _handle_widget_change must be built from separately callable all_stats and display-args builders. /load_expr and /reload_expr accept stats_tier and stats_delivery, replay them on reload, and keep them out of the warm short-circuit's has_config tuple. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

… s1) Add a dataflow-level stats_tier ("full" default, "schema"). The xorq schema tier builds identity and typing for every column from the expression's schema, with no data query beyond the cached row count, so a dataflow constructs in milliseconds rather than the stats' hundreds. The tier is part of _scope_cache_key, so a schema entry is never read as a full one, and _populate_sd_cache stores summary_sd under the filt key only if it was computed for the current frame, klass list and tier. add_analysis no longer builds DFStatsClass outside the hook when the tier is not full. The merged_sd observer body is extracted as the pure assemble_merged_sd, and _handle_widget_change is split into _build_df_data_dict and _build_df_display_args. /load_expr and /reload_expr accept stats_tier and stats_delivery, stored on the session beside dataflow_kwargs and replayed on reload. They stay out of the has_config tuple; the warm short-circuit compares the stored pair. stats_delivery="deferred" builds the schema-tier dataflow and publishes it. Defaults (full, inline) leave behaviour unchanged. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…esolve_stats_policy (rows-first p31) bytes and source_kind were documented as validated but are not: bytes=-1, 'lots', 1.5 and object() are accepted, a np.int64 is echoed unchanged so json.dumps of the result fails, a host tier passed positionally lands in the bytes slot, and source_kind=None or 5 is accepted. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…ation test (rows-first s1) The Max Versions jobs resolve pandas 3, which reports a string column's dtype as 'str' where pandas 2 says 'object'. The characterization test asserted 'object'. Verified in a Max Versions environment (pandas 3.0.6, polars 1.44.2, xorq 0.4.5): the unit suite passes. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…rows-first p31) bytes goes through the same integer check as rows and cols, so a negative, float or non-numeric value raises, a numpy integer is echoed as a plain int and the result stays JSON-serialisable, and a host tier passed positionally into the bytes slot raises instead of being echoed. source_kind must be a str; its vocabulary stays open for the callers that will read it. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…g on a skipped column (rows-first s1) A column in skip_stat_columns gets only name, dtype and length from the full-tier pipeline, so its _type comes from init_sd. The schema tier layers the schema-derived _type and is_* keys over it, so an int64 column that init_sd types as float merges as integer and renders with zero fraction digits instead of the float displayer init_sd asked for. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…ows-first s1) _get_schema_sd never read skip_stat_columns, so a skipped column's schema-derived _type and is_* keys overrode init_sd's _type once merged. The full tier gives a skipped column only name, dtype and length. The schema tier now does the same, so init_sd's _type decides the displayer at both tiers. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…er (rows-first s2) ServerDataflow and PolarsServerDataflow with stats_tier="schema" should publish the display state full stats give (column_config without stats-derived keys, pinned_rows including a host-supplied one, data_key, summary_stats_key) with no stat computed on the data, still apply init_sd, serve sorted windows, take a later full assignment into merged_sd for every scope, and assemble to the same sd as merged_sd. Both backends run through the same parametrized class, plus a pandas test that pins how an object column is typed from its dtype. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

ServerDataflow and PolarsServerDataflow now implement the _get_schema_sd hook, so stats_tier="schema" builds a dataflow that types every column from its dtype and runs no stat on the data. schema_sd (stat_pipeline.py) builds the sd the way process_df shapes it: an empty frame gives {}, a skipped column keeps only its names. pandas applies the existing typing_stats to a zero-row slice and derives _type through the _type stat; polars factors pl_dtype_typing out of pl_typing_stats and feeds it the dtype. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…s (rows-first p31b) The boundary tables now run on the module defaults and are written against the phase-0 proposals: full auto up to 12M rows and 520M cells, scalar auto up to 1.0B cells, full refused above 25M rows or 1.0B cells (force included), a scalar ceiling of 4.0B cells, and polars routed to xorq above 8M rows. Each threshold has an equal, a one-below and a one-above row, in rows and in cells. New tests pin each default to its literal value and cover the two new environment overrides, BUCKAROO_STATS_FULL_AUTO_CELLS and BUCKAROO_STATS_CEILING_FULL_CELLS. They fail on the current constants (10M rows, 500M cells, 50M-row ceiling, no scalar ceiling, R of 10M) and on the missing full_auto_cells and ceiling_full_cells fields. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…tats_request (rows-first s3) On a deferred /load_expr session a stats_request {stats_gen, scope} should return a stats_update with the matching stats_gen whose inline wide payload equals the all_stats an inline session sends, a stale stats_gen should get stats_aborted and run no query, and /load_expr and /reload_expr should bump the generation. df_meta.stats should be injected on every frame and survive a dataflow-field change, which returns the session to the schema tier. With a caps client and a legacy client on one session, the legacy client should keep getting complete messages through the websocket broadcast, the /load_expr, /reload_expr, /load and /load_compare pushes and the highlight overlay, while the caps client gets a stats-free frame and then pulls a stats_update. A spy telemetry sink should see a stats.request span. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…measurements (rows-first p31b) Full auto now needs at most 12M rows and 520M cells (was 10M rows), scalar auto goes to a 1.0B cell budget (was 500M), `full` is refused above 25M rows or 1.0B cells including for a forced request (was 50M rows), scalar gets a 4.0B cell ceiling by default (was unset; an extrapolation with no measurement behind it), and polars routes to xorq above 8M rows (was 10M; the figure assumes pre_limit False). Each DEFAULT_* constant carries the measurement it comes from. The two new bounds are full_auto_cells and ceiling_full_cells, appended to StatsLimits, with BUCKAROO_STATS_FULL_AUTO_CELLS and BUCKAROO_STATS_CEILING_FULL_CELLS as overrides under the same parsing rules. _size_tier and _ceiling_tier read them; signatures and the result shape are unchanged. Three existing tests that hard-coded the old boundaries are updated: the numpy-integer case (11M rows is now full), the reload case (a 40M-row entry is now refused full) and the empty-environment case for the scalar ceiling (an empty value now means the 4.0B default, not no ceiling). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…eration edge cases (rows-first s3) Four more cases for the stats wire format, kept in their own commit so each is seen failing on CI before the implementation lands. Returning to a state whose stats were completed once is answered from summary_stats_cache with no query. Completing the stats keeps the session's component_config on the refreshed display config. A warm /load_expr, which rebuilds nothing, leaves stats_gen alone. A stats_request on a session with no data is answered with stats_aborted. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…ed sessions (rows-first s3) A client that advertises ?caps=stats_update gets a stats-free initial_state on a deferred /load_expr session (df_meta.stats.status "pending") and pulls the stats with stats_request {stats_gen, scope}. The reply is a stats_update carrying the dataflow's all_stats as an inline wide envelope, or stats_aborted when the generation is stale. The request is the whole run: one synchronous call that computes the full stats, writes the full-tier summary_stats_cache entry, assigns summary_sd and refreshes the session snapshot through one helper. stats_gen is a server-owned counter bumped by every load handler and by a state change that touches a dataflow field, which also returns a deferred session to the schema tier. df_meta.stats is injected by build_state_message from the session, since the dataflow rebuilds df_meta wholesale. Every send site goes through build_state_message_for, so a client without the capability still gets complete messages (its missing stats run synchronously first); broadcast_state replaces the five copies of the send loop and sends to capable clients first. The stats_request branch binds the session's telemetry sink and emits a stats.request span. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…s (rows-first s4) Both pass on the current code. process_df is written out as the process_column loop it is, and each xorq histogram query's snapshot-cache key is compared with the key of a query built in the test, so the refactor into resumable units that follows can be checked against something that does not go through the units. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…tore (rows-first s4) Each stats class gets plan(state) and run(unit, acc): the fragments of the planned units, assembled through assemble_merged_sd, must equal the full-stats merged_sd on pandas, polars and xorq with init_sd, cleaning and overrides; a column group returns only its columns; skip_stat_columns columns get no unit; the xorq batch can be split by column chunk behind a default-off flag and is refused for anything but a plain parquet scan; the histogram cache keys stay put. A StatRun held on the session, keyed by (stats_gen, scope), keeps an append-only fragment list and accumulator, runs a unit only when asked, is dropped when stats_gen changes, and each connection keeps its own cursor. test_the_batch_is_one_query_by_default passes on the current code: it pins the default the flag must not change. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

StatPipeline and XorqStatPipeline get plan(state), new_accumulator(state) and run(unit, acc); process_df and process_table are those three in order, so the inline path and a caller running units one at a time compute the same stats. pandas and polars plan one unit per non-skipped column. xorq plans the scalar batch (which also yields histogram_bins), then one histogram query per column, visible columns first. The batch can be cut into column chunks sized by cells when a host sets stat_chunk_cells, and only for a plain parquet scan: any other source falls back to the single batch and logs why. A StatRun, kept on the session by (stats_gen, scope), holds the planned units, an append-only fragment list and the accumulator. It has no thread, timer or callback, begin_stats_generation drops it, and each WebSocket connection keeps its own StatCursor into the list. CustomizableDataflow.build_stats is the one place a stats class is built, with run=False for plan/run callers. Two of the new tests are corrected here: the default-path test now ignores the PERVERSE_DF self-check units, and the xorq comparisons round floats to nine digits because a parallel aggregate adds its partial sums in varying order. The new-module imports in the tests move to module level. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

… once (rows-first s4) A column group, the priority columns and StatRun's prefer hint accept original or rewritten (a, b, c) names and read each name in both. When an original name equals another column's rewritten name, a group returns the columns of both, priority ranks both first, and prefer picks the unit of whichever comes first in the frame. On a frame whose columns are c, b, a (rewritten a, b, c): - StatState(df, columns=('c',)) plans the units of c and a. - priority=('a',) plans c before a. - StatRun.next_unit(prefer=('a',)) picks the unit of c. The new tests also pin the namespace argument the fix adds, so a client that holds the rewritten names can ask for exactly those: StatState(namespace=) and StatRun.next_unit/run_next(namespace=), with "any" (the default), "original" and "rewritten". Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…rst s4) A column group, StatState.priority and StatRun's prefer hint matched a name against both the original and the rewritten (a, b, c) column names. When an original name equals another column's rewritten name, one name picked two columns: a group returned columns that were not asked for, and prefer ran the unit of whichever column came first in the frame. resolve_names reads each name once. In "any", the default and what the code did for names that do not collide, a name that is an original column name picks that column and only a name that is not one is read as a rewritten name. StatState(namespace=) and StatRun.next_unit/run_next(namespace=) also take "original" and "rewritten", so a caller that holds the client's rewritten names says so and gets exactly those columns. Priority and prefer names are resolved against every column of the frame, not only the columns in the group, so a name picks the same column whatever group is asked for. prioritized now takes the state, since it needs the frame's columns and the namespace to read the priority names. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…t cursors and the final assignment (rows-first s5) Covers the request branch that runs units for a time budget (at least one, one cold unit per budget), the per-handler cursors over a shared StatRun, the final assignment that frees the run and carries the rebuilt df_display_args, a legacy client finishing a run in progress, a concurrent infinite_request between units, the end-to-end order of first initial_state, rows and stats, /reload_expr mid-run, and the stats.unit and firstpull.stats_total spans. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…nt cursors and the final assignment (rows-first s5) A stats_request with incremental: true runs the units of the session's StatRun for about STATS_BUDGET_S (at least one) and answers with the fragments the asking client has not seen, read through the handler's own StatCursor. The request that runs the last unit does the final assignment in complete_stats: the full-tier summary_stats_cache entry, summary_sd, the session snapshot and the status in one step, the run freed, and the final reply carries the rebuilt df_display_args when its digest differs from the one the client holds. Without the field a request is still the whole run, now finishing the run's remaining units when a client has made progress. A legacy client's complete frame does the same. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…olicy module, PRs #1019 and #1023)

…handlers (rows-first p33) stats_tier takes auto, full, scalar or schema on /load_expr and /reload_expr and is stored with the pair. The policy resolves after the schema-tier dataflow and the count exist, is stored on the session, is reported in df_meta.stats (tier_target, reason, auto_request, requestable, estimate, omitted_keys, approx_keys, demand_columns, each with its documented default) and is applied at WebSocket open only for a client that sends ?caps=stats_update,stats_ondemand. The version-skew cases (no caps, stats_update only, both bits, an old server) and /load, which keeps resolving to full, are covered. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…and the compare reset under a stats policy (rows-first p33) Cases found untested after the first tests commit, each one a regression the first set did not catch: a stats_update client pulling units from a policy session gets full-tier updates and the final reply carries the rebuilt df_display_args, a scalar tier named with inline delivery builds a schema dataflow, and /load_compare clears the policy a session held. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…irst p33) /load_expr and /reload_expr accept stats_tier auto, full, scalar or schema, stored with the pair and kept out of has_config, so the warm short-circuit holds. When the dataflow is built at the schema tier the handler resolves the policy (resolve_stats_policy) from the count it already has, stores it on the session and starts the stats generation from it: a target below full is not_computed with the policy's reason. df_meta.stats reports the policy as tier_target and estimate, plus auto_request, requestable, omitted_keys, approx_keys and demand_columns where they differ from their documented defaults. It is applied at WebSocket open only for a client that sends ?caps=stats_update,stats_ondemand. A client with stats_update only is told the session is pending and pulls the stats, and a client with no caps gets them at connect, as for any deferred session. A tier the host named (scalar, schema) reaches every client as before. A session on an explicit stats_tier full within the ceiling sends the message it always has. /load keeps resolving to full: it does not read the field and clears a policy left on the session. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

… p34) A xorq stat run at the scalar tier is the scalar class of the batch without approx_median and distinct_count, and no histogram query. Its histogram_bins come from the batch's min and max. The run behaves like a filtered one: fragments go to the clients and nothing is assigned to the dataflow, the session snapshot or summary_stats_cache, so scalar stats are never served as the complete ones. Tests cover the plan, the queries a scalar run issues, the fragment union against the full stats, the bins, the chunked batch, the StatRun key and assignment rule, which clients are served the tier, and the stats_update messages of an ondemand client on a scalar target. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

github-actions · 2026-10-04T10:22:32Z

📦 TestPyPI package published

pip install --index-strategy unsafe-best-match --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ buckaroo==0.15.9.dev37196374384

or with uv:

uv pip install --index-strategy unsafe-best-match --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ buckaroo==0.15.9.dev37196374384

MCP server for Claude Code

claude mcp add buckaroo-table -- uvx --from "buckaroo[mcp]==0.15.9.dev37196374384" --index-strategy unsafe-best-match --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ buckaroo-table

📖 Docs preview

🎨 Storybook preview

…d generations (rows-first p34) A deliberate regression of the implementation showed four behaviours the first set of tests does not cover: a stat that reads a key the scalar tier leaves out is left out with it, a stat that reads distinct_count runs on the unknown, a scalar request emits the request and unit spans with the scalar tier and no completion span, and a dataflow-field change drops the scalar run and starts one over the new state. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…signed (rows-first p34) StatState takes a tier. A xorq run at the scalar tier is the batch alone, in the same column chunks, without the aggregates behind median and distinct_count and with no histogram unit. Its histogram_bins come out of the batch's min and max, so color_map needs no histogram query. The accumulator withholds median, distinct_count, distinct_per and histogram, so no fragment and no assembled sd carries a key the tier did not compute. The pandas and polars pipelines refuse a non-full state. StatRun takes a state, and reports its tier and whether it assigns (only the full tier over every column does). The full run keeps its (stats_gen, scope) key; every other run has a key of its own, so it never takes the full run's place. start_stat_run takes a tier and a column group. _run_summary refuses a run that does not assign, so a scalar or column-scoped sd is never written to summary_stats_cache as the complete one. A stats_request from a client that advertised stats_update and stats_ondemand, on a session whose policy target is scalar and has nothing computed, runs the scalar units and answers stats_update with tier scalar, final and remaining. The run behaves like a filtered one: the fragments go to the clients that ask (each through its own cursor) and nothing is assigned to the dataflow, the session snapshot, summary_stats_cache or the status. Other clients are served as before. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

paddymul · 2026-10-06T15:52:35Z

Closing. The scalar run is served as StatRun units that a client pulls, which the revised D2 of ADR-002 (#1043, 7943412) drops, and ADR-003 D3/D4 (#1044) implements tiers as a tier field on @stat with a filter in build_column_dag rather than a hard-coded list of omitted keys. The rule that only a full run over every column writes summary_stats_cache is ADR-003 D11.

The branch is kept as a reference for what the xorq scalar batch contains.

…count memo (rows-first p37) stats_policy gains GuardLimits (BUCKAROO_SORT_DISABLE_ROWS, default 25M rows; BUCKAROO_SEARCH_DISABLE_ROWS, default none) and resolve_source_guards, apart from the stats thresholds. A session a host opened with a stats policy resolves them at /load_expr and /reload_expr from the count load took. Above the sort threshold the xorq dataflow finishes its display config with disable_sorting, so no klass or override leaves a sortable column in a display infinite_request serves, and a sorted infinite_request is refused with error_code sort_disabled before any query runs. df_meta carries sort and search when one is disabled. handle_infinite_request_xorq holds the searched expression per (base expression, term), so the second window of a search is a hit in _expr_count's cache and issues no count. Rebased onto #1029 without the closed #1031 and #1033: the guard resets no longer call reset_stats_controls, session_dataflow stays in stats_wire, and the wire tests share a _LimitsWire base ported from #1033 without its unit and cost helpers. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

paddymul and others added 29 commits October 3, 2026 22:51

Merge origin/feat/rowsfirst-p31b-policy-calibration into p33 (stats p…

e552b40

…olicy module, PRs #1019 and #1023)

paddymul deployed to testpypi October 4, 2026 10:21 — with GitHub Actions Active

paddymul and others added 2 commits October 4, 2026 06:34

paddymul deployed to testpypi October 4, 2026 10:36 — with GitHub Actions Active

paddymul deployed to testpypi October 4, 2026 10:46 — with GitHub Actions Active

paddymul changed the base branch from main to adr-003-stats-tiers-and-size-policy October 6, 2026 14:40

paddymul closed this Oct 6, 2026

Navigation Menu

Repository navigation

feat(stats): xorq scalar tier units, served as fragments and never assigned (rows-first p34) - #1031

feat(stats): xorq scalar tier units, served as fragments and never assigned (rows-first p34)#1031
paddymul wants to merge 31 commits into
adr-003-stats-tiers-and-size-policyfrom
feat/rowsfirst-p34-scalar-tier-units

paddymul commented Oct 4, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 4, 2026 •

edited

Loading

Uh oh!

paddymul commented Oct 6, 2026

Uh oh!

This branch was successfully deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

paddymul commented Oct 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Phase and plan references

Approach

What changes

Tests

Why default behaviour is unchanged

Deviations from the plan

Not in this PR

Uh oh!

github-actions Bot commented Oct 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📦 TestPyPI package published

MCP server for Claude Code

📖 Docs preview

🎨 Storybook preview

Uh oh!

paddymul commented Oct 6, 2026

Uh oh!

This branch was successfully deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

paddymul commented Oct 4, 2026 •

edited

Loading

github-actions Bot commented Oct 4, 2026 •

edited

Loading