feat(server): let a host enable the xorq stats column-chunk split over HTTP (rows-first s6) by paddymul · Pull Request #1035 · buckaroo-data/buckaroo

paddymul · 2026-10-05T11:34:21Z

Stack

Stacked on #1034 (feat/rowsfirst-p37-huge-source-guards), which carries #1033, #1029, #1028, #1026, #1024, #1022, #1021 and the stats policy of #1019 and #1023. This PR is based on main so the repo's Checks workflow runs on it (its pull_request trigger uses branches: "*", which does not match a base branch containing a slash). The diff therefore includes the commits of those PRs until they merge.

The commits of this phase are the ones after 0181386e, the head of #1034 (git diff origin/feat/rowsfirst-p37-huge-source-guards...HEAD):

dbef9308, the failing tests, pushed alone. On that commit the eight Python / Test matrix jobs (3.11 to 3.14, and the four Max Versions jobs) failed and the other jobs passed, as expected; the base feat(server): sort and search guards for huge sources and a searched count memo (rows-first p37) #1034 has no failing check.
85271316, the implementation (handlers.py, session.py), pushed after that run had finished. On it all 28 checks completed: 27 SUCCESS and deploy SKIPPED, including the nine Python / Test jobs, Lint and Typecheck.

Problem

On the merged stack, the xorq batch aggregate is one stats unit. While it runs, the single IOLoop thread does nothing else: in an integration run against a real browser and xorq, on the 10.8M-row parking entry and a 12M-row slice of it, an infinite_request probe saw a maximum of 1743-2150 ms and /health a maximum of 2.0-3.0 s, 6 to 10 times the 250 ms budget plan 1 proposes.

XorqServerDataflow already has the column-chunk split behind stat_chunk_cells (#1026). With stat_chunk_cells=60M patched into the process, the same probes saw 216-254 ms and the total stats time did not change. Over HTTP the split cannot be reached: LoadExprHandler builds dataflow_kwargs from five body fields, and no body field or policy sets stat_chunk_cells, so a host cannot turn it on.

Phase and plan references

Rows-first s6, an integration finding rather than a numbered plan phase. From buckaroo2-reports/plans/: plan 2 (02-rows-first-xorq-and-lazy-polars.md) section 4.2 (the batch split behind a default-off flag, chunked only on scan-backed sources) and section 4.1 and phase 1's rule that host fields stay out of the truthiness-based has_config tuple; plan 1 (01-rows-first-stats-separate-plumbing.md) sections 3 and 4.0 for the session fields. The phase-0 gate that decides whether the split can become the default is separate and not touched here.

Approach

/load_expr accepts stat_chunk_cells: a positive integer cell count, or null for off. It is validated before anything else is built, like stats_tier and stats_delivery.
The value is stored on the session as SessionState.stat_chunk_cells (default None), beside stats_tier and stats_delivery and apart from dataflow_kwargs, and is passed to XorqServerDataflow(stat_chunk_cells=...). It is not in the has_config tuple, which tests truthiness: a host that sends a count on every POST would otherwise rebuild every time (feat(stats): expose summary-stat cache hit/miss + timing as a structured signal #944).
A warm re-POST short-circuits when the value equals the session's and rebuilds when it differs. Omitting the field keeps the session's value. An explicit null turns the split off, which is a change and rebuilds.
/reload_expr replays the stored value into the rebuilt dataflow. It takes no body field for it.
/load and /load_compare reset it to None with the rest of the stats fields, since they do not run the xorq batch.
The scan guard of feat(stats): resumable stat units and the StatRun store (rows-first s4) #1026 (XorqStatPipeline.chunk_refusal) stays authoritative. A host that sets the field on a join, an aggregate, a diff, a CSV read or a registered table gets the single batch, no error, and no change to df_meta.stats or the stats_update replies.

What changes

buckaroo/server/handlers.py: _stat_chunk_cells_from_body, its use in LoadExprHandler (validation, warm short-circuit, construction, storage) and ReloadExprHandler (replay); /load and /load_compare reset the field.
buckaroo/server/session.py: SessionState.stat_chunk_cells.
No change to the pipeline, the dataflow, the wire format or any client.

Tests

Added to the existing files of their kind: TestLoadExprStatChunkCells in tests/unit/server/test_load_expr.py (twelve tests, a twelve-row, six-column parquet scan and a join of it, built to build dirs and served over HTTP and WebSocket), one test each in test_server.py (/load) and test_load_compare.py.

The field reaches XorqServerDataflow.stat_chunk_cells and is stored on the session, outside dataflow_kwargs.
Default off; null equals absent, with the same frames.
On a parquet-scan build the first stats run has two batch units (a spy on XorqStatPipeline.run) against one without the field, and the assembled stats and the dataflow's merged_sd equal the unsplit run's. The same holds for an inline session: two batch queries, an identical initial_state.
On a join the run keeps one batch unit, df_meta.stats has only status, tier and gen, no reply carries a reason or status, and the final state is complete with the unsplit stats.
Invalid values (0, a negative, 1.5, 36.0, true, false, a string, a list, an object) answer 400 invalid_stat_chunk_cells, create no session and leave a stored value and its dataflow alone.
Warm re-POST: an unchanged value and an omitted one short-circuit; null sent on every POST does not defeat the exit; a new count, a first count and an explicit null rebuild.
/reload_expr replays the stored value (two batch queries in the rebuilt dataflow) and builds one batch when there is none.
/load and /load_compare drop a stored value.

On the base, ten of the twelve /load_expr tests and both reset tests fail (on assertions or on the missing session attribute); the other two, null on every POST and a reload with no stored value, pass on the earlier code and guard the default. Locally, with the implementation, the full unit suite (-m "not slow") gives 2192 passed, 5 skipped.

Measurements

A 2,000,000-row, 27-column parquet file (54M cells, the medium fixture of #1026) is loaded over HTTP with stats_delivery: "deferred", in a server process started with BUCKAROO_PERF=1, once per configuration and repetition (5 repetitions, new session each time, force_reload). One WebSocket plays the browser's scheduler: back-to-back incremental stats_requests until final. A second WebSocket sends an infinite_request for rows 0-50 every 100 ms, and a /health GET loop runs beside it; probe latency is counted from the scheduled send time, so a stalled loop is not hidden. The longest unit is the longest stats.unit span the server logged for the session. The machine was busy with other work (load average 4.3-5.2).

`stat_chunk_cells`	batch units	longest single stats unit	longest request (`elapsed_ms`)	probe p99 (pooled)	probe max	probes over 250 ms	`/health` max	total stats time
absent	1	0.879-0.891 s	884-895 ms	845.6 ms	848.8 ms	40 of 117	1.03-1.05 s	2.30-2.53 s
12,000,000	5	0.255-0.265 s	257-267 ms	219.5 ms	230.7 ms	0 of 116	227-238 ms	2.34-2.40 s
6,000,000	9	0.143-0.160 s	144-161 ms	154.0 ms	163.9 ms	0 of 123	117-134 ms	2.36-2.58 s

The ranges are over the five repetitions. A run sends about 23 probes, so the pooled p99 is close to the maximum. The split does not change the total stats time within the spread between repetitions. An earlier run of the same harness, started before the implementation commit was made (load average 5.4-6.5), gave a single batch of 0.58-0.60 s and a probe p99 of 528-554 ms without the field, against 119-153 ms at 12M cells and 111-124 ms at 6M cells. The absolute times differ between the two runs and I did not identify the variable behind that; the direction and the rough ratios are the same in both. The harness and the fixture (179 MB) are not part of this PR.

Why default behaviour is unchanged

The field defaults to None, which is what XorqServerDataflow already defaults to; nothing sets it unless a host sends it.
It is read outside has_config, so the warm exit is the same for every POST that omits it or sends null for a session that has none.
The existing tests pass unchanged.

Deviations from the plan

Explicit null on a session that has a value turns the split off. The stats fields treat null as "keep"; here null is the documented off value, and without this a host would have no way back short of a new session id. Omitting the field keeps the stored value, as the others do.
Integral floats such as 36.0 are refused with the other non-integers, since Python's json.dumps(60e6) writes 60000000.0 and a host should see the 400 and pass an int.
/reload_expr replays the value but does not accept a replacement in its body, as the scope says.

Not in this PR

Chunking by default or from the stats policy (the phase-0 measurements gate it separately).
Any client change.
CSV chunking (each chunk reparses the file, 2.1-2.2x).

🤖 Generated with Claude Code

…the _handle_widget_change split (rows-first s1) A default-tier XorqBuckarooWidget and BuckarooWidget publish df_data_dict, then df_display_args, then the rest of the widget_args_tuple observers on a search change, and merged_sd carries the full stat set. These pass on main and pin the behaviour the split must keep. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…(rows-first p31) New tests/unit/server/test_stats_policy.py for buckaroo/server/stats_policy.py, which does not exist yet: - resolve_stats_policy: a table of (backend, source_kind, rows, cols, host_tier) against (tier_target, auto_request, requestable, reason), including the boundaries of each threshold and the three tallyman entries over 70 s. - The ceiling holds for host_tier="full", for a reload that re-resolves against a larger row count, and for a force request, and every tier a result lists as requestable is one a force would be granted. - Probes add no data query: the parquet footer probe reads a small share of a counting file object and decodes no row group, the dtype probe never collects a LazyFrame or executes a xorq expression, and a known xorq count is an input. - route_polars_entry returns "xorq" above R and "eager" at or below it. - Threshold overrides from BUCKAROO_* environment variables. Nothing imports the module yet. The tests error at fixture setup on the missing module until the implementation lands. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

… p31) New buckaroo/server/stats_policy.py, pure and imported by nothing yet: - resolve_stats_policy(backend, source_kind, rows, cols, bytes, host_tier, limits) returns tier_target, auto_request, requestable, reason and estimate over the tiers schema < scalar < full. A ceiling is computed inside the function, so every caller gets the lower of the requested and ceiling tiers with reason "ceiling". A host tier lowers freely and raises only to the ceiling. Eager pandas and polars resolve to full. - route_polars_entry(rows, cols) returns "xorq" above R rows and "eager" at or below it. - probe_dtypes reads a schema without collecting or executing anything, and probe_parquet_rows reads a parquet footer's num_rows without decoding data. A xorq count is an input to the policy, never computed by it. - The thresholds are provisional constants gathered in StatsLimits, each with a BUCKAROO_* environment override read on every call. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…-handler fields (rows-first s1) A schema-tier XorqServerDataflow should match full stats on pinned_rows, data_key, summary_stats_key and (except the stats-derived minWidth) column_config, issue no data query besides the cached count, and keep init_sd hints and sorted windows working. A pending state must write nothing under a full-tier cache key, and a later full assignment must reach merged_sd for the raw, clean and filt scopes. assemble_merged_sd must equal merged_sd, and _handle_widget_change must be built from separately callable all_stats and display-args builders. /load_expr and /reload_expr accept stats_tier and stats_delivery, replay them on reload, and keep them out of the warm short-circuit's has_config tuple. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

… s1) Add a dataflow-level stats_tier ("full" default, "schema"). The xorq schema tier builds identity and typing for every column from the expression's schema, with no data query beyond the cached row count, so a dataflow constructs in milliseconds rather than the stats' hundreds. The tier is part of _scope_cache_key, so a schema entry is never read as a full one, and _populate_sd_cache stores summary_sd under the filt key only if it was computed for the current frame, klass list and tier. add_analysis no longer builds DFStatsClass outside the hook when the tier is not full. The merged_sd observer body is extracted as the pure assemble_merged_sd, and _handle_widget_change is split into _build_df_data_dict and _build_df_display_args. /load_expr and /reload_expr accept stats_tier and stats_delivery, stored on the session beside dataflow_kwargs and replayed on reload. They stay out of the has_config tuple; the warm short-circuit compares the stored pair. stats_delivery="deferred" builds the schema-tier dataflow and publishes it. Defaults (full, inline) leave behaviour unchanged. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…esolve_stats_policy (rows-first p31) bytes and source_kind were documented as validated but are not: bytes=-1, 'lots', 1.5 and object() are accepted, a np.int64 is echoed unchanged so json.dumps of the result fails, a host tier passed positionally lands in the bytes slot, and source_kind=None or 5 is accepted. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…ation test (rows-first s1) The Max Versions jobs resolve pandas 3, which reports a string column's dtype as 'str' where pandas 2 says 'object'. The characterization test asserted 'object'. Verified in a Max Versions environment (pandas 3.0.6, polars 1.44.2, xorq 0.4.5): the unit suite passes. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…rows-first p31) bytes goes through the same integer check as rows and cols, so a negative, float or non-numeric value raises, a numpy integer is echoed as a plain int and the result stays JSON-serialisable, and a host tier passed positionally into the bytes slot raises instead of being echoed. source_kind must be a str; its vocabulary stays open for the callers that will read it. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…g on a skipped column (rows-first s1) A column in skip_stat_columns gets only name, dtype and length from the full-tier pipeline, so its _type comes from init_sd. The schema tier layers the schema-derived _type and is_* keys over it, so an int64 column that init_sd types as float merges as integer and renders with zero fraction digits instead of the float displayer init_sd asked for. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…ows-first s1) _get_schema_sd never read skip_stat_columns, so a skipped column's schema-derived _type and is_* keys overrode init_sd's _type once merged. The full tier gives a skipped column only name, dtype and length. The schema tier now does the same, so init_sd's _type decides the displayer at both tiers. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…er (rows-first s2) ServerDataflow and PolarsServerDataflow with stats_tier="schema" should publish the display state full stats give (column_config without stats-derived keys, pinned_rows including a host-supplied one, data_key, summary_stats_key) with no stat computed on the data, still apply init_sd, serve sorted windows, take a later full assignment into merged_sd for every scope, and assemble to the same sd as merged_sd. Both backends run through the same parametrized class, plus a pandas test that pins how an object column is typed from its dtype. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

ServerDataflow and PolarsServerDataflow now implement the _get_schema_sd hook, so stats_tier="schema" builds a dataflow that types every column from its dtype and runs no stat on the data. schema_sd (stat_pipeline.py) builds the sd the way process_df shapes it: an empty frame gives {}, a skipped column keeps only its names. pandas applies the existing typing_stats to a zero-row slice and derives _type through the _type stat; polars factors pl_dtype_typing out of pl_typing_stats and feeds it the dtype. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…s (rows-first p31b) The boundary tables now run on the module defaults and are written against the phase-0 proposals: full auto up to 12M rows and 520M cells, scalar auto up to 1.0B cells, full refused above 25M rows or 1.0B cells (force included), a scalar ceiling of 4.0B cells, and polars routed to xorq above 8M rows. Each threshold has an equal, a one-below and a one-above row, in rows and in cells. New tests pin each default to its literal value and cover the two new environment overrides, BUCKAROO_STATS_FULL_AUTO_CELLS and BUCKAROO_STATS_CEILING_FULL_CELLS. They fail on the current constants (10M rows, 500M cells, 50M-row ceiling, no scalar ceiling, R of 10M) and on the missing full_auto_cells and ceiling_full_cells fields. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…tats_request (rows-first s3) On a deferred /load_expr session a stats_request {stats_gen, scope} should return a stats_update with the matching stats_gen whose inline wide payload equals the all_stats an inline session sends, a stale stats_gen should get stats_aborted and run no query, and /load_expr and /reload_expr should bump the generation. df_meta.stats should be injected on every frame and survive a dataflow-field change, which returns the session to the schema tier. With a caps client and a legacy client on one session, the legacy client should keep getting complete messages through the websocket broadcast, the /load_expr, /reload_expr, /load and /load_compare pushes and the highlight overlay, while the caps client gets a stats-free frame and then pulls a stats_update. A spy telemetry sink should see a stats.request span. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…measurements (rows-first p31b) Full auto now needs at most 12M rows and 520M cells (was 10M rows), scalar auto goes to a 1.0B cell budget (was 500M), `full` is refused above 25M rows or 1.0B cells including for a forced request (was 50M rows), scalar gets a 4.0B cell ceiling by default (was unset; an extrapolation with no measurement behind it), and polars routes to xorq above 8M rows (was 10M; the figure assumes pre_limit False). Each DEFAULT_* constant carries the measurement it comes from. The two new bounds are full_auto_cells and ceiling_full_cells, appended to StatsLimits, with BUCKAROO_STATS_FULL_AUTO_CELLS and BUCKAROO_STATS_CEILING_FULL_CELLS as overrides under the same parsing rules. _size_tier and _ceiling_tier read them; signatures and the result shape are unchanged. Three existing tests that hard-coded the old boundaries are updated: the numpy-integer case (11M rows is now full), the reload case (a 40M-row entry is now refused full) and the empty-environment case for the scalar ceiling (an empty value now means the 4.0B default, not no ceiling). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…eration edge cases (rows-first s3) Four more cases for the stats wire format, kept in their own commit so each is seen failing on CI before the implementation lands. Returning to a state whose stats were completed once is answered from summary_stats_cache with no query. Completing the stats keeps the session's component_config on the refreshed display config. A warm /load_expr, which rebuilds nothing, leaves stats_gen alone. A stats_request on a session with no data is answered with stats_aborted. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…ed sessions (rows-first s3) A client that advertises ?caps=stats_update gets a stats-free initial_state on a deferred /load_expr session (df_meta.stats.status "pending") and pulls the stats with stats_request {stats_gen, scope}. The reply is a stats_update carrying the dataflow's all_stats as an inline wide envelope, or stats_aborted when the generation is stale. The request is the whole run: one synchronous call that computes the full stats, writes the full-tier summary_stats_cache entry, assigns summary_sd and refreshes the session snapshot through one helper. stats_gen is a server-owned counter bumped by every load handler and by a state change that touches a dataflow field, which also returns a deferred session to the schema tier. df_meta.stats is injected by build_state_message from the session, since the dataflow rebuilds df_meta wholesale. Every send site goes through build_state_message_for, so a client without the capability still gets complete messages (its missing stats run synchronously first); broadcast_state replaces the five copies of the send loop and sends to capable clients first. The stats_request branch binds the session's telemetry sink and emits a stats.request span. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…s (rows-first s4) Both pass on the current code. process_df is written out as the process_column loop it is, and each xorq histogram query's snapshot-cache key is compared with the key of a query built in the test, so the refactor into resumable units that follows can be checked against something that does not go through the units. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…tore (rows-first s4) Each stats class gets plan(state) and run(unit, acc): the fragments of the planned units, assembled through assemble_merged_sd, must equal the full-stats merged_sd on pandas, polars and xorq with init_sd, cleaning and overrides; a column group returns only its columns; skip_stat_columns columns get no unit; the xorq batch can be split by column chunk behind a default-off flag and is refused for anything but a plain parquet scan; the histogram cache keys stay put. A StatRun held on the session, keyed by (stats_gen, scope), keeps an append-only fragment list and accumulator, runs a unit only when asked, is dropped when stats_gen changes, and each connection keeps its own cursor. test_the_batch_is_one_query_by_default passes on the current code: it pins the default the flag must not change. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

StatPipeline and XorqStatPipeline get plan(state), new_accumulator(state) and run(unit, acc); process_df and process_table are those three in order, so the inline path and a caller running units one at a time compute the same stats. pandas and polars plan one unit per non-skipped column. xorq plans the scalar batch (which also yields histogram_bins), then one histogram query per column, visible columns first. The batch can be cut into column chunks sized by cells when a host sets stat_chunk_cells, and only for a plain parquet scan: any other source falls back to the single batch and logs why. A StatRun, kept on the session by (stats_gen, scope), holds the planned units, an append-only fragment list and the accumulator. It has no thread, timer or callback, begin_stats_generation drops it, and each WebSocket connection keeps its own StatCursor into the list. CustomizableDataflow.build_stats is the one place a stats class is built, with run=False for plan/run callers. Two of the new tests are corrected here: the default-path test now ignores the PERVERSE_DF self-check units, and the xorq comparisons round floats to nine digits because a parallel aggregate adds its partial sums in varying order. The new-module imports in the tests move to module level. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

… once (rows-first s4) A column group, the priority columns and StatRun's prefer hint accept original or rewritten (a, b, c) names and read each name in both. When an original name equals another column's rewritten name, a group returns the columns of both, priority ranks both first, and prefer picks the unit of whichever comes first in the frame. On a frame whose columns are c, b, a (rewritten a, b, c): - StatState(df, columns=('c',)) plans the units of c and a. - priority=('a',) plans c before a. - StatRun.next_unit(prefer=('a',)) picks the unit of c. The new tests also pin the namespace argument the fix adds, so a client that holds the rewritten names can ask for exactly those: StatState(namespace=) and StatRun.next_unit/run_next(namespace=), with "any" (the default), "original" and "rewritten". Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…rst s4) A column group, StatState.priority and StatRun's prefer hint matched a name against both the original and the rewritten (a, b, c) column names. When an original name equals another column's rewritten name, one name picked two columns: a group returned columns that were not asked for, and prefer ran the unit of whichever column came first in the frame. resolve_names reads each name once. In "any", the default and what the code did for names that do not collide, a name that is an original column name picks that column and only a name that is not one is read as a rewritten name. StatState(namespace=) and StatRun.next_unit/run_next(namespace=) also take "original" and "rewritten", so a caller that holds the client's rewritten names says so and gets exactly those columns. Priority and prefer names are resolved against every column of the frame, not only the columns in the group, so a name picks the same column whatever group is asked for. prioritized now takes the state, since it needs the frame's columns and the namespace to read the priority names. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…t cursors and the final assignment (rows-first s5) Covers the request branch that runs units for a time budget (at least one, one cold unit per budget), the per-handler cursors over a shared StatRun, the final assignment that frees the run and carries the rebuilt df_display_args, a legacy client finishing a run in progress, a concurrent infinite_request between units, the end-to-end order of first initial_state, rows and stats, /reload_expr mid-run, and the stats.unit and firstpull.stats_total spans. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…nt cursors and the final assignment (rows-first s5) A stats_request with incremental: true runs the units of the session's StatRun for about STATS_BUDGET_S (at least one) and answers with the fragments the asking client has not seen, read through the handler's own StatCursor. The request that runs the last unit does the final assignment in complete_stats: the full-tier summary_stats_cache entry, summary_sd, the session snapshot and the status in one step, the run freed, and the final reply carries the rebuilt df_display_args when its digest differs from the one the client holds. Without the field a request is still the whole run, now finishing the run's remaining units when a client has made progress. A legacy client's complete frame does the same. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…olicy module, PRs #1019 and #1023)

…handlers (rows-first p33) stats_tier takes auto, full, scalar or schema on /load_expr and /reload_expr and is stored with the pair. The policy resolves after the schema-tier dataflow and the count exist, is stored on the session, is reported in df_meta.stats (tier_target, reason, auto_request, requestable, estimate, omitted_keys, approx_keys, demand_columns, each with its documented default) and is applied at WebSocket open only for a client that sends ?caps=stats_update,stats_ondemand. The version-skew cases (no caps, stats_update only, both bits, an old server) and /load, which keeps resolving to full, are covered. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…and the compare reset under a stats policy (rows-first p33) Cases found untested after the first tests commit, each one a regression the first set did not catch: a stats_update client pulling units from a policy session gets full-tier updates and the final reply carries the rebuilt df_display_args, a scalar tier named with inline delivery builds a schema dataflow, and /load_compare clears the policy a session held. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…irst p33) /load_expr and /reload_expr accept stats_tier auto, full, scalar or schema, stored with the pair and kept out of has_config, so the warm short-circuit holds. When the dataflow is built at the schema tier the handler resolves the policy (resolve_stats_policy) from the count it already has, stores it on the session and starts the stats generation from it: a target below full is not_computed with the policy's reason. df_meta.stats reports the policy as tier_target and estimate, plus auto_request, requestable, omitted_keys, approx_keys and demand_columns where they differ from their documented defaults. It is applied at WebSocket open only for a client that sends ?caps=stats_update,stats_ondemand. A client with stats_update only is told the session is pending and pulls the stats, and a client with no caps gets them at connect, as for any deferred session. A tier the host named (scalar, schema) reaches every client as before. A session on an explicit stats_tier full within the ceiling sends the message it always has. /load keeps resolving to full: it does not read the field and clears a policy left on the session. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

… p34) A xorq stat run at the scalar tier is the scalar class of the batch without approx_median and distinct_count, and no histogram query. Its histogram_bins come from the batch's min and max. The run behaves like a filtered one: fragments go to the clients and nothing is assigned to the dataflow, the session snapshot or summary_stats_cache, so scalar stats are never served as the complete ones. Tests cover the plan, the queries a scalar run issues, the fragment union against the full stats, the bins, the chunked batch, the StatRun key and assignment rule, which clients are served the tier, and the stats_update messages of an ondemand client on a scalar target. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…d generations (rows-first p34) A deliberate regression of the implementation showed four behaviours the first set of tests does not cover: a stat that reads a key the scalar tier leaves out is left out with it, a stat that reads distinct_count runs on the unknown, a scalar request emits the request and unit spans with the scalar tier and no completion span, and a dataflow-field change drops the scalar run and starts one over the new state. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…signed (rows-first p34) StatState takes a tier. A xorq run at the scalar tier is the batch alone, in the same column chunks, without the aggregates behind median and distinct_count and with no histogram unit. Its histogram_bins come out of the batch's min and max, so color_map needs no histogram query. The accumulator withholds median, distinct_count, distinct_per and histogram, so no fragment and no assembled sd carries a key the tier did not compute. The pandas and polars pipelines refuse a non-full state. StatRun takes a state, and reports its tier and whether it assigns (only the full tier over every column does). The full run keeps its (stats_gen, scope) key; every other run has a key of its own, so it never takes the full run's place. start_stat_run takes a tier and a column group. _run_summary refuses a run that does not assign, so a scalar or column-scoped sd is never written to summary_stats_cache as the complete one. A stats_request from a client that advertised stats_update and stats_ondemand, on a session whose policy target is scalar and has nothing computed, runs the scalar units and answers stats_update with tier scalar, final and remaining. The run behaves like a filtered one: the fragments go to the clients that ask (each through its own cursor) and nothing is assigned to the dataflow, the session snapshot, summary_stats_cache or the status. Other clients are served as before. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…demand scan (rows-first p36a) A stats_request from a client that advertised both capability bits can now say tier, force and columns. The tests pin what the server does with them: the ceiling refuses a tier for the cells asked for and runs nothing, requestable and the target decide which tier a whole-table request may name, force raises the target through a session override that survives a dataflow-field change for the unfiltered scope only, an automatic unit over the budget pauses the session (not_computed for cost) until a force request, and the pause survives a generation and /reload_expr. The demand scan reads the color_map rules of the built config (overrides and rules a klass adds at style time) with no query, and a scoped request computes those columns only. Every test here fails on the earlier code. The guard tests that pass there (a client without both bits is served as before, a session with no policy ignores the fields) go in with the implementation. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…the demand scan (rows-first p36a) A stats_request from a client that advertised both capability bits reads tier, force and, with a tier, columns. The server judges it: the ceiling for the cells asked for (resolve_stats_policy with the tier as the host's) answers a refusal and runs nothing, a whole-table tier must be the target or above it with force, and a paused session refuses a request that is not forced. A force above the target stores stats_override on the session, and effective_stats_policy raises the target for the unfiltered scope and for the generation it was forced in. begin_stats_generation, stats_meta and serves_scalar_tier read the effective policy. An automatic request that ran a unit over STATS_COST_BUDGET_S and left units to run sets cost_paused, which begin_stats_generation reads, so the pause survives a stats_gen bump and /reload_expr until a force request. The override and the pause are forgotten by /load, /load_compare, another expression and a body that names a tier. stats_policy.demand_columns scans the built display config for color_map rules and names the columns they read; the snapshot refresh writes them to stats_policy["demand_columns"] for a schema target, and a scoped request computes those columns only, with a run that is never assigned. The final reply of a run that does not assign now carries the session's status. Guard tests that pass on the earlier code go in with this commit, and the elapsed_ms test has wider margins on both sides. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…n it continues after a cost pause (rows-first p36a) A client that continues a cost-paused session (force with no tier, or a tier equal to the target) never has its display_args_hash recorded, so the final stats_update carries no df_display_args and the client keeps the config its not_computed frame carried. Two clients are in that state: one that connected while the session was paused, and one that changed state (a search) while it was. Both complete the session with force and expect the rebuilt config in the final reply. The client that connected pending before the pause is covered by the existing tests. Both tests fail on the earlier code: the final reply has no df_display_args. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…sed run (rows-first p36a) _apply_force recorded the client's display_args_hash only when a force raised the target above the policy's. A continue after a cost pause names no tier, or the target, so the digest was never recorded: build_state_message_for had already cleared it for the paused not_computed frame, and the final stats_update left out df_display_args. A client that connected while the session was paused, or changed state while it was, kept the schema-tier config its frame carried after the session completed. The digest is now recorded for any whole-table force that runs the full tier, by raising the target or by continuing to it. A force for the scalar tier or for named columns still records none, since neither run ends in an assignment. A client that already holds a digest keeps it. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…arched count memo (rows-first p37) Above the sort threshold every column of the grid served by infinite_request has ag_grid_specs sortable false, including a column a klass or a host override styles, and a sorted infinite_request gets error_code sort_disabled. df_meta carries sort and search flags. A repeated searched xorq window issues one count. The thresholds come from the stats policy module. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…count memo (rows-first p37) A regression check on the first tests commit left cases that no test pinned. A searched expression a base holds is bounded and the least recently asked term goes first, a /load_compare ends the guard like /load does, and the pass that turns sorting off leaves a column entry that is not a dict as it is and replaces ag_grid_specs that is not a dict. The wire test of a state change reads the three rows first_three leaves, not five. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…count memo (rows-first p37) stats_policy gains GuardLimits (BUCKAROO_SORT_DISABLE_ROWS, default 25M rows; BUCKAROO_SEARCH_DISABLE_ROWS, default none) and resolve_source_guards, apart from the stats thresholds. A session a host opened with a stats policy resolves them at /load_expr and /reload_expr from the count load took. Above the sort threshold the xorq dataflow finishes its display config with disable_sorting, so no klass or override leaves a sortable column in a display infinite_request serves, and a sorted infinite_request is refused with error_code sort_disabled before any query runs. df_meta carries sort and search when one is disabled. handle_infinite_request_xorq holds the searched expression per (base expression, term), so the second window of a search is a hit in _expr_count's cache and issues no count. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…load_expr (rows-first s6) /load_expr should accept stat_chunk_cells (a positive integer cell count, or null for off) and /reload_expr should replay it, so a host can turn on the xorq stats column-chunk split over HTTP. Tests cover the field reaching XorqServerDataflow.stat_chunk_cells, several batch units and unchanged stats on a parquet scan, one batch unit on a join, the warm short-circuit rules, 400 invalid_stat_chunk_cells, the reload replay, and /load and /load_compare dropping a stored value. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

github-actions · 2026-10-05T11:35:51Z

📦 TestPyPI package published

pip install --index-strategy unsafe-best-match --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ buckaroo==0.15.9.dev37353046864

or with uv:

uv pip install --index-strategy unsafe-best-match --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ buckaroo==0.15.9.dev37353046864

MCP server for Claude Code

claude mcp add buckaroo-table -- uvx --from "buckaroo[mcp]==0.15.9.dev37353046864" --index-strategy unsafe-best-match --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ buckaroo-table

📖 Docs preview

🎨 Storybook preview

…r /load_expr (rows-first s6) /load_expr accepts stat_chunk_cells, a positive integer cell count or null for off, and passes it to XorqServerDataflow, whose scan guard still keeps a join, aggregate, diff, CSV read or registered table on the single batch. It is stored on the session beside stats_tier and stats_delivery and outside has_config, so a warm re-POST with the same value short-circuits and a changed value (null included) rebuilds; omitting the field keeps the stored value. /reload_expr replays it, and /load and /load_compare reset it. A zero, negative, non-integer or bool value answers 400 invalid_stat_chunk_cells. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

paddymul and others added 30 commits October 3, 2026 22:51

Merge origin/feat/rowsfirst-p31b-policy-calibration into p33 (stats p…

e552b40

…olicy module, PRs #1019 and #1023)

paddymul and others added 9 commits October 4, 2026 06:35

paddymul deployed to testpypi October 5, 2026 11:35 — with GitHub Actions Active

paddymul deployed to testpypi October 5, 2026 18:04 — with GitHub Actions Active

paddymul mentioned this pull request Oct 6, 2026

docs(adr): ADR-002 rows-first delivery with server-pushed summary stats #1043

Merged

paddymul changed the base branch from main to adr-002-rows-first-stats-delivery October 6, 2026 14:40

paddymul mentioned this pull request Oct 6, 2026

feat(xorq): stats tier machinery and the xorq schema tier (rows-first s1) #1021

Merged

paddymul force-pushed the adr-002-rows-first-stats-delivery branch from c500780 to f61df3d Compare October 6, 2026 18:27

Navigation Menu

Repository navigation

feat(server): let a host enable the xorq stats column-chunk split over HTTP (rows-first s6) - #1035

feat(server): let a host enable the xorq stats column-chunk split over HTTP (rows-first s6)#1035
paddymul wants to merge 40 commits into
adr-002-rows-first-stats-deliveryfrom
feat/rowsfirst-s6-chunk-cells-http

paddymul commented Oct 5, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 5, 2026 •

edited

Loading

Uh oh!

This branch was successfully deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

paddymul commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Stack

Problem

Phase and plan references

Approach

What changes

Tests

Measurements

Why default behaviour is unchanged

Deviations from the plan

Not in this PR

Uh oh!

github-actions Bot commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📦 TestPyPI package published

MCP server for Claude Code

📖 Docs preview

🎨 Storybook preview

Uh oh!

This branch was successfully deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

paddymul commented Oct 5, 2026 •

edited

Loading

github-actions Bot commented Oct 5, 2026 •

edited

Loading