feat(server): stats request limits, session override, cost guard and the demand scan (rows-first p36a)

[paddymul]

Stacked on #1031 (feat/rowsfirst-p34-scalar-tier-units), which carries #1021, #1022, #1024, #1026, #1028, #1029 and the stats policy of #1019 and #1023. This PR is based on main so the repo's Checks workflow runs on it (its pull_request trigger uses branches: "*", which does not match a base branch containing a slash). The diff therefore includes the commits of those PRs until they merge. The commits of this phase are d4477c46 (failing tests) and 9af4042c (implementation); read only those (git diff origin/feat/rowsfirst-p34-scalar-tier-units...HEAD).

Problem

A client that advertised stats_ondemand can be told a session has no stats (not_computed) and what it could ask for (requestable), but it cannot ask. A stats_request carries no tier, so the only runs a server can start are the ones the policy target names, and a request for anything else is not_requestable. Three things the plan puts on the server are missing:

• The ceiling and requestable are advertised but not enforced on a request, so a client that sends a tier the policy did not offer has nothing stopping it.
• Nothing remembers that a user asked for more than the policy gave. A forced tier would be lost at the next dataflow-field change, and a run that was too slow to continue automatically would be restarted by the next stats_gen bump or /reload_expr.
• A color map reads histogram_bins of its val_column. When the policy computes no stats, those columns get no bins, and nothing finds out which columns they are.

Phase and plan references

Rows-first p36a, from buckaroo2-reports/plans/: plan 3 (03-no-summary-stats-for-large-files.md) section 6 "Phase 6a", with section 3.2 (the cost guard, elapsed_ms), section 3.3 (the demand-driven minimum) and section 3.4 (the control's server side: force, the ceiling answer, the session override). Plan 1 sections 3 and 4.0, plan 2 sections 3 and 4.1.

Approach

• Request fields. For a client that advertised both bits, on a session that has a policy, stats_request reads tier (scalar or full), force (a boolean) and, together with a tier, columns (the grid's a, b, c names) that scope the run to those columns. Without a tier, columns is only the ordering hint it was. A value the server cannot read is stats_aborted with reason bad_request. Every other client, and a session with no policy, ignores all three fields.
• Judging a request (_gate_request), in this order. The ceiling: resolve_stats_policy is called with the requested tier as the host's, over the cells asked for (all of them, or rows times the named columns), so the ceiling is the same function load uses; a tier it lowers is answered stats_update {final: true, status: "not_computed", reason: "ceiling"} with no payload and nothing run. Then, for a whole-table request, the tier must be the target or above it with force (below the target, or above it without force, is not_requestable, as is a schema target with no tier). Then the cost pause: a request without force on a paused session is answered with reason cost. A column-scoped request needs no force, since it is how the demand columns are asked for.
• The override. A force above the target stores stats_override and the generation it was made in on the session. effective_stats_policy(session) is the policy in force: the one resolved at load, or the one the override tier resolves to for the same entry (never lower than the target, still under the ceiling, thresholds read again). It applies to an unfiltered state and to the generation it was forced in. "Filtered" is a quick command (a search) on the operations, which is the difference between the dataflow's filt and clean chains; a cleaning method or a post-processor does not make a state filtered. begin_stats_generation, stats_meta and serves_scalar_tier read the effective policy, so a forced full makes the session pending for full again after a dataflow-field change, and a client that connects later is told so.
• The cost guard. run_units can report how long each unit took (from the run's own timer). An automatic request, which is a request without force, that ran a unit over STATS_COST_BUDGET_S and whose reply leaves units to run sets cost_paused and the status not_computed for cost. A unit cannot be cut short, so the unit that went over has run; the guard stops the next one. A request that finishes the run pauses nothing, and a forced request never pauses. A force clears the pause, and the run continues where it stopped (ran is kept). The pause is on the session, so begin_stats_generation reads it (initial_stats_status(..., cost_paused)) and it survives a stats_gen bump and /reload_expr. A client without both bits is not held by it: it is told pending and gets the stats when it pulls, as before.
• When the controls are forgotten. reset_stats_controls clears the override and the pause on /load, /load_compare, a /load_expr of another expression, and a /load_expr or /reload_expr whose body names a stats_tier. A rebuild of the same expression, or a reload that names none, keeps them.
• The demand scan. stats_policy.demand_columns(display_args, pairs) collects the val_column of every color_map_config whose color_rule is color_map, in any display, and returns the rewritten names in the table's order. It reads the built config, so it covers rules in the column overrides and rules a klass adds at style time, and it sends no query. A val_column may be the rewritten name or an original one (a rewritten name wins), and one that names no column is dropped. A categorical rule, a tooltip and a color_map with no val_column create no demand. refresh_demand_columns writes the result to session.stats_policy["demand_columns"] at load and with every snapshot refresh (a dataflow-field change, /reload_expr), for a schema target only, since a higher target computes those columns anyway; df_meta.stats reports it to an ondemand client through the existing _policy_fields.
• Scoped runs. A column-scoped request is served by _serve_run from start_stat_run(tier=..., columns=...), the run p34 added: its key is (stats_gen, scope, tier, columns), two requests for the same columns in any order share it, it never assigns, and a client that asks later reads its fragments with no query. Its final reply says what the session still is (status, and reason when there is one). The scalar run of p34 is served by the same function and now says the same on its final reply.

What changes

• buckaroo/server/session.py: stats_override, stats_override_gen and cost_paused on SessionState; initial_stats_status(..., cost_paused); effective_stats_policy, reset_stats_controls, restore_stats_status; begin_stats_generation and stats_meta read the effective policy; session_dataflow moved here from stats_wire (still importable from there).
• buckaroo/server/stats_wire.py: STATS_COST_BUDGET_S, REQUEST_TIERS, TierRequest, _tier_request, _gate_request, _apply_force, _pause_if_slow, _refused; _serve_scalar is now _serve_run (tier and columns); run_units(timings=); refresh_demand_columns, called from refresh_session_snapshot; serves_scalar_tier reads the effective policy.
• buckaroo/server/stats_policy.py: demand_columns; the module docstring no longer says nothing imports it.
• buckaroo/server/handlers.py: the resets above, same_expression at /load_expr, the demand scan after the snapshot is stored.
• No change to websocket_handler.py, dataflow.py or any client.

Tests

Two commits: d4477c46 (failing tests, pushed alone) and 9af4042c (implementation and guard tests). Tests are in the existing files of their kind.

• test_stats_policy.py (pure, every Python and Windows): TestRequestsOverTheCeiling, TestMalformedTierRequests, TestRequestableTiers, TestCostPausedStatus, TestEffectiveStatsPolicy, TestDemandColumns. They cover the ceiling answer for six combinations of host tier, limits and request, each unreadable field, which tiers a whole-table request may name, the status a paused session starts a generation with and the order of the checks, what a forced tier raises the policy to and where it applies, and the demand scan on every shape of config.
• test_load_expr.py (xorq, over HTTP and WebSocket): TestForcedRequestsWire, TestCostGuardWire, TestCostGuardUnits, TestDemandScanWire, TestTierFieldsAndOlderClients. A forced scalar and a forced full run for real; the ceiling answer runs no query; the override across a field change, a search, a reload and a new expression; the pause across a generation, a reconnect and a reload; a unit under the budget, a forced unit and a request that finishes the run do not pause; elapsed_ms on updates and refusals; the demand columns of a klass at style time, of an override, and none for categorical rules, with zero stat queries at load; a demand request runs the named columns only, assigns nothing and is replayed to a second client with no query; the ceiling judged per named column.
• Added with the implementation because they pass on the earlier code: nine tests that pin what does not change (a stats_update-only client has the fields ignored and a host-named schema target still refuses it, a session with no policy ignores them, an explicit full session serves plain requests as before, a client without both bits is not held by a pause, a session that computes the columns anyway has no demand, categorical rules alone have none, a session nobody forced stays not_computed through a field change, a scoped request on a complete session is answered from the dataflow).

The elapsed_ms test has bounds of 50 ms in d4477c46: a request that ran a 50 ms query takes at least that long, and a refusal takes less. 9af4042c widens both sides (a 300 ms query, bounds of 250 ms), because an upper bound of 50 ms on a refusal can fail on a loaded machine (the load average here reached 35 during the full suite). The test fails on the earlier code for the same reason in both versions: the scoped request is refused with not_requestable.

On d4477c46 all nine Python / Test jobs failed (3.11 to 3.14, Max Versions 3.11 to 3.14, Windows), with all 28 checks completed; the other 18 succeeded (Docs, Lint, Typecheck, the JS job, the wheel build and the Playwright jobs among them) and deploy was skipped. CI logs were not read (the REST log API is rate-limited for this account), so the reasons rest on the local run of the same commit: 107 of the new tests fail, each on a missing session attribute or function, on a reply with no tier, final or elapsed_ms where a stats_update is expected, or on a request refused with not_requestable or stats_aborted where an answer is expected, and the other 1940 unit tests pass.

CI on 9af4042c: all 28 checks completed, 27 succeeded (one of them the Read the Docs status) and deploy was skipped. That includes all nine Python / Test jobs, Python / Lint, Python / Typecheck, Docs / Build + Check Links, the JS job, the wheel build and the Playwright jobs.

Local checks on 9af4042c:

• The full unit suite (pytest ./tests/unit -m "not slow") gives 2056 passed and 5 skipped: the 1940 of feat(stats): xorq scalar tier units, served as fragments and never assigned (rows-first p34) #1031, the 107 failing tests, and the 9 guard tests. The same 2056 and 5 in an environment resolved the way the Max Versions job does it (pandas 3.0.6, polars 1.44.2, xorq 0.4.5, numpy 2.5.3), in a copy of the tree with the lockfile removed and uv sync --resolution=highest. That copy left out packages/, so one test that reads a committed JS fixture (test_xorq_window_fixture.py) failed there until the fixture file was copied in; it then passed.
• Forty-five deliberate regressions of the implementation (each check of _gate_request removed, the pause never set, set on a final reply, set on a forced request, never cleared by force, the override not recorded, ignoring the filtered scope or lasting only one generation, the reset removed from each of its triggers, the demand scan not refreshed, not limited to a schema target, including categorical rules, letting an original name win, the fields applying to every client or to a session with no policy, a scoped request served as the whole run, and others). On the first set of tests 40 were killed and five survived. One survivor was an equivalent mutation of a redundant check, which is gone now (simplifying it also exposed that an unhashable val_column raised TypeError in the scan, which is fixed and tested). The other four led to two new tests (a paused schema target has nothing to request; /load and /load_compare forget the controls) and two tighter ones (a forced unit is checked right after its reply, not after a later forced request has cleared the pause; a session with no policy ignores unreadable fields too). With those, all the others are killed.
• Real server processes (python -m buckaroo.server --no-browser, started and stopped by me, killed by listener PID, ports free afterwards), on a 60,000-row, three-column xorq build with a deferred auto session. On port 8941 with BUCKAROO_STATS_FULL_AUTO_ROWS=1000, BUCKAROO_STATS_SCALAR_AUTO_CELLS=100000 and BUCKAROO_STATS_CEILING_FULL_ROWS=50000 and a color_map override naming price: the first frame of a client with both bits was not_computed, reason size, tier_target schema, requestable ["scalar"] and demand_columns ["a"]. A request with no tier was not_requestable; the demand request (tier: "scalar", columns: ["a"]) was a stats_update with the scalar rows of column a only in 16.3 ms; a bad tier was bad_request; scalar without force was not_requestable; full with force and a scoped full over three columns were refused with reason ceiling in 0.0 and 0.1 ms; scalar with force ran for all three columns (24.3 ms) and a client that connected afterwards was told tier_target: "scalar". A stats_update-only client was told pending and a no-caps client got the complete stats. On port 8942 with the cost budget set to 0 and one unit per request by a small launcher script (a 10 s unit cannot be produced on this table): the first automatic request returned a partial update and paused the session; a client that connected then was told not_computed, reason cost, tier_target full, while a stats_update-only client was told pending; the next automatic request was refused with reason cost; a force request continued the run where it stopped and the last one was final.

Why default behaviour is unchanged

• Every new request behaviour is reached only through tier_fields_apply: a client that sent both stats_update and stats_ondemand, on a session that has a policy. No released client sends stats_ondemand, and a session has a policy only when it is deferred or a host named scalar or schema.
• A request without the new fields takes the path it took before, unless the session is paused or the thresholds in the environment have changed since load (the ceiling is judged again at request time). The cost guard acts only for the clients the fields apply to, and nothing sets stats_override or cost_paused without them, so begin_stats_generation, stats_meta and serves_scalar_tier return what they returned.
• df_meta.stats changes only for an ondemand client of a schema target whose config has a color_map rule (demand_columns), of a forced session (the effective target) and of a paused one.
• One status rule changed: a session with a policy whose target is full is pending whatever its delivery. That case could not occur before this PR, since a policy exists only for a dataflow built at the schema tier and a host naming full there is the explicit-full case feat(server): stats policy wire contract and handler plumbing (rows-first p33) #1029 pins.
• The existing tests pass unchanged. The only edit to an existing test file's text is the module docstring of test_stats_policy.py.

Deviations from the plan

• STATS_COST_BUDGET_S is 10 s, a module constant and not an environment setting. The plan gives no number; this is the tallyman client's base timeout and is marked provisional beside the constant with the measurements it can be compared with.
• The guard measures the slowest unit of a request and acts after the reply, not before the unit: units cannot be cancelled (plan 3 section 3.4), so the first unit over the budget has already run and the next one is what the pause prevents. A request that leaves nothing to run, and any forced request, pause nothing. A pause is cleared by one force, and a later automatic request can pause again.
• "The unfiltered scope" (plan 3 question 3) is read as: no quick command on the operations. The plan leaves the answer open ("proposed: yes for the unfiltered scope only"); a cleaning method is on the clean chain, which survives a filter flip, so it does not end the override. A force made in a filtered state serves that state and does not reach another filtered one.
• A whole-table force above the target sets the override; a force that only continues the target (a "Continue" after a pause) sets nothing. A force with no tier on a schema target is not_requestable and clears nothing.
• The ceiling is judged at request time with the thresholds read again, over the cells of the request, with the count load took (estimate). Named columns are judged by rows times their number, so a ceiling that refuses the whole table can still allow a few columns.
• A refusal is a stats_update with tier set to the tier asked for (the target when none was named), status: "not_computed" always, and no payload or df_display_args.
• The final reply of a run that does not assign (the scalar run of p34 included) carries the session's status and reason. p34's scalar final reply carried neither; no existing test pinned that.
• df_meta.stats reports tier_target, estimate and the other policy fields for an explicit full session while it is paused, which it does not otherwise (feat(server): stats policy wire contract and handler plumbing (rows-first p33) #1029). A client that reads reason: "cost" needs to know which tier a force continues.
• The demand scan runs at load and with every snapshot refresh, and writes demand_columns to the session policy (the p33 interface note). It is empty unless the target is schema. Nothing on the server requests the demand columns: they are a scoped stats_request, which the client sends (phase 5), under the same ceiling and pause.
• reset_stats_controls and its triggers are mine; the plan says only that the override and the pause survive a generation and a reload.
• session_dataflow moved from stats_wire to session, because the effective policy needs it and stats_wire imports session.

Not in this PR

• Client rendering of the control, demand_columns and the cost state (phase 5); sort and search guards (phase 7); the config upgrade after stats complete (phase 6b); default flips (phase 9).
• Calibrating STATS_COST_BUDGET_S.
• A bound on the scoped runs a session holds. Each distinct column group keeps a StatRun (an accumulator and its fragments, no data) until the generation changes; a client that asks for many different groups grows that dict.
• Carrying filtered stats on the wire (unchanged from p34: a filtered state's stats_update holds schema rows only).

🤖 Generated with Claude Code

[github-actions]

📦 TestPyPI package published

pip install --index-strategy unsafe-best-match --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ buckaroo==0.15.9.dev37240210873

or with uv:

uv pip install --index-strategy unsafe-best-match --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ buckaroo==0.15.9.dev37240210873

MCP server for Claude Code

claude mcp add buckaroo-table -- uvx --from "buckaroo[mcp]==0.15.9.dev37240210873" --index-strategy unsafe-best-match --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ buckaroo-table

📖 Docs preview

🎨 Storybook preview

[paddymul]

Review pass on this PR: one finding, confirmed and fixed.

Finding (medium): a Continue after a cost pause never records the client's display digest. _apply_force set client.display_args_hash only inside the branch for a tier above the target. A Continue names no tier, or the tier the session is already headed for, so the digest was not recorded. build_state_message_for had already cleared it for the paused not_computed frame. The final stats_update then carried no df_display_args, and the client kept the schema-tier config after the session completed. Two clients were in that state: one that connected while the session was paused, and one that changed state (a search) while it was. The client that connected pending before the pause was not affected, which is why the existing tests passed.

I reproduced both cases before changing anything. In each, the final reply had no df_display_args, the session was complete, and the frame's config differed from the complete one.

How it was addressed

• 6e85bd0 adds two tests to TestCostGuardWire: a client that connects while paused and continues (with force alone, and with tier: "full"), and a client that sends a search while paused and then continues. Each expects the final reply to carry the rebuilt config (with the client's highlight, in the search case). Locally they failed on the missing df_display_args and the rest of the suite passed. On CI, Python / Test (3.11), (3.12), (3.13) and the three matching Max Versions jobs failed on that commit, where the previous head had passed them. The 3.14 jobs passed because xorq is gated to Python below 3.14 and this test module is skipped without it.
• 909ed59 changes _apply_force to record the digest for any whole-table force that runs the full tier, whether it raises the target or continues to it, and keeps the guard that a client already holding a digest keeps it. The override condition is the same as before. A forced scalar request or one scoped to columns still records nothing, since neither run ends in an assignment. The docstring now says this.

Every check on 909ed59 is complete and green (the deploy job is skipped), and the local unit suite is 2058 passed. No existing test was changed.

Not covered: the digest is taken when the client sends force. A client that was told not_computed for cost and then sends a request without force, after another client's force has cleared the pause and run the session to completion, would still get no df_display_args. The finding did not include that case and I did not change it here.

[paddymul]

Closing. The cost guard pauses between units and the demand columns are served as scoped StatRuns, both pull machinery that the revised D2 of ADR-002 (#1043, 7943412) drops.

These still apply to an explicit stats_request under the push and are the parts to carry forward: _gate_request (the ceiling and requestable checks), the override (effective_stats_policy, reset_stats_controls) and the demand_columns scan. The branch is kept for them. #1034 is stacked on this branch and needs a rebase.