feat(server): sort and search guards for huge sources and a searched count memo (rows-first p37)

[paddymul]

Stack

Stacked on #1033 (feat/rowsfirst-p36a-limits-override-cost-guard), which carries #1021, #1022, #1024, #1026, #1028, #1029, #1031 and the stats policy of #1019 and #1023. This PR is based on main so the repo's Checks workflow runs on it (its pull_request trigger uses branches: "*", which does not match a base branch containing a slash). The diff therefore includes the commits of those PRs until they merge.

The commits of this phase are the three after 909ed59c, the head of #1033 (git diff origin/feat/rowsfirst-p36a-limits-override-cost-guard...HEAD):

• c7fb1ff9 and d9be4e90, failing tests, each pushed alone.
• 0181386e, the implementation.

Problem

A session whose source is very large can skip its summary stats (#1029, #1033), but nothing keeps the grid from running the two requests that stay expensive without them:

• A sorted window on a xorq source of tens of millions of rows blocks the loop for the whole sort (261-287 ms warm at 10.8M rows x 43 columns, 326-337 ms at 3M CSV rows, from the phase-0 measurements; the 42M to 78M entries are extrapolations). The grid offers a sort on every column, and a klass or a host override can turn it back on.
• A searched window pays a fresh count() per request. search_expr builds a new expression on every call and _expr_count caches by expression object, so the filtered expression always misses (504-1063 ms for a "NY" search on 10.8M rows, each window).

Phase and plan references

Rows-first p37, from buckaroo2-reports/plans/: plan 3 (03-no-summary-stats-for-large-files.md) section 6 "Phase 7" with section 3.5, and plan 2 (02-rows-first-xorq-and-lazy-polars.md) section 4.3 (the filtered count memo). Plan 1 sections 3 and 4.0 and plan 2 sections 3 and 4.1 for the session and wire contract it extends. The detail file's section 4.7 has the measurements.

Approach

• Thresholds (stats_policy.py). GuardLimits(sort_disable_rows, search_disable_rows) with from_env(), and resolve_source_guards(backend, source_kind, rows, cols, limits=None), which returns {"sort": "enabled" | "disabled", "search": "enabled" | "disabled"}. They are separate from StatsLimits, which is unchanged: a sort scales with rows, the stats tiers with cells. BUCKAROO_SORT_DISABLE_ROWS (provisional default 25,000,000 rows) and BUCKAROO_SEARCH_DISABLE_ROWS (default none) are read on every call, like the stats ones. Eager backends are always enabled. 25M is also the ceiling on full stats (the midpoint of the telemetry gap between 11.8M and 42.3M rows). It is a guess marked provisional, with the extrapolated costs written beside it.
• Who is guarded. Only a session a host opened with a stats policy: resolve_session_guards(dataflow_tier, rows, cols) returns None unless the dataflow was built at the schema tier, as resolve_session_policy does. A session built the way every host builds one today, with its stats inline, has no guard. The count is the one load took (metadata["rows"]), so the guard describes the source, not a search over it, and does not flip while a user types.
• The final pass. XorqServerDataflow.set_sort_enabled(False) makes _build_df_display_args finish with stats_policy.disable_sorting(display_args). That sets ag_grid_specs.sortable false on every column and every left (index) column of each display whose data_key is main, the display infinite_request serves, and keeps the column's other specs. The summary display is a client-side grid and keeps its sort. The pass runs after the klasses and after column_config_overrides are merged, so neither can turn sorting back on. The flag lives on the dataflow, so every rebuild keeps it (a field change, completed stats, /reload_expr).
• The refusal. session.sort_refusal(session, payload_args) returns infinite_resp {key, length: 0, error_info, error_code: "sort_disabled"} for a window that has a sort on a session whose sort is off. DataStreamHandler._dispatch asks it first, so both the first and the second_request window are judged and no query runs. No parquet frame follows.
• The flags. session.source_guards holds the resolved flags. build_state_message adds df_meta.sort and df_meta.search when one is "disabled", to every client (the server refuses the sort whoever asks). An enabled flag is left out, as the stats fields are, so a session under the thresholds sends the message it always has. session.guards_with_defaults(df_meta) reads them with the default filled in. search is advice: there is no server switch for search (plan 3 question 7), so a window with a term is served either way, and a client that wants to hide the search cell reads the flag.
• The count memo. handle_infinite_request_xorq holds the searched expression per (base expression, term) in a weak-keyed map, with a 16-term LRU per base. The second request for the same term gets the same expression object, so _expr_count's own cache serves the count, and a count that failed is still not cached. A state change builds a new base expression, which is a new key.

What changes

• buckaroo/server/stats_policy.py: DEFAULT_SORT_DISABLE_ROWS, DEFAULT_SEARCH_DISABLE_ROWS, GuardLimits, resolve_source_guards, disable_sorting.
• buckaroo/server/session.py: SessionState.source_guards, SOURCE_GUARD_DEFAULTS, guards_with_defaults, sort_refusal; build_state_message adds the flags.
• buckaroo/server/stats_wire.py: resolve_session_guards, apply_source_guards.
• buckaroo/server/xorq_loading.py: XorqServerDataflow.sort_enabled, set_sort_enabled and the _build_df_display_args override; the searched-expression memo in handle_infinite_request_xorq. search_expr is imported at module top instead of inside the function.
• buckaroo/server/handlers.py: /load_expr and /reload_expr resolve and apply the guards; /load and /load_compare clear them.
• buckaroo/server/websocket_handler.py: _dispatch asks sort_refusal first.
• No change to dataflow.py, the widgets or any client.

Tests

120 new tests, in the existing files of their kind.

• test_stats_policy.py (pure, runs on every Python and on Windows): TestGuardLimits, TestResolveSourceGuards (a boundary table in rows, the two thresholds apart from each other and from the stats ones, eager backends, bad inputs), TestGuardFlagsInDfMeta, TestSortRefusal, TestResolveSessionGuards, TestDisableSorting.
• test_load_expr.py (xorq): TestSortGuardDataflow, TestSortGuardWire (every column across a klass and an override, the refusal with no query and no stray frame, second_request, the flags for each kind of client, a reload resolving the guards again in both directions, /load and /load_compare ending the guard), TestSearchedCountMemo and TestSearchedWindowCountsWire (one CountStar for a repeated searched window, for other windows of the same search and for two clients; the bound, the LRU order, a failed count not remembered, and the base expression not kept alive).

The tests came in two tests-only commits, as in #1029 and #1031. c7fb1ff9 has the first set. A deliberate-regression pass over it left cases no test pinned (the memo's bound and order, /load_compare, non-dict column entries), and d9be4e90 adds those and corrects one wire test that expected five rows after first_three, which leaves three. Guard tests that already pass on the earlier code (an inline session is not guarded, flags that are enabled change nothing, the stats policy ignores the guard thresholds, the memo does not hold the base expression alive) went in with the implementation, because they cannot be seen failing.

Verification

• CI on c7fb1ff9 (tests only): all nine Python / Test jobs failed (3.11-3.14, Max Versions 3.11-3.14, Windows), 28 checks completed.
• CI on d9be4e90 (tests only): all nine Python / Test jobs failed again, 28 checks completed, deploy skipped. Locally, against the earlier source, the tests of the first commit and the six new or corrected ones of the second fail on assertions, not on collection.
• Full unit suite on 0181386e: 2178 passed, 5 skipped. The same 2178 passed and 5 skipped in a newest-versions environment (pandas 3.0.6, polars 1.44.2, xorq 0.4.5, numpy 2.5.3).
• 44 deliberate regressions of the implementation (a threshold compared with >=, each flag reading the other's threshold, a pass that skips the left columns or also changes the summary display, a refusal that ignores the flag or the sort key, guards applied after the snapshot, /load keeping the guard, a memo that never hits, is unbounded, evicts the newest, is not LRU, or holds the base alive) each fail at least one test. One survived the first set (the memo not refreshing a term on a hit) and is killed by d9be4e90.
• A real server on ports 8951 (this branch) and 8952 (the earlier source), both started and stopped by me, on a 2M-row, 4-column parquet entry opened with stats_tier auto, stats_delivery deferred and both thresholds at 1M rows:
  • this branch: df_meta carried sort and search as disabled; every column of the main display had sortable false and the summary display none; a sorted window got sort_disabled in 0.3 ms with no frame, for a client with the stats capabilities and for one with none; an unsorted window was served; searched windows took 53 ms, then 8.3, 8.1 and 7.7 ms.
  • earlier source: no flags, no sortable setting, the sorted window was served (19.6 ms on this table), and every searched window took 46-49 ms.
• CI on 0181386e: all 28 checks completed, 27 success and deploy skipped, including all nine Python / Test jobs, Python / Lint and Python / Typecheck.

Why default behaviour is unchanged

• A session is guarded only if the host opened it with a stats policy, and only above a threshold. Inline sessions, /load and /load_compare get no guard and the same messages (test_a_session_built_the_old_way_is_not_guarded).
• An enabled flag is not sent, so df_meta is the same for every session under the thresholds (test_enabled_flags_leave_the_message_as_it_was).
• The memo changes the number of backend counts. A response has the same rows and the same length.
• The existing tests pass unchanged. The only edits to existing test text are the module docstring of test_stats_policy.py and three import lines in test_load_expr.py (Expr for counting queries, gc and weakref).

What the client does on sort_disabled

Nothing new; the client is not changed. getKeySmartRowCache sees error_info on the infinite_resp, calls addErrorResponse, which fails the waiting block, and passes error_info to its error callback. In BuckarooView, the server-view client, that callback is a console.error; in BuckarooWidgetInfinite the respError state that would hold the text is commented out and the grid gets error_info="". So the refusal is logged, the block fails, and the grid keeps its sort model. With sortable: false on every column the header offers no sort, so a refusal reaches only a client that holds an older config. Reverting the sort model on error_code, and showing the message, is client work for a later phase.

Deviations from the plan

• The final pass is in XorqServerDataflow._build_df_display_args, not in style_columns. merge_column_config applies the host overrides after style_columns returns, so a pass inside it would be bypassed by {"qty": {"ag_grid_specs": {"sortable": true}}}. A test pins that case.
• The pass covers displays served by infinite_request (data_key == "main"), not the client-side summary grid.
• Guards apply only to sessions opened with a stats policy, a rule the plan does not state. It keeps this phase opt-in.
• search is a flag with a threshold that defaults to none and that nothing enforces; the plan has no server switch for search.
• The memo holds the searched expression per (base, term) rather than the integer count, so the failure rule of _expr_count (a failed count is not cached) stays in one place.
• The guard judges the source's row count, not the count of a search over it, so a sort that was allowed stays allowed while a user searches.
• A refusal has error_info as well as error_code, because the existing client path acts on error_info and ignores the code.
• search_expr is imported at the top of xorq_loading.py, where it was imported inside handle_infinite_request_xorq, to follow the repo's rule on imports.
• Two tests-only commits instead of one (see Tests).

Not in this PR

• The position cache of the L-track, the client revert of the sort model, default flips.
• A search switch on the server. The flag is advice.
• A sort guard that follows a searched count.

🤖 Generated with Claude Code

[github-actions]

📦 TestPyPI package published

pip install --index-strategy unsafe-best-match --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ buckaroo==0.15.9.dev37494092809

or with uv:

uv pip install --index-strategy unsafe-best-match --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ buckaroo==0.15.9.dev37494092809

MCP server for Claude Code

claude mcp add buckaroo-table -- uvx --from "buckaroo[mcp]==0.15.9.dev37494092809" --index-strategy unsafe-best-match --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ buckaroo-table

📖 Docs preview

🎨 Storybook preview