feat(client): not_computed states, compute control and demand units for the stats policy (rows-first c5)

[paddymul]

Problem

Plan 3 (plans/03-no-summary-stats-for-large-files.md) lets the server leave a large entry without summary stats. The wire for that is in the server branch #1029 (p33): df_meta.stats gains tier_target, reason (size | host | cost | ceiling), auto_request, requestable, estimate, demand_columns, omitted_keys and approx_keys, and a client that sends ?caps=stats_update,stats_ondemand is told not_computed with those fields instead of being sent complete stats at connect. The client of c0a to c4b knows not_computed only as a status. It has no empty state for the summary view, no message for a refusal, no tier on the control's request, never asks for demand_columns, does not continue a forced run, and advertises only stats_update, so a p33 server would never apply the policy to it.

Phase and plan references

Client phase c5: plan 3 section 6 "Phase 5", with sections 3.2 (capabilities, cost guard), 3.3 (what the grid shows, demand-driven minimum) and 3.4 (the on-demand control). Also plan 1 sections 3 and 4.0 and plan 2 sections 3 and 4.1. The field names and defaults follow the interface notes of p33 (#1029): auto_request absent means true, requestable absent means ["full"].

Approach

Capabilities. withStatsCapability adds stats_update and stats_ondemand to ?caps=. Both wiring copies call it (BuckarooServerView.tsx and packages/js/standalone.tsx), so both advertise the second bit. This client can render not_computed with its reason and send tiered requests, which is what the bit promises.

Types and helpers (WidgetTypes.tsx). DFMetaStats carries the policy fields. statsRequestable and statsAutoRequest read the documented defaults. nextRequestTier returns the smallest tier in requestable above the tier reached, so scalar goes before full, and scalar having been reached the next one is full. canRequestStats is true for a not_computed session with such a tier and a reason other than ceiling. demandTier is the smallest tier from scalar up that the policy allows, which is the one that carries min and max.

Reply handling (StatsChannel.ts). A final stats_update may carry status and reason. {final: true, status: "not_computed", reason: "ceiling"} with no payload sets df_meta.stats to the same stats with the new reason and leaves all_stats alone, so the session ends in the ceiling message. A reply that carries a payload and a status merges the payload and takes the status, which is the shape a scoped run for some columns needs. The columns the run's replies filled go in df_meta.stats.computed_columns, a field the client sets and the server never sends: a column counts where some stat row holds a value for it, a final reply or a not_requestable refusal that leaves the session not computed adds them, and a new gen, or a frame that replaced the stats, starts the list over. A final reply with no status still completes the session and drops the policy fields, as before. The reply's tier counts only when the session completes.

The control (StateOrchestrator.ts). forceStats(model, opts?) sends stats_request {stats_gen, scope: "raw", incremental: true, force: true, tier} with the tier from nextRequestTier, and sends nothing when no tier is left. opts.columns is the per-column form of plan 3 section 3.4 and adds columns. The whole-table request carries no columns: on a forced request columns names what the request is for, so the grid's visible columns (an ordering hint on the scheduler's requests) are not added to it. requestStats takes tier and columns options. forceStats records the run on the model under stats_forced.

Scheduler. It acts on two kinds of work besides the pending run of c4:

• With auto_request false it is idle, except that it asks for demand_columns: one scoped request {incremental: true, columns: demand_columns, tier} per gen and list of columns, after the first rows, at the tier demandTier gives, with no force. A reply that is not final (a new df_data_dict under the same df_meta) is answered with the next request, a final reply or a refusal (a new df_meta) ends it, and a changed list or a new gen starts it again. A pending session with auto_request false is not run whole.
• A forced run recorded by forceStats is continued: each reply that is not final is answered with the same force, tier and columns, and a final reply, a refusal or a new gen ends it. c4b left this open because a forced request had nothing to continue it.

Sessions that auto-request keep c4's behaviour. A not_computed session with auto_request true (a scalar target before the server has scalar units) is left alone.

What the user sees. In the status bar the stats cell for not_computed shows the "Compute summary stats" button when canRequestStats, "Continue computing stats" when the reason is cost, the message "Summary stats unavailable: size limit" and no button for ceiling, and the old label when nothing is requestable. The button calls the host's callback with no arguments. In the summary view, StatsEmptyState replaces the grid while the status is not_computed and no run has computed a column (computed_columns is empty): it says why (size, host, cost, ceiling), with the size from estimate ("12.4M rows x 44 columns"), and offers one button for the next tier ("Compute basic stats" for scalar, "Compute full stats", "Continue computing stats" after a cost pause) with a per-column picker. For ceiling, or when nothing is requestable, it is the message alone. Once a run for some columns (the per-column form, or the demand columns) has filled some, the summary view shows its grid with those stats in place of the empty state. The per-column picker is then off screen, and the status bar's control still asks for the whole table. The main view keeps its grid, with the typed columns and, from c0a, no blank pinned rows. A session whose df_meta.stats is absent behaves as before.

on_compute_stats now takes optional {columns}. BuckarooView and the standalone BuckarooApp pass (opts) => forceStats(model, opts); forceStats is exported next to requestStats.

Status bar geometry (dcf-npm.css). The control's button is 16px high inside a container that fills the status bar's one row, and the status bar's invisible scrollbar overlay takes no pointer events. See Tests for the CI failure that led to this.

What changes

• src/components/WidgetTypes.tsx: the policy fields and the helpers.
• src/server/StatsChannel.ts: the capability pair; the status and reason of a final reply; computed_columns.
• src/server/StateOrchestrator.ts: requestStats options, forceStats, FORCED_RUN_KEY, demand and forced runs in the scheduler.
• src/components/StatusBar.tsx, src/components/StatsEmptyState.tsx (new), src/components/BuckarooWidgetInfinite.tsx, src/style/dcf-npm.css: the states and the control's geometry.
• src/server/BuckarooView.tsx, packages/js/standalone.tsx, src/index.ts: the wiring and the export.

Tests

Two tests-only commits, pushed before any fix: 01609fa3 and f2c4c0a4. On each, exactly three checks failed (JS / Build + Test, Storybook Playwright Tests, Server Playwright Tests) and every Python check and the other jobs passed. f2c4c0a4 pins cases the first commit left untested: the control's pending state, a size the server chose with nothing above it reading as the ceiling, and demand runs that have finished. Where a test needs a symbol that does not exist yet, the commit carries a stub that keeps the old behaviour, so the failures are assertions.

Review follow-up, one more pair: 30bf7c02 (tests) and 9992b2e2 (fix). A run for some columns ends with its stats merged into all_stats and the status still not_computed, and the summary view showed the empty state for that status, so no view showed them. On 30bf7c02, JS / Build + Test failed. On 9992b2e2 all 28 entries completed, 27 SUCCESS and deploy SKIPPED.

• Jest: WidgetTypes.test.ts (defaults, nextRequestTier, canRequestStats, demandTier), StatsChannel.test.ts (the capability pair, a final update with a status), StateOrchestrator.test.ts (demand runs, forceStats, continued forced runs, a not_computed session end to end), StatsEmptyState.test.tsx (new), StatusBar.stats.test.tsx, BuckarooInfiniteWidget.flash.test.tsx (the summary view's empty state, and its grid after a run for some columns), BuckarooView.stats.test.tsx, BuckarooServerView.caps.test.tsx.
• Storybook Playwright, stats-scheduler-states.spec.ts: five new tests (the control's message, typed columns with no blank pinned rows, the empty state, the ceiling, a paused run) plus the status bar's control, on the StatsSchedulerStates story.
• Server Playwright, server.spec.ts: four new tests against the standalone page (the URL carries both capabilities; a policy session asks for nothing and the control asks for the basic tier; a request over the ceiling ends in the ceiling message; a forced run is asked for again after each partial reply and ends with the final one).

Fix commits:

• 17376859, the implementation. On CI, JS / Build + Test and Server Playwright Tests passed and Storybook Playwright Tests failed: the click on "Compute summary stats" timed out in headless Linux Chromium with "ag-body-horizontal-scroll-container from ag-body-horizontal-scroll ag-scrollbar-scrolling ag-scrollbar-invisible subtree intercepts pointer events". It passed on macOS. The condition is deterministic: the button was 20px high in a 20px row, so the browser scrolled the grid to reveal it, AG Grid put ag-scrollbar-scrolling on the status bar's horizontal scrollbar overlay, and the overlay (positioned over the last 16px of a one-row grid, not drawn) sat above the button.
• 7b30aad1, CSS only. Reproduced locally by holding the overlay in its scrolling state and clicking the button: it fails with the CI message on the CSS of 17376859 and lands on this one.

CI on 7b30aad1: all 28 entries completed, 27 SUCCESS (26 check runs and the Read the Docs status) and deploy SKIPPED.

Local, before the push of 7b30aad1: jest 557 passed, tsc -b clean, the five Storybook Playwright files CI runs pass with one worker and no retries (stats-scheduler-states 9 of 9), server Playwright 54 of 54 on the second full run. The first full run after rebuilding failed its first test, server-buckaroo-search.spec.ts "searching filters the table data": it reads the text of .df-viewer once the first .ag-cell is visible and found no row text. The same test failed in two earlier full runs on this branch, before the CSS change, passes alone three times in a row, and passed in the next full run and on CI for both fix commits (CI retries twice). I did not pin its governing variable. I suspect the first infinite_resp arriving after the first pinned-row cell on a freshly started server; delaying that frame would decide it. The test is not changed here.

Why default behaviour is unchanged

Nothing new is requested unless df_meta.stats carries the new fields or a status other than pending, which only a session on a server with the policy reports, and only to a client that advertises stats_ondemand. A session with no df_meta.stats sends no stats_request and renders as before. A session at pending with no auto_request field runs whole, as in c4. The summary view keeps its grid for every status but not_computed. The grid registers no new AG Grid module.

The one change a default session can see is the URL: the client now asks for stats_update,stats_ondemand. A server that predates the policy ignores the second bit. A server with the p33 policy (#1029) applies it to this client, and a session it resolves below full is shown as not_computed. That needs the host to opt in with stats_tier or a server that resolves auto; the server default is unchanged. The status bar CSS applies to the stats control only and to the status bar's invisible scrollbar overlay.

Existing tests changed, because the behaviour they pin changes by design: the ?caps= assertions (StatsChannel.test.ts, BuckarooServerView.caps.test.tsx, and the standalone URL test in server.spec.ts) expect stats_update,stats_ondemand instead of stats_update, and the assertions of the control's stats_request (BuckarooView.stats.test.tsx, StateOrchestrator.test.ts, server.spec.ts, stats-scheduler-states.spec.ts) no longer expect the c4b columns hint on that request (see the first deviation). The StatusBar.stats.test.tsx helpers take the callback's new optional argument, routeStatsRequests in server.spec.ts takes an optional first-frame stats argument with the old value as its default, and the story's control calls forceStats. Two assertions of this branch's own tests that read the end state of a scoped run now expect computed_columns (StatsChannel.test.ts, "merges a payload it carries...", and StateOrchestrator.test.ts, "a run for some columns ends not computed..."), because that end state changed by design. No assertion about other behaviour was loosened.

Deviations from the plan

• The whole-table control sends no columns. Plan 3 section 3.4 gives the control's message as {stats_gen, scope, tier, force} with a columns form per column. c4b added the visible-columns hint to every request, the control's included. On a forced request columns is a scope, so the hint is dropped from it.
• The tier of the control is chosen on the client from requestable and the tier reached (smallest first). Plan 3 does not say how.
• A forced run is continued by the scheduler, from a model key (stats_forced) that forceStats sets. c4b's interface notes list this as an open gap. How the server answers a forced request (a partial reply, then a final one) is the p33 and phase 6a contract; against a server that refuses it with not_requestable the run ends at once.
• The demand request is {incremental, columns, tier} with no force. Plan 3 section 3.3 says "an automatic scoped stats_request with a columns hint, under the same policy". The wire shape of the reply to it (a final reply with status: "not_computed" and a payload) is my reading, for phase 6a to confirm.
• A reply's status and reason on a final stats_update is a new part of the wire. Plan 3 section 3.4 gives the ceiling reply as stats_update {final: true, status: "not_computed", reason: "ceiling"}; taking status on any final reply is the general form.
• The summary-view empty state replaces the grid instead of overlaying it. With no data rows the summary view has only its pinned rows, which are all omitted for not_computed. The empty state replaces the grid only while no run has computed a column. After a run for some columns the grid is shown, and the per-column picker, which lives in the empty state, is no longer on screen; asking for more columns goes through the status bar's whole-table control.
• cost is handled as a reason on df_meta.stats (the button reads Continue). The server half, cost_paused and elapsed_ms, is phase 6a.
• omitted_keys and approx_keys are typed but not shown anywhere.
• Three tests-only commits and three fix commits instead of one of each. The second fix is the CSS fix for a CI-only failure, above. The third pair follows the review finding about the per-column form.

Not in this PR

• Server changes, including phase 6a (limits, force override, cost guard, demand scan) and phase 4 (scalar units). Against today's server a forced request on a not_computed session is answered stats_aborted not_requestable, so the control does nothing visible until 6a.
• The optional config upgrade (plan 3 phase 6b): the final reply's df_display_args is still not applied.
• The tallyman repo (plan 3 phase 8). This needs a buckaroo-js-core release and a tallyman bump, which are not part of this PR.

Stack

Built on feat/rowsfirst-c4b-client-incremental-requests (#1030), which contains c4 (#1027), c2 (#1025) and c0a (#1020). The PR targets main, so the diff includes their commits until they merge: c0a 29e1e42c and 405f4ded; c2 fe046acc, 3118435a, 1fd9cff8 and 5935432d; c4 dd6ebe6a, d6afbf98 and 95dcceec; c4b d7eda002, 5d1e0467 and e5649605. This phase's commits are 01609fa3 and f2c4c0a4 (tests), 17376859 (implementation), 7b30aad1 (status bar CSS), and 30bf7c02 (tests) with 9992b2e2 (fix) for the review finding about the per-column form. Read only those. The server branches (#1024, #1028, #1029) are not part of this PR.

🤖 Generated with Claude Code

[github-actions]

📦 TestPyPI package published

pip install --index-strategy unsafe-best-match --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ buckaroo==0.15.9.dev37206648816

or with uv:

uv pip install --index-strategy unsafe-best-match --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ buckaroo==0.15.9.dev37206648816

MCP server for Claude Code

claude mcp add buckaroo-table -- uvx --from "buckaroo[mcp]==0.15.9.dev37206648816" --index-strategy unsafe-best-match --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ buckaroo-table

📖 Docs preview

🎨 Storybook preview

[paddymul]

Review of the not_computed control: one finding, fixed

Finding (medium): the per-column compute form produced stats that no view showed. A scoped run (the per-column picker, or the demand columns) ends with status: "not_computed" and a payload. StatsChannel merged the payload into all_stats, but the summary view showed StatsEmptyState instead of the grid for as long as the status was not_computed, so the stats of the picked column were in the model and on no screen. The reviewer's probe rendered the widget with df_meta.stats not computed and min and max rows in summary_stats and got stats-empty-state with the grid mounted zero times. I checked the cause in BuckarooWidgetInfinite.tsx (statsEmptyState was true for every not_computed session in a view whose data_key is not main), and the new tests below fail on it.

Addressed in 30bf7c02 (tests) and 9992b2e2 (fix).

• The widget cannot tell "nothing computed" from "some columns computed" by looking at all_stats: the schema tier already puts dtype and length rows there, and this PR treats that as nothing computed (the story and its Playwright test depend on it). So StatsChannel records what its payloads filled. Each applied payload adds the columns that hold a value in some stat row (a null cell fills nothing). The final reply, or a not_requestable refusal, that leaves the session not computed puts them in df_meta.stats.computed_columns. A reply that is not final leaves df_meta alone, so c0a's inFlight rule is unaffected. A new gen, or a frame that replaced df_data_dict under the same gen, starts the list over; A complete session drops it, and an error discards the run's pending columns and adds none.
• The summary view shows its grid, with the merged rows, when computed_columns is not empty. An empty or missing list keeps the empty state, so the schema-tier case, the ceiling message and the paused-run message are unchanged.
• computed_columns is typed on DFMetaStats with a note that the server never sends it.

Verification. 30bf7c02 adds 13 jest tests and amends 2 assertions across StatsChannel.test.ts, StateOrchestrator.test.ts and BuckarooInfiniteWidget.flash.test.tsx, including one that feeds a scoped reply through a real StatsChannel and renders the widget from the resulting model state. 13 of those 15 failed on assertions (the type stub keeps the failures from being compile errors), and JS / Build + Test failed on that head. The two that pass pin what must stay: an error carries no list, and an empty list keeps the empty state. After the fix: jest 570 of 570, tsc -b clean, pnpm run build clean, the Python unit suite 1185 passed and 5 skipped. On 9992b2e2 all 28 check entries completed: 27 SUCCESS and deploy SKIPPED, including the Max Versions jobs and the Storybook and Server Playwright jobs.

Existing tests changed. Two assertions in this branch's own c5 tests read the end state of a scoped run with toEqual on the unchanged policy stats: StatsChannel.test.ts ("merges a payload it carries...") and StateOrchestrator.test.ts ("a run for some columns ends not computed..."). They pinned the behaviour the finding is about, so they now expect computed_columns. Nothing from main changed, and the PR body says so.

Limits, not changed here.

• After a first scoped run the grid replaces the empty state, and the per-column picker lives in the empty state, so it is no longer on screen. More columns go through the status bar's whole-table control. Keeping the picker next to the grid is a layout change that I left out of this fix.
• If the server sends a new frame under the same gen after a scoped run, the frame replaces df_meta and all_stats together, and the empty state returns with the merged stats. Whether a p33 server carries them in that frame is for phase 6a to settle.
• No Playwright test was added. The NotComputed story does not route replies through StatsChannel, and the switch between the empty state and the grid is plain React, which the jest test covers with the real channel and widget.

Unrelated failure seen once. In the first full Python run after the test commit, tests/unit/file_cache/mp_timeout_decorator_test.py::test_is_running_in_mp_timeout failed. It spawns a subprocess under mp_timeout(TIMEOUT * 3), and the run overlapped with other suites on the same machine (load average 33). It passed alone, and in the next full run, and this PR touches nothing under file_cache.

🤖 Generated with Claude Code

[paddymul]

Closing. The scheduler half (demand-column runs, continuing a forced run) is client pull, which the revised D2 of ADR-002 (#1043, 7943412) drops.

The UI half still applies: StatsEmptyState, the reason messages, the tiered Compute control and the stats_ondemand capability. Those are the parts to carry into a client UI PR on top of #1025. The branch is kept for them.