feat(client): scheduler and control request incremental stats with a columns hint (rows-first c4b)

[paddymul]

Problem

The server phases s4 and s5 (#1026, #1028) added stat units and an opt-in incremental: true on stats_request: such a request runs units for about 75 ms and answers stats_update {final: false, remaining, payload} with the new fragments, the final reply carries the complete all_stats, and a columns hint (the grid's rewritten column names) runs the units that cover those columns first. The client scheduler of c4 (#1027) was written in parallel and still sends whole-run requests: a request with no incremental field runs every unit in one synchronous call, however long the longest unit is, and the hint is never sent. The Compute summary stats control sends a whole-run request too.

Phase and plan references

Client phase c4b, the follow-up to c4. Plan 1 (plans/01-rows-first-stats-separate-plumbing.md) section 9 "Phase 4" and section 5 (Strategy C), with section 3 and section 4.0. Plan 2 (plans/02-rows-first-xorq-and-lazy-polars.md) section 3 and section 4.1 ("Request"). Message names and shapes follow the server branches #1024 and #1028.

Approach

Request shape. requestStats in StateOrchestrator.ts is the one place that builds a stats_request, and the scheduler and the control both use it. It now sends {type, stats_gen, scope: "raw", incremental: true}, plus columns when the grid has reported the columns it shows (an empty or missing list sends no hint), plus force: true for the control. The hint is read from the model's visible_columns key at the moment a request goes out, so each request of a run carries the columns on screen then.

The run. The scheduler's chain was already one request per reply that leaves the status pending, read off the model (a new df_data_dict under the same df_meta). It needed no new logic for partial replies: c2's StatsChannel merges a non-final stats_update into all_stats and leaves df_meta alone, so the status stays pending and the placeholder pinned rows stay until the final reply, which carries the complete all_stats and sets the status to complete. A gen change (a local dataflow-field change or a frame with a new gen) cancels the run and starts a new one after the delay, and StatsChannel drops a late reply for the old gen. A server that does not know the fields ignores them, runs the whole run and answers final: true, which ends the run the same way. Tests pin each of these.

Where the hint comes from. DFViewerInfinite takes on_visible_columns and calls it with the data columns that have a header cell in the grid's viewport, when the grid is ready and on onVirtualColumnsChanged, skipping an unchanged list. BuckarooInfiniteWidget forwards the prop. BuckarooView and the standalone BuckarooApp pass a callback that stores the list on the model with setVisibleColumns(model, columns) (new, exported next to requestStats). The key is client-side: the server never sends it, and WebSocketModel keeps it across initial_state frames.

The columns are read from the grid's rendered header cells (.ag-header-viewport .ag-header-cell[col-id], one frame after the event), not from api.getAllDisplayedVirtualColumns. That function lives in AG Grid's ColumnApiModule, which this package does not register. On main api.getColumn (the active-cell refresh in onCellClicked) and api.getColumnState and applyColumnState (the per-view column state in the view_name effect) are already inert for the same reason: AG Grid logs error #200 for each. Registering the module would switch them on in every grid, which changes default behaviour, so this PR does not. I did not touch those call sites.

What changes

• src/server/StateOrchestrator.ts: requestStats sends incremental: true and the hint; VISIBLE_COLUMNS_KEY and setVisibleColumns; the header comment describes the run.
• src/components/DFViewerParts/DFViewerInfinite.tsx: on_visible_columns (the header-cell read, onGridReady and onVirtualColumnsChanged).
• src/components/BuckarooWidgetInfinite.tsx: forwards on_visible_columns in the buckaroo widget (viewer mode has no stats).
• src/server/BuckarooView.tsx, packages/js/standalone.tsx: the callback. src/index.ts: exports setVisibleColumns (named and on the default srt object).

Tests

All new tests are in existing files of the same kind. The tests-only commit is d7eda002, pushed and watched on CI before the fix. A second tests-only commit (5d1e0467) adds a two-frame wait to the wide-table test, which failed once in a full local run: it released the reply as soon as the header cells had changed, a frame before the page reports its columns. No assertion changed. The fix is e5649605.

CI on d7eda002: all 28 entries in the rollup completed and exactly three failed, JS / Build + Test, Storybook Playwright Tests and Server Playwright Tests; every Python / Test job passed (this phase has no Python change). CI logs were not read (REST rate limit); the reasons are from running the same files locally against the stubs of that commit, where 44 jest tests failed on assertions, five of the new server Playwright tests failed and the Storybook control test failed.

• src/server/StateOrchestrator.test.ts: a block "incremental requests" (the shape with no hint, with a hint, with an empty list, the hint re-read for each request of a run, the same shape for the first request of the next gen); requestStats with the hint and force; in the block that builds a real WebSocketModel from a fake socket: partial replies walked to the final one with df_meta unchanged (the same object) until the last, a whole-run final from a server that ignores the field, a gen change mid-run (late reply for the old gen dropped, one request for the new gen, which then chains).
• src/components/DFViewerParts/DFViewerInfinite.test.tsx: the grid reports the data columns of its viewport (not the index column), again when they change, not when they stay the same, and [] before layout. The AG Grid mock now renders header cells for the columns it is told are in view. BuckarooInfiniteWidget.flash.test.tsx: the widget hands the grid the callback. BuckarooView.stats.test.tsx: the callback stores the list and the control's request carries it.
• pw-tests/server.spec.ts (server Playwright, "WebSocket data flow"): the real standalone page against a real session whose first frame is rewritten to pending, with a harness answering stats_request (the real server's own all_stats as the final payload, so no xorq is needed): several partial replies lead to repeated requests and the status stays pending until the final; a gen change mid-run stops the run for the old gen and starts one for the new gen; a 60-column table sends only the columns on screen, and after a horizontal scroll the next request carries the new ones.
• pw-tests/stats-scheduler-states.spec.ts (Storybook): the control's request carries incremental, force and the columns the grid reports; a stat row that has arrived for one column leaves the other columns' cells empty while the status is pending. The second one passes on the earlier code, since AG Grid does not render a cell whose value is absent. It is a characterization test of what a partial merge shows.

Existing tests changed, and why. The scheduler's request() helper in StateOrchestrator.test.ts gained incremental: true, and the existing assertions on the sent message in BuckarooView.stats.test.tsx, server.spec.ts (two) and stats-scheduler-states.spec.ts (one) gained incremental: true and the hint where the grid reports one: the request shape changes by design, and every one of those asserts the exact message. No assertion about behaviour was loosened.

Two on_visible_columns props are unused stubs in the tests-only commit so that the new tests compile and fail on assertions.

Run locally on the fix: jest 431 passed (417 before this phase); tsc -b; the five Storybook Playwright files, 16 passed; the server Playwright files, 42 passed in server.spec.ts on three runs in a row and 17 in the other four files on three runs in a row (in one earlier full run server-buckaroo-search.spec.ts failed once, on the race that c4's description names, and did not fail in those three runs); tests/unit 1185 passed, 5 skipped. Twelve deliberate regressions of the implementation (no incremental, no hint, an empty hint sent, another model key, the index column not filtered, no dedupe, no onVirtualColumnsChanged, no report on grid ready, the widget, BuckarooView and the control each dropping their hop, the callback a no-op) each make at least one jest test fail, and dropping the callback from standalone.tsx makes five server Playwright tests fail.

I also ran the built standalone page against a scratch checkout of #1028 (detached at ad26fdf4) on a free port, with a real deferred /load_expr session over a 200,000-row, 30-column xorq memtable. The observed sequence: initial_state (pending, gen 1), infinite_request, infinite_resp, then stats_request {gen 1, incremental: true, columns: a-g} 15 ms later; 16 partial stats_update replies with remaining 30, 28, ... 2, each answered at once by the next stats_request (the hint widening to a-j after the grid laid out); the 17th reply final: true, remaining: 0 (carrying df_display_args, which the client does not apply), and the status bar reading "Summary stats ready". The whole run took about 2.0 s, in requests of 120 to 200 ms. In a second run a search term was typed 700 ms in: the next reply for gen 1 arrived after the buckaroo_state_change and changed nothing, the server sent initial_state pending gen 2, one stats_request for gen 2 went out 315 ms after that frame, and the run chained to a final reply. No request for gen 1 followed the change. That check is not committed because it needs xorq and the other branch's server.

Why default behaviour is unchanged

Nothing is requested unless df_meta.stats says pending, which only a deferred /load_expr session reports, so a default session still sends no stats_request (the existing server Playwright test asserts it). The hint is reported only when a host passes on_visible_columns; the Jupyter widget and viewer mode do not, so their grids add nothing (the one new prop on DFViewerInfinite is optional and, when absent, the read is skipped). The grid registers no new AG Grid module. A server that predates incremental ignores the field.

Deviations from the plan

• The hint is the columns with a rendered header cell, read from the DOM, for the reason under "Where the hint comes from". The plan did not say how the client learns them.
• The hint is stored on the model under visible_columns instead of through a setter on the scheduler, so a host that runs its own scheduler on its own IModel reads it too and requestStats needs no extra argument.
• The control sends one incremental request, as the scheduler's first. It does not chain: on a not_computed session the status is not pending, the scheduler ignores it, and feat(server): stats_request, stats_update and df_meta.stats on deferred sessions (rows-first s3) #1024 and feat(stats): stats_request runs units for a time budget, with per-client cursors and the final assignment (rows-first s5) #1028 answer every force request stats_aborted not_requestable today. How a forced run continues is plan 3's (force semantics), so I left it.
• The final reply's df_display_args (the rebuilt config, sent by feat(stats): stats_request runs units for a time budget, with per-client cursors and the final assignment (rows-first s5) #1028 when its digest differs) is not applied by the client. The scope names no config upgrade, and plan 3 lists "an upgraded df_display_args neither remounts nor purges rows" as unmeasured.

Not in this PR

• Policy fields (auto_request, requestable, demand_columns, ceiling) and the tier-aware control: the next client phase.
• Any server change. A buckaroo-js-core release and a tallyman bump.

Stack

Built on feat/rowsfirst-c4-client-stats-scheduler (#1027), which contains c2 (#1025) and c0a (#1020). The PR targets main, so the diff includes their commits until they merge: c0a 29e1e42c and 405f4ded; c2 fe046acc, 3118435a, 1fd9cff8 and 5935432d; c4 dd6ebe6a, d6afbf98 and 95dcceec. This phase's commits are d7eda002 and 5d1e0467 (tests) and e5649605 (fix). Read only those. The server branches #1024 and #1028 are not part of this PR.

🤖 Generated with Claude Code

[github-actions]

📦 TestPyPI package published

pip install --index-strategy unsafe-best-match --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ buckaroo==0.15.9.dev37193367183

or with uv:

uv pip install --index-strategy unsafe-best-match --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ buckaroo==0.15.9.dev37193367183

MCP server for Claude Code

claude mcp add buckaroo-table -- uvx --from "buckaroo[mcp]==0.15.9.dev37193367183" --index-strategy unsafe-best-match --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ buckaroo-table

📖 Docs preview

🎨 Storybook preview

[paddymul]

Closing. Incremental requests and the visible-columns hint drive the client-pull loop that the revised D2 of ADR-002 (#1043, 7943412) replaces with a server push after the first row reply. Nothing here carries over.