feat(client): merge stats_update into all_stats and advertise the capability (rows-first c2)

[paddymul]

Problem

The server side of the stats wire is on its own branch (#1024): a client that connects with ?caps=stats_update to a deferred session gets a first initial_state without data-derived stats (df_meta.stats.status == "pending"), asks for them with stats_request {stats_gen, scope}, and receives a stats_update {stats_gen, scope, tier, final, payload, elapsed_ms} or a stats_aborted. No client handles any of that yet:

1. WebSocketModel drops unknown message types, so a stats_update is lost.
2. Nothing tells the client which stats generation it is looking at, so a stats_update for a state it has left (a /load, a dataflow-field change) could not be told from a current one.
3. The standalone page and BuckarooServerView open their WebSocket without ?caps=, so the server treats them as legacy clients and computes the stats synchronously before every message built for them.

Phase and plan references

Plan 1 (plans/01-rows-first-stats-separate-plumbing.md) section 9 "Phase 2" (the jest half: WebSocketModel merges stats_update, drops a mismatched stats_gen, advances expectedGen on a broadcast frame with no reply_seq, ignores unknown types) and section 4.0 (client). Plan 2 sections 3 and 4.1 for the request and reply shapes; plan 3 section 3.2 for elapsed_ms. Message names and shapes follow the server branch feat/rowsfirst-s3-stats-wire-server (#1024), read from its stats_wire.py and checked against frames captured from that branch (see Tests).

Approach

A new module, src/server/StatsChannel.ts, holds the client half. WebSocketModel gets one field and one line in onmessage: it constructs new StatsChannel(this) and returns early when stats.handle(msg) consumes a frame. Nothing else in WebSocketModel.ts changes, so the rewrites in #1013, #1014 and #1015 conflict with it as little as possible.

• Expected gen. StatsChannel.expectedGen is the df_meta.stats.gen of the df_meta the model currently holds, read each time. It starts from the frame the model was built from, and every initial_state the model applies moves it, a broadcast frame with no reply_seq included. Reading the model rather than the initial_state branch means it follows the frames that were actually applied, whichever of the server mode: a stale initial_state reverts the client's buckaroo_state when dataflow changes overlap #998 fixes lands. A df_meta with no stats.gen leaves nothing expected, so a server that stops reporting stats cannot have a late stats_update merged.
• Merge. A stats_update whose stats_gen is not expectedGen is dropped, on arrival and again after the payload decodes. Otherwise the payload (a wide DFEnvelope, decoded with decodeDFData) is key-merged into df_data_dict.all_stats by mergeStatRows: for each stat row the payload names, each column it carries replaces that cell, columns it does not carry keep their cells, and a stat row the table lacks is appended. A null in the payload never replaces a value that is already there, because the wide pivot pads a stat a column did not carry with null. The merge builds new row objects, a new all_stats array and a new df_data_dict object. It never mutates the decoder's cached arrays, and the new reference is what the makeLatestDictDecoder from c0a needs to see a change. all_stats may be a decoded array (the seed) or an undecoded envelope (a later initial_state), so the base is decoded first. Updates are applied one at a time, in arrival order. If a frame replaces df_data_dict while the base decodes, the merge is redone on the new dict; if the gen moved on, it is dropped.
• Status. A final update also sets df_meta.stats to {status: "complete", tier: <update tier>, gen} in a new df_meta object, which is what c0a's inFlight rule and pinned-row placeholders read. A non-final update merges and leaves the status alone.
• stats_aborted. For the expected gen, reason error sets the status to error (reason stats_failed) and not_requestable sets it to not_computed. Other reasons (stale, unsupported_scope, no_data) and any other gen change nothing. A stale reply is answered by the initial_state that carries the new gen; adopting current_gen without that frame would let stats for a state the client has not seen merge into the one it shows.
• Unknown types are ignored as before.
• Capability. withStatsCapability(wsUrl) adds caps=stats_update to a WebSocket URL: ?caps= on a bare URL, &caps= after an existing query, a comma-joined value when the host already passes caps, and the fragment stays last. Other parameters are left as the host wrote them. BuckarooServerView applies it where it opens the socket, so any host's wsUrl gets it (buckarooWsUrl is unchanged), and standalone.tsx applies it to its own URL. BuckarooView takes an IModel and builds no URL.

Jupyter and anywidget are untouched: merge semantics are WebSocket-only.

What changes

• New packages/buckaroo-js-core/src/server/StatsChannel.ts (StatsChannel, mergeStatRows, withStatsCapability, STATS_UPDATE_CAP).
• WebSocketModel.ts (the field and the dispatch line), BuckarooServerView.tsx (the URL), src/index.ts (exports withStatsCapability, which standalone.tsx calls), packages/js/standalone.tsx (the URL).

Tests

Three tests-only commits come first (fe046acc, 3118435a, 1fd9cff8), each pushed and watched on CI before the fix. On fe046acc and on 1fd9cff8 all 27 checks completed and exactly two failed, JS / Build + Test and Server Playwright Tests (the deploy job was skipped). On 3118435a both had already failed when the next push cancelled the rest. 1fd9cff8 contains every test that fails before the fix. The fix is 5935432d: all 27 checks completed on it, 26 succeeded (JS / Build + Test, Server Playwright Tests and all nine Python / Test jobs among them) and deploy was skipped. CI logs were not read (REST rate limit); the reasons are from running the same commits locally, where every one of those tests fails on an assertion against the stub StatsChannel.ts of the first commit.

• src/server/StatsChannel.test.ts (new, drives a WebSocketModel with a fake socket): withStatsCapability; the merge (per-column key-merge, null padding, new row objects, an undecoded envelope as the base, other df_data_dict keys kept, a real wide parquet_b64 payload from the shared summary-stats fixture, a model with no dict or no all_stats); update order; df_meta.stats after a final and a non-final update; stats_gen (a mismatched gen dropped, the expected gen advancing on a broadcast frame, a merge discarded when a newer-gen frame arrives during the decode, a merge redone when a same-gen frame replaces the dict during the decode); stats_aborted; unknown types and infinite_resp pairing still working. The two in-flight cases and the order case hold a decode open through a mocked decodeDFData.
• src/server/BuckarooServerView.caps.test.tsx (new): the socket opens with ?caps=stats_update and keeps a query string the host passed.
• pw-tests/server.spec.ts: the standalone page connects with caps=stats_update (the server Playwright job).
• src/stories/StatsUpdateChannel.stories.tsx and pw-tests/stats-update-channel.spec.ts (d6959824, Storybook Playwright job, no server): a real WebSocketModel and BuckarooView on a fake in-page server, with stats_update frames sent by button. Column chunks merge into the pinned rows and the final one completes df_meta.stats; after a search moves the client to gen 2, gen 1's late chunks are dropped and gen 2's merge. Each test fails under a matching StatsChannel mutation (frames ignored, gen checks removed, a null allowed to overwrite, no status change on final).

The new jest files are separate from WebSocketModel.test.ts, which the #998 PRs create.

Tests that pass before the fix (the guards: caps already advertised, no stats reported, an initial_state without df_meta.stats, ignored stats_aborted cases, unknown types, infinite_resp pairing) went into the fix commit with the implementation.

Run locally on the fix: jest 365 passed (334 from c0a plus 31 here); tsc -b; the five server Playwright files against a rebuilt standalone.js, 45 passed. Twelve deliberate regressions of the implementation (the arrival-time gen check, the null rule, copying rows, serialising updates, the dict re-check, the final flip, the error status, the aborted gen check, decoding the base, reading the gen off df_meta, the dispatch line, the URL) were tried: eleven make at least one test fail. The twelfth, dropping the arrival-time gen check, passes, because the check after the decode catches the same updates; the early check only skips decoding a stale payload. Dropping the post-decode check instead makes the merge loop spin; I saw that once and did not rerun it.

I also replayed frames captured from the #1024 server (a deferred /load_expr session over a xorq memtable, a capable client, one stats_request, a post_processing change, a stale and a current request) through a real WebSocketModel in a throwaway jest test: the schema-tier all_stats (one dtype row) merged into the full set, the status went to complete at tier full for gen 1 and again for gen 2, the stale reply changed nothing, and a replayed gen-1 update was dropped. That test is not committed, because it needs the other branch's server to regenerate its frames.

Why default behaviour is unchanged

No default server sends df_meta.stats, stats_update or stats_aborted, so expectedGen is undefined and the channel consumes nothing. A client that now advertises caps=stats_update is treated by the #1024 server as capable. That only matters on a deferred session (opt-in per /load_expr body field), where it is not completed synchronously. On any other session the capability changes nothing except the order broadcast_state sends in (capable clients first). A server that predates #1024 ignores the query parameter.

Before a host turns deferred delivery on: a client that advertises stats_update and never sends stats_request stays on the stats-free frame with the pinned rows as placeholders. The scheduler that sends the request is the next client phase and has to ship before any host selects deferred delivery.

Deviations from the plan

• The expected gen is read off the model's df_meta instead of being set in the initial_state branch, so one dispatch line in WebSocketModel serves the merge and the gen tracking and the branch is not touched.
• stats_aborted is acted on for error and not_requestable; the plan says only "handle stats_aborted".
• Only the gen is checked, not the scope; the server accepts only raw today.
• The plan lists the jest cases under WebSocketModel; they are in StatsChannel.test.ts to avoid an add/add conflict with the server mode: a stale initial_state reverts the client's buckaroo_state when dataflow changes overlap #998 PRs.

Not in this PR

• The scheduler that sends stats_request, and any "Compute summary stats" control (plan 1 phase 4).
• Model-before-decode (plan 1 phase 0b, gated on server mode: a stale initial_state reverts the client's buckaroo_state when dataflow changes overlap #998): a second initial_state that arrives while the first is still being decoded, before the model exists, is still lost.
• A jest test for packages/js/standalone.tsx, which has no harness (importing it runs main()); the server Playwright test above covers its URL.
• A buckaroo-js-core release and the tallyman bump.

Stack

Built on feat/rowsfirst-c0a-client-hardening (#1020), which this PR contains until it merges, so the diff includes c0a's two commits (29e1e42c tests, 405f4ded fix). This phase's commits are the three tests-only commits fe046acc, 3118435a and 1fd9cff8, and the fix commit 5935432d. The server branch #1024 is not part of this PR; its message shapes are an interface, not a dependency, and the tests use a fake socket.

🤖 Generated with Claude Code

[github-actions]

📦 TestPyPI package published

pip install --index-strategy unsafe-best-match --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ buckaroo==0.15.9.dev37478765398

or with uv:

uv pip install --index-strategy unsafe-best-match --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ buckaroo==0.15.9.dev37478765398

MCP server for Claude Code

claude mcp add buckaroo-table -- uvx --from "buckaroo[mcp]==0.15.9.dev37478765398" --index-strategy unsafe-best-match --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ buckaroo-table

📖 Docs preview

🎨 Storybook preview

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.