feat(server): stats_request, stats_update and df_meta.stats on deferred sessions (rows-first s3)

[paddymul]

Stacked on #1022 (feat/rowsfirst-s2-schema-tier-pandas-polars), which is stacked on #1021 (feat/rowsfirst-s1-stats-tier-core). This PR is based on main so the repo's Checks workflow runs on it (its pull_request trigger uses branches: "*", which does not match a base branch containing a slash). The diff therefore includes the commits of #1021 and #1022 until they merge. The commits of this phase are d7ba7bd9 and 6bd63ed7 (failing tests) and 9bfebfb9 (implementation); read only those three.

Problem

#1021 lets a host load a xorq expression with stats_delivery="deferred", which publishes a schema-tier dataflow and stops there: the stats never arrive, and nothing on the wire says so. A client cannot tell a session that is waiting for stats from one that will never have them, cannot ask for them, and cannot tell a reply for the state it is looking at from one for a state it has left (#998, for stats). DataStreamHandler.on_message drops unknown message types, so a stats_request is ignored today. The dataflow also rebuilds df_meta wholesale on each state change, so any flag the dataflow wrote there would be lost at the next change.

A deferred session must also keep serving clients that know nothing about this protocol (older embeds, MCP and Python callers, tallyman on buckaroo-js-core 0.15.8). The connection is seen only in open(), but the session snapshot is pushed from five other sites and the highlight overlay, so a check at open() protects the first message only.

Phase and plan references

Rows-first s3, from buckaroo2-reports/plans/: plan 1 (01-rows-first-stats-separate-plumbing.md) section 9 "Phase 2", server half (the jest half is a separate client phase), with sections 3, 4.0, 5, 6 and 8; plan 2 (02-rows-first-xorq-and-lazy-polars.md) sections 3 and 4.1 for the final assignment and the snapshot refresh; plan 3 (03-no-summary-stats-for-large-files.md) section 3.2 for the field names (reason, elapsed_ms, not_computed).

Approach

• SessionState gains stats_gen, stats_status and stats_reason. begin_stats_generation(session) bumps the counter and restarts the status from the session's policy pair: pending for a deferred session headed for the full tier, not_computed (reason host) for one headed for the schema tier, complete otherwise. It is called by /load, /load_expr, /load_compare, /reload_expr and by a buckaroo_state_change that changes a _DATAFLOW_FIELDS field. It is independent of fix(server): sequence token on buckaroo_state_change so a stale initial_state is dropped (#998) #1014's state_seq.
• build_state_message injects df_meta.stats = {status, tier, gen, reason?} into a copy of df_meta, because the dataflow rebuilds df_meta wholesale. tier is the tier reached so far, so it is schema until the session is complete. A session on the default policy (inline delivery, full tier, complete) gets no stats key and sends the message it always has; a client reads a missing stats as complete.
• open() records ?caps=a,b as self.caps. A new module, buckaroo/server/stats_wire.py, holds the rest. build_state_message_for(session, client, metadata=None) is the one builder every send site uses: for a client without stats_update on a pending deferred session it first runs the missing stats synchronously (today's cost, paid on the loop), and a capable client gets the snapshot as it is, stats-free. broadcast_state(session, metadata=None, reset_search=False) replaces the four copies of the per-client send loop (LoadHandler._push_state_to_clients, /load_expr, /load_compare, /reload_expr) and the one in the WebSocket state-change handler, and sends to capable clients first: a legacy client's message completes the session, and a message built after that would carry the stats, so the capable client would never get the pending frame that describes its own state.
• stats_request {stats_gen, scope, columns?} is a synchronous branch in on_message. A request for another generation gets stats_aborted {stats_gen, current_gen, scope, reason: "stale"} before any query runs. Otherwise the request is the whole run: complete_stats(session) computes the full stats in one call and applies the final assignment. It writes the full-tier entry into summary_stats_cache (or finds the one an earlier visit to the same state left there), assigns summary_sd, refreshes the session snapshot through refresh_session_snapshot and sets the status to complete in the same step. The reply is stats_update {type, stats_gen, scope, tier, final: true, payload, elapsed_ms} with payload the dataflow's own all_stats, an inline parquet_b64 envelope with layout: "wide", so binary pairing stays single-slot. A repeated request on a complete session is answered from the dataflow with no query.
• A dataflow-field change on a deferred session puts the dataflow back at the schema tier before the new traits are set, so the cascade costs the schema rerun and not the stats, bumps stats_gen and broadcasts. The stats then follow as requests, or run synchronously for a legacy client.
• Telemetry: the stats_request branch enters telemetry_context with session.tele_sink, and handle_stats_request wraps each request in a stats.request span (stats_gen, scope, columns as a count, tier, outcome). outcome is update or the abort reason, so updates sent, requests dropped as stale and errors are counted from the span records. complete_stats adds a firstpull.stats_total span per completed run.

What changes

• buckaroo/server/session.py: stats_gen, stats_status, stats_reason, STATS_STATUSES, initial_stats_status, begin_stats_generation, stats_meta; build_state_message injects df_meta.stats.
• buckaroo/server/stats_wire.py (new): parse_caps, client_has_cap, session_dataflow, refresh_session_snapshot, assign_full_stats, complete_stats, build_state_message_for, broadcast_state, handle_stats_request.
• buckaroo/server/websocket_handler.py: caps at open(), the stats_request branch, the tier reset and begin_stats_generation in _handle_buckaroo_state_change, refresh_session_snapshot and broadcast_state in place of the copies, the overlay built through build_state_message_for.
• buckaroo/server/handlers.py: begin_stats_generation in the four load handlers, broadcast_state in place of the copies, refresh_session_snapshot in /reload_expr; /load and /load_compare reset the stored stats policy to the defaults.

Tests

Three commits: failing tests (d7ba7bd9), a second batch of failing tests (6bd63ed7), then the implementation (9bfebfb9). The second batch is four cases I found untested after the first was seen failing on CI, so they got their own commit and their own failing run. All new tests are in tests/unit/server/test_load_expr.py, in a class TestStatsWire next to TestLoadExprStatsPolicy, which already holds the deferred-session tests. They load a 5-row table whose float column has a 1e9 maximum, so full stats change its minWidth and a stats-free message differs from a complete one there.

• test_stats_request_returns_a_stats_update_that_completes_the_stats: the reply carries the matching stats_gen, an inline wide parquet_b64 payload, and rows equal to the all_stats an inline session of the same build sends. test_rows_are_served_before_and_after_the_stats_request checks that no stray binary frame follows it.
• test_a_stale_stats_gen_gets_stats_aborted_and_runs_nothing (a spy on XorqStatPipeline._execute sees no query), test_an_unsupported_scope_gets_stats_aborted.
• test_load_expr_and_reload_expr_bump_stats_gen, test_df_meta_stats_survives_a_dataflow_field_change (the frame after a search carries df_meta.stats with the next generation next to the rebuilt df_meta, runs no stat query, and a request for the old generation is stale).
• test_a_legacy_client_stays_complete_while_a_caps_client_gets_a_stats_free_frame: two caps clients (one with ?caps=other,stats_update) and two legacy clients (one with ?caps=unknown) on one session. A legacy client's post_processing change gives both legacy clients a message equal to the inline session's apart from df_meta.stats, gives both caps clients a frame with the schema tier only, and a caps client's stats_request is then answered with no further query.
• One test per remaining send site, each with a caps client and a legacy client: test_load_expr_push_keeps_a_legacy_client_complete, test_reload_expr_push_keeps_a_legacy_client_complete, test_load_push_leaves_both_clients_complete, test_load_compare_push_leaves_both_clients_complete, and test_highlight_overlay_is_complete_for_a_legacy_client_and_stats_free_for_a_caps_client. test_overlay_for_a_legacy_client_is_built_after_its_stats_are_completed calls _send_highlight_overlay directly, because every send completes a session that has a legacy client connected, so the overlay never meets a pending session through the socket.
• test_a_client_connecting_after_completion_gets_the_complete_state, test_a_legacy_client_connecting_to_a_pending_session_completes_it: stats computed once are not computed again for a later client.
• test_a_session_targeting_the_schema_tier_is_not_computed, test_a_failed_stats_run_reports_error_until_the_next_generation.
• test_stats_request_emits_a_stats_request_span: a spy sink sees stats.request for a stale and a served request, and firstpull.stats_total.
• The second batch: test_returning_to_a_completed_state_is_answered_from_the_cache (search, complete, clear the search, and the original state's request runs no query and returns the original rows), test_completing_the_stats_keeps_component_config, test_a_warm_load_expr_keeps_the_generation, test_a_stats_request_with_no_data_loaded_is_aborted.
• test_inline_sessions_send_no_df_meta_stats, test_load_push_leaves_both_clients_complete and test_load_compare_push_leaves_both_clients_complete pass on the base branch by design: they pin that the default policy sends the message it always has, and that /load and /load_compare leave no deferred policy behind.

On the first tests commit 16 of its 19 tests fail locally (15 on initial_state carries no df_meta.stats, one on the session having no stats_status) and the three above pass; with the second batch 20 of 23 fail. I checked the tests against seven deliberate regressions of the implementation (no capable-first order, no tier reset on a state change, /load keeping the stored policy, the overlay copying the display config before the message is built, no summary_stats_cache lookup, a snapshot refresh that drops component_config, a warm /load_expr that bumps the generation); each makes at least one test fail.

CI on the tests commits: on d7ba7bd9 and again on 6bd63ed7 (both without the implementation) six of the nine Python / Test jobs fail (3.11, 3.12 and 3.13, and Max Versions 3.11, 3.12 and 3.13); the rest of the checks pass. The other three (3.14, Max Versions 3.14, Windows) pass because xorq is gated to Python below 3.14 and test_load_expr.py is skipped on Windows, so they never ran the new tests. CI's annotation only says the step exited 1 and the REST log API is rate-limited for this account, so the reason for the failures rests on the local run of the same commit.

CI on the implementation commit 9bfebfb9: all 27 check runs completed, 26 succeeded and deploy was skipped. That includes all nine Python / Test jobs, Python / Lint, the JS job, the wheel build and the Playwright jobs. Locally the full unit suite (pytest ./tests/unit -m "not slow") gives 1268 passed and 5 skipped.

Why default behaviour is unchanged

• Nothing is opt-in by default: a session gets stats_delivery="deferred" or stats_tier="schema" only from a /load_expr or /reload_expr body field (feat(xorq): stats tier machinery and the xorq schema tier (rows-first s1) #1021). On the default policy stats_meta returns None, so df_meta is the dataflow's own object and initial_state is byte-for-byte what it was.
• build_state_message_for runs stats only for a deferred session whose status is pending. Every other session takes the path build_state_message always took, with the client's own search_string.
• broadcast_state is the old loop in one place. The only change for a session on the default policy is the send order (capable clients first), which has no effect when no client has a capability.
• stats_gen is bumped on every session, but it is read only through df_meta.stats, which a default session does not send.
• The existing tests pass unchanged (full unit suite: 1268 passed, 5 skipped locally, which is the 1245 of feat(server): pandas and polars schema stats tier (rows-first s2) #1022 plus the 23 new tests).

Deviations from the plan

• A new module, stats_wire.py, holds the builder, broadcast and request handling. session.py stays free of data_loading imports, and the plan's three files would each have needed to import the others.
• broadcast_state replaces the five send loops outright, where the plan routes each loop through build_state_message_for. Capable-first ordering needs one place to live.
• stats_gen is also bumped by /load_expr and /load_compare, not only /load, /reload_expr and state changes: all of them replace the data the stats describe.
• /load and /load_compare reset the session's stored stats policy to full and inline. feat(xorq): stats tier machinery and the xorq schema tier (rows-first s1) #1021 noted that /load left the pair in place. That is harmless while nothing reads it, but a stale deferred would now make a pandas session look pending and send it through complete_stats.
• df_meta.stats is also sent for a session headed for the schema tier with inline delivery (not_computed, reason host), not only for deferred ones. The status follows from the policy pair, so the rule is the same for both.
• A legacy client on a session headed for the schema tier gets the schema tier, as it does from an inline schema session today. Nothing is missing relative to the target, so the plan's rule (run what is missing) has nothing to run. Plan 3's stats_ondemand capability decides this later.
• stats_update carries elapsed_ms (plan 3 section 3.2), which the scope's message shape does not list. stats_aborted is {type, stats_gen, current_gen, scope, reason}, with stats_gen the request's so a client can pair them; reasons are stale, unsupported_scope, not_requestable, error and no_data.
• Only scope: "raw" is accepted. Filtered stats exist only through a dataflow-field change, which bumps the generation (plan 1 section 6). columns is read for the span and otherwise ignored by a whole-run request.
• A failed run is the session's state for that generation (error, reason stats_failed) and is not retried by the next request. The plan does not say.
• refresh_session_snapshot replaces two of the three copies named in plan 2 section 4.1 (websocket_handler.py state change, /reload_expr). The /load_expr copy sets more fields and reads the dataflow's attributes directly, so it stays.
• Counters for updates sent, dropped as stale and errors are the outcome attribute of the stats.request span, not separate counters.

Not in this PR

• Resumable units, StatRun, the time budget and per-unit spans: the next phases generalize complete_stats. stats_update has no remaining field yet.
• The config upgrade on final: the final reply does not carry a rebuilt df_display_args, so a caps client keeps its schema-tier display config (a float column's minWidth, for one) until its next initial_state. A client that connects after completion gets the refreshed config.
• Client code (the jest half of plan 1 phase 2, the scheduler and the pending UI), ?caps=stats_ondemand, the policy fields of plan 3 and any default flip.
• Deferred sessions for /load (pandas and polars): the fields exist on /load_expr only.

🤖 Generated with Claude Code

[github-actions]

📦 TestPyPI package published

pip install --index-strategy unsafe-best-match --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ buckaroo==0.15.9.dev37513303397

or with uv:

uv pip install --index-strategy unsafe-best-match --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ buckaroo==0.15.9.dev37513303397

MCP server for Claude Code

claude mcp add buckaroo-table -- uvx --from "buckaroo[mcp]==0.15.9.dev37513303397" --index-strategy unsafe-best-match --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ buckaroo-table

📖 Docs preview

🎨 Storybook preview

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.