Skip to content

docs(adr): ADR-003 ordered stat tiers and a size policy for degrading summary stats - #1044

Open
paddymul wants to merge 2 commits into
mainfrom
adr-003-stats-tiers-and-size-policy
Open

paddymul wants to merge 2 commits into
mainfrom
adr-003-stats-tiers-and-size-policy

Conversation

@paddymul

@paddymul paddymul commented Oct 6, 2026

Copy link
Copy Markdown
Collaborator

Problem

Nothing on main decides, from the size of a dataframe, how much of the stats pipeline to run, and the hard-coded gates that exist apply a 50,000-row sample to every stat, including length, null_count, min and max, which are then wrong above 1M cells and 50,000 rows. On large xorq entries stats take 75 to 215 s. The policy PRs (#1019, #1023, #1029, #1031, #1033, #1034, and the client PRs #1032 and #1036) implement a three-tier subset (schema, scalar, full). The wider design, with an approx tier, a per-stat tier declaration and a footer provider, exists only in design notes outside the repo.

Change

Adds docs/plans/ADR-003-stats-tiers-and-size-policy.md, status Proposed. It records twelve decisions: four ordered tiers; exact scalars that never read the 50,000-row sample; tier and approx on StatFunc with a fail-closed default; the filter at the start of build_column_dag; one StatsLimits table with a fixed configuration order; a server-side policy with a ceiling no caller can exceed; omitted_keys and approx_keys in df_meta.stats; the compute control, cost pause and demand columns; sort and search guards; the parquet footer rung; tier in every cache key; runtime-added stats. It also lists the alternatives rejected and the 13 decisions that are still open, each with a recommendation where there is one.

No code changes.

Stack

The PRs listed above are retargeted to this branch as their base. Merging this PR retargets them to main. The branch name has no slash because the Checks workflow's pull_request trigger uses branches: "*", which does not match a base branch containing a slash.

Companion ADRs: ADR-001 (#1040, the stat cache) and ADR-002 (how stats reach the client).

🤖 Generated with Claude Code

… summary stats

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@github-actions

github-actions Bot commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

📦 TestPyPI package published

pip install --index-strategy unsafe-best-match --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ buckaroo==0.15.9.dev37492679921

or with uv:

uv pip install --index-strategy unsafe-best-match --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ buckaroo==0.15.9.dev37492679921

MCP server for Claude Code

claude mcp add buckaroo-table -- uvx --from "buckaroo[mcp]==0.15.9.dev37492679921" --index-strategy unsafe-best-match --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ buckaroo-table

📖 Docs preview

🎨 Storybook preview

…and narrows the D11 cache-id gap

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

This branch was successfully deployed

1 active deployment
testpypi — 41bfcab0 Deployed Oct 6, 2026 by paddymul via Publish to TestPyPI #1750
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant