fix(trim): bound per-coordinate trimming so history-heavy columns don't stall

[clawd131662]

Problem

co:trim's nightly cron ran for ~23h a night while advancing its checkpoint not at
all for days: db/trim_state.yml stayed frozen while every run re-scanned the same
window and died on one coordinate. Two distinct issues, in layers:

1. rowid filesort timeout. To keep a hot coordinate's newest --keep rows, trim ran
   ... ORDER BY rowid DESC .pluck(:rowid). The (wid, x, z, time) index can't satisfy
   ORDER BY rowid, so a million-row column sorted its whole history in one statement,
   exceeded max_statement_time, and raised before save_trim_state — the checkpoint
   never advanced.
2. table look-up on the delete path. Even after fixing the cutoff, the per-(y, action)
   victim delete filters on y/action, which aren't in the index — so every victim
   batch on a million-row machine column table-looks-up its way through the column. A
   single batch there exceeds max_statement_time too; keyset paging removes the O(n²)
   re-scan but not the per-row look-up.

Fix

bfcb7a9 — bound the per-coordinate trim. Cutoff via ORDER BY time DESC (the
index's own order, O(keep) look-ups; measured 0.118s vs >3600s). Delete below it in a
(time, rowid) keyset walk (no filesort, no re-scan). A StatementInvalid on one
coordinate is logged and skipped, never aborting the segment or forfeiting the checkpoint.

1a73f55 — route machine columns to a covering whole-column trim. A (wid, x, z)
column with a COVERING_THRESHOLD-th newest row (machine-scale) is trimmed whole through
the covering (wid, x, z, time) index — cutoff and delete both ride the index with no
table look-up — and its other (y, action) hot keys in the segment are skipped. Column
size is probed index-only (offset/limit), not COUNTed. Smaller coordinates keep the
precise per-(y, action) keyset trim. Adds --throttle: sleep throttle*(batch seconds)
between delete batches so a large covering cleanup yields the disk (duty ~ 1/(1+throttle));
0 (default) for daily incremental runs.

Semantics: normal coordinates still keep the newest --keep rows per (wid, x, y, z, action);
a machine column collapses to per-(x, z) keep, which the --threshold gate confines to
automated machines (normal play never piles up COVERING_THRESHOLD rows at one column).

Validation

No automated suite in this repo, so validated against the production DB:

• ruby -c and rubocop clean (only the pre-existing Metrics/ClassLength).
• Dry-run routing: machine columns report "whole column", normal coordinates report
  "action=…"; victims = total − keep on every line.
• Index-only size probe ~1.8s; covering cutoff ~0.7ms; covering DELETE … LIMIT and
  keyset precise delete both verified running, consumer healthy throughout.

Note

The host carries a ~100M-row backlog at one already-stopped end-portal machine array; it
is being cleaned incrementally by the throttled path (the machines are stopped, so this is
reclaiming space, not unblocking the consumer). Follow-up: docs still use the old "pluck"
wording.

🤖 Generated with Claude Code

[claude]

Read through both commits and the surrounding co:trim code. The diagnosis in the description holds up: ORDER BY rowid DESC against a (wid, x, z, time) index really is a filesort over the whole column, and moving the cutoff into the index's own order plus the keyset walk is the right shape of fix. The per-coordinate StatementInvalid rescue is a good call — forfeiting the checkpoint over one bad coordinate was the thing that made this stall for days. delete_in_batches cleanly unifies both delete paths, and big_column?'s index-only probe is a nice touch.

My concerns are all with the second commit's covering path, where the index's inability to filter y/action is traded for scope that's broader than the docs describe:

• column_scope ignores --action (co.thor:415) — the whole-column delete removes kill rows even when the run excluded them, orphaning co_entity, and the purge_orphaned_entities reminder is gated on --action so it never fires. Given the backlog in question is an end-portal machine array, this is likely to bite on the first real run.
• @columns_trimmed/@big_column are run-scoped, not segment-scoped (co.thor:456) — contradicts the comment and the PR description, and means a multi-segment run trims a machine column once and advances the checkpoint past the rest.
• COVERING_THRESHOLD keys on all-time column size, not in-window heat (co.thor:41) — a long-lived public coordinate plus one ordinary hot (y, action) key is enough to collapse a column's whole build history, which is the case co:trim is meant to be safe for.
• Dry-run COUNT is unbounded (co.thor:448) — the preview does the expensive thing the design avoids elsewhere, and the new rescue hides the failure.

One more, outside the diff: the semantics change here is user-visible and CLAUDE.md's co:trim section is now wrong — it still states the invariant as "newest N rows per (wid, x, y, z, action)" and doesn't list --throttle. The description flags docs as a follow-up; the --throttle option and the collapse semantics seem worth carrying in this PR rather than deferring, since they're what someone reads before running a destructive command.

Nothing here is a rewrite — the first commit looks ready as-is, and the second mostly needs its scope narrowed to match what the comments already claim it does.

[claude]

column_scope carries no action filter, so the covering path deletes every row in the column below the cutoff — including actions the run explicitly excluded. --action defaults to -block,+block,click precisely because deleting kill rows orphans co_entity (CLAUDE.md), and this path is the most likely to hit them: the backlog described in the PR note is an end-portal machine array, i.e. enderman kills, which are exactly the Block::Tile → Entity (via data) rows.

Compounding it, trim_notice_messages only prints the co:purge_orphaned_entities reminder when option_action_ids.include?(3) — so a default-action run that collapses a machine column silently orphans entities and says nothing about it.

The index genuinely can't filter action, so I don't think restricting the delete is on the table; but the notice should fire whenever a whole-column trim actually ran, and the action collapse deserves to be stated alongside the y collapse in the docs.

[claude]

@columns_trimmed (and @big_column above) are run-scoped, but the comment on trim_coordinate — and the PR description — say the skip covers a column's other hot keys in the same segment. trim_proc never clears either hash, so once a machine column is trimmed in segment N it is skipped in every later segment of the same run.

On a multi-segment run (a catch-up over the ~100M-row backlog, but also an ordinary daily run on a busy server) the column is trimmed once and the rest of the window's accumulation is left behind while the checkpoint advances past it — which breaks the invariant CLAUDE.md states, on exactly the columns this path exists for. A stale big_column? == false has the same shape in the worse direction: a column that crosses COVERING_THRESHOLD mid-run keeps taking the slow precise path this commit was written to avoid.

Clearing both at the top of the segment block in trim_proc restores the documented behaviour. If you do, note that report_planned_column then needs the planned_victims de-dup treatment that report_planned_trim has, or a dry run will count the same column once per segment.

[claude]

The routing gate is the column's all-time size, not how hot it is in the scan window — and once it trips, the whole (wid, x, z) column collapses to --keep rows across every y and every action, for all history.

Those two conditions are weaker in combination than the comment reads. A single (y, action) key only has to gain --threshold (default 100) rows within the window to be flagged hot, which ordinary play reaches without any machine involved (a spawn door or shop chest clicked 100 times in a window). And 50k rows at one (x, z) across the full y-range is reachable for a long-lived public coordinate. When both happen at the same column, the trim deletes that column's entire build history — the thing co:trim promises to preserve, as distinct from co:purge.

Gating on in-window growth of the (wid, x, z) group instead of total column size would confine the path to actually-active machines (hot_coordinates already has the window; a second grouped count would give it). Short of that, making the threshold an option rather than a constant would at least let it be raised without a code edit when a false positive shows up.

[claude]

This is the unbounded COUNT that big_column? was deliberately written to avoid, and it runs only on columns already known to hold ≥ COVERING_THRESHOLD rows (per the PR note, up to ~100M). At that size it's a real risk of exceeding max_statement_time — and the new rescue in trim_proc will then report it as a skipped coordinate, so --dry-run quietly under-reports on precisely the columns most worth previewing before a destructive run.

A bounded count (.limit(n).count, which AR wraps in a subquery) reported as "≥ n" would keep the preview honest and cheap.