Skip to content

feat: prefill the server before measured phases - #14

Open
vishr wants to merge 2 commits into
mainfrom
feat/prefill-before-reads
Open

vishr wants to merge 2 commits into
mainfrom
feat/prefill-before-reads

Conversation

@vishr

@vishr vishr commented Nov 18, 2026

Copy link
Copy Markdown
Member

Why

Read cost grows with retained data. Two Hetzner ccx33 runs at the same 48k items/s mixed ingest with 20 UI reads/s on v2026.11.9-alpha and v2026.11.8-alpha gave very different results:

Server state Reads completed/s Pitot CPU (of 8) Drill-down and trace-detail successes
Fresh, about 15M rows 20.0 / 20 3.25 all
Prefilled for 10 min, about 180M rows 2.6 / 20 7.3 0

In the prefilled run, every failure was a 10-second timeout on /api/queries/exemplars. A read gate measured on a nearly empty server therefore overstates interactive performance.

What

  • --prefill <duration> and --prefill-processes <n> (default 16) add an ingest-only mixed phase after warmup. Prefill runs without read load and uses the preset's workers and rate.
  • After prefill, the runner waits up to 10 minutes. It continues once pitot_ingest_queue_depth and pitot_rollup_backlog_chunks are exported and read zero in two consecutive scrapes. A missing gauge never counts as drained. The wait is skipped for compaction backlog, which counts underfilled groups and need not reach zero.
  • A new prefill-settled quality gate fails when the backlog does not drain.
  • Prefill rows are excluded from totals, peaks, topology selection and classification.
  • The flags are passed through the local, remote and managed paths, the reproduction command and the workload identity. A run without prefill keeps its existing workload hash.

Tests

go test ./... passes with 177 tests. New tests cover the plan, the gate, the settle loop (two clean scrapes, a missing gauge, cancellation), flag plumbing and the workload hash.

Read cost grows with retained data, so a read gate measured on a nearly
empty server is optimistic. --prefill runs an ingest-only mixed phase after
warmup, waits for the ingest queue and rollup backlog to drain, and records
a prefill-settled gate. Prefill rows are excluded from totals, peaks and
topology selection.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant