Skip to content

docs(proposals): 0012 bounded trace ranking for drill-down - #504

Draft
vishr wants to merge 2 commits into
mainfrom
docs/proposal-0012-bounded-trace-ranking
Draft

vishr wants to merge 2 commits into
mainfrom
docs/proposal-0012-bounded-trace-ranking

Conversation

@vishr

@vishr vishr commented Nov 18, 2026

Copy link
Copy Markdown
Member

Summary

Drill-down ranks every retained trace in its window to return the top 21, so its cost grows with retained data. A server that has ingested for a while times out on /api/queries/exemplars. This draft proposes exactly maintained per-bucket rank lists in the read cache, so drill-down cost depends on window length, not retained traces.

Evidence

  • pitot-bench, Hetzner ccx33 pair, v2026.11.9-alpha, 48k items/s mixed ingest with 20 UI reads/s: a fresh server passes at 20/20 ops/s. After a 10-minute prefill (about 180M rows) it completes 2.6/20 ops/s. All 1,716 failures are exemplar 504s.
  • The new filebench scale benchmark, fully cached: 400k traces take 21 ms (single-service) and 93 ms (two-service); 1.6M traces take 197 ms and 633 ms.
  • A join rewrite gave no gain once the index was fully cached, so this needs a structural change.

Contents

  • docs/proposals/0012-bounded-trace-ranking.md: design, exactness argument, maintenance under publication, compaction and retention, four phases with gates (the last is a full-day prefill on Hetzner), alternatives and risks.
  • internal/engine/trace_candidate_scale_bench_test.go: build tag filebench. PITOT_SCALE_BATCHES sets cached batches of 20k traces; PITOT_SCALE_MIXED=1 spreads traces over two services.

No product code changes. Related: #480, labstack/pitot-bench#14.

Drill-down ranks every retained trace in its window, so it times out once
the server holds data. Propose exactly maintained per-bucket rank lists in
the read cache, and add the filebench scale benchmark that measures the
current cost.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant