You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
perf: read admission is neither global nor cost-aware #480
Only panels, exemplars and trace detail pass through admission (internal/query/exec.go:120-126). The observability readers (internal/observability/{overview,topology,dependencies,trace,health_trend}.go), annotations, and variable-option queries (internal/query/variables.go:70-73,107) call the engine directly and compete for the two spare connections and all DuckDB threads. A slot is one connection, with no notion of thread or memory cost. backgroundSlots has capacity 1 (internal/engine/duck.go:540), so a rollup pass blocks refreshReadCaches for its full duration, which contradicts the stated intent that completed-batch writes have their own gate and slot.
Direction: one admission layer at the engine boundary with classes (interactive / panel / background / agent), shedding by deadline; one pinned snapshot and connection per panel run shared by its sub-reads.
Found in a read-only engine audit. Reasoned from source.
Benchmark evidence from v2026.11.8-alpha (current main)
Ran the read-under-write suite on two Hetzner ccx33 (8 vCPU / 32 GiB, fsn1): pitot-bench --preset publish --query-rate 20. The read gate fails, and the numbers point straight at the unadmitted-reader path this issue describes.
Sustained read gate (5 min, target 20 ops/s):
Completed
3.0 / 20 ops/s
Attempted
6,010 (errors 1546, shed 3533)
Latency
p50 132 ms · p95 2370 ms · p99 8644 ms
dashboard-refresh
837 / 1000 successful
drill-traces
19 / 1000
trace-detail
75 / 1000
Smoking gun for "not cost-aware": the mixed-p1-r1 phase offers only 60,000 rows/s of ingest plus 20 read ops/s, yet Pitot pegs 7.90 / 8 cores and 19.4 GiB RSS. For comparison, single-signal logs-p4-r1 ingests 80,000 rows/s on 0.56 cores / 6.7 GiB. The entire cost jump is the concurrent read mix — System-overview dashboard refresh (batched panels + annotations) plus exemplar drill-down and trace-detail — scanning the growing Parquet snapshot. dashboard-refresh (the overview + annotations readers, which bypass admission per this issue) is the worst hit, confirming that a slot = one connection with no thread/memory cost accounting lets these readers saturate the box and starve everything else.
Ingest is unaffected: per-signal and mixed ingest match v2026.11.0-alpha within noise (traces 266k, logs 496k, metrics 558k rows/s; no dropped rows, no restarts, storage verified). The regression is entirely on the read path under concurrent ingest.
Note: #423's trace-index drill-down fix is present in this build, but drill-traces still lands 19/1000 under load — it narrowed one query shape without addressing the admission/cost root cause here.
Activity
vishr commentedon Nov 18, 2026
Benchmark evidence from
v2026.11.8-alpha(currentmain)Ran the read-under-write suite on two Hetzner
ccx33(8 vCPU / 32 GiB, fsn1):pitot-bench --preset publish --query-rate 20. The read gate fails, and the numbers point straight at the unadmitted-reader path this issue describes.Sustained read gate (5 min, target 20 ops/s):
Smoking gun for "not cost-aware": the
mixed-p1-r1phase offers only 60,000 rows/s of ingest plus 20 read ops/s, yet Pitot pegs 7.90 / 8 cores and 19.4 GiB RSS. For comparison, single-signallogs-p4-r1ingests 80,000 rows/s on 0.56 cores / 6.7 GiB. The entire cost jump is the concurrent read mix — System-overview dashboard refresh (batched panels + annotations) plus exemplar drill-down and trace-detail — scanning the growing Parquet snapshot.dashboard-refresh(theoverview+annotationsreaders, which bypass admission per this issue) is the worst hit, confirming that a slot = one connection with no thread/memory cost accounting lets these readers saturate the box and starve everything else.Ingest is unaffected: per-signal and mixed ingest match
v2026.11.0-alphawithin noise (traces 266k, logs 496k, metrics 558k rows/s; no dropped rows, no restarts, storage verified). The regression is entirely on the read path under concurrent ingest.Note: #423's trace-index drill-down fix is present in this build, but
drill-tracesstill lands 19/1000 under load — it narrowed one query shape without addressing the admission/cost root cause here.Repro:
docs(proposals): add 0011 research validation and phased build sequence