Skip to content

perf: read admission is neither global nor cost-aware #480

Open

Description

@vishr

Type: design, performance

Only panels, exemplars and trace detail pass through admission (internal/query/exec.go:120-126). The observability readers (internal/observability/{overview,topology,dependencies,trace,health_trend}.go), annotations, and variable-option queries (internal/query/variables.go:70-73,107) call the engine directly and compete for the two spare connections and all DuckDB threads. A slot is one connection, with no notion of thread or memory cost. backgroundSlots has capacity 1 (internal/engine/duck.go:540), so a rollup pass blocks refreshReadCaches for its full duration, which contradicts the stated intent that completed-batch writes have their own gate and slot.

Direction: one admission layer at the engine boundary with classes (interactive / panel / background / agent), shedding by deadline; one pinned snapshot and connection per panel run shared by its sub-reads.

Found in a read-only engine audit. Reasoned from source.

Activity

  1. vishr commented on Nov 18, 2026

    @vishr
    MemberAuthor

    Benchmark evidence from v2026.11.8-alpha (current main)

    Ran the read-under-write suite on two Hetzner ccx33 (8 vCPU / 32 GiB, fsn1): pitot-bench --preset publish --query-rate 20. The read gate fails, and the numbers point straight at the unadmitted-reader path this issue describes.

    Sustained read gate (5 min, target 20 ops/s):

    Completed 3.0 / 20 ops/s
    Attempted 6,010 (errors 1546, shed 3533)
    Latency p50 132 ms · p95 2370 ms · p99 8644 ms
    dashboard-refresh 837 / 1000 successful
    drill-traces 19 / 1000
    trace-detail 75 / 1000

    Smoking gun for "not cost-aware": the mixed-p1-r1 phase offers only 60,000 rows/s of ingest plus 20 read ops/s, yet Pitot pegs 7.90 / 8 cores and 19.4 GiB RSS. For comparison, single-signal logs-p4-r1 ingests 80,000 rows/s on 0.56 cores / 6.7 GiB. The entire cost jump is the concurrent read mix — System-overview dashboard refresh (batched panels + annotations) plus exemplar drill-down and trace-detail — scanning the growing Parquet snapshot. dashboard-refresh (the overview + annotations readers, which bypass admission per this issue) is the worst hit, confirming that a slot = one connection with no thread/memory cost accounting lets these readers saturate the box and starve everything else.

    Ingest is unaffected: per-signal and mixed ingest match v2026.11.0-alpha within noise (traces 266k, logs 496k, metrics 558k rows/s; no dropped rows, no restarts, storage verified). The regression is entirely on the read path under concurrent ingest.

    Note: #423's trace-index drill-down fix is present in this build, but drill-traces still lands 19/1000 under load — it narrowed one query shape without addressing the admission/cost root cause here.

    Repro:

    pitot-bench run --host 'root@<host>' --pitot-version v2026.11.8-alpha --preset publish \
      --generator-host 'root@<generator-host>' --query-rate 20
    
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions