Skip to content

perf(query): admit the observability and annotation readers - #502

Merged
vishr merged 2 commits into
mainfrom
perf/admission-global-readers
Nov 18, 2026
Merged

vishr merged 2 commits into
mainfrom
perf/admission-global-readers

Conversation

@vishr

@vishr vishr commented Nov 18, 2026

Copy link
Copy Markdown
Member

Phase 0 of proposal 0011 (#480): make read admission global.

Problem

The observability readers (overview, topology, dependencies, trace, health_trend) and the annotation reader call the engine directly through the injected queryrows.Queryer, bypassing the executor's read admission. Only panels, exemplars, and trace detail are admitted today. Under a saturating dashboard the bypassing readers oversubscribe the DuckDB read pool (admitted panels + unbounded direct reads → DB.Conn blocks), which defeats the deadline shedding the pool size exists to enforce. A benchmark of v2026.11.8-alpha on two Hetzner ccx33 (publish preset, --query-rate 20) reproduces the collapse: 3/20 ops/s completed, 1546 errors, 3533 shed, p99 8.6 s.

Change

Route those readers through the existing interactive admission lane with an admittingQueryer wrapper (defined in package query, which already imports observability, so there is no import cycle). The wrapper:

  • holds the interactive slot across row iteration and releases it exactly once on Close (via sync.Once);
  • bounds the admission wait by the batch timeout, so a deadline-less caller (background or MCP read) cannot sit in the bounded queue indefinitely;
  • forwards DefaultNamespace() and queryrows.ReadTransactioner, so annotations keep their single-snapshot read and default namespace (both probed by optional-interface assertion in annotations/service.go);
  • counts refusals in the executor's interactive-pressure counter, so a read flood also feeds ingest backpressure.

Excess read load now sheds as a bounded ErrBusy (503 + Retry-After) instead of piling up inside the engine; the annotation route maps ErrBusy to 503 to match the trace route.

The panel-internal rollup reader stays on the raw engine: its reads already run inside an admitted panel slot, so admitting them again would deadlock a one-slot pool (the panel holds the only slot while its nested read waits for one).

Scope

This makes read load degrade gracefully (errors → ~0, no thrash pegging, dashboard-refresh goodput up, ingest flat). It does not by itself meet the sustained read-rate gate — that is a read-cost problem owned by proposals 0007 (additive rollups) and 0010 (result-cache extent), per 0011's research validation. Later phases (background-lane split, cost-aware classes, per-panel shared snapshot) build on this.

Tests

New behavioral tests cover slot-held-until-close, release-once-on-double-close, shed-when-full, read-transaction-admitted-as-a-unit, and default-namespace forwarding, plus an ErrBusy → 503 case on the annotation route. just vet and just check-static pass.

The observability readers (overview, topology, dependencies, trace,
health_trend) and the annotation reader call the engine directly, bypassing
the executor's read admission. Under a saturating dashboard they oversubscribe
the DuckDB read pool, block on DB.Conn, and defeat the deadline shedding the
pool size is meant to enforce, so reads collapse under concurrent ingest
(issue #480).

Route those readers through the existing interactive admission lane with an
admittingQueryer wrapper. The wrapper holds the interactive slot across row
iteration and releases it on Close, bounds the wait by the batch timeout, and
forwards DefaultNamespace and ReadTransactioner so annotations keep their
single-snapshot read. Excess load now sheds as a bounded ErrBusy (503 with
Retry-After) instead of piling up inside the engine.

The panel-internal rollup reader stays on the raw engine: its reads already
run inside an admitted panel slot, so admitting them again would deadlock a
one-slot pool.

This is Phase 0 of proposal 0011. It makes read load degrade gracefully; it
does not by itself meet the sustained read-rate gate, which depends on the
read-cost work in proposals 0007 and 0010.
…ssion

Trace detail is already admitted by the cached-trace run, so routing its
candidate and log reads through the admitting observability reader acquired a
second interactive slot while holding the first and could deadlock a one-slot
pool. Use the raw engine for every reader reached through the executor (the
dashboard rollup reader and trace detail) and admit only the readers that
bypass it (the MCP observability tools and the annotation route).
@vishr
vishr added this pull request to the merge queue Nov 18, 2026
Merged via the queue into main with commit 1977a39 Nov 18, 2026
31 checks passed
@vishr
vishr deleted the perf/admission-global-readers branch November 18, 2026 11:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant