Repository navigation
perf(query): admit the observability and annotation readers - #502
Merged
Merged
Conversation
The observability readers (overview, topology, dependencies, trace, health_trend) and the annotation reader call the engine directly, bypassing the executor's read admission. Under a saturating dashboard they oversubscribe the DuckDB read pool, block on DB.Conn, and defeat the deadline shedding the pool size is meant to enforce, so reads collapse under concurrent ingest (issue #480). Route those readers through the existing interactive admission lane with an admittingQueryer wrapper. The wrapper holds the interactive slot across row iteration and releases it on Close, bounds the wait by the batch timeout, and forwards DefaultNamespace and ReadTransactioner so annotations keep their single-snapshot read. Excess load now sheds as a bounded ErrBusy (503 with Retry-After) instead of piling up inside the engine. The panel-internal rollup reader stays on the raw engine: its reads already run inside an admitted panel slot, so admitting them again would deadlock a one-slot pool. This is Phase 0 of proposal 0011. It makes read load degrade gracefully; it does not by itself meet the sustained read-rate gate, which depends on the read-cost work in proposals 0007 and 0010.
…ssion Trace detail is already admitted by the cached-trace run, so routing its candidate and log reads through the admitting observability reader acquired a second interactive slot while holding the first and could deadlock a one-slot pool. Use the raw engine for every reader reached through the executor (the dashboard rollup reader and trace detail) and admit only the readers that bypass it (the MCP observability tools and the annotation route).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 0 of proposal 0011 (#480): make read admission global.
Problem
The observability readers (
overview,topology,dependencies,trace,health_trend) and the annotation reader call the engine directly through the injectedqueryrows.Queryer, bypassing the executor's read admission. Only panels, exemplars, and trace detail are admitted today. Under a saturating dashboard the bypassing readers oversubscribe the DuckDB read pool (admitted panels + unbounded direct reads →DB.Connblocks), which defeats the deadline shedding the pool size exists to enforce. A benchmark ofv2026.11.8-alphaon two Hetznerccx33(publishpreset,--query-rate 20) reproduces the collapse: 3/20 ops/s completed, 1546 errors, 3533 shed, p99 8.6 s.Change
Route those readers through the existing interactive admission lane with an
admittingQueryerwrapper (defined in packagequery, which already importsobservability, so there is no import cycle). The wrapper:Close(viasync.Once);DefaultNamespace()andqueryrows.ReadTransactioner, so annotations keep their single-snapshot read and default namespace (both probed by optional-interface assertion inannotations/service.go);Excess read load now sheds as a bounded
ErrBusy(503 +Retry-After) instead of piling up inside the engine; the annotation route mapsErrBusyto 503 to match the trace route.The panel-internal rollup reader stays on the raw engine: its reads already run inside an admitted panel slot, so admitting them again would deadlock a one-slot pool (the panel holds the only slot while its nested read waits for one).
Scope
This makes read load degrade gracefully (errors → ~0, no thrash pegging, dashboard-refresh goodput up, ingest flat). It does not by itself meet the sustained read-rate gate — that is a read-cost problem owned by proposals 0007 (additive rollups) and 0010 (result-cache extent), per 0011's research validation. Later phases (background-lane split, cost-aware classes, per-panel shared snapshot) build on this.
Tests
New behavioral tests cover slot-held-until-close, release-once-on-double-close, shed-when-full, read-transaction-admitted-as-a-unit, and default-namespace forwarding, plus an
ErrBusy→ 503 case on the annotation route.just vetandjust check-staticpass.