fix(collector): switch PeriodicReader on async runtime to enable tonic metrics logs - #50
Open
rodonile wants to merge 1 commit into
Open
fix(collector): switch PeriodicReader on async runtime to enable tonic metrics logs#50rodonile wants to merge 1 commit into
rodonile wants to merge 1 commit into
Conversation
Switch to periodic_reader_with_async_runtime since the default PeriodicReader has no I/O reactor on its background thread and cannot reliably drive the grpc-tonic exporter. Run it on a separate single-thread tokio runtime so metrics export doesn't compete with the collection/publishing pipeline for worker threads. Also wire up the previously-unused reader_timeout config, and flush/shutdown the meter provider before exit.
rodonile
force-pushed
the
otel-exp-metrics
branch
from
September 2, 2026 13:58
c4bbd4a to
e36ae33
Compare
There was a problem hiding this comment.
🟢 Approval recommended
The runtime separation, timeout wiring, and explicit meter provider shutdown are implemented cleanly and align with the stated goal of surfacing OTLP export failures.
Pull request overview
This PR improves OpenTelemetry metrics export reliability/observability in the collector by moving the metrics PeriodicReader onto a Tokio async runtime so the tonic-based OTLP exporter can complete RPCs and emit meaningful export error logs, while also ensuring a final flush on shutdown.
Changes:
- Switch metrics export to
periodic_reader_with_async_runtime::PeriodicReaderusingopentelemetry_sdk::runtime::Tokio, wiringconfig.reader_timeoutinto the reader. - Run OTEL metrics exporting on a dedicated single-worker Tokio runtime to avoid contention with the main collection/publishing runtime.
- Return the
SdkMeterProviderfrom telemetry initialization and callshutdown()before exit to flush the final metrics batch.
File summaries
| File | Description |
|---|---|
| crates/collector/src/main.rs | Moves OTEL metrics periodic export onto a dedicated Tokio runtime, wires reader timeout, and ensures provider shutdown flushes final metrics. |
| crates/collector/Cargo.toml | Enables the opentelemetry_sdk experimental feature required for the async-runtime periodic reader implementation. |
Review details
- Files reviewed: 2/2 changed files
- Comments generated: 0
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
OTEL metric export failures were essentially invisible.
The default
PeriodicReaderhas no export error logging at all:it runs the exporter via
futures_executor::block_onon its own barestd::thread, with no tokio reactor on that thread. The exporter isgrpc-tonic(hyper/tonic), which needs a live tokio I/O driver to complete an RPC, so on
that thread it can't reliably do so, and any failure only shows up as a
generic debug-level log with no gRPC status.
The async-runtime reader schedules export via
self.runtime.spawn(...)inside an actual tokio task, so tonic can complete the round-trip and
return the real gRPC status. On failure it emits a proper structured
error log with the failure reason, giving us visibility into export
errors (e.g. ResourceExhausted) that we previously had no way to detect.
Changes
periodic_reader_with_async_runtime::PeriodicReaderbound toopentelemetry_sdk::runtime::Tokio(requires enablingexperimental_metrics_periodicreader_with_async_runtimeonopentelemetry_sdk).export doesn't compete with the flow/BMP/UDP-notif pipeline for
worker threads.
config.reader_timeoutvia.with_timeout(...)(previously adead config field, since the default reader's builder had no such method).
shutdown()the meter provider before process exit so thelast metrics batch flushes instead of being dropped.