Skip to content

fix(collector): switch PeriodicReader on async runtime to enable tonic metrics logs - #50

Open
rodonile wants to merge 1 commit into
network-analytics:mainfrom
rodonile:otel-exp-metrics
Open

fix(collector): switch PeriodicReader on async runtime to enable tonic metrics logs#50
rodonile wants to merge 1 commit into
network-analytics:mainfrom
rodonile:otel-exp-metrics

Conversation

@rodonile

@rodonile rodonile commented Sep 2, 2026

Copy link
Copy Markdown
Member

Problem

OTEL metric export failures were essentially invisible.

The default PeriodicReader has no export error logging at all:
it runs the exporter via futures_executor::block_on on its own bare
std::thread, with no tokio reactor on that thread. The exporter is grpc-tonic
(hyper/tonic), which needs a live tokio I/O driver to complete an RPC, so on
that thread it can't reliably do so, and any failure only shows up as a
generic debug-level log with no gRPC status.

The async-runtime reader schedules export via self.runtime.spawn(...)
inside an actual tokio task, so tonic can complete the round-trip and
return the real gRPC status. On failure it emits a proper structured
error log with the failure reason, giving us visibility into export
errors (e.g. ResourceExhausted) that we previously had no way to detect.

Changes

  • Switch to periodic_reader_with_async_runtime::PeriodicReader bound to
    opentelemetry_sdk::runtime::Tokio (requires enabling
    experimental_metrics_periodicreader_with_async_runtime on opentelemetry_sdk).
  • Run it on its own dedicated single-worker-thread runtime so metrics
    export doesn't compete with the flow/BMP/UDP-notif pipeline for
    worker threads.
  • Wire up config.reader_timeout via .with_timeout(...) (previously a
    dead config field, since the default reader's builder had no such method).
  • Explicitly shutdown() the meter provider before process exit so the
    last metrics batch flushes instead of being dropped.

@rodonile rodonile changed the title fix(collector): isolate OTEL metrics PeriodicReader on its own runtime to enable tonic metrics logs fix(collector): switch PeriodicReader on async runtime to enable tonic metrics logs Sep 2, 2026
Switch to periodic_reader_with_async_runtime since the default
PeriodicReader has no I/O reactor on its background thread and
cannot reliably drive the grpc-tonic exporter. Run it on a separate
single-thread tokio runtime so metrics export doesn't compete with
the collection/publishing pipeline for worker threads. Also wire up
the previously-unused reader_timeout config, and flush/shutdown the
meter provider before exit.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The runtime separation, timeout wiring, and explicit meter provider shutdown are implemented cleanly and align with the stated goal of surfacing OTLP export failures.

Pull request overview

This PR improves OpenTelemetry metrics export reliability/observability in the collector by moving the metrics PeriodicReader onto a Tokio async runtime so the tonic-based OTLP exporter can complete RPCs and emit meaningful export error logs, while also ensuring a final flush on shutdown.

Changes:

  • Switch metrics export to periodic_reader_with_async_runtime::PeriodicReader using opentelemetry_sdk::runtime::Tokio, wiring config.reader_timeout into the reader.
  • Run OTEL metrics exporting on a dedicated single-worker Tokio runtime to avoid contention with the main collection/publishing runtime.
  • Return the SdkMeterProvider from telemetry initialization and call shutdown() before exit to flush the final metrics batch.
File summaries
File Description
crates/collector/src/main.rs Moves OTEL metrics periodic export onto a dedicated Tokio runtime, wires reader timeout, and ensures provider shutdown flushes final metrics.
crates/collector/Cargo.toml Enables the opentelemetry_sdk experimental feature required for the async-runtime periodic reader implementation.
Review details
  • Files reviewed: 2/2 changed files
  • Comments generated: 0
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants