Skip to content

feat(eventhubs): add distributed tracing to the clients - #7339

Open
Johnathan W (j7nw4r) wants to merge 12 commits into
Azure:mainfrom
j7nw4r:feat/7336-eventhubs-tracing
Open

feat(eventhubs): add distributed tracing to the clients#7339
Johnathan W (j7nw4r) wants to merge 12 commits into
Azure:mainfrom
j7nw4r:feat/7336-eventhubs-tracing

Conversation

@j7nw4r

@j7nw4r Johnathan W (j7nw4r) commented Aug 15, 2026

Copy link
Copy Markdown
Member

Summary

The Event Hubs C++ SDK emits no distributed tracing spans. Producer and consumer operations stay invisible to OpenTelemetry-based observability pipelines. This change adds provider-neutral spans on send and receive, built only on the azure-core tracing abstraction.

Motivation

Distributed tracing is a GA requirement in the Azure SDK guidelines. Without spans on send and receive, users cannot correlate Event Hubs traffic with the rest of a traced application. The azure-core tracing abstraction already provides a provider-neutral API, so the instrumentation adds no new dependency.

Changes

  • Adds a public std::shared_ptr<Azure::Core::Tracing::TracerProvider> TracingProvider field to ProducerClientOptions and ConsumerClientOptions.
  • Every ProducerClient::Send overload creates exactly one span ProducerClient.Send with SpanKind::Producer. The span covers the batch creation, which opens the AMQP link, and the whole retry loop, so a connection failure is recorded and retries add no extra top-level span.
  • PartitionClient::ReceiveEvents creates one span PartitionClient.ReceiveEvents with SpanKind::Client; the receive count is the size of the returned vector, not the internal prefetch count.
  • Span attributes use the OpenTelemetry semantic conventions 1.17.0 names, because the azure-core-tracing-opentelemetry adapter pins schema 1.17.0: az.namespace, messaging.system, messaging.destination.name, messaging.operation, messaging.batch.message_count, net.peer.name. server.address is not used because it post-dates schema 1.17.0.
  • The instrumentation scope is azure-messaging-eventhubs-cpp with the package version.
  • On failure or cancellation the span records the exception and the method rethrows unchanged. Failures that the retry loop absorbs set no final span status.
  • With no TracingProvider set, no span is created and no tracing data is recorded. The span helper returns before it adds any attribute, so the path costs 2 heap allocations for each operation. Both come from the child context that azure-core derives, and issue TracingContextFactory allocates when no tracer is configured #7340 tracks removing them.
  • Adds a shared helper in src/private/eventhubs_tracing.hpp and src/eventhubs_tracing.cpp.
  • EventDataBatch::NumberOfEvents() becomes const and its mutex becomes mutable, because Send takes EventDataBatch const& and must read the batch count. This is source-compatible.
  • messaging.batch.message_count goes out as an unsigned integer attribute, which matches the type the 1.17.0 convention defines.
  • Adds 16 unit tests in test/ut/eventhubs_tracing_test.cpp with local test doubles; they need no OpenTelemetry and no live service.
  • Adds one live-only test for the receive span in test/ut/consumer_client_test.cpp, because a PartitionClient needs a live AMQP link.
  • No opentelemetry-cpp dependency is added; vcpkg.json is untouched.

Deferred to follow-up work: cross-message trace-context propagation, remote-parent extraction, and span links for batches are not in this pull request. Azure Core has no API for them today; the only propagation API is Span::PropagateToHttpHeaders, and CreateSpanOptions has no Links member. Issue #7336 asks for these to be designed in Azure Core or tracked as a follow-up rather than bypassed with Event Hubs-specific OpenTelemetry calls.

Cross-language note: the .NET SDK emits a send span for Event Hubs and sets az.namespace to Microsoft.EventHub, which this change matches. The .NET SDK creates no receive span on the pull path, so the PartitionClient.ReceiveEvents span follows the OpenTelemetry messaging conventions rather than a .NET precedent. The Go azeventhubs package ships no tracing today.

Known limitations: the PartitionClient::ReceiveEvents span is covered only by a live-only test, because a PartitionClient is reachable only through a factory that opens a real AMQP link. CI runs in playback and skips that test, so the receive span has no coverage in CI. An offline seam needs a change in azure-core-amqp, tracked in issue #7341.

Test plan

  • Local unit test suite with AZURE_TEST_MODE=PLAYBACK: 100% tests passed, 0 tests failed out of 111 (the 94-test baseline plus the 17 new tests). The live-only receive-span test is skipped in playback and is unverified.
  • clang-format-11 --dry-run -Werror is clean over every changed C++ file.

Closes #7336

Add eventhubs_tracing_test.cpp with the recording tracer, span, and
attribute set doubles for the Event Hubs tracing work in issue 7336.

The tests pin the TracingProvider option on ProducerClientOptions and
ConsumerClientOptions, the eager tracer creation in both clients, the
ProducerClient.Send span shape on a cancelled context and on a
non-cancel exception, and the shape of the shared StartSpan helper for
PartitionClient.ReceiveEvents.

Tests 1 to 7 do not compile until the tracing source lands. The
TracingProvider member does not exist yet, and the private header
src/private/eventhubs_tracing.hpp does not exist yet.

SendWithoutProviderIsUnchanged is a characterization test. It passes
today. It guards the no-provider path against a null dereference after
the source lands.
ProducerClientOptions and ConsumerClientOptions gain a TracingProvider
field. Each client builds a TracingContextFactory from that field at
construction time. When the field is empty, the factory has no tracer
and the clients create no spans.

ProducerClient::Send creates one Producer span named
ProducerClient.Send. PartitionClient::ReceiveEvents creates one Client
span named PartitionClient.ReceiveEvents. Both spans carry the
messaging attributes, the az.namespace attribute, and the message
count. Both spans record an exception event and an error status, then
rethrow the original exception.

A new private helper, src/private/eventhubs_tracing.hpp, holds the
factory creation and the span shape, so the producer and the partition
client emit the same attributes.

EventDataBatch::NumberOfEvents is now const, and its mutex is mutable,
because Send takes the batch by const reference.

Trace-context propagation and span links stay deferred, because Azure
Core has no API for them.
Run clang-format-11 over eventhubs_tracing_test.cpp. The declarations
of AddAttributes and SetStatus on the recording span double broke the
Validate Clang Format step in CI.

The change is formatting only. No assertion, no test name, and no
behavior changed.
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 3 pipeline(s).
7 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds provider-neutral distributed tracing to Event Hubs producer and consumer operations.

Changes:

  • Adds tracing-provider configuration and shared span helpers.
  • Instruments send and receive operations with messaging attributes and error recording.
  • Adds tracing tests, build integration, and changelog documentation.

Public API and tracing architecture changes require maintainer review.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
CHANGELOG.md Documents distributed tracing support.
CMakeLists.txt Builds tracing helper sources.
inc/azure/messaging/eventhubs/consumer_client.hpp Adds consumer tracing configuration.
inc/azure/messaging/eventhubs/event_data_batch.hpp Makes event-count access const-safe.
inc/azure/messaging/eventhubs/partition_client.hpp Stores receive tracing context.
inc/azure/messaging/eventhubs/producer_client.hpp Adds producer tracing configuration.
src/consumer_client.cpp Propagates tracing into partition clients.
src/eventhubs_tracing.cpp Implements span creation and attributes.
src/partition_client.cpp Instruments receive operations.
src/private/eventhubs_tracing.hpp Declares shared tracing helpers.
src/private/eventhubs_utilities.hpp Extends partition-client factory inputs.
src/producer_client.cpp Instruments send operations.
test/ut/CMakeLists.txt Registers tracing tests.
test/ut/eventhubs_tracing_test.cpp Tests tracing options and span behavior.

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

Comment thread sdk/eventhubs/azure-messaging-eventhubs/src/producer_client.cpp
Comment thread sdk/eventhubs/azure-messaging-eventhubs/src/eventhubs_tracing.cpp Outdated
Comment thread sdk/eventhubs/azure-messaging-eventhubs/src/eventhubs_tracing.cpp
Comment thread sdk/eventhubs/azure-messaging-eventhubs/test/ut/eventhubs_tracing_test.cpp Outdated
Comment thread sdk/eventhubs/azure-messaging-eventhubs/src/partition_client.cpp
ProducerClient::Send converted the batch to an AMQP message before it
started the span. An empty batch makes that conversion throw, so the
send failed before a span existed and nothing recorded the failure.

The conversion now runs at the top of the try block, after the span
starts. The existing catch clause adds the exception to the span and
rethrows, the same as every other send failure.
The OpenTelemetry semantic conventions at schema 1.17.0 define
messaging.batch.message_count as an int. The Event Hubs tracing helper
records it as a string, and the test doubles could not show the
difference, because they stringified every overload into one map.

TestAttributeSet and TestSpan now record a type tag for each attribute.
The tag names the overload that delivered the value. Three tests use it:
the receive span count and the send span count must arrive through the
uint64 overload, and SetMessageCount must stay quiet and safe when the
factory has no tracer. The first two tests are red today. The third one
is a characterization test that guards the null attribute set hazard.
The batch message count now goes out as an unsigned integer attribute,
which matches the OpenTelemetry semantic conventions at schema 1.17.0.
Before this change the helper wrote the count as a string.

SetMessageCount takes the tracing context factory, because the typed
attribute needs an attribute set. The helper guards the null attribute
set on the no-tracer path, because the factory returns nullptr when it
has no tracer and that pointer is not null safe.
The span helper built a std::string temporary for each attribute name
before it added the attribute. Two of the names are longer than the
libc++ small-string limit, so each call heap allocated even when the
client had no tracer and dropped every attribute.

The span helper now returns the tracing context immediately when the
factory has no tracer. The message count helper keeps its own null
attribute set guard, because PartitionClient::ReceiveEvents calls it
directly.

The remaining two allocations per operation come from azure-core and
are tracked in issue 7340.
The SingleSpan helper called front() on the tracer list after a
non-fatal EXPECT_EQ on the list size. When no tracer was created, that
call dereferenced an empty std::list and the test process ended in a
segmentation fault. The crash stopped the run, so the tests after the
first affected one never reported at all.

The helper now returns null when the tracer list is not the expected
size, so a failure reports through the caller's ASSERT_NE assertion
rather than a crash. This mirrors the guard that already protects the
spans list two lines later.
The receive span of PartitionClient::ReceiveEvents has no offline
coverage. A partition client comes only from a factory that opens a
live AMQP link, so no unit test can reach the span, the message count,
or the exception path.

This test runs against a real namespace. It gives the consumer client
a recording tracing provider, receives events from a partition, and
makes sure the span name, the span kind, the messaging attributes, and
the message count are correct. The count assertion compares the span
attribute with the size of the vector that ReceiveEvents returned.

The recording tracing test doubles move from eventhubs_tracing_test.cpp
into a shared header, because the live test lives in a different file.
The two convenience Send overloads build the batch before they send it,
and CreateBatch opens the connection, the session and the AMQP link. A
bad host fails there. Only the batch overload starts a span today, so
that failure unwinds with no span at all.

Add two offline tests that send one event and a vector of three events
against a fake connection string. Each test asserts one span named
ProducerClient.Send with the kind Producer, the message count as a
uint64 attribute, one recorded event and the Error status. Both fail
today because the provider holds zero spans.

Add a characterization test that keeps the no-provider path unchanged,
and a live test that sends one event and asserts a single span. The
live test catches a nested second span, which an offline test cannot
reach because CreateBatch always throws offline.
The convenience Send overloads made the batch first, and the batch
creation opens the connection, the session, and the AMQP link. A bad
host or a failed authentication threw there, and the stack unwound
before a span existed. A caller then saw no producer span for the
failed send.

Each public Send overload now starts one span, and one try/catch
records the exception on that span. A new private method,
SendBatchInSpan, holds the send body and starts no span, so one
logical send still makes one span. The convenience overloads pass the
span context to CreateBatch, which puts the link open under the span.

@j7nw4r Johnathan W (j7nw4r) left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Every review finding is fixed, and each fix is in its own commit. The empty-batch send now records a span, the batch message count goes out as an unsigned integer, the no-tracer path skips the attribute work, the test helper no longer crashes on an empty tracer list, and every ProducerClient::Send overload now creates exactly one span that covers the batch creation. The playback suite went from 94 tests to 111, all passing.

Two findings need work that does not belong here. Azure Core injects the tracing factory into the context even with no tracer, which costs 2 heap allocations for each operation, and the short-circuit cannot be written from a service package because ServiceSpan's default constructor is private. That is issue #7340. The send and receive paths also have no offline test seam, because MessageSender and MessageReceiver are both final types that only a live AMQP link produces, so azure-core-amqp has to publish its existing mock server before those spans can be checked in CI. That is issue #7341.

Two gaps stay open in this pull request, and I would rather state them than leave them implied. The receive span is covered only by a live-only test that CI skips. The rule that one logical Send makes one span is also live-only, because CreateBatch throws offline before the send logic runs, so no test that CI runs can tell a correct implementation from one that nests a second span. Both close when #7341 lands.

The measurements behind these claims are real rather than reasoned. I counted the allocations with a probe against the compiled library, reproduced the test crash and then removed it, and deleted the receive span block to confirm the suite never noticed.

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 3 pipeline(s).
7 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

The tracing change added a TracingProvider field to
ProducerClientOptions and ConsumerClientOptions, but no document told
a user how to set it. The only repo document on tracing,
doc/DistributedTracing.md, showed clientOptions.Telemetry.TracingProvider.
That line does not compile for Event Hubs, because the two option
structs do not derive from Azure::Core::_internal::ClientOptions and
have no Telemetry field.

- Adds a "Distributed tracing" section to the Event Hubs README with a
  snippet that builds an OpenTelemetryProvider and sets TracingProvider
  on both client options.
- Lists the span names, the span kinds, and the attributes that the
  clients emit, and names the 1.17.0 semantic conventions schema.
- States that the client creates no spans when TracingProvider is not
  set, because there is no global fallback provider.
- Adds a note to doc/DistributedTracing.md for the clients that declare
  TracingProvider at the top level of their own options structure.

@sagar0207 Sagar Patel (sagar0207) left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the tracing design end to end, and verified the semantic-convention claims against the spec at tag v1.17.0 rather than the summary tables, plus the azure-core headers in this repo.

The design is sound. _detail::StartSpan correctly guards HasTracer() and the null return from CreateAttributeSet() (tracing.cpp:118-126) - that null is real and would crash. Error status is handled: ServiceSpan::AddEvent(std::exception const&) sets SpanStatus::Error internally (service_tracing.hpp:142-149), so the bare AddEvent(ex) in the catch blocks is sufficient. NumberOfEvents() const + mutable mutex is correct and ABI-safe (inline, no exported symbol). SendBatchInSpan genuinely yields one span per logical Send.

I also confirmed two of your defensive claims and would let them stand:

  • Propagation genuinely requires azure-core changes. Span (tracing_impl.hpp:170-236) exposes only PropagateToHttpHeaders with no trace-id/span-id accessor, and CreateSpanOptions (line 242) has only ParentSpan, no Links. Not a dodge.
  • ServiceSpan's private default ctor does block a service-side short-circuit (service_tracing.hpp:34-35).

Two blocking items and three worth discussing, inline. All five are convention/scope issues, not correctness bugs.

One follow-up not worth an inline comment: neither client creates spans for GetEventHubProperties / GetPartitionProperties, and CreateBatch / CreatePartitionClient also open links without spans. #7336 scopes this PR to send/receive so that is defensible, but it is a real completeness gap for a later PR.

Things I checked and am explicitly not raising, so you do not have to re-defend them: missing SetStatus(Ok) on success (correct - OTel wants Unset), -Wreorder (declaration order matches init order in both clients), the raw this stored in the Context by CreateTracingContext (the derived context never escapes the synchronous call, and Processor holds the client behind a unique_ptr so the factory address is stable), and the static_cast in the test doubles (the provider is the only AttributeSet source).

"publish",
m_eventHub,
m_fullyQualifiedNamespace,
size_t{1},

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking (convention). This is the single-message overload, so per the v1.17.0 convention this PR targets, messaging.batch.message_count should not be set here.

From the v1.17.0 messaging attributes table, note [2] on messaging.batch.message_count:

Instrumentations SHOULD NOT set messaging.batch.message_count on spans that operate with a single message. When a messaging client library supports both batch and single-message API for the same operation, instrumentations SHOULD use messaging.batch.message_count for batching APIs and SHOULD NOT use it for single-message APIs.

ProducerClient is exactly the library shape that note describes: Send(EventData const&) is the single-message API, while Send(EventDataBatch const&) and Send(std::vector<EventData> const&) are the batching APIs. That this overload batches internally is an implementation detail; the public API is single-message.

Suggest passing Azure::Nullable<size_t>{} here. The other two overloads are correct as-is and should keep the count even when it happens to be 1.

Two tests currently pin the non-conforming value and would need to flip to asserting absence:

  • test/ut/eventhubs_tracing_test.cpp:219 (SendEventSpanRecordsCreateBatchFailure)
  • test/ut/producer_client_test.cpp:137 (SendEventSpan_LIVEONLY_)

Azure::Core::Amqp::Models::_internal::AmqpError>
result;
// The message count is known only when the loop ends, so the span gets it later.
auto tracingContext = _detail::StartSpan(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking (convention). The receive span sets messaging.destination.name but not messaging.source.name. v1.17.0 has this the other way round for consumer spans.

In the v1.17.0 Consumer attributes table, messaging.source.name is Conditionally Required ("If the value applies to all messages in the batch" - it does here, every returned event comes from this Event Hub), while messaging.destination.name on a consumer is only Recommended: If known on consumer.

The spec's own "Batch receiving" example is precisely this operation's shape, and shows the split clearly for Span Recv1:

Attribute Span Recv1
messaging.source.name "Q"
messaging.destination.name (blank)
messaging.operation "receive"
messaging.batch.message_count 2

So the required attribute is the missing one. Keeping messaging.destination.name as well is permitted, so the minimal fix is to also emit messaging.source.name = the Event Hub name on the receive path. That likely means giving StartSpan a destination/source role, or adding a small receive-specific helper, since the two paths now diverge.

Separately, and clearly beyond what 1.17.0 requires so treat it as optional: the receive span carries no partition ID, which is the single most useful dimension when debugging Event Hubs. v1.17.0 defines no partition attribute for this system (only messaging.kafka.* equivalents), so there is no conformant name to use yet - but PartitionClient knows it, and it may deserve a documented package-specific attribute or a note for when the conventions catch up.

auto tracingContext = _detail::StartSpan(
m_tracingFactory,
"PartitionClient.ReceiveEvents",
Azure::Core::Tracing::_internal::SpanKind::Client,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Worth flagging (question, not a defect). What drove SpanKind::Client for the receive span rather than Consumer?

Asking because the v1.17.0 examples consistently use CONSUMER for the receive span. In "Batch receiving", Span Recv1 - a pull-style batch receive with no parent, which is exactly ReceiveEvents - is listed as SpanKind: CONSUMER, and the same holds in "Batch processing" for Span Recv1/Recv2.

Client is a defensible reading for a synchronous pull receive and other Azure SDKs have made that call, so I am not asserting it is wrong. But the README states the attributes follow 1.17.0, and this is the one place the span shape visibly departs from the spec's worked examples. Either aligning to Consumer or recording the rationale would stop this being re-litigated in future reviews.


- [[#7250]](https://github.com/Azure/azure-sdk-for-cpp/issues/7250) Restored connection-string authentication for `ProducerClient` and `ConsumerClient`, including support for the Event Hubs emulator.
- [[#7295]](https://github.com/Azure/azure-sdk-for-cpp/issues/7295) Connection-string authentication now works on the Rust AMQP backend. `ProducerClient` and `ConsumerClient` no longer throw when the caller passes a connection string.
- [[#7336]](https://github.com/Azure/azure-sdk-for-cpp/issues/7336) Added distributed tracing. `ProducerClientOptions` and `ConsumerClientOptions` gained a `TracingProvider` field. `ProducerClient::Send` and `PartitionClient::ReceiveEvents` create one span per call when a tracing provider is set.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Worth flagging (scope/tracking, not code). Two related concerns about this entry.

1. Closes #7336 retires the only issue tracking propagation. #7336 is not scoped to spans alone - it has a dedicated "Cross-message propagation and batch links" section requiring message context injection/extraction, span links for batches, and a cross-language convention for W3C trace context in Event Hubs application properties. It says these "should be designed in Azure Core, or tracked as a prerequisite/follow-up."

I agree that work does not belong in this PR, and I verified it is genuinely blocked on azure-core. But merging with Closes #7336 auto-closes it, and neither companion issue covers the gap - #7340 is the no-tracer allocations, #7341 is the offline AMQP test seam. That leaves producer-to-consumer correlation untracked entirely. Suggest either dropping to Towards #7336 or filing a dedicated propagation issue and referencing it here.

2. Document the limitation where users will hit it. "Added distributed tracing" is accurate about the spans, but a user reading it will reasonably expect producer and consumer spans to correlate, and they will not - there is no traceparent/Diagnostic-Id on the message. A one-line note in the README tracing section (something like "trace context is not yet propagated in messages, so producer and consumer spans are not linked") would prevent bug reports that are really this known gap.

/**@brief The tracer provider used to create distributed tracing spans. When this field is
* empty, the client creates no spans.
*/
std::shared_ptr<Azure::Core::Tracing::TracerProvider> TracingProvider;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Worth flagging (public API shape). Worth an explicit decision from API review before this locks in, because #7336 asked for something different.

#7336 says: "Expose Azure Core telemetry/tracing configuration through ProducerClientOptions and ConsumerClientOptions, following the established client-options pattern", and notes the work "should receive API and architecture review before implementation."

The established pattern is Azure::Core::_internal::ClientOptions, where this lives at Telemetry.TracingProvider (policy.hpp:74). This PR instead adds a bare top-level TracingProvider. Your reasoning is documented and correct on the facts - these option structs do not derive from ClientOptions, so Telemetry does not exist here, and doc/DistributedTracing.md would otherwise be wrong for Event Hubs.

The concern is forward-compatibility rather than correctness. TelemetryOptions is a grouping that has already grown (ApplicationId, TracingProvider), and Event Hubs separately carries its own ApplicationID at top level. If a second tracing knob is ever needed, a flat field cannot absorb it without another top-level addition, whereas a nested Telemetry sub-struct could - and would make Event Hubs options read the same as every other Azure SDK client.

Not asking you to change this unilaterally; asking that the architects sign off on the flat field, given #7336 explicitly called for the established pattern and for API review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add distributed tracing instrumentation to the Event Hubs SDK

3 participants