feat(eventhubs): add distributed tracing to the clients - #7339
feat(eventhubs): add distributed tracing to the clients#7339Johnathan W (j7nw4r) wants to merge 12 commits into
Conversation
Add eventhubs_tracing_test.cpp with the recording tracer, span, and attribute set doubles for the Event Hubs tracing work in issue 7336. The tests pin the TracingProvider option on ProducerClientOptions and ConsumerClientOptions, the eager tracer creation in both clients, the ProducerClient.Send span shape on a cancelled context and on a non-cancel exception, and the shape of the shared StartSpan helper for PartitionClient.ReceiveEvents. Tests 1 to 7 do not compile until the tracing source lands. The TracingProvider member does not exist yet, and the private header src/private/eventhubs_tracing.hpp does not exist yet. SendWithoutProviderIsUnchanged is a characterization test. It passes today. It guards the no-provider path against a null dereference after the source lands.
ProducerClientOptions and ConsumerClientOptions gain a TracingProvider field. Each client builds a TracingContextFactory from that field at construction time. When the field is empty, the factory has no tracer and the clients create no spans. ProducerClient::Send creates one Producer span named ProducerClient.Send. PartitionClient::ReceiveEvents creates one Client span named PartitionClient.ReceiveEvents. Both spans carry the messaging attributes, the az.namespace attribute, and the message count. Both spans record an exception event and an error status, then rethrow the original exception. A new private helper, src/private/eventhubs_tracing.hpp, holds the factory creation and the span shape, so the producer and the partition client emit the same attributes. EventDataBatch::NumberOfEvents is now const, and its mutex is mutable, because Send takes the batch by const reference. Trace-context propagation and span links stay deferred, because Azure Core has no API for them.
Run clang-format-11 over eventhubs_tracing_test.cpp. The declarations of AddAttributes and SetStatus on the recording span double broke the Validate Clang Format step in CI. The change is formatting only. No assertion, no test name, and no behavior changed.
|
Azure Pipelines: Successfully started running 3 pipeline(s). 7 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
There was a problem hiding this comment.
Pull request overview
Adds provider-neutral distributed tracing to Event Hubs producer and consumer operations.
Changes:
- Adds tracing-provider configuration and shared span helpers.
- Instruments send and receive operations with messaging attributes and error recording.
- Adds tracing tests, build integration, and changelog documentation.
Public API and tracing architecture changes require maintainer review.
Reviewed changes
Copilot reviewed 14 out of 14 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
CHANGELOG.md |
Documents distributed tracing support. |
CMakeLists.txt |
Builds tracing helper sources. |
inc/azure/messaging/eventhubs/consumer_client.hpp |
Adds consumer tracing configuration. |
inc/azure/messaging/eventhubs/event_data_batch.hpp |
Makes event-count access const-safe. |
inc/azure/messaging/eventhubs/partition_client.hpp |
Stores receive tracing context. |
inc/azure/messaging/eventhubs/producer_client.hpp |
Adds producer tracing configuration. |
src/consumer_client.cpp |
Propagates tracing into partition clients. |
src/eventhubs_tracing.cpp |
Implements span creation and attributes. |
src/partition_client.cpp |
Instruments receive operations. |
src/private/eventhubs_tracing.hpp |
Declares shared tracing helpers. |
src/private/eventhubs_utilities.hpp |
Extends partition-client factory inputs. |
src/producer_client.cpp |
Instruments send operations. |
test/ut/CMakeLists.txt |
Registers tracing tests. |
test/ut/eventhubs_tracing_test.cpp |
Tests tracing options and span behavior. |
💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.
ProducerClient::Send converted the batch to an AMQP message before it started the span. An empty batch makes that conversion throw, so the send failed before a span existed and nothing recorded the failure. The conversion now runs at the top of the try block, after the span starts. The existing catch clause adds the exception to the span and rethrows, the same as every other send failure.
The OpenTelemetry semantic conventions at schema 1.17.0 define messaging.batch.message_count as an int. The Event Hubs tracing helper records it as a string, and the test doubles could not show the difference, because they stringified every overload into one map. TestAttributeSet and TestSpan now record a type tag for each attribute. The tag names the overload that delivered the value. Three tests use it: the receive span count and the send span count must arrive through the uint64 overload, and SetMessageCount must stay quiet and safe when the factory has no tracer. The first two tests are red today. The third one is a characterization test that guards the null attribute set hazard.
The batch message count now goes out as an unsigned integer attribute, which matches the OpenTelemetry semantic conventions at schema 1.17.0. Before this change the helper wrote the count as a string. SetMessageCount takes the tracing context factory, because the typed attribute needs an attribute set. The helper guards the null attribute set on the no-tracer path, because the factory returns nullptr when it has no tracer and that pointer is not null safe.
The span helper built a std::string temporary for each attribute name before it added the attribute. Two of the names are longer than the libc++ small-string limit, so each call heap allocated even when the client had no tracer and dropped every attribute. The span helper now returns the tracing context immediately when the factory has no tracer. The message count helper keeps its own null attribute set guard, because PartitionClient::ReceiveEvents calls it directly. The remaining two allocations per operation come from azure-core and are tracked in issue 7340.
The SingleSpan helper called front() on the tracer list after a non-fatal EXPECT_EQ on the list size. When no tracer was created, that call dereferenced an empty std::list and the test process ended in a segmentation fault. The crash stopped the run, so the tests after the first affected one never reported at all. The helper now returns null when the tracer list is not the expected size, so a failure reports through the caller's ASSERT_NE assertion rather than a crash. This mirrors the guard that already protects the spans list two lines later.
The receive span of PartitionClient::ReceiveEvents has no offline coverage. A partition client comes only from a factory that opens a live AMQP link, so no unit test can reach the span, the message count, or the exception path. This test runs against a real namespace. It gives the consumer client a recording tracing provider, receives events from a partition, and makes sure the span name, the span kind, the messaging attributes, and the message count are correct. The count assertion compares the span attribute with the size of the vector that ReceiveEvents returned. The recording tracing test doubles move from eventhubs_tracing_test.cpp into a shared header, because the live test lives in a different file.
The two convenience Send overloads build the batch before they send it, and CreateBatch opens the connection, the session and the AMQP link. A bad host fails there. Only the batch overload starts a span today, so that failure unwinds with no span at all. Add two offline tests that send one event and a vector of three events against a fake connection string. Each test asserts one span named ProducerClient.Send with the kind Producer, the message count as a uint64 attribute, one recorded event and the Error status. Both fail today because the provider holds zero spans. Add a characterization test that keeps the no-provider path unchanged, and a live test that sends one event and asserts a single span. The live test catches a nested second span, which an offline test cannot reach because CreateBatch always throws offline.
The convenience Send overloads made the batch first, and the batch creation opens the connection, the session, and the AMQP link. A bad host or a failed authentication threw there, and the stack unwound before a span existed. A caller then saw no producer span for the failed send. Each public Send overload now starts one span, and one try/catch records the exception on that span. A new private method, SendBatchInSpan, holds the send body and starts no span, so one logical send still makes one span. The convenience overloads pass the span context to CreateBatch, which puts the link open under the span.
Johnathan W (j7nw4r)
left a comment
There was a problem hiding this comment.
Every review finding is fixed, and each fix is in its own commit. The empty-batch send now records a span, the batch message count goes out as an unsigned integer, the no-tracer path skips the attribute work, the test helper no longer crashes on an empty tracer list, and every ProducerClient::Send overload now creates exactly one span that covers the batch creation. The playback suite went from 94 tests to 111, all passing.
Two findings need work that does not belong here. Azure Core injects the tracing factory into the context even with no tracer, which costs 2 heap allocations for each operation, and the short-circuit cannot be written from a service package because ServiceSpan's default constructor is private. That is issue #7340. The send and receive paths also have no offline test seam, because MessageSender and MessageReceiver are both final types that only a live AMQP link produces, so azure-core-amqp has to publish its existing mock server before those spans can be checked in CI. That is issue #7341.
Two gaps stay open in this pull request, and I would rather state them than leave them implied. The receive span is covered only by a live-only test that CI skips. The rule that one logical Send makes one span is also live-only, because CreateBatch throws offline before the send logic runs, so no test that CI runs can tell a correct implementation from one that nests a second span. Both close when #7341 lands.
The measurements behind these claims are real rather than reasoned. I counted the allocations with a probe against the compiled library, reproduced the test crash and then removed it, and deleted the receive span block to confirm the suite never noticed.
|
Azure Pipelines: Successfully started running 3 pipeline(s). 7 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
The tracing change added a TracingProvider field to ProducerClientOptions and ConsumerClientOptions, but no document told a user how to set it. The only repo document on tracing, doc/DistributedTracing.md, showed clientOptions.Telemetry.TracingProvider. That line does not compile for Event Hubs, because the two option structs do not derive from Azure::Core::_internal::ClientOptions and have no Telemetry field. - Adds a "Distributed tracing" section to the Event Hubs README with a snippet that builds an OpenTelemetryProvider and sets TracingProvider on both client options. - Lists the span names, the span kinds, and the attributes that the clients emit, and names the 1.17.0 semantic conventions schema. - States that the client creates no spans when TracingProvider is not set, because there is no global fallback provider. - Adds a note to doc/DistributedTracing.md for the clients that declare TracingProvider at the top level of their own options structure.
Sagar Patel (sagar0207)
left a comment
There was a problem hiding this comment.
Reviewed the tracing design end to end, and verified the semantic-convention claims against the spec at tag v1.17.0 rather than the summary tables, plus the azure-core headers in this repo.
The design is sound. _detail::StartSpan correctly guards HasTracer() and the null return from CreateAttributeSet() (tracing.cpp:118-126) - that null is real and would crash. Error status is handled: ServiceSpan::AddEvent(std::exception const&) sets SpanStatus::Error internally (service_tracing.hpp:142-149), so the bare AddEvent(ex) in the catch blocks is sufficient. NumberOfEvents() const + mutable mutex is correct and ABI-safe (inline, no exported symbol). SendBatchInSpan genuinely yields one span per logical Send.
I also confirmed two of your defensive claims and would let them stand:
- Propagation genuinely requires azure-core changes.
Span(tracing_impl.hpp:170-236) exposes onlyPropagateToHttpHeaderswith no trace-id/span-id accessor, andCreateSpanOptions(line 242) has onlyParentSpan, noLinks. Not a dodge. ServiceSpan's private default ctor does block a service-side short-circuit (service_tracing.hpp:34-35).
Two blocking items and three worth discussing, inline. All five are convention/scope issues, not correctness bugs.
One follow-up not worth an inline comment: neither client creates spans for GetEventHubProperties / GetPartitionProperties, and CreateBatch / CreatePartitionClient also open links without spans. #7336 scopes this PR to send/receive so that is defensible, but it is a real completeness gap for a later PR.
Things I checked and am explicitly not raising, so you do not have to re-defend them: missing SetStatus(Ok) on success (correct - OTel wants Unset), -Wreorder (declaration order matches init order in both clients), the raw this stored in the Context by CreateTracingContext (the derived context never escapes the synchronous call, and Processor holds the client behind a unique_ptr so the factory address is stable), and the static_cast in the test doubles (the provider is the only AttributeSet source).
| "publish", | ||
| m_eventHub, | ||
| m_fullyQualifiedNamespace, | ||
| size_t{1}, |
There was a problem hiding this comment.
Blocking (convention). This is the single-message overload, so per the v1.17.0 convention this PR targets, messaging.batch.message_count should not be set here.
From the v1.17.0 messaging attributes table, note [2] on messaging.batch.message_count:
Instrumentations SHOULD NOT set
messaging.batch.message_counton spans that operate with a single message. When a messaging client library supports both batch and single-message API for the same operation, instrumentations SHOULD usemessaging.batch.message_countfor batching APIs and SHOULD NOT use it for single-message APIs.
ProducerClient is exactly the library shape that note describes: Send(EventData const&) is the single-message API, while Send(EventDataBatch const&) and Send(std::vector<EventData> const&) are the batching APIs. That this overload batches internally is an implementation detail; the public API is single-message.
Suggest passing Azure::Nullable<size_t>{} here. The other two overloads are correct as-is and should keep the count even when it happens to be 1.
Two tests currently pin the non-conforming value and would need to flip to asserting absence:
test/ut/eventhubs_tracing_test.cpp:219(SendEventSpanRecordsCreateBatchFailure)test/ut/producer_client_test.cpp:137(SendEventSpan_LIVEONLY_)
| Azure::Core::Amqp::Models::_internal::AmqpError> | ||
| result; | ||
| // The message count is known only when the loop ends, so the span gets it later. | ||
| auto tracingContext = _detail::StartSpan( |
There was a problem hiding this comment.
Blocking (convention). The receive span sets messaging.destination.name but not messaging.source.name. v1.17.0 has this the other way round for consumer spans.
In the v1.17.0 Consumer attributes table, messaging.source.name is Conditionally Required ("If the value applies to all messages in the batch" - it does here, every returned event comes from this Event Hub), while messaging.destination.name on a consumer is only Recommended: If known on consumer.
The spec's own "Batch receiving" example is precisely this operation's shape, and shows the split clearly for Span Recv1:
| Attribute | Span Recv1 |
|---|---|
messaging.source.name |
"Q" |
messaging.destination.name |
(blank) |
messaging.operation |
"receive" |
messaging.batch.message_count |
2 |
So the required attribute is the missing one. Keeping messaging.destination.name as well is permitted, so the minimal fix is to also emit messaging.source.name = the Event Hub name on the receive path. That likely means giving StartSpan a destination/source role, or adding a small receive-specific helper, since the two paths now diverge.
Separately, and clearly beyond what 1.17.0 requires so treat it as optional: the receive span carries no partition ID, which is the single most useful dimension when debugging Event Hubs. v1.17.0 defines no partition attribute for this system (only messaging.kafka.* equivalents), so there is no conformant name to use yet - but PartitionClient knows it, and it may deserve a documented package-specific attribute or a note for when the conventions catch up.
| auto tracingContext = _detail::StartSpan( | ||
| m_tracingFactory, | ||
| "PartitionClient.ReceiveEvents", | ||
| Azure::Core::Tracing::_internal::SpanKind::Client, |
There was a problem hiding this comment.
Worth flagging (question, not a defect). What drove SpanKind::Client for the receive span rather than Consumer?
Asking because the v1.17.0 examples consistently use CONSUMER for the receive span. In "Batch receiving", Span Recv1 - a pull-style batch receive with no parent, which is exactly ReceiveEvents - is listed as SpanKind: CONSUMER, and the same holds in "Batch processing" for Span Recv1/Recv2.
Client is a defensible reading for a synchronous pull receive and other Azure SDKs have made that call, so I am not asserting it is wrong. But the README states the attributes follow 1.17.0, and this is the one place the span shape visibly departs from the spec's worked examples. Either aligning to Consumer or recording the rationale would stop this being re-litigated in future reviews.
|
|
||
| - [[#7250]](https://github.com/Azure/azure-sdk-for-cpp/issues/7250) Restored connection-string authentication for `ProducerClient` and `ConsumerClient`, including support for the Event Hubs emulator. | ||
| - [[#7295]](https://github.com/Azure/azure-sdk-for-cpp/issues/7295) Connection-string authentication now works on the Rust AMQP backend. `ProducerClient` and `ConsumerClient` no longer throw when the caller passes a connection string. | ||
| - [[#7336]](https://github.com/Azure/azure-sdk-for-cpp/issues/7336) Added distributed tracing. `ProducerClientOptions` and `ConsumerClientOptions` gained a `TracingProvider` field. `ProducerClient::Send` and `PartitionClient::ReceiveEvents` create one span per call when a tracing provider is set. |
There was a problem hiding this comment.
Worth flagging (scope/tracking, not code). Two related concerns about this entry.
1. Closes #7336 retires the only issue tracking propagation. #7336 is not scoped to spans alone - it has a dedicated "Cross-message propagation and batch links" section requiring message context injection/extraction, span links for batches, and a cross-language convention for W3C trace context in Event Hubs application properties. It says these "should be designed in Azure Core, or tracked as a prerequisite/follow-up."
I agree that work does not belong in this PR, and I verified it is genuinely blocked on azure-core. But merging with Closes #7336 auto-closes it, and neither companion issue covers the gap - #7340 is the no-tracer allocations, #7341 is the offline AMQP test seam. That leaves producer-to-consumer correlation untracked entirely. Suggest either dropping to Towards #7336 or filing a dedicated propagation issue and referencing it here.
2. Document the limitation where users will hit it. "Added distributed tracing" is accurate about the spans, but a user reading it will reasonably expect producer and consumer spans to correlate, and they will not - there is no traceparent/Diagnostic-Id on the message. A one-line note in the README tracing section (something like "trace context is not yet propagated in messages, so producer and consumer spans are not linked") would prevent bug reports that are really this known gap.
| /**@brief The tracer provider used to create distributed tracing spans. When this field is | ||
| * empty, the client creates no spans. | ||
| */ | ||
| std::shared_ptr<Azure::Core::Tracing::TracerProvider> TracingProvider; |
There was a problem hiding this comment.
Worth flagging (public API shape). Worth an explicit decision from API review before this locks in, because #7336 asked for something different.
#7336 says: "Expose Azure Core telemetry/tracing configuration through ProducerClientOptions and ConsumerClientOptions, following the established client-options pattern", and notes the work "should receive API and architecture review before implementation."
The established pattern is Azure::Core::_internal::ClientOptions, where this lives at Telemetry.TracingProvider (policy.hpp:74). This PR instead adds a bare top-level TracingProvider. Your reasoning is documented and correct on the facts - these option structs do not derive from ClientOptions, so Telemetry does not exist here, and doc/DistributedTracing.md would otherwise be wrong for Event Hubs.
The concern is forward-compatibility rather than correctness. TelemetryOptions is a grouping that has already grown (ApplicationId, TracingProvider), and Event Hubs separately carries its own ApplicationID at top level. If a second tracing knob is ever needed, a flat field cannot absorb it without another top-level addition, whereas a nested Telemetry sub-struct could - and would make Event Hubs options read the same as every other Azure SDK client.
Not asking you to change this unilaterally; asking that the architects sign off on the flat field, given #7336 explicitly called for the established pattern and for API review.
Summary
The Event Hubs C++ SDK emits no distributed tracing spans. Producer and consumer operations stay invisible to OpenTelemetry-based observability pipelines. This change adds provider-neutral spans on send and receive, built only on the azure-core tracing abstraction.
Motivation
Distributed tracing is a GA requirement in the Azure SDK guidelines. Without spans on send and receive, users cannot correlate Event Hubs traffic with the rest of a traced application. The azure-core tracing abstraction already provides a provider-neutral API, so the instrumentation adds no new dependency.
Changes
std::shared_ptr<Azure::Core::Tracing::TracerProvider> TracingProviderfield toProducerClientOptionsandConsumerClientOptions.ProducerClient::Sendoverload creates exactly one spanProducerClient.SendwithSpanKind::Producer. The span covers the batch creation, which opens the AMQP link, and the whole retry loop, so a connection failure is recorded and retries add no extra top-level span.PartitionClient::ReceiveEventscreates one spanPartitionClient.ReceiveEventswithSpanKind::Client; the receive count is the size of the returned vector, not the internal prefetch count.az.namespace,messaging.system,messaging.destination.name,messaging.operation,messaging.batch.message_count,net.peer.name.server.addressis not used because it post-dates schema 1.17.0.azure-messaging-eventhubs-cppwith the package version.TracingProviderset, no span is created and no tracing data is recorded. The span helper returns before it adds any attribute, so the path costs 2 heap allocations for each operation. Both come from the child context that azure-core derives, and issue TracingContextFactory allocates when no tracer is configured #7340 tracks removing them.src/private/eventhubs_tracing.hppandsrc/eventhubs_tracing.cpp.EventDataBatch::NumberOfEvents()becomesconstand its mutex becomesmutable, becauseSendtakesEventDataBatch const&and must read the batch count. This is source-compatible.messaging.batch.message_countgoes out as an unsigned integer attribute, which matches the type the 1.17.0 convention defines.test/ut/eventhubs_tracing_test.cppwith local test doubles; they need no OpenTelemetry and no live service.test/ut/consumer_client_test.cpp, because aPartitionClientneeds a live AMQP link.vcpkg.jsonis untouched.Deferred to follow-up work: cross-message trace-context propagation, remote-parent extraction, and span links for batches are not in this pull request. Azure Core has no API for them today; the only propagation API is
Span::PropagateToHttpHeaders, andCreateSpanOptionshas noLinksmember. Issue #7336 asks for these to be designed in Azure Core or tracked as a follow-up rather than bypassed with Event Hubs-specific OpenTelemetry calls.Cross-language note: the .NET SDK emits a send span for Event Hubs and sets
az.namespacetoMicrosoft.EventHub, which this change matches. The .NET SDK creates no receive span on the pull path, so thePartitionClient.ReceiveEventsspan follows the OpenTelemetry messaging conventions rather than a .NET precedent. The Goazeventhubspackage ships no tracing today.Known limitations: the
PartitionClient::ReceiveEventsspan is covered only by a live-only test, because aPartitionClientis reachable only through a factory that opens a real AMQP link. CI runs in playback and skips that test, so the receive span has no coverage in CI. An offline seam needs a change in azure-core-amqp, tracked in issue #7341.Test plan
AZURE_TEST_MODE=PLAYBACK: 100% tests passed, 0 tests failed out of 111 (the 94-test baseline plus the 17 new tests). The live-only receive-span test is skipped in playback and is unverified.clang-format-11 --dry-run -Werroris clean over every changed C++ file.Closes #7336