fix: avoid OTel "Failed to detach context" error on early run_async close - #6560
Open
GanziMan wants to merge 1 commit into
Open
fix: avoid OTel "Failed to detach context" error on early run_async close#6560GanziMan wants to merge 1 commit into
GanziMan wants to merge 1 commit into
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
…lose The schema-v1 invocation span in record_invocation wrapped runners._run_node_async (an async generator) with start_as_current_span. When a caller stops iterating early, the generator is finalized in a different execution context than where the span was attached, so the automatic context detach raised "Token was created in a different Context" -- OpenTelemetry swallows it but logs it at ERROR on every early-terminated run, creating log noise and false error-rate spikes. Manage the span/context explicitly and detach only on normal completion; skip detach on early close. The span is always ended, so trace data stays complete. Fixes google#6559
GanziMan
force-pushed
the
fix/otel-invocation-span-detach
branch
from
August 3, 2026 07:33
2cfb75c to
04c6708
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #6559.
Problem
record_invocation(schema v1, the default off Agent Engine) wrapsrunners._run_node_async— anasyncgenerator — withstart_as_current_span("invocation"). When a caller stops iteratingRunner.run_async()early (a commonbreak/returnas soon as the finalresponse arrives), the generator is finalized (
GeneratorExit/CancelledError) in a different execution context than the one where thespan was attached. The automatic context
detach()then raisesValueError: <Token ...> was created in a different Context.OpenTelemetry swallows that exception inside
context.detach()but logs it atERROR (
Failed to detach context) on every early-terminated run. Theinvocation still completes correctly, so this is cosmetic — but it creates
significant log noise and false error-rate spikes (e.g. Datadog). Note that
OTEL_SDK_DISABLED=truedoes not suppress it, sinceattach()/detach()livein the context API, not the SDK.
Fix
Manage the span/context explicitly and
detach()only when the generator bodycompleted normally; skip
detach()on early close. The span is alwaysend()ed, so trace data stays complete. Full consumption is unchanged.Tests
Adds
tests/unittests/telemetry/test_instrumentation.py:test_record_invocation_no_detach_error_on_early_close— reproduces theearly-close path (via
asyncio.run→loop.shutdown_asyncgens(), whichfinalizes the still-open generator in a different context) and asserts no
Failed to detach contextERROR is logged, while theinvocationspan isstill recorded and ended.
test_record_invocation_full_consumption_still_records_span— fullconsumption keeps recording the span with no error.
Verified the new early-close test fails without this change (with the exact
ValueError: <Token ...> was created in a different Context) and passeswith it.
Scope
Intentionally minimal: only the default schema-v1
invocationspan that usershit today. The same
start_as_current_span-around-async-generator pattern alsoexists on the schema-v2 path (
node_tracing._use_invoke_workflow_span) and canget the same treatment in a follow-up.