feat(vm): export driver traces over OTLP - #2564
Merged
Merged
Conversation
krishicks
requested review from
a team,
derekwaynecarr,
maxamillion and
mrunalp
as code owners
July 30, 2026 21:13
|
🌿 Preview your docs: https://nvidia-preview-pr-2564.docs.buildwithfern.com/openshell |
krishicks
changed the title
krishicks
changed the title
krishicks
force-pushed
the
hicks/push-nkmtsusunlnk
branch
from
July 30, 2026 21:39
498b495 to
32b368b
Compare
krishicks
marked this pull request as draft
July 30, 2026 22:09
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
krishicks
force-pushed
the
hicks/push-nkmtsusunlnk
branch
from
July 30, 2026 22:57
32b368b to
47cd4eb
Compare
krishicks
force-pushed
the
hicks/push-nkmtsusunlnk
branch
2 times, most recently
from
July 31, 2026 20:05
72dfb6a to
43cce78
Compare
krishicks
marked this pull request as ready for review
July 31, 2026 20:06
krishicks
force-pushed
the
hicks/push-nkmtsusunlnk
branch
4 times, most recently
from
August 3, 2026 15:50
5faf418 to
c3de0c8
Compare
SDAChess
reviewed
Aug 3, 2026
krishicks
force-pushed
the
hicks/push-nkmtsusunlnk
branch
3 times, most recently
from
August 3, 2026 16:46
9b888d6 to
d4ae496
Compare
SDAChess
reviewed
Aug 3, 2026
krishicks
force-pushed
the
hicks/push-nkmtsusunlnk
branch
from
August 3, 2026 16:54
d4ae496 to
2bad42f
Compare
matthewgrossman
left a comment
matthewgrossman
left a comment
Contributor
There was a problem hiding this comment.
Generated with help from Codex.
matthewgrossman
left a comment
matthewgrossman
left a comment
Contributor
There was a problem hiding this comment.
Generated with help from Codex.
krishicks
force-pushed
the
hicks/push-nkmtsusunlnk
branch
from
August 3, 2026 18:09
2bad42f to
98659db
Compare
Continue distributed traces across the gateway-to-driver process boundary and export VM driver spans to the same OTLP/gRPC collector. The driver reports as the distinct openshell-driver-vm service. Updated the gateway architecture and configuration reference with a generic external-driver forwarding contract. Instrumented: - Every RemoteComputeDriver RPC injects the active W3C trace context into tonic metadata. Managed VM readiness and runtime initialization give startup capability probes stable parent operations rather than isolated root spans. - A tonic service layer creates fixed, low-cardinality server spans for every ComputeDriver RPC. New handlers inherit tracing automatically; failures record OpenTelemetry error status and the gRPC status code. - Background provisioning remains attached to CreateSandbox after the RPC returns without extending the RPC span lifetime. - Provisioning records image preparation, bootstrap image resolution, overlay preparation, lifecycle configuration, pre-launch hooks, guest preparation, and launcher spawn as child spans. - VM startup reconciliation roots one trace for the persisted-sandbox scan, with per-sandbox restore and provision operations beneath it. The root remains open until all spawned restore tasks finish. - Delete cleanup records its own child operation. Design notes: - The gateway forwards its configured OTLP endpoint to managed external drivers. SDK `OTEL_*` variables continue to own sampling, batching, limits, headers, and transport tuning. - The VM driver has its own tracer provider and service resource so trace backends preserve the service boundary. - RPC operation names come from an explicit method mapping, keeping cardinality bounded without parsing the protobuf descriptor set at runtime. - Propagation uses a remote SpanContext for spawned provisioning. This keeps one trace while allowing the CreateSandbox server span to finish when the RPC response is sent. - Startup restoration is independent of gateway requests. It begins at the VM driver reconciliation span rather than attaching to an unrelated RPC. - Existing tracing events remain on the logging path. The OpenTelemetry layer exports spans only and excludes the SDK exporter callsites to avoid recursive traces. - Export configuration failures do not prevent the driver from serving, and buffered spans are drained during graceful shutdown. - Trace fields identify drivers, sandboxes, images, lifecycle phases, and gRPC outcomes without recording credentials, sandbox tokens, or request query parameters. Refs #2507 Signed-off-by: Kris Hicks <khicks@nvidia.com>
krishicks
force-pushed
the
hicks/push-nkmtsusunlnk
branch
from
August 3, 2026 18:40
98659db to
d50ac12
Compare
Collaborator
Author
|
I think all the feedback has been addressed. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Continue distributed traces across the gateway-to-driver process boundary and export VM driver spans to the same OTLP/gRPC collector. The driver reports as the distinct openshell-driver-vm service.
Updated the gateway architecture and configuration reference with a generic external-driver forwarding contract.
Related Issue
Refs #2507
Changes
Instrumented:
Design notes:
OTEL_*variables continue to own sampling, batching, limits, headers, and transport tuning.Testing
mise run pre-commitpassesChecklist