[Sunghoon Choi (hanbitmyths)]

Summary

• Pair CUDA capture ownership by graph ID; safely discard failed captures and release executions with their staged host buffers.
• Retain caller-owned buffers for the provider lifetime; reuse session-owned contexts across serialized callers and select the correct device.
• Serialize graph-enabled sessions. Reject cudaStreamPerThread only with graphs enabled, in bundled and V2/plugin factories; preserve offload behavior.

Scope: lifecycle/capture only; no kernel changes or external record-only Session::Run. Kernel companion: ORT #33952.

Reported validation: 17 actual CUDA capture/provider-option cases, no skips; V2 factory/test units compiled; scoped format/lint passed. Standalone plugin runtime execution is not claimed.

Performance: Qwen3-VL-2B

Combined package/runtime results, not isolated gains from this PR. Production uses decoder INT4/group32; optimized uses 66 INT8 + 131 INT4 projections, FP16 vision/embeddings/KV, pruned prefill and decoder capture.

A100-SXM4-80GB

Historical production: Oct 28 (b679913a2310); optimized: Nov 5 frozen clean gate.

Tokens	Production TTFT ms	Optimized TTFT ms	Production decode tok/s	Optimized decode tok/s	Change
128	21.25	10.73	232.16	349.41	+50.50%
512	29.52	17.66	230.47	340.51	+47.75%
2048	62.12	47.78	219.57	325.15	+48.08%
4096	122.50	100.42	212.16	304.94	+43.73%
7680	241.71	206.10	202.21	288.68	+42.76%

H200

Baseline: official qwen3-vl-2b-instruct-cuda-gpu:2 on published ORT 1.31.1 / GenAI 0.19.0, CUDA 13. Optimized: retained Nov 8 FPA1 results, not rerun.

Workload	Baseline TTFT ms	Optimized TTFT ms	Baseline decode tok/s	Optimized decode tok/s	Change (unpaired)
Text 128	9.06	8.04	220.0	514.9	+134.08%
Text 512	12.29	11.08	216.0	516.7	+139.25%
Text 2048	26.07	25.98	211.7	504.8	+138.42%
Text 4096	52.39	51.55	205.3	481.0	+134.26%
Text 7680	104.96	103.56	201.6	467.3	+131.85%
Image + text	30.55	40.90	216.9	514.4	+137.16%

Controls: batch 1, context 8192, FP16 KV, chunk 2048, 128 outputs, 8 inference threads, no prefix reuse. TTFT is HTTP first-token latency; decode uses 127 first-to-last intervals. A100: 7 production / 14 optimized samples. H200 baseline: 14 samples/row after warmups, GPU 1 / CPU cores 8-15; pre-rendered prompts, tokenization/image processing inside HTTP.

Limits: H200 is unpaired: prompts/harness differ; the new 448x448 image has 244 input tokens versus 356 historically. Ratios are descriptive, not causal speedups. A100 is not matched to the new H200 runtime; no cross-GPU hardware claim. Optimized ORT 5481f20b / GenAI 4e51307d + 41c96365 are frozen, not current PR-head benchmarks.

Review required: historical optimized export loses 9 AI2D answers versus the accepted native recipe; padded/batched capture limitations remain. No default promotion.

[Copilot]

🟡 Changes recommended

Critical defects leave failed CUDA captures marked active and allow reused contexts to run on the wrong GPU, blocking capture recovery and risking invalid-resource failures.

2 open issues

• Select the provider device when reusing the shared context · New
• Clear the capture annotation after discarding a failed capture · New

What changed in this PR

This PR strengthens CUDA graph ownership and lifetime management in ONNX Runtime and adds an opt-in single-warp INT4 GEMV path for selected decode shapes, preserving default dispatch.

Changes:

• Track capture ownership by graph ID, release owned graphs and staged buffers, and share a CUDA context across serialized callers.
• Add a shape-restricted FP16/group32 INT4 GEMV specialization with explicit environment-control precedence.
• Add ownership, cleanup, dispatch, and numerical regressions, with fault injection confined to the test module.

File	Description
onnxruntime/​test/​providers/​cuda/​test_cases/​matmul_nbits_gemv_test.cc	Tests eligibility, launch geometry, and control precedence.
onnxruntime/​test/​providers/​cuda/​test_cases/​cuda_graph_test.cc	Tests capture ownership, release, external-buffer retention, and cleanup failures.
onnxruntime/​test/​contrib_ops/​matmul_4bits_test.cc	Adds numerical cases for specialization controls and shapes.
onnxruntime/​core/​providers/​cuda/​cuda_kernel.h	Updates comments describing captured-buffer lifetimes.
onnxruntime/​core/​providers/​cuda/​cuda_graph.h	Declares graph release and capture-owner tracking.
onnxruntime/​core/​providers/​cuda/​cuda_graph.cc	Implements capture pairing, failure cleanup, and synchronized release.
onnxruntime/​core/​providers/​cuda/​cuda_execution_provider.h	Adds ownership metadata and graph-enabled serialization policy.
onnxruntime/​core/​providers/​cuda/​cuda_execution_provider.cc	Shares graph contexts and releases buffers by owning graph ID.
onnxruntime/​contrib_ops/​cuda/​quantization/​matmul_nbits.cuh	Defines specialization eligibility and launcher declaration.
onnxruntime/​contrib_ops/​cuda/​quantization/​matmul_4bits.cu	Adds opt-in dispatch and environment-control precedence.
onnxruntime/​contrib_ops/​cuda/​quantization/​matmul_4bits_m1_impl.cuh	Parameterizes column tiling and reduction unrolling while preserving defaults.
onnxruntime/​contrib_ops/​cuda/​quantization/​matmul_4bits_half.cu	Implements single-warp FP16 kernel launches.
docs/​contrib_ops/​cuda/​matmul_nbits.md	Documents supported shapes, opt-in controls, and the legacy alias.
cmake/​onnxruntime_unittests.cmake	Adds test-only CUDA fault-injection linker wrappers on supported Linux builds.

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The tests omit host-buffer lifetime checks for ORT-owned captures, leaving per-ID reclamation and continued replay of the other graph unverified.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Test per-ID host-buffer reclamation for ORT-owned graphs

onnxruntime/​core/​providers/​cuda/​cuda_execution_provider.cc:790

The new tests cover graph release and external-buffer retention, but none retains host buffers during an ORT-owned capture, leaving per-ID reclamation untested. Add a provider-level regression with two owned graph IDs and weak-pointer lifetime checks: both buffers should survive capture/replay, and releasing one ID should free only its buffer while the other graph remains replayable. Releasing the second ID should then free its buffer.

2 resolved since last review

• Clear the capture annotation after discarding a failed capture
• Select the provider device when reusing the shared context

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

The regressions do not cover retained-buffer cleanup across two ORT-owned captures, leaving ownership tagging and selective reclamation unverified.

1 open issue

• Add regression test for selective owned-buffer cleanup · New

_{Review effort: Balanced}

[Tianlei Wu (tianleiwu)]

It's better to split to two PRs since they are independent.
For GEMV change, please run some benchmark on consumer GPUs to see whether the gain still hold.

[Sunghoon Choi (hanbitmyths)]

Ownership review response: #33932 (comment)

Fixed in 52b23f9, now published in capture-only head 0cbdbe5. OwnedBuffersReleaseOnlyWithTheirGraph captures graph IDs 7 and 8 with separate retained shared payloads, drops caller references, verifies both survive and replay, releases only ID 7 and checks only its weak pointer expires, replays ID 8 successfully, then releases it and checks final expiration. The provider/internal runner executed this case and all 16 capture-only CUDA regressions without skips. Both upstream owning-stream self-copy cases also pass. No new production ownership code was needed for this coverage request.

The exact capture-only ORT tree passed the matched 1,080-request gate with GEMV off in both variants: +0.2194% paired decode, 95% interval [+0.0617%, +0.3773%], and unchanged sampled 5,055 MiB peak VRAM. A final GenAI-boundary follow-up is being checked again using fresh immutable runtimes before the consolidated report.

[Sunghoon Choi (hanbitmyths)]

Follow-up to the split / consumer-GPU request: #33932 (comment)

The GEMV kernel, controls, documentation, and tests are now isolated in draft #33952 (main-based b38a219). The live capture branch 0cbdbe5 contains none of those seven math paths; its title and description now identify only capture/context ownership, retained-buffer lifetime, and recovery. All six affected CUDA units, including FP32/BF16 and batched callers, compiled against the independent branch/main-only headers. The math files are byte-identical to the natively validated combined tree.

Consumer-GPU measurements are still blocked: this host has eight A100-SXM4-80GB GPUs and no consumer device. The independent PR remains draft and explicitly labels the +3.99% ablation as historical A100-only data, not a fresh-head or consumer result. The shared +46.54% Oct 28 production comparison remains historical combined-stack context, not isolated attribution. All six companion descriptions retain the production/evaluation caveats; the new GEMV companion has been linked from the existing stack.

[Copilot]

🟡 Changes recommended

Releasing a captured graph from another thread can synchronize the wrong per-thread CUDA stream, freeing pinned staging memory while an asynchronous replay is still using it.

1 open issue

• Prevent freeing staging buffers before async replay completes · New

1 resolved since last review

• Add regression test for selective owned-buffer cleanup

_{Review effort: Balanced}

[Sunghoon Choi (hanbitmyths)]

Stream-safety review follow-up

Addressed the cross-thread asynchronous replay/release comment in 368aae9e.

• Both the bundled CUDA EP and plugin/V2 factory reject user_compute_stream=cudaStreamPerThread when CUDA graphs are enabled, with an explicit error. Non-graph use of this pseudo-handle and supported explicit streams remain unchanged.
• The asynchronous-release regression now retains an actual cudaHostAlloc buffer. Caller A stays alive after asynchronous replay; serialized caller B releases the graph while its captured host callback is blocked. The test verifies that release waits, the pinned buffer remains alive until completion, and it is reclaimed afterward. Callbacks make no CUDA API calls, and controller handshakes use bounded waits.
• This is setup-time validation; no per-replay event tracking or additional decode synchronization was introduced.

Validation: 17 actual inner CUDA capture/provider-option cases passed without skips, with the separate outer runner also passing. The changed plugin factory and real V2 regression translation units compiled with the plugin adapter configuration. This local build uses the bundled provider: standalone plugin runtime execution is not claimed and remains for CI.

No new throughput or quality measurements are claimed. The existing matched-performance sections retain their frozen source revisions and limitations. Consumer-GPU GEMV measurements remain outstanding in #33952; only A100 hardware is available here.

[Copilot]

🟢 Approval recommended

No active review issues remain for the CUDA graph lifecycle changes.

No open issues 🎉

1 resolved since last review

• Prevent freeing staging buffers before async replay completes

_{Review effort: Balanced}

[Akshay Sonawane (apsonawane)]

AI-assisted source review at 0345f08514a75dc9750fd2269eb3700c94e7118b.

Traced graph-enabled provider/context serialization, capture-owner transitions, successful/failed/discarded capture, error preservation and partial-executable cleanup, per-graph retained-buffer tagging/release, external captures, and per-thread-stream rejection in bundled/plugin factories. The earlier wrong-device, stale-owner, selective-buffer, and cross-thread-stream findings are addressed at this head.

No new actionable findings in this pass. This was source-level review, not independent CUDA execution. The existing 17-case lifecycle/provider-option report was inspected; standalone plugin runtime, Windows fault injection, exception-disabled builds, and multi-device behavior were not independently exercised here. No claim is made that every CUDA cleanup path has been hardware-validated.