VitisAI EP silently leaves batch slots > 0 unwritten under static batch > 1 (Phoenix/XDNA1, XINT8)

[jdominick05]

Environment

• Ryzen AI Software 1.7.1 (Windows), VitisAI ONNX Runtime EP
• Target: Phoenix (XDNA1), X1 backend, XINT8 quantization (via Quark, enable_npu_cnn), static batch export (no dynamic_axes), opset 17

Problem

Exporting and quantizing a standard CNN (ResNet50, ImageNet) at a genuinely static batch size of 2 instead of 1 does not fail or throw when run through the VitisAI EP. Instead it silently produces a fragment of correct output and a fragment of stale, unwritten output, with nothing in the logs, the returned tensors, or the operator assignment report indicating which is which:

• The EP accepts only a fragment of the graph at batch 2: 80 of 395 nodes go to the NPU (vs. 393/395 at batch 1); Conv itself falls entirely to CPU.
• The session builds and runs to completion, reporting a plausible-looking latency (71.6 ms/image — 12.6x the batch-1 per-image latency, not an obviously broken number in isolation).
• Batch slot 0's output is correct: 72.00% top-1, matching the batch-1 accuracy within noise.
• Every subsequent batch slot (slot 1 of 2 tested) is wrong in a specific way — not merely inaccurate, but numerically identical across every input image tested, regardless of what's actually in that slot. This is the signature of an unwritten or stale output buffer, not a miscomputation.
• A single pooled accuracy number across both slots (36.0%) looks like ordinary quantization degradation on its own and gives no indication that a slot is unwritten — this only becomes visible when accuracy is checked per batch-index rather than as an aggregate.
• Running the identical exported batch-2 model on CPUExecutionProvider (same weights) produces correct results in both slots (76.0% top-1), confirming the model/export itself is fine and the failure is specific to the VitisAI EP's batch>1 handling on this backend.

Reproduction

1. Export a standard CNN (e.g. torchvision/timm ResNet50) to ONNX with a genuinely static batch dimension of 2 (not dynamic_axes), opset 17.
2. Quantize XINT8 via Quark with enable_npu_cnn enabled.
3. Build a VitisAI EP session targeting PHX/HPT (X1 backend / XINT8) and run inference with a real batch of 2 different images.
4. Compare per-slot outputs against a CPUExecutionProvider run on the same weights and the same batch, split by batch index rather than pooled into a single accuracy number.

Impact

This is a silent correctness bug, not a silent fallback. It's a step worse than the already-known silent-CPU-fallback behavior on unsupported quantization configs (A8W8 falls back to CPU, which is at least numerically correct) — here the EP claims to run on-device, returns a tensor of the expected shape, and reports a plausible latency, while quietly leaving part of that tensor stale. A pooled accuracy metric can look like ordinary degradation rather than a hard failure, which is exactly the shape of bug that ships unnoticed.

Ask

Is static batch > 1 expected to be supported at all on the X1 (PHX/HPT INT8) backend? If not, the EP should refuse to build the session or throw at inference time rather than silently returning an unwritten buffer for batch slots beyond the first — an explicit error is a much safer failure mode than a plausible-looking wrong answer.

[jdominick05]

Related: #400 (same underlying shape of bug — the EP reports success while quietly doing something else: CPU in an NPU costume there, an unwritten output buffer here).

Adding what I should have included the first time: exact versions, and a smaller reproduction that doesn't require ImageNet or a labeled eval.

Versions

• Ryzen AI Software 1.7.1 (Windows)
• onnxruntime-vitisai 1.23.3.dev20260320
• voe 1.7.1
• amd-quark 0.11rc1 (quantized with get_default_config("XINT8"), enable_npu_cnn)
• NPU driver: 32.0.20101.3760 ("NPU Compute Accelerator Device" in Device Manager)
• xclbin: voe-4.0-win_amd64\xclbins\phoenix\4x4.xclbin, target: "X1"
• Chip: Phoenix (Ryzen 7 8700G)

Smaller reproduction, no labels/ImageNet needed

The original report used a real ImageNet eval to get a percentage, which is more setup than the bug needs. This version only checks numerical identity: fill every slot of two batches with independent random noise, run both, and check per slot whether the output actually changed. Two independent draws of continuous random floats landing on the same output is a probability-zero coincidence, so "identical" only means one thing here.

import numpy as np
import onnxruntime as ort

model = "resnet50_b2_xint8_c64.onnx"  # any static-batch-N XINT8 CNN
opts = {
    "cacheDir": ".", "cacheKey": "probecache",
    "enable_cache_file_io_in_mem": "0", "target": "X1",
    "xlnx_enable_py3_round": "0",
    "xclbin": r"C:\Program Files\RyzenAI\1.7.1\voe-4.0-win_amd64\xclbins\phoenix\4x4.xclbin",
}
sess = ort.InferenceSession(model, providers=["VitisAIExecutionProvider"], provider_options=[opts])
name = sess.get_inputs()[0].name
n, chw = sess.get_inputs()[0].shape[0], sess.get_inputs()[0].shape[1:]  # N read off the graph

rng = np.random.default_rng(0)
batch_a = rng.standard_normal((n, *chw), dtype=np.float32)
batch_b = rng.standard_normal((n, *chw), dtype=np.float32)

out_a, out_b = sess.run(None, {name: batch_a})[0], sess.run(None, {name: batch_b})[0]
for i in range(n):
    same = np.array_equal(out_a[i], out_b[i])
    print(f"slot {i}: identical despite different input = {same}"
          f"  (max abs diff {np.abs(out_a[i]-out_b[i]).max():.6f})")

Ran this just now (2026-09-06) against the same weights as the original report:

=== NPU EP, batch 2 ===
slot 0: identical despite different input = False  (max abs diff 2.125000)
slot 1: identical despite different input = True   (max abs diff 0.000000)  <-- STALE/UNWRITTEN
stale slots: [1] of 2

=== CPU EP, batch 2 (control, same weights) ===
slot 0: identical despite different input = False  (max abs diff 1.625000)
slot 1: identical despite different input = False  (max abs diff 2.500000)
no stale slots detected out of 2

Full script and log (with the exact model-build provider options this repo uses): https://gitlab.com/jdominick05/ryzen-ai-xdna1-quantization/-/blob/main/tools/probe_batch_slot_write.py and https://gitlab.com/jdominick05/ryzen-ai-xdna1-quantization/-/blob/main/results/batch/slot_probe_b2.log

Scope, addressing the obvious next question directly: only static batch 2 was tested — I have not exported or run a batch-4 model. The script above isn't hardcoded to 2, though: it reads the batch size off the graph and reports a per-slot table for whatever N you give it, so it's ready to answer "slot 0 only, or slots 0..k?" the moment a batch-4 XINT8 export exists. I just don't have one built yet (the export/calibration toolchain isn't set up on the machine I ran this from). If a batch-4 (or batch-8) XINT8 model already exists on your end, pointing this script at it is a 30-second answer; otherwise let me know if it's worth me building one and I will.