[Discussion / Feedback] ONNX Backend: Benchmarks, Questions, and Issues

This issue is to collect discussions and feedback regarding the ONNX backend. Feel free to post your thoughts.

#1234 has been merged, bringing large speedups on Intel GPUs and NPUs, as well as a generic DirectML backend that may perform better than OpenCL on the latest transformer models.

Prebuilt DirectML binaries are available at https://github.com/lightvector/KataGo/actions/workflows/onnx-backend.yml. Open the latest run and download the files you need from artifacts.

The OpenVINO backend is currently only built when lightvector manually triggers a run or tags a commit. If you want to test it, you can use https://github.com/seniorfish/KataGo/releases/tag/v1.17.2-onnx-openvino, but note that this is not the latest version. For configuration, you can refer to gtp_example.cfg.

Future plans:

There are reports that the NPU backend is not working on the largest b40 net; I will investigate this. If the backend crashes on your device, please let me know.
Optimize the OpenVINO batching strategy and fine-tune config options, which may require more benchmark data across various processors.
onnxbackend.cpp seems a bit messy with many special cases in the code. I might look into refactoring it for better maintainability as more execution providers might be added in the future.
Discuss the possibility of adding more Execution Providers supported by ONNX Runtime. Examples have already come up: MIGraphX (Add MIGraphX backend for AMD GPUs (ROCm) — 2.94x over OpenCL on MI300X #1235) and WebAssembly (Add ONNX Runtime backend support (OpenVINO / DirectML / TensorRT / CPU, Windows & Linux) #1222 (comment)). I think running KataGo in the browser would be very interesting.
Take suggestions posted under this issue.

@seniorfish

@seniorfish

On the NPU, for b11c768h12nbt3, onnxTransformerNHWC=false provides a 3× speedup; for b28c512nbt and b18c384nbt, onnxSkipScale8=true provides a 4× speedup.

On the GPU, for b11c768h12nbt3, onnxTransformerNHWC=true provides a 3× speedup; for b40c768nbt, b28c512nbt, and b18c384nbt, onnxSkipScale8=true provides a 2× speedup. The performance starts to decline with 15 threads or more, and the speed with 20 threads is below 5 v/s.

I used https://github.com/seniorfish/KataGo/releases/tag/v1.17.2-onnx-openvino to run the tests, but it still does not include the configuration file (match, human).

katago-onnx-openvino-c025a30 b11c768h12nbt3 b40c768nbt b28c512nbt b18c384nbt b40c256
NPU_t1_onnxSkipScale8=false_onnxTransformerNHWC=false 34 --- 32 61 276
NPU_t1_onnxSkipScale8=false_onnxTransformerNHWC=true 10 --- 32 59 266
NPU_t1_onnxSkipScale8=true_onnxTransformerNHWC=false 35 --- 141 253 249
NPU_t1_onnxSkipScale8=true_onnxTransformerNHWC=true 10 --- 135 244 267
GPU_onnxSkipScale8=false_onnxTransformerNHWC=false 26 30 90 182 334
GPU_onnxSkipScale8=false_onnxTransformerNHWC=true 105 33 88 189 320
GPU_onnxSkipScale8=true_onnxTransformerNHWC=false 29 56 169 427 330
GPU_onnxSkipScale8=true_onnxTransformerNHWC=true 100 56 165 431 321
I tried adjusting the parameters, but b40c768nbt still does not work properly on the NPU backend. I’m guessing it’s because it’s too large? Has anyone figured this out?
PS C:\Users\foxrain> C:\Users\foxrain\Downloads\katago-onnx-openvino-c025a30\katago.exe benchmark -model C:\Users\foxrain\Downloads\kata1-zhizi-b40c768nbt-s11472M-d5982M.bin.gz -config C:\Users\foxrain\Downloads\katago-onnx-openvino-c025a30\gtp_example.cfg -t 1
2026-08-14 12:57:10+0800: Running with following config:
allowResignation = true
lagBuffer = 1.0
logAllGTPCommunication = true
logDir = gtp_logs
logSearchInfo = true
logSearchInfoForChosenMove = false
logToStderr = false
maxTimePondering = 60.0
maxVisits = 500
numSearchThreads = 1
onnxDeviceToUse = 0
onnxOpenVINOCacheDir = C://temp//katago_ov_cache
onnxOpenVINODeviceType = NPU
onnxOpenVINOPrecision = FP16
onnxProvider = openvino
onnxSkipScale8 = false
onnxTransformerNHWC = false
ponderingEnabled = false
resignConsecTurns = 3
resignThreshold = -0.90
rules = tromp-taylor
searchFactorAfterOnePass = 0.50
searchFactorAfterTwoPass = 0.25
searchFactorWhenWinning = 0.40
searchFactorWhenWinningThreshold = 0.95

2026-08-14 12:57:10+0800: Loading model and initializing benchmark...
2026-08-14 12:57:10+0800: Testing with default positions for board size: 19
2026-08-14 12:57:10+0800: nnRandSeed0 = 9550974731077147000
2026-08-14 12:57:10+0800: After dedups: nnModelFile0 = C:\Users\foxrain\Downloads\kata1-zhizi-b40c768nbt-s11472M-d5982M.bin.gz useFP16 auto
2026-08-14 12:57:10+0800: Initializing neural net buffer to be size 19 * 19 exactly
2026-08-14 12:57:24+0800: ONNX backend: creating compute context for 19x19 with provider 'openvino'
2026-08-14 12:57:24+0800: ONNX backend thread 0: Model version 15
2026-08-14 12:57:24+0800: ONNX backend thread 0: Model name: kata1-zhizi-b40c768nbt-s11472M-d5982M (nbt convnet, 232532105 params)
2026-08-14 12:57:24+0800: ONNX backend thread 0: provider=openvino deviceIdx=NPU
2026-08-14 12:57:24+0800: ONNX backend: building ONNX graph from model weights...
2026-08-14 12:57:24+0800: Building internal onnx model, requireExactNNLen=true transformerNHWC=false
2026-08-14 12:57:25+0800: ONNX backend: ONNX graph ready (930619286 bytes)
2026-08-14 12:57:25+0800: ONNX backend: OpenVINO execution provider enabled for thread 0, device_type=NPU, cache_dir=C://temp//katago_ov_cache, precision=FP16
2026-08-14 12:57:28+0800: ONNX backend: graph input order: [0]InputSpatial [1]InputGlobal [2]InputMask
2026-08-14 12:57:28+0800: ONNX backend: graph output order: [0]OutputPolicyPass [1]OutputPolicy [2]OutputValue [3]OutputScoreValue [4]OutputOwnership
2026-08-14 12:57:28+0800: ONNX backend: session created, inputs=3 outputs=5
2026-08-14 12:57:28+0800: ONNX backend: OpenVINO warmup for thread 0: precompiling batch sizes 1..1 on device_type=NPU (cache_dir=C://temp//katago_ov_cache, compiled graphs are reused by later runs)
PS C:\Users\foxrain>

from #1234 (comment)
@foxrainowo It looks like a problem related to model size, since only the largest network crashed. What processor model are you using? How much memory do you have for npu?

To answer your question: the purpose of onnxSkipScale8 is to work around the workaround(means disables it) that skips FP16 activation scaling. If the intermediate activations of a large convnet get too large, they overflow to inf/NaN under FP16 inference, or their precision degrades sharply near the upper limit (FP16 only has about 3 decimal digits of precision). KataGo uses the mathematically equivalent transformation of "dividing the whole network's activations by 8" to keep activations in a safe range — that's the scale8 workaround. However, after deep profiling of the code's performance, we found that when this option is enabled, it blocks operator fusion and significantly lowers performance on convnets (which you measured). The cost of enabling it is that under extreme conditions (very large boards) it might overflow and produce invalid values, but in that case KataGo should fail loud, and testing shows this problem is hardly hit on a normal 19x19 board. This option defaults to false, meaning slower speed but higher stability; you can enable it in daily use for better performance, and it is forcibly disabled when contributing. This option does not affect transformer models. See 4f86efa and c2e5552 for details.

The purpose of onnxTransformerNHWC is that for models containing transformer blocks, the entire trunk stack of blocks runs in NHWC (channel-last, [N,H,W,C]) layout instead of NCHW. This should improve memory access locality. It defaults to true. See onnxmodelbuilder.cpp:1190-1204 for details. According to your benchmarks, this option seems to slow down NPU performance when enabled; I'll look into that when I have time.

Note: I believe I did include seniorfish's directml fixes, so I think that means we should be good on windows 10 too?

from #1222 (comment)
yes, any modern GPU that supports DirectX 12 on Windows 10/11.

@seniorfish

@seniorfish
Thank you for your development and answer! I am using Intel® Core™ Ultra 5 Processor 228V, Intel® Arc™ 130V GPU, Intel® AI Boost. The GPU and NPU share 18 GB of memory. I have never met memory lack when I running match, I suspect that it is more likely to be a software issue than a hardware problem.

@seniorfish

@seniorfish
Thank you very much. Here are some debug logs, but I do not have katago_crash.dmp.

run_debug.log
katago_crash.log
debug_log.txt
Windows_error_log.txt

@seniorfish

@seniorfish
The issue still happens. The program hangs and does not exit. I also asked an AI about the possible cause (which may not be accurate), and I’m sending its analysis to you for reference. I can run b40 with https://github.com/Looong01/KataGo-Multi-backends/releases/tag/v1.17.1-openvino.
debug_logs_2.zip

No, it still don't work and hangs again.

PS C:\Users\foxrain> C:\Users\foxrain\Downloads\katago-onnx-openvino-staticbatch-690811a9\katago.exe benchmark -model C:\Users\foxrain\Downloads\kata1-zhizi-b40c768nbt-s11472M-d5982M.bin.gz -config C:\Users\foxrain\Downloads\katago-onnx-openvino-staticbatch-690811a9\test_npu_b40.cfg -t 1 -v 100

2026-08-19 11:35:14+0800: Running with following config:
logDir = logs
logToStderr = true
maxVisits = 100
nnMaxBatchSize = 1
numSearchThreads = 1
onnxOpenVINOCacheDir = C://temp//katago_ov_cache
onnxOpenVINODeviceType = NPU
onnxOpenVINOPrecision = FP16
onnxOpenVINOStaticBatchSize = 1
onnxProvider = openvino
onnxSkipScale8 = false
onnxTransformerNHWC = false

2026-08-19 11:35:14+0800: Loading model and initializing benchmark...
2026-08-19 11:35:14+0800: Loading model and initializing benchmark...
2026-08-19 11:35:14+0800: Testing with default positions for board size: 19
WARNING: Your nnMaxBatchSize is hardcoded to 1, ignoring it and assuming it is >= threads, for this benchmark.
2026-08-19 11:35:14+0800: nnRandSeed0 = 10713308195463582077
2026-08-19 11:35:14+0800: After dedups: nnModelFile0 = C:\Users\foxrain\Downloads\kata1-zhizi-b40c768nbt-s11472M-d5982M.bin.gz useFP16 auto
2026-08-19 11:35:14+0800: Initializing neural net buffer to be size 19 * 19 exactly
2026-08-19 11:35:22+0800: ONNX backend: creating compute context for 19x19 with provider 'openvino'
2026-08-19 11:35:22+0800: ONNX backend thread 0: Model version 15
2026-08-19 11:35:22+0800: ONNX backend thread 0: Model name: kata1-zhizi-b40c768nbt-s11472M-d5982M (nbt convnet, 232532105 params)
2026-08-19 11:35:22+0800: ONNX backend thread 0: provider=openvino deviceIdx=NPU
2026-08-19 11:35:22+0800: ONNX backend: building ONNX graph from model weights...
2026-08-19 11:35:22+0800: Building internal onnx model, requireExactNNLen=true transformerNHWC=false
2026-08-19 11:35:23+0800: ONNX backend: graph emitted with fixed batch size 1 for OpenVINO device NPU
2026-08-19 11:35:23+0800: ONNX backend: ONNX graph ready (930619246 bytes)
2026-08-19 11:35:23+0800: ONNX backend: OpenVINO execution provider enabled for thread 0, device_type=NPU, cache_dir=C://temp//katago_ov_cache, precision=FP16

This is a separate issue from the ones discussed above. The way "onnxProvider" is currently declared in "default_gtp.cfg" is confusing. I'm referring to the config file in "katago-v1.18.1-onnx-openvino-linux-x64.zip".

At the end of the config file, we find the lines:

# This package runs on Intel GPUs and NPUs via OpenVINO.
onnxProvider = openvino

However, the proper place to define the onnxProvider appears to be in the "ONNX Runtime backend settings" block, where we find:

# onnxProvider = ...

It is commented out, so does not conflict. But trying to follow the setup instructions, I actually uncommented this line to define openvino as the provider and was then confronted with an error message about defining twice.

I suggest to remove the lines at the end and to declare onnxProvider in the "ONNX Runtime backend settings" block.

@Yodo13

@Yodo13 Sorry about that, it was an issue with the preparation of the files packaging for that specific build. I attempted to reupload fixed versions now, I think it should be fixed.

@foxrainowo

I can run b40 with https://github.com/Looong01/KataGo-Multi-backends/releases/tag/v1.17.1-openvino.

@foxrainowo Can #1171 resolve this issue?

@ChinChangYang

Thx for ur ONNS Speedup!
Few questions to ask:

As there are many ONNX for KataGo, e.g. @ChinChangYang @Looong01 @seniorfish , what's the diff?
Is 2~3 speedup true in pure CPU or integrated GPU? If false, what's the speedup rate?
If a device no OpenCL drive, it can not run KataGo OpenCL version. How about ONNX?

If ONNX need to install before to run KataGo ONNX version, how to install? Are there guide in the repo?

@HackYardo

@HackYardo This should answer your first two questions. In short:

This implementation reuses existing modules. Also, Looong01's branch has been synced with this one.
Performance gains are hardware-dependent, so there is no fixed "speedup rate." If you are using an Intel GPU, the OpenVINO backend brings a significant speedup to the transformer model (see the benchmark results in the link for details).
DirectML requires DirectX 12, while OpenVINO requires Intel hardware. You don't need to install ONNX separately, as the required libraries are already shipped with the release and automatically loaded at runtime. Just ensure your hardware meets the requirements, download the corresponding backend build, and configure the settings properly.

Thx reply, I'll check it out the next time I play with KataGo.

Navigation Menu

Repository navigation

[Discussion / Feedback] ONNX Backend: Benchmarks, Questions, and Issues #1238

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

[Discussion / Feedback] ONNX Backend: Benchmarks, Questions, and Issues #1238

Description

Activity

seniorfish commented on Aug 14, 2026

seniorfish commented on Aug 14, 2026

foxrainowo commented on Aug 14, 2026

seniorfish commented on Aug 15, 2026

foxrainowo commented on Aug 15, 2026

seniorfish commented on Aug 16, 2026

foxrainowo commented on Aug 16, 2026

seniorfish commented on Aug 18, 2026

foxrainowo commented on Aug 19, 2026

seniorfish commented on Aug 23, 2026

Yodo13 commented on Aug 25, 2026

lightvector commented on Aug 25, 2026

seniorfish commented on Aug 28, 2026

HackYardo commented on Sep 7, 2026

seniorfish commented on Sep 10, 2026

HackYardo commented on Sep 12, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions