[Discussion / Feedback] ONNX Backend: Benchmarks, Questions, and Issues

[seniorfish]

This issue is to collect discussions and feedback regarding the ONNX backend. Feel free to post your thoughts.

#1234 has been merged, bringing large speedups on Intel GPUs and NPUs, as well as a generic DirectML backend that may perform better than OpenCL on the latest transformer models.

Prebuilt DirectML binaries are available at https://github.com/lightvector/KataGo/actions/workflows/onnx-backend.yml. Open the latest run and download the files you need from artifacts.

The OpenVINO backend is currently only built when lightvector manually triggers a run or tags a commit. If you want to test it, you can use https://github.com/seniorfish/KataGo/releases/tag/v1.17.2-onnx-openvino, but note that this is not the latest version. For configuration, you can refer to gtp_example.cfg.

Future plans:

1. There are reports that the NPU backend is not working on the largest b40 net; I will investigate this. If the backend crashes on your device, please let me know.
2. Optimize the OpenVINO batching strategy and fine-tune config options, which may require more benchmark data across various processors.
3. onnxbackend.cpp seems a bit messy with many special cases in the code. I might look into refactoring it for better maintainability as more execution providers might be added in the future.
4. Discuss the possibility of adding more Execution Providers supported by ONNX Runtime. Examples have already come up: MIGraphX (Add MIGraphX backend for AMD GPUs (ROCm) — 2.94x over OpenCL on MI300X #1235) and WebAssembly (Add ONNX Runtime backend support (OpenVINO / DirectML / TensorRT / CPU, Windows & Linux) #1222 (comment)). I think running KataGo in the browser would be very interesting.
5. Take suggestions posted under this issue.

[seniorfish]

@seniorfish

On the NPU, for b11c768h12nbt3, onnxTransformerNHWC=false provides a 3× speedup; for b28c512nbt and b18c384nbt, onnxSkipScale8=true provides a 4× speedup.

On the GPU, for b11c768h12nbt3, onnxTransformerNHWC=true provides a 3× speedup; for b40c768nbt, b28c512nbt, and b18c384nbt, onnxSkipScale8=true provides a 2× speedup. The performance starts to decline with 15 threads or more, and the speed with 20 threads is below 5 v/s.

I used https://github.com/seniorfish/KataGo/releases/tag/v1.17.2-onnx-openvino to run the tests, but it still does not include the configuration file (match, human).

katago-onnx-openvino-c025a30 b11c768h12nbt3 b40c768nbt b28c512nbt b18c384nbt b40c256
NPU_t1_onnxSkipScale8=false_onnxTransformerNHWC=false 34 --- 32 61 276
NPU_t1_onnxSkipScale8=false_onnxTransformerNHWC=true 10 --- 32 59 266
NPU_t1_onnxSkipScale8=true_onnxTransformerNHWC=false 35 --- 141 253 249
NPU_t1_onnxSkipScale8=true_onnxTransformerNHWC=true 10 --- 135 244 267
GPU_onnxSkipScale8=false_onnxTransformerNHWC=false 26 30 90 182 334
GPU_onnxSkipScale8=false_onnxTransformerNHWC=true 105 33 88 189 320
GPU_onnxSkipScale8=true_onnxTransformerNHWC=false 29 56 169 427 330
GPU_onnxSkipScale8=true_onnxTransformerNHWC=true 100 56 165 431 321
I tried adjusting the parameters, but b40c768nbt still does not work properly on the NPU backend. I’m guessing it’s because it’s too large? Has anyone figured this out?

PS C:\Users\foxrain> C:\Users\foxrain\Downloads\katago-onnx-openvino-c025a30\katago.exe benchmark -model C:\Users\foxrain\Downloads\kata1-zhizi-b40c768nbt-s11472M-d5982M.bin.gz -config C:\Users\foxrain\Downloads\katago-onnx-openvino-c025a30\gtp_example.cfg -t 1
2026-08-14 12:57:10+0800: Running with following config:
allowResignation = true
lagBuffer = 1.0
logAllGTPCommunication = true
logDir = gtp_logs
logSearchInfo = true
logSearchInfoForChosenMove = false
logToStderr = false
maxTimePondering = 60.0
maxVisits = 500
numSearchThreads = 1
onnxDeviceToUse = 0
onnxOpenVINOCacheDir = C://temp//katago_ov_cache
onnxOpenVINODeviceType = NPU
onnxOpenVINOPrecision = FP16
onnxProvider = openvino
onnxSkipScale8 = false
onnxTransformerNHWC = false
ponderingEnabled = false
resignConsecTurns = 3
resignThreshold = -0.90
rules = tromp-taylor
searchFactorAfterOnePass = 0.50
searchFactorAfterTwoPass = 0.25
searchFactorWhenWinning = 0.40
searchFactorWhenWinningThreshold = 0.95

2026-08-14 12:57:10+0800: Loading model and initializing benchmark...
2026-08-14 12:57:10+0800: Testing with default positions for board size: 19
2026-08-14 12:57:10+0800: nnRandSeed0 = 9550974731077147000
2026-08-14 12:57:10+0800: After dedups: nnModelFile0 = C:\Users\foxrain\Downloads\kata1-zhizi-b40c768nbt-s11472M-d5982M.bin.gz useFP16 auto
2026-08-14 12:57:10+0800: Initializing neural net buffer to be size 19 * 19 exactly
2026-08-14 12:57:24+0800: ONNX backend: creating compute context for 19x19 with provider 'openvino'
2026-08-14 12:57:24+0800: ONNX backend thread 0: Model version 15
2026-08-14 12:57:24+0800: ONNX backend thread 0: Model name: kata1-zhizi-b40c768nbt-s11472M-d5982M (nbt convnet, 232532105 params)
2026-08-14 12:57:24+0800: ONNX backend thread 0: provider=openvino deviceIdx=NPU
2026-08-14 12:57:24+0800: ONNX backend: building ONNX graph from model weights...
2026-08-14 12:57:24+0800: Building internal onnx model, requireExactNNLen=true transformerNHWC=false
2026-08-14 12:57:25+0800: ONNX backend: ONNX graph ready (930619286 bytes)
2026-08-14 12:57:25+0800: ONNX backend: OpenVINO execution provider enabled for thread 0, device_type=NPU, cache_dir=C://temp//katago_ov_cache, precision=FP16
2026-08-14 12:57:28+0800: ONNX backend: graph input order: [0]InputSpatial [1]InputGlobal [2]InputMask
2026-08-14 12:57:28+0800: ONNX backend: graph output order: [0]OutputPolicyPass [1]OutputPolicy [2]OutputValue [3]OutputScoreValue [4]OutputOwnership
2026-08-14 12:57:28+0800: ONNX backend: session created, inputs=3 outputs=5
2026-08-14 12:57:28+0800: ONNX backend: OpenVINO warmup for thread 0: precompiling batch sizes 1..1 on device_type=NPU (cache_dir=C://temp//katago_ov_cache, compiled graphs are reused by later runs)
PS C:\Users\foxrain>

from #1234 (comment)
@foxrainowo It looks like a problem related to model size, since only the largest network crashed. What processor model are you using? How much memory do you have for npu?

To answer your question: the purpose of onnxSkipScale8 is to work around the workaround(means disables it) that skips FP16 activation scaling. If the intermediate activations of a large convnet get too large, they overflow to inf/NaN under FP16 inference, or their precision degrades sharply near the upper limit (FP16 only has about 3 decimal digits of precision). KataGo uses the mathematically equivalent transformation of "dividing the whole network's activations by 8" to keep activations in a safe range — that's the scale8 workaround. However, after deep profiling of the code's performance, we found that when this option is enabled, it blocks operator fusion and significantly lowers performance on convnets (which you measured). The cost of enabling it is that under extreme conditions (very large boards) it might overflow and produce invalid values, but in that case KataGo should fail loud, and testing shows this problem is hardly hit on a normal 19x19 board. This option defaults to false, meaning slower speed but higher stability; you can enable it in daily use for better performance, and it is forcibly disabled when contributing. This option does not affect transformer models. See 4f86efa and c2e5552 for details.

The purpose of onnxTransformerNHWC is that for models containing transformer blocks, the entire trunk stack of blocks runs in NHWC (channel-last, [N,H,W,C]) layout instead of NCHW. This should improve memory access locality. It defaults to true. See onnxmodelbuilder.cpp:1190-1204 for details. According to your benchmarks, this option seems to slow down NPU performance when enabled; I'll look into that when I have time.

[seniorfish]

Note: I believe I did include seniorfish's directml fixes, so I think that means we should be good on windows 10 too?

from #1222 (comment)
yes, any modern GPU that supports DirectX 12 on Windows 10/11.

[foxrainowo]

@seniorfish
Thank you for your development and answer! I am using Intel® Core™ Ultra 5 Processor 228V, Intel® Arc™ 130V GPU, Intel® AI Boost. The GPU and NPU share 18 GB of memory. I have never met memory lack when I running match, I suspect that it is more likely to be a software issue than a hardware problem.

[seniorfish]

@foxrainowo
I still can't find the problem, so I prepared a diagnostic build based on the version you tested, with full-stack debug logging enabled by default and an automatic crash log + minidump: https://github.com/seniorfish/KataGo/releases/tag/v1.17.2-onnx-b40-npu-debug. Please run your b40c768nbt with it following the included instructions (Chinese and English), and share run_debug.log, katago_crash.log/katago_crash.dmp, the exit code.

[foxrainowo]

@seniorfish
Thank you very much. Here are some debug logs, but I do not have katago_crash.dmp.

run_debug.log
katago_crash.log
debug_log.txt
Windows_error_log.txt

[seniorfish]

@foxrainowo Thanks for the logs, The crash is a stack overflow. The NPU translates the whole b40 model on the NN thread before the first forward, and on Lunar Lake that translation needs more stack room than the 8 MB KataGo reserved, so the process died. New builds raise the thread stack to 256 MB (just reserved address space, no extra RAM used), which should fix it.
https://github.com/seniorfish/KataGo/releases/tag/v1.17.2-onnx-b40-npu-fix
Please try them: the regular build (for normal play/benchmarks) and, if anything still fails, the diagnostic build. Paste any new katago_crash.log here. If the regular build works, please also try NPU_COMPILER_TYPE=DRIVER once and check whether the LiveKernelEvent 141 entries stop appearing.

[foxrainowo]

@seniorfish
The issue still happens. The program hangs and does not exit. I also asked an AI about the possible cause (which may not be accurate), and I’m sending its analysis to you for reference. I can run b40 with https://github.com/Looong01/KataGo-Multi-backends/releases/tag/v1.17.1-openvino.
debug_logs_2.zip

[seniorfish]

@foxrainowo Thanks for all the logs, they helped a lot.

I looked at your npccheck output again together with how the Looong01 build does it, and my current best guess is this: KataGo's ONNX graph leaves the batch dimension as a dynamic, unbounded one. The NPU compiler seems to have trouble planning for that on a model as big as b40, which is probably what led to the crash and the hangs. The earlier 256 MB stack build treated a symptom and was not enough.

So I put together a new build that fixes the batch to 1 right in the graph, so the whole graph is static, plus a small NPU test config.

Could you try it and let me know how it goes?

https://github.com/seniorfish/KataGo/releases/download/v1.17.2-onnx-b40-npu-static-batch/katago-onnx-openvino-staticbatch-690811a9.zip

If it still fails, please share the new log and the exit code. I am not fully sure this is the whole story, so any result helps.

[foxrainowo]

No, it still don't work and hangs again.

PS C:\Users\foxrain> C:\Users\foxrain\Downloads\katago-onnx-openvino-staticbatch-690811a9\katago.exe benchmark -model C:\Users\foxrain\Downloads\kata1-zhizi-b40c768nbt-s11472M-d5982M.bin.gz -config C:\Users\foxrain\Downloads\katago-onnx-openvino-staticbatch-690811a9\test_npu_b40.cfg -t 1 -v 100

2026-08-19 11:35:14+0800: Running with following config:
logDir = logs
logToStderr = true
maxVisits = 100
nnMaxBatchSize = 1
numSearchThreads = 1
onnxOpenVINOCacheDir = C://temp//katago_ov_cache
onnxOpenVINODeviceType = NPU
onnxOpenVINOPrecision = FP16
onnxOpenVINOStaticBatchSize = 1
onnxProvider = openvino
onnxSkipScale8 = false
onnxTransformerNHWC = false

2026-08-19 11:35:14+0800: Loading model and initializing benchmark...
2026-08-19 11:35:14+0800: Loading model and initializing benchmark...
2026-08-19 11:35:14+0800: Testing with default positions for board size: 19
WARNING: Your nnMaxBatchSize is hardcoded to 1, ignoring it and assuming it is >= threads, for this benchmark.
2026-08-19 11:35:14+0800: nnRandSeed0 = 10713308195463582077
2026-08-19 11:35:14+0800: After dedups: nnModelFile0 = C:\Users\foxrain\Downloads\kata1-zhizi-b40c768nbt-s11472M-d5982M.bin.gz useFP16 auto
2026-08-19 11:35:14+0800: Initializing neural net buffer to be size 19 * 19 exactly
2026-08-19 11:35:22+0800: ONNX backend: creating compute context for 19x19 with provider 'openvino'
2026-08-19 11:35:22+0800: ONNX backend thread 0: Model version 15
2026-08-19 11:35:22+0800: ONNX backend thread 0: Model name: kata1-zhizi-b40c768nbt-s11472M-d5982M (nbt convnet, 232532105 params)
2026-08-19 11:35:22+0800: ONNX backend thread 0: provider=openvino deviceIdx=NPU
2026-08-19 11:35:22+0800: ONNX backend: building ONNX graph from model weights...
2026-08-19 11:35:22+0800: Building internal onnx model, requireExactNNLen=true transformerNHWC=false
2026-08-19 11:35:23+0800: ONNX backend: graph emitted with fixed batch size 1 for OpenVINO device NPU
2026-08-19 11:35:23+0800: ONNX backend: ONNX graph ready (930619246 bytes)
2026-08-19 11:35:23+0800: ONNX backend: OpenVINO execution provider enabled for thread 0, device_type=NPU, cache_dir=C://temp//katago_ov_cache, precision=FP16