[Intel NPU/GPU] Add Windows & Linux Intel NPU & GPU support

Compiling.md

@@ -277,14 +277,16 @@ Set `onnxProvider` in your config to choose the execution provider:
      cmake --build KataGo/cpp/build -j
      ```
    * `ONNXRUNTIME_ROOT` is the directory containing ONNX Runtime's `include/` and `lib/`.
+   * For an Intel NPU or iGPU build, `-DUSE_ONNX_EP=OPENVINO` defaults `ONNXRUNTIME_ROOT` to `KataGo/cpp/external/onnxruntime-win-x64-openvino` (or `-linux-x64-` on Linux), so unpacking your OpenVINO-enabled ONNX Runtime there lets you drop the flag. It changes nothing else: the execution provider is still chosen at runtime by `onnxProvider`.
    * If CMake does not find protobuf on its own, also pass `-DProtobuf_PROTOC_EXECUTABLE=<protoc>`, `-DProtobuf_INCLUDE_DIR=<include-dir>`, and `-DProtobuf_LIBRARY=<library>`.
    * As with other backends, `-DNO_GIT_REVISION=1` avoids embedding the git hash, and `-DBUILD_DISTRIBUTED=1` enables contributing to distributed training.
 
 ### Runtime
    * The `onnxruntime` shared library must be next to the executable or on your library path.
-   * For the OpenVINO provider, the OpenVINO runtime DLLs (`openvino.dll`, `openvino_intel_gpu_plugin.dll`, `tbb12.dll`, `cache.json`, etc.) must also be next to the executable or on the system path.
+   * For the OpenVINO provider, the OpenVINO runtime DLLs (`openvino.dll`, `tbb12.dll`, `cache.json`, etc.) must also be next to the executable or on the system path, along with the plugin for each device you want to use: `openvino_intel_gpu_plugin.dll` for Intel GPUs and `openvino_intel_npu_plugin.dll` for Intel NPUs. A device whose plugin is missing is simply not available, so shipping only the GPU plugin makes an NPU machine behave like a GPU-only one.
    * For the DirectML provider, `DirectML.dll` 1.8.0 or newer (from the Microsoft.AI.DirectML package) must be next to `onnxruntime.dll`. Without it, Windows 10 falls back to its much older inbox DirectML and the provider fails at startup.
-   * Choose the provider and its options with the `onnx*` keys in your config, e.g. `onnxProvider = openvino` and `onnxOpenVINODeviceType = GPU`. The ONNX section of `configs/gtp_example.cfg` documents all the options.
+   * Choose the provider with the `onnxProvider` key in your config, e.g. `onnxProvider = openvino`. The ONNX section of `configs/gtp_example.cfg` documents all the options.
+   * For OpenVINO you normally do not need to name a device. With `onnxOpenVINODeviceType` unset, KataGo probes the machine and picks the first device present, NPU before GPU, logging which one it chose, and also picks the transformer trunk layout to match it (NCHW for the NPU, NHWC for the GPU) since the two plugins want opposite layouts. Set `onnxOpenVINODeviceType` explicitly only to override that, e.g. to force `GPU` on a machine that also has an NPU; an explicit device is never second-guessed, so naming one that is not there fails at startup rather than falling back.
 
 ### Working with .onnx files
 This backend and the TensorRT backend can also write out the ONNX graph they build (`katago dumponnx`) and load a `.onnx` file as a model in place of the `.bin.gz`, including one produced by other tooling. See **[ONNX_Model_Files.md](docs/ONNX_Model_Files.md)** for the commands and the model file format.

cpp/CMakeLists.txt

@@ -1670,7 +1670,23 @@ elseif(USE_BACKEND STREQUAL "ONNX")
   # ONNX Runtime install tree (include/ lib/ bin/). The official prebuilt ORT packages do
   # NOT ship the OpenVINO execution provider, so for Intel GPU acceleration ORT must be
   # built from source with --use_openvino GPU (see Compiling.md).
-  set(ONNXRUNTIME_ROOT "" CACHE PATH "Path to ONNX Runtime package root (containing include/, lib/, bin/)")
+  # Which execution provider this build is being set up for. ONNX Runtime selects the provider at
+  # runtime from the onnxProvider config key, so this affects nothing but the default
+  # ONNXRUNTIME_ROOT below. With -DUSE_ONNX_EP=OPENVINO it points at an OpenVINO-enabled ONNX
+  # Runtime unpacked under cpp/external, which is where the Intel NPU / iGPU instructions in
+  # Compiling.md put it, so those builds configure without further flags. Left blank, there is no
+  # default and ONNXRUNTIME_ROOT must be given explicitly.
+  set(USE_ONNX_EP "" CACHE STRING "ONNX Runtime execution provider this build targets: blank or OPENVINO")
+  set_property(CACHE USE_ONNX_EP PROPERTY STRINGS "" OPENVINO)
+  set(_onnx_default_root "")
+  if(USE_ONNX_EP STREQUAL "OPENVINO")
+    if(WIN32)
+      set(_onnx_default_root "${CMAKE_CURRENT_SOURCE_DIR}/external/onnxruntime-win-x64-openvino")
+    elseif(UNIX AND NOT APPLE)
+      set(_onnx_default_root "${CMAKE_CURRENT_SOURCE_DIR}/external/onnxruntime-linux-x64-openvino")
+    endif()
+  endif()
+  set(ONNXRUNTIME_ROOT "${_onnx_default_root}" CACHE PATH "Path to ONNX Runtime package root (containing include/, lib/, bin/)")
   if(NOT IS_DIRECTORY "${ONNXRUNTIME_ROOT}")
     message(FATAL_ERROR "ONNXRUNTIME_ROOT does not exist: ${ONNXRUNTIME_ROOT}. Set -DONNXRUNTIME_ROOT=<ort install dir>.")
   endif()

cpp/configs/gtp_example.cfg

@@ -485,13 +485,26 @@ searchFactorWhenWinningThreshold = 0.95
 # onnxDeviceToUseThread0 = 0
 # onnxDeviceToUseThread1 = 1
 
-# OpenVINO options (only used when onnxProvider = openvino):
-
-# Which device to use: GPU (default), NPU, a specific device like GPU.0 or GPU.1 if you
-# have several, or an OpenVINO multi-device string such as AUTO:GPU,CPU.
-# (For CPU inference use onnxProvider = cpu instead.)
-# NOTE: avoid setting nnMaxBatchSize = 1 in this config - a known OpenVINO bug produces
-# garbage NPU output at batch size 1. KataGo's automatic batch sizing is not affected.
+# OpenVINO provider options (only used when onnxProvider = openvino):
+# Target device: NPU, GPU, GPU.0, GPU.1, or an OpenVINO multi-device string such as
+# AUTO:GPU,CPU. Only GPU/NPU acceleration is supported here, use onnxProvider = cpu for CPU inference
+#
+# LEAVE THIS UNSET unless you have a reason not to. When it is unset KataGo probes the machine
+# and picks the first device that is actually there, NPU before GPU, and logs which one it chose.
+# That is the right answer on almost every Intel machine, and it also picks the trunk layout to
+# match (see onnxTransformerNHWC below), which a hand-set device does not do for you.
+#   - Machine has an NPU (with or without an Intel GPU) -> NPU
+#   - Machine has an Intel GPU but no NPU              -> GPU
+#   - Neither                                          -> startup fails with OpenVINO's
+#                                                         "Device GPU is not available"
+# Setting this key explicitly always wins, including when the device does not exist: KataGo will
+# not quietly run somewhere other than where you asked, it fails with that same OpenVINO error.
+# NOTE: NPU inference recompiles the whole graph for each new NN batch size, which caused long
+# stalls whenever numSearchThreads > 1. onnxPadBatch (below) holds the batch size constant and is
+# enabled automatically for NPU devices, which fixes this.
+# NOTE: nnMaxBatchSize = 1 makes NPU inference produce non-finite outputs (an OpenVINO provider
+# bug, observed with OpenVINO 2026.2.1). KataGo's automatic batch sizing never picks 1, so this
+# only matters if you set nnMaxBatchSize = 1 in this config yourself.
 # onnxOpenVINODeviceType = GPU
 
 # Optionally override the device per neural net server thread, e.g. to use GPU and NPU
@@ -519,12 +532,27 @@ searchFactorWhenWinningThreshold = 0.95
 # extremely slow without padding. Auto (the default) enables it exactly for those.
 # onnxPadBatch = auto
 
-# Keep neural net activations safely inside FP16 numeric range, at some speed cost.
-# Setting this to true is faster but risks garbage output from FP16 overflow, mainly
-# for convolutional nets on larger boards.
+# The scale8 FP16-range workaround is applied by default (onnxSkipScale8 = false).
+# scale8 keeps convnet activations 8x smaller so they stay inside the FP16 range OpenVINO
+# infers in. It is emitted as a fusable Mish subgraph, so OpenVINO still fuses it and it is
+# not measurably slower than skipping it. There is normally no reason to turn it off.
+# Contributing to distributed training forces it on regardless, so that an FP16 overflow
+# cannot put NaN rows into uploaded training data.
 # onnxSkipScale8 = false
 
-# Run transformer models channel-last (NHWC). Convolutional nets ignore this.
+# Run the trunk block stack channel-last (NHWC) for transformer models.
+# Only takes effect for models with transformer blocks, and convnets ignore it.
+#
+# There is no single right answer here: the two OpenVINO plugins want opposite layouts.
+# Measured on b11c768h12nbt3tflrs at 8 search threads, the GPU is ~1.5x faster on NHWC, and
+# the NPU is ~1.75x faster on NCHW. So when this key is unset KataGo picks per device:
+#   - onnxProvider = openvino and any thread's device might be an NPU  -> NCHW (false)
+#       This includes explicit NPU, composite strings that mention NPU such as AUTO:GPU,NPU,
+#       and a bare AUTO, since OpenVINO can resolve that to an NPU and only tells you at
+#       runtime. Guessing NCHW costs a GPU ~1.5x when wrong but saves an NPU ~1.75x when right.
+#   - onnxProvider = openvino on a GPU-only device string             -> NHWC (true)
+#   - any other provider                                              -> NHWC (true)
+# Set it explicitly only to override that, e.g. if you pinned AUTO and know it lands on a GPU.
 # onnxTransformerNHWC = true
 
 # ------------------------------

cpp/neuralnet/onnxbackend.cpp

@@ -194,6 +194,32 @@ struct ComputeContext {
 
 static std::vector<std::string> parseDeviceNames(const std::string& deviceType);
 
+//--------------------------------------------------------------
+// Helper: is an OpenVINO device_type string usable on this machine?
+//
+// AppendExecutionProvider_OpenVINO_V2 validates device_type as soon as it is called and throws
+// "[OpenVINO] Device X is not available" for anything the installed plugins cannot provide, so a
+// throwaway SessionOptions is enough to probe. No model, session or inference is involved and the
+// call costs a plugin lookup, which is why this can run unconditionally at context creation.
+//
+// ONNX Runtime's own device enumeration (Ort::Env::GetEpDevices, OrtHardwareDeviceType) is
+// deliberately not used here: on a provider-bridge OpenVINO build it reports the CPU only, even on
+// a machine that does have an NPU and an integrated GPU, so it would steer exactly the setups this
+// is meant to help onto the slowest device available.
+//--------------------------------------------------------------
+static bool openvinoDeviceAvailable(const string& deviceType) {
+  try {
+    Ort::SessionOptions probeOpts;
+    std::unordered_map<std::string, std::string> probeEpOpts;
+    probeEpOpts["device_type"] = deviceType;
+    probeOpts.AppendExecutionProvider_OpenVINO_V2(probeEpOpts);
+    return true;
+  }
+  catch(const std::exception&) {
+    return false;
+  }
+}
+
 ComputeContext* NeuralNet::createComputeContext(
   const std::vector<int>& gpuIdxs,
   Logger* logger,
@@ -224,7 +250,29 @@ ComputeContext* NeuralNet::createComputeContext(
   ctx->providerName = getRequiredProviderLowercase(cfg);
 
   // OpenVINO EP options.
-  ctx->openvinoDeviceType = cfg.contains("onnxOpenVINODeviceType") ? cfg.getString("onnxOpenVINODeviceType") : "GPU";
+  // Device selection. An explicit onnxOpenVINODeviceType is always honored as-is, including when
+  // the device turns out not to exist: silently running somewhere other than where the user asked
+  // would be worse than the EP's "Device X is not available" error.
+  //
+  // When the key is absent, probe for a device instead of assuming one. NPU first, then GPU: on the
+  // Intel parts this backend targets the NPU is both the faster and the more power-efficient of the
+  // two for KataGo's trunk, and a machine that has one almost always wants it. CPU is not a
+  // candidate, since the OpenVINO provider rejects CPU-only device strings below.
+  if(cfg.contains("onnxOpenVINODeviceType")) {
+    ctx->openvinoDeviceType = cfg.getString("onnxOpenVINODeviceType");
+  }
+  else if(ctx->providerName == "openvino") {
+    static const char* const candidates[] = {"NPU", "GPU"};
+    for(const char* cand : candidates) {
+      if(openvinoDeviceAvailable(cand)) {
+        ctx->openvinoDeviceType = cand;
+        break;
+      }
+    }
+    if(logger != NULL)
+      logger->write(
+        string("ONNX backend: onnxOpenVINODeviceType not set, auto-selected '") + ctx->openvinoDeviceType + "'");
+  }
   ctx->openvinoCacheDir = cfg.contains("onnxOpenVINOCacheDir") ? cfg.getString("onnxOpenVINOCacheDir") : "";
   ctx->openvinoPrecision = cfg.contains("onnxOpenVINOPrecision") ? cfg.getString("onnxOpenVINOPrecision") : "";
   ctx->openvinoNumStreams = cfg.contains("onnxOpenVINONumStreams") ? cfg.getString("onnxOpenVINONumStreams") : "";
@@ -240,25 +288,6 @@ ComputeContext* NeuralNet::createComputeContext(
       logger->write("ONNX backend: useFP16 = false, forcing OpenVINO precision = FP32");
   }
 
-  // Trunk layout for transformer models. Default NHWC (channel-last), matching the TensorRT
-  // backend's trtTransformerNHWC default. NHWC is markedly faster for transformer trunks on
-  // OpenVINO GPU/NPU, and is ignored entirely for models without transformer blocks.
-  ctx->transformerNHWC = cfg.contains("onnxTransformerNHWC") ? cfg.getBool("onnxTransformerNHWC") : true;
-  if(loadedModel->isExternalOnnx && logger != NULL && cfg.contains("onnxTransformerNHWC") &&
-     ctx->transformerNHWC != loadedModel->externalOnnx.buildParams.transformerNHWC &&
-     loadedModel->modelDesc.hasAnyTransformerBlocks())
-    logger->write(
-      "ONNX backend: WARNING - onnxTransformerNHWC = " + Global::boolToString(ctx->transformerNHWC) +
-      " has no effect on a model loaded from a .onnx file. The trunk layout is baked into the graph "
-      "(transformerNHWC=" + Global::boolToString(loadedModel->externalOnnx.buildParams.transformerNHWC) + ").");
-
-  // Skip the scale8 FP16-range workaround. Default false, meaning the workaround is applied.
-  // See the onnxSkipScale8 documentation in configs/gtp_example.cfg for the tradeoff.
-  ctx->skipScale8 = cfg.contains("onnxSkipScale8") ? cfg.getBool("onnxSkipScale8") : false;
-
-  // Must happen here rather than at compute-handle creation. See LoadedModel::scale8Resolved.
-  loadedModel->maybeApplyScale8(ctx->skipScale8, cfg.contains("onnxSkipScale8"), logger);
-
   // --- Per-thread device type assignment ---
   // Pre-parse onnxOpenVINODeviceTypeThread<N> keys so ComputeHandle can look up
   // the device type for each server thread without reaching back into ConfigParser.
@@ -274,6 +303,49 @@ ComputeContext* NeuralNet::createComputeContext(
     }
   }
 
+  // Trunk layout for transformer models, ignored entirely for models without transformer blocks.
+  //
+  // There is no layout that is right for both OpenVINO plugins. Measured on one transformer model,
+  // b11c768h12nbt3tflrs at 8 search threads: the GPU plugin runs NHWC about 1.5x faster than NCHW,
+  // while the NPU plugin runs NCHW about 1.75x faster than NHWC. So pick per device rather than by
+  // a fixed default, and default to NCHW as soon as any server thread might land on an NPU.
+  //
+  // "Might" is the operative word for composite device strings. As with padsBatchForDevice, a
+  // string like AUTO:GPU,NPU leaves the choice to OpenVINO at runtime and it is not reported back
+  // through ORT's API, so anything mentioning NPU counts. A bare AUTO counts too: it can resolve to
+  // an NPU, and the asymmetry above makes guessing NCHW the better bet either way, costing a GPU
+  // 1.5x when wrong but saving an NPU 1.75x when right.
+  //
+  // Non-OpenVINO providers keep NHWC, matching the TensorRT backend's trtTransformerNHWC default.
+  auto mayBeNPU = [](const string& dev) {
+    string upper = Global::toUpper(Global::trim(dev));
+    if(upper.find("NPU") != string::npos)
+      return true;
+    return upper == "AUTO";
+  };
+  bool anyMayBeNPU = false;
+  if(ctx->providerName == "openvino") {
+    for(const string& dev : ctx->perThreadDeviceType) {
+      if(mayBeNPU(dev))
+        anyMayBeNPU = true;
+    }
+  }
+  ctx->transformerNHWC = cfg.contains("onnxTransformerNHWC") ? cfg.getBool("onnxTransformerNHWC") : !anyMayBeNPU;
+  if(loadedModel->isExternalOnnx && logger != NULL && cfg.contains("onnxTransformerNHWC") &&
+     ctx->transformerNHWC != loadedModel->externalOnnx.buildParams.transformerNHWC &&
+     loadedModel->modelDesc.hasAnyTransformerBlocks())
+    logger->write(
+      "ONNX backend: WARNING - onnxTransformerNHWC = " + Global::boolToString(ctx->transformerNHWC) +
+      " has no effect on a model loaded from a .onnx file. The trunk layout is baked into the graph "
+      "(transformerNHWC=" + Global::boolToString(loadedModel->externalOnnx.buildParams.transformerNHWC) + ").");
+
+  // Skip the scale8 FP16-range workaround. Default false, meaning the workaround is applied.
+  // See the onnxSkipScale8 documentation in configs/gtp_example.cfg for the tradeoff.
+  ctx->skipScale8 = cfg.contains("onnxSkipScale8") ? cfg.getBool("onnxSkipScale8") : false;
+
+  // Must happen here rather than at compute-handle creation. See LoadedModel::scale8Resolved.
+  loadedModel->maybeApplyScale8(ctx->skipScale8, cfg.contains("onnxSkipScale8"), logger);
+
   // The OpenVINO provider is only used for GPU/NPU acceleration here. For CPU inference the
   // plain cpu provider (or the Eigen backend) is the right tool, so reject any device string
   // that resolves to CPU alone (CPU, cpu, CPU.0, AUTO:CPU, ...). Composite strings that also
@@ -1306,7 +1378,8 @@ void NeuralNet::printDevices() {
   cout << "Set onnxProvider (e.g. 'openvino') plus provider-specific options in the config." << endl;
   cout << endl;
   cout << "OpenVINO provider options:" << endl;
-  cout << "  onnxOpenVINODeviceType = GPU            (default; GPU, NPU, GPU.0, GPU.1, etc.)" << endl;
+  cout << "  onnxOpenVINODeviceType = NPU            (GPU, NPU, GPU.0, GPU.1, etc.)" << endl;
+  cout << "    Leave it unset to auto-select: NPU if this machine has one, else GPU." << endl;
   cout << "  Also supports OpenVINO multi-device strings:" << endl;
   cout << "    AUTO:GPU,CPU  MULTI:GPU,NPU  HETERO:GPU,CPU" << endl;
   cout << endl;