landing/opensource summit korea deck

docs/faq/faq.md

@@ -178,7 +178,7 @@ If the official Device Plugin cannot provide the required information, HAMi deve
 
 ## How does HAMi enforce GPU memory and compute limits?
 
-HAMi injects `libvgpu.so` into containers via `/etc/ld.so.preload`. The library intercepts CUDA memory allocation calls and returns OOM when the `nvidia.com/gpumem` limit is exceeded; compute limits use a token-bucket throttle on kernel launch calls. Applications that bypass the CUDA library (Docker-in-Docker, direct driver API) are not covered. For the full interception flow, see [GPU Virtualization](../core-concepts/gpu-virtualization).
+HAMi injects `libvgpu.so` into containers via `/etc/ld.so.preload`. The library intercepts CUDA memory allocation calls and returns OOM when the `nvidia.com/gpumem` limit is exceeded; compute limits use a token-bucket throttle on kernel launch calls. Applications that bypass the CUDA library (Docker-in-Docker, direct driver API) are not covered. For the full interception flow, see [GPU Virtualization](./core-concepts/gpu-virtualization).
 
 ## How does HAMi vGPU differ from NVIDIA MIG? When should I use each?
 
@@ -194,19 +194,19 @@ HAMi vGPU is software-only with no hardware requirements. NVIDIA MIG is hardware
 | Dynamic reconfiguration | Yes, no node drain needed | Requires MIG profile reconfiguration |
 | Multi-tenant noise isolation | Best-effort | Strong |
 
-Use HAMi vGPU when the GPU does not support MIG, workloads need flexible memory sizes, or dynamic repacking without node drains is needed. Use MIG when hard hardware isolation is a compliance or SLA requirement. HAMi also supports dynamic MIG via `mig-parted`; see [Dynamic MIG Support](../userguide/nvidia-device/dynamic-mig-support).
+Use HAMi vGPU when the GPU does not support MIG, workloads need flexible memory sizes, or dynamic repacking without node drains is needed. Use MIG when hard hardware isolation is a compliance or SLA requirement. HAMi also supports dynamic MIG via `mig-parted`; see [Dynamic MIG Support](./userguide/nvidia-device/dynamic-mig-support).
 
 ## Why does nvidia-smi inside my container show less memory than on the host?
 
-`libvgpu.so` intercepts `nvmlDeviceGetMemoryInfo` and related calls, returning the `nvidia.com/gpumem` limit instead of physical VRAM. This is intentional: workloads that size their allocations based on reported memory (such as vLLM) will use only their budget. The host’s `nvidia-smi` always shows physical memory. See [GPU Virtualization](../core-concepts/gpu-virtualization).
+`libvgpu.so` intercepts `nvmlDeviceGetMemoryInfo` and related calls, returning the `nvidia.com/gpumem` limit instead of physical VRAM. This is intentional: workloads that size their allocations based on reported memory (such as vLLM) will use only their budget. The host’s `nvidia-smi` always shows physical memory. See [GPU Virtualization](./core-concepts/gpu-virtualization).
 
 ## Why is my nvidia.com/gpumem limit not enforced? {#why-is-my-nvidiagpumem-limit-not-enforced}
 
 The four most common causes: `CUDA_DISABLE_CONTROL=true` is set, the workload runs inside Docker-in-Docker, the application calls the GPU driver directly (bypassing `libvgpu.so`), or `nvidia-container-runtime` is not the default runtime on the node. See [Troubleshooting](../troubleshooting/troubleshooting.md) for resolution steps.
 
 ## Does HAMi replace kube-scheduler or run alongside it?
 
-HAMi runs alongside kube-scheduler as a [scheduler extender](https://github.com/kubernetes/design-proposals-archive/blob/main/scheduling/scheduler_extender.md) - it does not replace it. The MutatingWebhook sets `schedulerName: hami-scheduler` only on pods requesting HAMi resources; all other pods follow the default scheduler path unchanged. See [Architecture](../core-concepts/architecture).
+HAMi runs alongside kube-scheduler as a [scheduler extender](https://github.com/kubernetes/design-proposals-archive/blob/main/scheduling/scheduler_extender.md) - it does not replace it. The MutatingWebhook sets `schedulerName: hami-scheduler` only on pods requesting HAMi resources; all other pods follow the default scheduler path unchanged. See [Architecture](./core-concepts/architecture).
 
 ## Does HAMi work with vLLM, and what are the known limitations for multi-GPU tensor parallelism?
 
@@ -221,8 +221,10 @@ devicePlugin:
   enabled: false
 ```
 
-DCGM Exporter is not affected and continues to report physical-level counters normally. HAMi’s per-container virtual metrics are separate; see [GPU Utilization Metrics](../developers/gpu-utilization-metrics).
+DCGM Exporter is not affected and continues to report physical-level counters normally. HAMi’s per-container virtual metrics are separate; see [GPU Utilization Metrics](./developers/gpu-utilization-metrics).
+
+If the HAMi Device Plugin or workloads fail after upgrading to GPU Operator 25.10+, see [NVIDIA containers fail with GPU Operator 25.10+](../troubleshooting/troubleshooting.md#nvidia-toolkit-gpu-operator-25-10).
 
 ## How do I set up Prometheus and Grafana monitoring for HAMi vGPU metrics?
 
-The `hami-device-plugin` pod on each node exposes per-container vGPU metrics on port `31992` (configurable via `devicePlugin.service.httpPort`). See [Grafana Dashboard](../userguide/monitoring/grafana-dashboard) for the full setup including Prometheus scrape config and dashboard import.
+The `hami-device-plugin` pod on each node exposes per-container vGPU metrics on port `31992` (configurable via `devicePlugin.service.httpPort`). See [Grafana Dashboard](./userguide/monitoring/grafana-dashboard) for the full setup including Prometheus scrape config and dashboard import.

docs/troubleshooting/troubleshooting.md

@@ -15,7 +15,7 @@ If a container exceeds its `nvidia.com/gpumem` limit, check the following causes
   containerd config dump | grep default_runtime_name
   ```
 
-  The output must show `nvidia`. If not, follow the [Prerequisites](../installation/online-installation) guide.
+  The output must show `nvidia`. If not, follow the [Prerequisites](./installation/online-installation) guide.
 
 - If you don’t explicitly request vGPUs when using the device plugin with NVIDIA images, all GPUs on the host may be exposed to your container.
 - Currently, A100 MIG can be supported in only "none" and "mixed" modes.
@@ -33,3 +33,130 @@ If a container exceeds its `nvidia.com/gpumem` limit, check the following causes
     ```
 
     This will apply the fix automatically.
+
+## NVIDIA containers fail with GPU Operator 25.10+ {#nvidia-toolkit-gpu-operator-25-10}
+
+Use this section when the HAMi Device Plugin or a HAMi-scheduled NVIDIA workload stops starting after GPU Operator is installed or upgraded.
+
+### Problem 1: The HAMi Device Plugin fails to start
+
+#### Identify the cause
+
+Check the Device Plugin logs:
+
+```bash
+kubectl logs -n kube-system \
+  -l app.kubernetes.io/component=hami-device-plugin \
+  --all-containers --tail=200
+```
+
+Match the output to one of these errors:
+
+| Error in the log | Cause |
+| --- | --- |
+| `Incompatible strategy detected auto` | The Device Plugin cannot discover NVML devices because its container did not receive the NVIDIA driver and devices. |
+| `invalid device discovery strategy` | The Device Plugin could not initialize NVIDIA device discovery, usually for the same runtime-injection reason. |
+| `failed to locate libcuda.so` or `failed to locate libnvidia-ml.so` | HAMi cannot find the driver libraries under the configured driver root while generating a CDI specification. |
+
+Confirm the runtime and CDI configuration:
+
+```bash
+kubectl get clusterpolicy -o yaml | grep -A 5 'cdi:'
+kubectl get runtimeclass nvidia
+
+kubectl get pods -n kube-system \
+  -l app.kubernetes.io/component=hami-device-plugin \
+  -o custom-columns=NAME:.metadata.name,RUNTIMECLASS:.spec.runtimeClassName
+```
+
+With GPU Operator 25.10+, CDI is normally enabled, the `nvidia` RuntimeClass must exist, and the HAMi Device Plugin must show `nvidia` in the `RUNTIMECLASS` column.
+
+#### Solution
+
+Configure the `nvidia` RuntimeClass for HAMi and restart the Device Plugin:
+
+```bash
+helm upgrade hami hami-charts/hami \
+  --namespace kube-system \
+  --reuse-values \
+  --set devicePlugin.runtimeClassName=nvidia
+
+kubectl rollout restart daemonset/hami-device-plugin -n kube-system
+kubectl rollout status daemonset/hami-device-plugin -n kube-system
+```
+
+If the logs report missing driver libraries while HAMi CDI is enabled, use the GPU Operator paths:
+
+```yaml
+devicePlugin:
+  runtimeClassName: nvidia
+  deviceListStrategy: cdi-annotations
+  nvidiaDriverRoot: /run/nvidia/driver
+  nvidiaHookPath: /usr/local/nvidia/toolkit/nvidia-ctk
+```
+
+Wait until the NVIDIA driver and Toolkit DaemonSets are ready, then restart the HAMi Device Plugin. For host-installed drivers, set `devicePlugin.nvidiaDriverRoot` to `/` instead.
+
+### Problem 2: A HAMi-scheduled Pod fails to start
+
+#### Identify the cause
+
+Inspect the Pod events and its assigned RuntimeClass:
+
+```bash
+kubectl describe pod <pod-name> -n <namespace>
+kubectl get pod <pod-name> -n <namespace> \
+  -o custom-columns=NAME:.metadata.name,RUNTIMECLASS:.spec.runtimeClassName
+```
+
+Use the error text to select the correct path:
+
+| Error in the Pod events | Cause |
+| --- | --- |
+| `libcuda.so.1: cannot open shared object file` | The container started without the NVIDIA driver libraries. |
+| `unresolvable CDI devices management.nvidia.com/gpu=GPU-...` | The NVIDIA runtime selected a GPU management CDI device, but the corresponding device could not be generated or resolved. |
+| `unresolvable CDI devices k8s.device-plugin.nvidia.com/gpu=GPU-...` | HAMi returned a CDI device, but the runtime cannot find a matching HAMi-generated CDI specification. |
+
+#### Solution
+
+First determine which HAMi injection mode is configured:
+
+```bash
+helm get values hami -n kube-system | grep -A 5 'devicePlugin:'
+```
+
+- For the default `devicePlugin.deviceListStrategy=envvar` mode, set `devicePlugin.runtimeClassName=nvidia` by using the Helm command from Problem 1. This makes the NVIDIA runtime process the UUID returned through `NVIDIA_VISIBLE_DEVICES`.
+- For `devicePlugin.deviceListStrategy=cdi-annotations`, apply all four CDI values shown in Problem 1. Then inspect `/var/run/cdi/k8s.device-plugin.nvidia.com-gpu.json` on the node and verify that it contains the allocated GPU UUID.
+- For a host-installed Container Toolkit, confirm that the `nvidia` runtime is present in the active container runtime configuration. Restart the container runtime after correcting its configuration.
+
+Do not mix HAMi CDI annotations with a missing or stale HAMi CDI specification. See [NVIDIA CDI support](../installation/configure-cdi.md) for the complete setup and verification procedure.
+
+### Why this happens
+
+Starting with [GPU Operator 25.10.0](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/25.10/cdi.html), CDI is enabled by default and the Operator no longer makes the `nvidia` runtime the default runtime.
+
+Before 25.10.0, GPU Operator normally configured the NVIDIA runtime as the default. As a result, the NVIDIA runtime hook processed every Pod and injected the devices and driver libraries selected through `NVIDIA_VISIBLE_DEVICES`.
+
+With 25.10.0 and later, `runc` remains the default runtime and the container runtime uses native CDI for standard Device Plugin workloads. GPU management containers that access GPUs through `NVIDIA_VISIBLE_DEVICES`, including the HAMi Device Plugin, must explicitly use `runtimeClassName: nvidia`.
+
+HAMi supports two device-injection paths:
+
+| HAMi mode | Allocation result | Runtime requirement |
+| --- | --- | --- |
+| `envvar` (default) | HAMi writes the allocated GPU UUID to `NVIDIA_VISIBLE_DEVICES`. | On GPU Operator 25.10+, the Pod must use the `nvidia` RuntimeClass. |
+| `cdi-annotations` | HAMi returns a CDI device named `k8s.device-plugin.nvidia.com/gpu=GPU-...` and generates its CDI specification on the node. | The container runtime must have CDI enabled and be able to read the current HAMi specification. |
+
+The HAMi chart applies `devicePlugin.runtimeClassName` both to the Device Plugin and to NVIDIA workloads mutated by the HAMi scheduler. This is why setting it to `nvidia` fixes the management container and keeps the workload runtime path consistent.
+
+For new clusters, GPU Operator is recommended because it provides one entry point for configuring and upgrading the driver, Container Toolkit, and monitoring components. If these components are already installed on the hosts and you maintain their runtime configuration yourself, GPU Operator is optional; follow [Prerequisites](../installation/prerequisites.md).
+
+:::warning
+
+Disable the GPU Operator Device Plugin when using HAMi. Both plugins advertise `nvidia.com/gpu` and must not run on the same nodes.
+
+```yaml
+devicePlugin:
+  enabled: false
+```
+
+:::

i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/local-fake-gpu.md

@@ -242,8 +242,6 @@ echo "NODE_NAME=${NODE_NAME}"
 | macOS | `orbstack`               |
 | Linux | `hami-lab-control-plane` |
 
----
-
 :::info[步骤 2–7 在两个平台上完全相同]
 
 从这里开始，所有命令在 macOS 和 Linux 上完全一致。你唯一会注意到的区别是示例输出中的节点名称。

i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/nvml-mock.md

@@ -138,8 +138,6 @@ Windows 用户请使用 [WSL2](https://learn.microsoft.com/zh-cn/windows/wsl/ins
 
 :::
 
----
-
 ## 步骤 1：创建 kind 集群
 
 ```bash
@@ -159,8 +157,6 @@ echo "NODE_NAME=${NODE_NAME}"
 NODE_NAME=nvml-mock-test-control-plane
 ```
 
----
-
 ## 步骤 2：构建并部署 nvml-mock
 
 nvml-mock 提供一个虚假的 `libnvidia-ml.so`、虚拟的 `/dev/nvidia*` 设备节点以及 PCI 拓扑条目，让 HAMi 的 device-plugin 在节点上看到 8 张 A100 GPU。
@@ -210,8 +206,6 @@ NAME                             GPU_PRESENT
 nvml-mock-test-control-plane     true
 ```
 
----
-
 ## 步骤 3：基于 `main` 分支构建 HAMi
 
 `main` 分支包含一个修复：当未启用 MIG 时，阻止调用 `nvidia-mig-parted`。从源码构建可确保该修复已包含在内，无需等待正式发布版本。
@@ -245,8 +239,6 @@ kind load docker-image hami:local --name nvml-mock-test
 
 调度器和 device-plugin 二进制文件都打包在单个 `hami:local` 镜像中。
 
----
-
 ## 步骤 4：部署 HAMi
 
 ### 4.1 通过 Helm 安装
@@ -332,8 +324,6 @@ hami-scheduler-7858c744cc-7pb79   2/2     Running            0          13m
 
 :::
 
----
-
 ## 步骤 5：验证 GPU 资源
 
 HAMi 将每张物理 GPU 切分成 10 个虚拟槽位。节点有 8 张物理 GPU，因此应该对外提供 **80** 个可分配的虚拟 GPU。
@@ -353,8 +343,6 @@ kubectl describe node ${NODE_NAME} | grep nvidia.com/gpu
 
 `Capacity` 和 `Allocatable` 都显示 `80`，确认 device-plugin 已注册全部虚拟 GPU 槽位。最后一行是 `Allocated resources` 表，当前为 `0`，因为还没有 Pod 申请 GPU。
 
----
-
 ## 步骤 6：测试基础 GPU 调度
 
 部署一个最小化的 Pod，申请一张 GPU。`CUDA_DISABLE_CONTROL=true` 阻止 HAMi 注入的 CUDA shim 尝试真实设备访问：
@@ -406,8 +394,6 @@ hami.io/vgpu-devices-allocated: GPU-12345678-1234-1234-1234-123456780006,NVIDIA,
 
 > 注解格式为 `<UUID>,<厂商>,<显存MiB>,<算力>`。A100 GPU 有 40960 MiB 显存，看到这个注解即确认调度器分配并记录了一个虚拟 GPU。
 
----
-
 ## 步骤 7：测试 GPU 共享（时间片）
 
 再部署三个 Pod，每个申请 1 张 GPU：
@@ -457,8 +443,6 @@ gpu-test-4   1/1     Running   0          9s
 
 四个 Pod 并发运行在 80 个虚拟 GPU 槽位的资源池上。调度器通过各自独立的 `vgpu-devices-allocated` 注解独立跟踪每次分配。
 
----
-
 ## 步骤 8：测试显存和算力限制
 
 ```bash
@@ -485,7 +469,7 @@ EOF
 
 :::info
 
-资源限制格式 `nvidia.com/gpumem` 接受**以 MiB 为单位的绝对值**：`"10"` 表示 10 MiB。`nvidia.com/gpucores: "30"` 表示在所选 GPU 上申请 30 个计算核心。
+资源限制格式 `nvidia.com/gpumem` 接受**以 MiB 为单位的绝对值**：`"10"` 表示 10 MiB。`nvidia.com/gpucores: "30"` 表示在所选 GPU 上申请 30% 的计算算力。
 
 :::
 
@@ -501,9 +485,7 @@ kubectl describe pod gpu-limits | grep vgpu-devices-allocated
 hami.io/vgpu-devices-allocated: GPU-12345678-1234-1234-1234-123456780002,NVIDIA,10,30:;
 ```
 
-注解记录了 `10` MiB 和 `30` 个核心，正是所申请的值。
-
----
+注解记录了 `10` MiB 和 `30`（即 30% 的计算算力），正是所申请的值。
 
 ## 步骤 9：测试百分比显存申请
 
@@ -566,8 +548,6 @@ GPU-12345678-1234-1234-1234-123456780003,NVIDIA,12288,100:;
 
 > 第三个字段显示 `12288` MiB（即 40960 MiB 的 30%），确认调度器正确地将百分比转换为本次分配的绝对显存预算。
 
----
-
 ## 步骤 10：测试多 GPU 分配
 
 ```bash
@@ -637,16 +617,14 @@ kubectl get pod gpu-multi \
 
 :::
 
----
-
 ## 已验证功能总结
 
 | 功能 | 测试 Pod | 如何验证 |
 | --- | --- | --- |
 | 基础 GPU 调度 | `gpu-test-1` | 注解显示 1 个 vGPU UUID + 40960 MiB |
 | GPU 共享（时间片） | `gpu-test-1` 到 `gpu-test-4` | 4 个 Pod 并发运行 |
 | 显存限制（`gpumem`） | `gpu-limits` | 注解显示 `10` MiB |
-| 算力限制（`gpucores`） | `gpu-limits` | 注解显示 `30` 个核心 |
+| 算力限制（`gpucores`） | `gpu-limits` | 注解显示 `30`（30% 的计算算力） |
 | 百分比显存（`gpumem-percentage`） | `gpu-mem-30pct` | 注解显示 `12288` MiB（A100 的 30%） |
 | 多 GPU 分配 | `gpu-multi` | hami-scheduler 事件显示 `BindingSucceed` |
 
@@ -659,8 +637,6 @@ kubectl get pod gpu-multi \
 
 :::
 
----
-
 ## 清理
 
 删除所有测试 Pod：
@@ -700,8 +676,6 @@ kind delete cluster --name nvml-mock-test
 
 :::
 
----
-
 ## 下一步
 
 - 切换到真实 GPU 集群（参见 [实验 1：在线安装 HAMi](/tutorials/labs/online-install)），用真实 CUDA 工作负载测试显存和算力隔离。

i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/online-install.md

@@ -54,8 +54,6 @@ flowchart LR
 
 > 费用提示：`n1-standard-4` + T4 虚拟机约 $0.55/小时。[实验 3](./gpu-partitioning.md) 和[实验 4](./hami-dra.md) 直接复用这套集群，一次开机即可完成全部三个实验。实验结束后请删除虚拟机。
 
----
-
 ## 步骤 1: 创建 GCP 虚拟机
 
 ### 目的
@@ -324,8 +322,6 @@ prometheus-prometheus-node-exporter-xxxxx              1/1     Running   0
 
 > 如果安装失败，需要先卸载再重装：`helm uninstall -n monitoring prometheus`
 
----
-
 ## 步骤 5: 安装 GPU Operator
 
 ### 目的
@@ -396,8 +392,6 @@ kubectl -n gpu-operator exec -it $(kubectl get pods -n gpu-operator -l app=nvidi
 +-----------------------------------------------------------------------------------------+
 ```
 
----
-
 ## 步骤 6: 安装 HAMi
 
 ### 目的

i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/topology-aware-scheduling.md

@@ -203,8 +203,6 @@ NODE_NAME=$(kubectl get nodes -o jsonpath='{.items[0].metadata.name}')
 echo "NODE_NAME=${NODE_NAME}"
 ```
 
----
-
 :::info[后续大部分步骤在两个平台上是一致的]
 
 从这里开始，大多数命令在 macOS 和 Linux 上是一致的。仍有少数步骤存在差异 —— 步骤 2.3 和步骤 3.1 中分别有一条仅 Linux 需要执行的额外 `kind load` 命令 —— 其余部分完全相同，只是示例输出中的节点名不同。