From 389611109cf0a41678ab99566f7411bec5eef17c Mon Sep 17 00:00:00 2001 From: Jifei Wang Date: Tue, 4 Aug 2026 14:43:26 +0800 Subject: [PATCH 1/7] add troubleshoot for GPU-Operator 25.10+ Signed-off-by: Jifei Wang --- docs/faq/faq.md | 2 + docs/troubleshooting/troubleshooting.md | 127 ++++++++++++++++++ .../current/faq/faq.md | 2 + .../troubleshooting/troubleshooting.md | 127 ++++++++++++++++++ 4 files changed, 258 insertions(+) diff --git a/docs/faq/faq.md b/docs/faq/faq.md index ad1a102fa..d94b959ff 100644 --- a/docs/faq/faq.md +++ b/docs/faq/faq.md @@ -223,6 +223,8 @@ devicePlugin: DCGM Exporter is not affected and continues to report physical-level counters normally. HAMi’s per-container virtual metrics are separate; see [GPU Utilization Metrics](../developers/gpu-utilization-metrics). +If the HAMi Device Plugin or workloads fail after upgrading to GPU Operator 25.10+, see [NVIDIA containers fail with GPU Operator 25.10+](../troubleshooting/troubleshooting.md#nvidia-toolkit-gpu-operator-25-10). + ## How do I set up Prometheus and Grafana monitoring for HAMi vGPU metrics? The `hami-device-plugin` pod on each node exposes per-container vGPU metrics on port `31992` (configurable via `devicePlugin.service.httpPort`). See [Grafana Dashboard](../userguide/monitoring/grafana-dashboard) for the full setup including Prometheus scrape config and dashboard import. diff --git a/docs/troubleshooting/troubleshooting.md b/docs/troubleshooting/troubleshooting.md index 882815c22..101509cac 100644 --- a/docs/troubleshooting/troubleshooting.md +++ b/docs/troubleshooting/troubleshooting.md @@ -33,3 +33,130 @@ If a container exceeds its `nvidia.com/gpumem` limit, check the following causes ``` This will apply the fix automatically. + +## NVIDIA containers fail with GPU Operator 25.10+ {#nvidia-toolkit-gpu-operator-25-10} + +Use this section when the HAMi Device Plugin or a HAMi-scheduled NVIDIA workload stops starting after GPU Operator is installed or upgraded. + +### Problem 1: The HAMi Device Plugin fails to start + +#### Identify the cause + +Check the Device Plugin logs: + +```bash +kubectl logs -n kube-system \ + -l app.kubernetes.io/component=hami-device-plugin \ + --all-containers --tail=200 +``` + +Match the output to one of these errors: + +| Error in the log | Cause | +| --- | --- | +| `Incompatible strategy detected auto` | The Device Plugin cannot discover NVML devices because its container did not receive the NVIDIA driver and devices. | +| `invalid device discovery strategy` | The Device Plugin could not initialize NVIDIA device discovery, usually for the same runtime-injection reason. | +| `failed to locate libcuda.so` or `failed to locate libnvidia-ml.so` | HAMi cannot find the driver libraries under the configured driver root while generating a CDI specification. | + +Confirm the runtime and CDI configuration: + +```bash +kubectl get clusterpolicy -o yaml | grep -A 5 'cdi:' +kubectl get runtimeclass nvidia + +kubectl get pods -n kube-system \ + -l app.kubernetes.io/component=hami-device-plugin \ + -o custom-columns=NAME:.metadata.name,RUNTIMECLASS:.spec.runtimeClassName +``` + +With GPU Operator 25.10+, CDI is normally enabled, the `nvidia` RuntimeClass must exist, and the HAMi Device Plugin must show `nvidia` in the `RUNTIMECLASS` column. + +#### Solution + +Configure the `nvidia` RuntimeClass for HAMi and restart the Device Plugin: + +```bash +helm upgrade hami hami-charts/hami \ + --namespace kube-system \ + --reuse-values \ + --set devicePlugin.runtimeClassName=nvidia + +kubectl rollout restart daemonset/hami-device-plugin -n kube-system +kubectl rollout status daemonset/hami-device-plugin -n kube-system +``` + +If the logs report missing driver libraries while HAMi CDI is enabled, use the GPU Operator paths: + +```yaml +devicePlugin: + runtimeClassName: nvidia + deviceListStrategy: cdi-annotations + nvidiaDriverRoot: /run/nvidia/driver + nvidiaHookPath: /usr/local/nvidia/toolkit/nvidia-ctk +``` + +Wait until the NVIDIA driver and Toolkit DaemonSets are ready, then restart the HAMi Device Plugin. For host-installed drivers, set `devicePlugin.nvidiaDriverRoot` to `/` instead. + +### Problem 2: A HAMi-scheduled Pod fails to start + +#### Identify the cause + +Inspect the Pod events and its assigned RuntimeClass: + +```bash +kubectl describe pod -n +kubectl get pod -n \ + -o custom-columns=NAME:.metadata.name,RUNTIMECLASS:.spec.runtimeClassName +``` + +Use the error text to select the correct path: + +| Error in the Pod events | Cause | +| --- | --- | +| `libcuda.so.1: cannot open shared object file` | The container started without the NVIDIA driver libraries. | +| `unresolvable CDI devices management.nvidia.com/gpu=GPU-...` | The NVIDIA runtime selected a GPU management CDI device, but the corresponding device could not be generated or resolved. | +| `unresolvable CDI devices k8s.device-plugin.nvidia.com/gpu=GPU-...` | HAMi returned a CDI device, but the runtime cannot find a matching HAMi-generated CDI specification. | + +#### Solution + +First determine which HAMi injection mode is configured: + +```bash +helm get values hami -n kube-system | grep -A 5 'devicePlugin:' +``` + +- For the default `devicePlugin.deviceListStrategy=envvar` mode, set `devicePlugin.runtimeClassName=nvidia` by using the Helm command from Problem 1. This makes the NVIDIA runtime process the UUID returned through `NVIDIA_VISIBLE_DEVICES`. +- For `devicePlugin.deviceListStrategy=cdi-annotations`, apply all four CDI values shown in Problem 1. Then inspect `/var/run/cdi/k8s.device-plugin.nvidia.com-gpu.json` on the node and verify that it contains the allocated GPU UUID. +- For a host-installed Container Toolkit, confirm that the `nvidia` runtime is present in the active container runtime configuration. Restart the container runtime after correcting its configuration. + +Do not mix HAMi CDI annotations with a missing or stale HAMi CDI specification. See [NVIDIA CDI support](../installation/configure-cdi.md) for the complete setup and verification procedure. + +### Why this happens + +Starting with [GPU Operator 25.10.0](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/25.10/cdi.html), CDI is enabled by default and the Operator no longer makes the `nvidia` runtime the default runtime. + +Before 25.10.0, GPU Operator normally configured the NVIDIA runtime as the default. As a result, the NVIDIA runtime hook processed every Pod and injected the devices and driver libraries selected through `NVIDIA_VISIBLE_DEVICES`. + +With 25.10.0 and later, `runc` remains the default runtime and the container runtime uses native CDI for standard Device Plugin workloads. GPU management containers that access GPUs through `NVIDIA_VISIBLE_DEVICES`, including the HAMi Device Plugin, must explicitly use `runtimeClassName: nvidia`. + +HAMi supports two device-injection paths: + +| HAMi mode | Allocation result | Runtime requirement | +| --- | --- | --- | +| `envvar` (default) | HAMi writes the allocated GPU UUID to `NVIDIA_VISIBLE_DEVICES`. | On GPU Operator 25.10+, the Pod must use the `nvidia` RuntimeClass. | +| `cdi-annotations` | HAMi returns a CDI device named `k8s.device-plugin.nvidia.com/gpu=GPU-...` and generates its CDI specification on the node. | The container runtime must have CDI enabled and be able to read the current HAMi specification. | + +The HAMi chart applies `devicePlugin.runtimeClassName` both to the Device Plugin and to NVIDIA workloads mutated by the HAMi scheduler. This is why setting it to `nvidia` fixes the management container and keeps the workload runtime path consistent. + +For new clusters, we recommend GPU Operator because it provides one entry point for configuring and upgrading the driver, Container Toolkit, and monitoring components. If these components are already installed on the hosts and you maintain their runtime configuration yourself, GPU Operator is optional; follow [Prerequisites](../installation/prerequisites.md). + +:::warning + +Disable the GPU Operator Device Plugin when using HAMi. Both plugins advertise `nvidia.com/gpu` and must not run on the same nodes. + +```yaml +devicePlugin: + enabled: false +``` + +::: diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/faq/faq.md b/i18n/zh/docusaurus-plugin-content-docs/current/faq/faq.md index 244cf28db..029c2fc13 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/faq/faq.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/faq/faq.md @@ -220,6 +220,8 @@ devicePlugin: DCGM Exporter 不受影响,继续正常上报物理级计数器。HAMi 的每容器虚拟指标是独立的;参见 [GPU 利用率指标](../developers/gpu-utilization-metrics)。 +如果升级到 GPU Operator 25.10+ 后 HAMi Device Plugin 或工作负载启动失败,请参见[使用 GPU Operator 25.10+ 时 NVIDIA 容器启动失败](../troubleshooting/troubleshooting.md#nvidia-toolkit-gpu-operator-25-10)。 + ## 如何为 HAMi vGPU 指标设置 Prometheus 和 Grafana 监控? 每个节点上的 `hami-device-plugin` Pod 在端口 `31992`(可通过 `devicePlugin.service.httpPort` 配置)上暴露每容器 vGPU 指标。完整的设置步骤(包括 Prometheus 采集配置和 Dashboard 导入)参见 [Grafana Dashboard](../userguide/monitoring/grafana-dashboard)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/troubleshooting/troubleshooting.md b/i18n/zh/docusaurus-plugin-content-docs/current/troubleshooting/troubleshooting.md index 318d9f221..ebbd2efa8 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/troubleshooting/troubleshooting.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/troubleshooting/troubleshooting.md @@ -25,3 +25,130 @@ translated: true - 从 v2.3.10 起,HAMi 将 `device-plugin` 环境变量名称从 `NodeName` 更改为 `NODE_NAME`。如果你使用早于 v2.3.10 的镜像版本,`device-plugin` 可能无法启动,有两种方法可以解决: - 手动执行 `kubectl edit daemonset` 修改 `device-plugin` 环境变量从 `NodeName` 为 `NODE_NAME`。 - 使用 Helm 升级到最新版本:执行 `helm upgrade hami hami/hami -n kube-system`,它将自动修复。 + +## 使用 GPU Operator 25.10+ 时 NVIDIA 容器启动失败 {#nvidia-toolkit-gpu-operator-25-10} + +安装或升级 GPU Operator 后,如果 HAMi Device Plugin 或由 HAMi 调度的 NVIDIA 工作负载无法启动,请按照本节排查。 + +### 问题一:HAMi Device Plugin 无法启动 + +#### 定位原因 + +查看 Device Plugin 日志: + +```bash +kubectl logs -n kube-system \ + -l app.kubernetes.io/component=hami-device-plugin \ + --all-containers --tail=200 +``` + +根据错误日志定位原因: + +| 错误日志 | 原因 | +| --- | --- | +| `Incompatible strategy detected auto` | Device Plugin 容器没有获得 NVIDIA 驱动和设备,因而无法通过 NVML 发现 GPU。 | +| `invalid device discovery strategy` | Device Plugin 无法初始化 NVIDIA 设备发现,通常也是 runtime 未正确注入设备所致。 | +| `failed to locate libcuda.so` 或 `failed to locate libnvidia-ml.so` | HAMi 生成 CDI spec 时,无法从配置的驱动根目录找到驱动库。 | + +确认 runtime 和 CDI 配置: + +```bash +kubectl get clusterpolicy -o yaml | grep -A 5 'cdi:' +kubectl get runtimeclass nvidia + +kubectl get pods -n kube-system \ + -l app.kubernetes.io/component=hami-device-plugin \ + -o custom-columns=NAME:.metadata.name,RUNTIMECLASS:.spec.runtimeClassName +``` + +使用 GPU Operator 25.10+ 时,CDI 通常处于开启状态,集群中必须存在 `nvidia` RuntimeClass,HAMi Device Plugin 的 `RUNTIMECLASS` 列应显示 `nvidia`。 + +#### 解决方案 + +为 HAMi 配置 `nvidia` RuntimeClass,然后重启 Device Plugin: + +```bash +helm upgrade hami hami-charts/hami \ + --namespace kube-system \ + --reuse-values \ + --set devicePlugin.runtimeClassName=nvidia + +kubectl rollout restart daemonset/hami-device-plugin -n kube-system +kubectl rollout status daemonset/hami-device-plugin -n kube-system +``` + +如果 HAMi 已开启 CDI,且日志显示驱动库缺失,请使用 GPU Operator 对应的路径: + +```yaml +devicePlugin: + runtimeClassName: nvidia + deviceListStrategy: cdi-annotations + nvidiaDriverRoot: /run/nvidia/driver + nvidiaHookPath: /usr/local/nvidia/toolkit/nvidia-ctk +``` + +等待 NVIDIA 驱动和 Toolkit DaemonSet 就绪后,再重启 HAMi Device Plugin。如果驱动直接安装在宿主机,请将 `devicePlugin.nvidiaDriverRoot` 设为 `/`。 + +### 问题二:HAMi 调度的 Pod 无法启动 + +#### 定位原因 + +检查 Pod 事件和分配到的 RuntimeClass: + +```bash +kubectl describe pod -n +kubectl get pod -n \ + -o custom-columns=NAME:.metadata.name,RUNTIMECLASS:.spec.runtimeClassName +``` + +根据错误内容选择对应的排障路径: + +| Pod 事件中的错误 | 原因 | +| --- | --- | +| `libcuda.so.1: cannot open shared object file` | 容器启动时没有注入 NVIDIA 驱动库。 | +| `unresolvable CDI devices management.nvidia.com/gpu=GPU-...` | NVIDIA runtime 选择了 GPU 管理 CDI 设备,但无法生成或解析对应设备。 | +| `unresolvable CDI devices k8s.device-plugin.nvidia.com/gpu=GPU-...` | HAMi 返回了 CDI 设备,但容器运行时找不到匹配的 HAMi CDI spec。 | + +#### 解决方案 + +先确认 HAMi 使用的设备注入模式: + +```bash +helm get values hami -n kube-system | grep -A 5 'devicePlugin:' +``` + +- 默认的 `devicePlugin.deviceListStrategy=envvar` 模式:使用问题一中的 Helm 命令设置 `devicePlugin.runtimeClassName=nvidia`,让 NVIDIA runtime 处理通过 `NVIDIA_VISIBLE_DEVICES` 返回的 UUID。 +- `devicePlugin.deviceListStrategy=cdi-annotations` 模式:同时应用问题一列出的四个 CDI 参数,然后在节点上检查 `/var/run/cdi/k8s.device-plugin.nvidia.com-gpu.json`,确认其中包含分配到的 GPU UUID。 +- Container Toolkit 直接安装在宿主机:确认当前容器运行时配置中存在 `nvidia` runtime;修正配置后重启容器运行时。 + +不要混用 HAMi CDI annotation 与缺失或过期的 HAMi CDI spec。完整配置和验证步骤参见 [NVIDIA CDI 支持](../installation/configure-cdi.md)。 + +### 为什么会出现这个问题 + +从 [GPU Operator 25.10.0](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/25.10/cdi.html) 开始,CDI 默认开启,Operator 不再将 `nvidia` 设为默认 runtime。 + +在 25.10.0 之前,GPU Operator 通常会把 NVIDIA runtime 配置为默认 runtime。因此,每个 Pod 都会经过 NVIDIA runtime hook,并根据 `NVIDIA_VISIBLE_DEVICES` 注入指定的设备和驱动库。 + +从 25.10.0 开始,`runc` 保持为默认 runtime,普通 Device Plugin 工作负载由容器运行时通过原生 CDI 注入设备。通过 `NVIDIA_VISIBLE_DEVICES` 访问 GPU 的管理容器必须显式使用 `runtimeClassName: nvidia`,HAMi Device Plugin 就属于此类管理容器。 + +HAMi 支持两条设备注入路径: + +| HAMi 模式 | 分配结果 | runtime 要求 | +| --- | --- | --- | +| `envvar`(默认) | HAMi 将分配到的 GPU UUID 写入 `NVIDIA_VISIBLE_DEVICES`。 | 使用 GPU Operator 25.10+ 时,Pod 必须使用 `nvidia` RuntimeClass。 | +| `cdi-annotations` | HAMi 返回名为 `k8s.device-plugin.nvidia.com/gpu=GPU-...` 的 CDI 设备,并在节点上生成对应 CDI spec。 | 容器运行时必须开启 CDI,并能够读取当前的 HAMi CDI spec。 | + +HAMi Chart 会把 `devicePlugin.runtimeClassName` 同时应用到 Device Plugin,以及由 HAMi scheduler 修改的 NVIDIA 工作负载。因此,将其设为 `nvidia` 既能修复管理容器,也能保持工作负载的 runtime 路径一致。 + +对于新集群,推荐使用 GPU Operator,因为它提供统一的驱动、Container Toolkit 和监控组件配置与升级入口。如果这些组件已经安装在宿主机,并由你自行维护容器运行时配置,也可以不使用 GPU Operator;这种情况请参考[前置条件](../installation/prerequisites.md)。 + +:::warning + +使用 HAMi 时必须关闭 GPU Operator 自带的 Device Plugin。两者都会注册 `nvidia.com/gpu`,不能在同一节点同时运行。 + +```yaml +devicePlugin: + enabled: false +``` + +::: From 2267dc6c4da73bfc6126a59ff3c7a7a4843f3cd8 Mon Sep 17 00:00:00 2001 From: Jifei Wang Date: Wed, 5 Aug 2026 11:59:31 +0800 Subject: [PATCH 2/7] update Signed-off-by: Jifei Wang --- docs/troubleshooting/troubleshooting.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/troubleshooting/troubleshooting.md b/docs/troubleshooting/troubleshooting.md index 101509cac..5bb7d2e9b 100644 --- a/docs/troubleshooting/troubleshooting.md +++ b/docs/troubleshooting/troubleshooting.md @@ -148,7 +148,7 @@ HAMi supports two device-injection paths: The HAMi chart applies `devicePlugin.runtimeClassName` both to the Device Plugin and to NVIDIA workloads mutated by the HAMi scheduler. This is why setting it to `nvidia` fixes the management container and keeps the workload runtime path consistent. -For new clusters, we recommend GPU Operator because it provides one entry point for configuring and upgrading the driver, Container Toolkit, and monitoring components. If these components are already installed on the hosts and you maintain their runtime configuration yourself, GPU Operator is optional; follow [Prerequisites](../installation/prerequisites.md). +For new clusters, GPU Operator is recommended because it provides one entry point for configuring and upgrading the driver, Container Toolkit, and monitoring components. If these components are already installed on the hosts and you maintain their runtime configuration yourself, GPU Operator is optional; follow [Prerequisites](../installation/prerequisites.md). :::warning From 427ad129d0fc00abef521dd3f78b5454e50815ed Mon Sep 17 00:00:00 2001 From: Jifei Wang Date: Thu, 6 Aug 2026 18:14:26 +0800 Subject: [PATCH 3/7] sync to v2.9.0 Signed-off-by: Jifei Wang --- .../version-v2.9.0/faq/faq.md | 14 +- .../troubleshooting/troubleshooting.md | 133 +++++++++++++++++- versioned_docs/version-v2.9.0/faq/faq.md | 22 +-- .../troubleshooting/troubleshooting.md | 129 ++++++++++++++++- 4 files changed, 278 insertions(+), 20 deletions(-) diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/faq/faq.md b/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/faq/faq.md index 261d08363..029c2fc13 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/faq/faq.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/faq/faq.md @@ -175,7 +175,7 @@ HAMi 的调度器需要从节点获取足够的 GPU 信息来完成资源调度 ## HAMi 如何强制执行 GPU 显存和算力限制? -HAMi 通过 `/etc/ld.so.preload` 将 `libvgpu.so` 注入容器。该库拦截 CUDA 显存分配调用,当超过 `nvidia.com/gpumem` 限制时返回 OOM;算力限制通过令牌桶算法对 kernel launch 调用进行节流。绕过 CUDA 库的应用(如 Docker-in-Docker、直接调用驱动 API)不受管控。完整的拦截流程参见 [GPU 虚拟化](./core-concepts/gpu-virtualization)。 +HAMi 通过 `/etc/ld.so.preload` 将 `libvgpu.so` 注入容器。该库拦截 CUDA 显存分配调用,当超过 `nvidia.com/gpumem` 限制时返回 OOM;算力限制通过令牌桶算法对 kernel launch 调用进行节流。绕过 CUDA 库的应用(如 Docker-in-Docker、直接调用驱动 API)不受管控。完整的拦截流程参见 [GPU 虚拟化](../core-concepts/gpu-virtualization)。 ## HAMi vGPU 与 NVIDIA MIG 有何区别?各适用于什么场景? @@ -191,11 +191,11 @@ HAMi vGPU 是纯软件方案,无硬件要求。NVIDIA MIG 是硬件分区, | 动态重配置 | 支持,无需排空节点 | 需要重新配置 MIG 配置文件 | | 多租户噪声隔离 | 尽力而为 | 强隔离 | -当 GPU 不支持 MIG、工作负载需要灵活的显存大小、或需要无需排空节点的动态重打包时,使用 HAMi vGPU。当硬隔离是合规或 SLA 要求时,使用 MIG。HAMi 也通过 `mig-parted` 支持动态 MIG;参见[动态 MIG 支持](./userguide/nvidia-device/dynamic-mig-support)。 +当 GPU 不支持 MIG、工作负载需要灵活的显存大小、或需要无需排空节点的动态重打包时,使用 HAMi vGPU。当硬隔离是合规或 SLA 要求时,使用 MIG。HAMi 也通过 `mig-parted` 支持动态 MIG;参见[动态 MIG 支持](../userguide/nvidia-device/dynamic-mig-support)。 ## 为什么容器内 nvidia-smi 显示的显存比宿主机少? -`libvgpu.so` 拦截了 `nvmlDeviceGetMemoryInfo` 及相关调用,返回 `nvidia.com/gpumem` 限制值而非物理显存。这是预期行为:根据上报显存大小进行分配的工作负载(如 vLLM)将只使用其配额。宿主机的 `nvidia-smi` 始终显示物理显存。参见 [GPU 虚拟化](./core-concepts/gpu-virtualization)。 +`libvgpu.so` 拦截了 `nvmlDeviceGetMemoryInfo` 及相关调用,返回 `nvidia.com/gpumem` 限制值而非物理显存。这是预期行为:根据上报显存大小进行分配的工作负载(如 vLLM)将只使用其配额。宿主机的 `nvidia-smi` 始终显示物理显存。参见 [GPU 虚拟化](../core-concepts/gpu-virtualization)。 ## 为什么 nvidia.com/gpumem 限制未生效? {#why-is-my-nvidiagpumem-limit-not-enforced} @@ -203,7 +203,7 @@ HAMi vGPU 是纯软件方案,无硬件要求。NVIDIA MIG 是硬件分区, ## HAMi 是替换 kube-scheduler 还是与其并行运行? -HAMi 作为 [scheduler extender](https://github.com/kubernetes/design-proposals-archive/blob/main/scheduling/scheduler_extender.md) 与 kube-scheduler 并行运行,不会替换它。MutatingWebhook 仅在请求 HAMi 资源的 Pod 上设置 `schedulerName: hami-scheduler`;所有其他 Pod 仍走默认调度器路径。参见[架构](./core-concepts/architecture)。 +HAMi 作为 [scheduler extender](https://github.com/kubernetes/design-proposals-archive/blob/main/scheduling/scheduler_extender.md) 与 kube-scheduler 并行运行,不会替换它。MutatingWebhook 仅在请求 HAMi 资源的 Pod 上设置 `schedulerName: hami-scheduler`;所有其他 Pod 仍走默认调度器路径。参见[架构](../core-concepts/architecture)。 ## HAMi 是否支持 vLLM?多 GPU 张量并行有哪些已知限制? @@ -218,8 +218,10 @@ devicePlugin: enabled: false ``` -DCGM Exporter 不受影响,继续正常上报物理级计数器。HAMi 的每容器虚拟指标是独立的;参见 [GPU 利用率指标](./developers/gpu-utilization-metrics)。 +DCGM Exporter 不受影响,继续正常上报物理级计数器。HAMi 的每容器虚拟指标是独立的;参见 [GPU 利用率指标](../developers/gpu-utilization-metrics)。 + +如果升级到 GPU Operator 25.10+ 后 HAMi Device Plugin 或工作负载启动失败,请参见[使用 GPU Operator 25.10+ 时 NVIDIA 容器启动失败](../troubleshooting/troubleshooting.md#nvidia-toolkit-gpu-operator-25-10)。 ## 如何为 HAMi vGPU 指标设置 Prometheus 和 Grafana 监控? -每个节点上的 `hami-device-plugin` Pod 在端口 `31992`(可通过 `devicePlugin.monitorPort` 配置)上暴露每容器 vGPU 指标。完整的设置步骤(包括 Prometheus 采集配置和 Dashboard 导入)参见 [Grafana Dashboard](./userguide/monitoring/grafana-dashboard)。 +每个节点上的 `hami-device-plugin` Pod 在端口 `31992`(可通过 `devicePlugin.service.httpPort` 配置)上暴露每容器 vGPU 指标。完整的设置步骤(包括 Prometheus 采集配置和 Dashboard 导入)参见 [Grafana Dashboard](../userguide/monitoring/grafana-dashboard)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/troubleshooting/troubleshooting.md b/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/troubleshooting/troubleshooting.md index 4b581fa88..ebbd2efa8 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/troubleshooting/troubleshooting.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/troubleshooting/troubleshooting.md @@ -16,12 +16,139 @@ translated: true containerd config dump | grep default_runtime_name ``` - 输出必须显示 `nvidia`。如未显示,请按照[前置条件](./installation/online-installation)指南操作。 + 输出必须显示 `nvidia`。如未显示,请按照[前置条件](../installation/online-installation)指南操作。 - 如果在使用 NVIDIA 镜像的设备插件时不请求 vGPU,机器上的所有 GPU 可能会在容器内暴露。 - 目前,A100 MIG 仅支持 "none" 和 "mixed" 模式。 - 目前无法调度带有 "nodeName" 字段的任务;请改用 "nodeSelector"。 - 目前仅支持计算任务;不支持视频编解码处理。 -- 我们将 `device-plugin` 环境变量名称从 `NodeName` 更改为 `NODE_NAME`,如果你使用镜像版本 `v2.3.9`,可能会遇到 `device-plugin` 无法启动的情况,有两种方法可以解决: +- 从 v2.3.10 起,HAMi 将 `device-plugin` 环境变量名称从 `NodeName` 更改为 `NODE_NAME`。如果你使用早于 v2.3.10 的镜像版本,`device-plugin` 可能无法启动,有两种方法可以解决: - 手动执行 `kubectl edit daemonset` 修改 `device-plugin` 环境变量从 `NodeName` 为 `NODE_NAME`。 - - 使用 helm 升级到最新版本,`device-plugin` 镜像的最新版本是 `v2.3.10`,执行 `helm upgrade hami hami/hami -n kube-system`,它将自动修复。 + - 使用 Helm 升级到最新版本:执行 `helm upgrade hami hami/hami -n kube-system`,它将自动修复。 + +## 使用 GPU Operator 25.10+ 时 NVIDIA 容器启动失败 {#nvidia-toolkit-gpu-operator-25-10} + +安装或升级 GPU Operator 后,如果 HAMi Device Plugin 或由 HAMi 调度的 NVIDIA 工作负载无法启动,请按照本节排查。 + +### 问题一:HAMi Device Plugin 无法启动 + +#### 定位原因 + +查看 Device Plugin 日志: + +```bash +kubectl logs -n kube-system \ + -l app.kubernetes.io/component=hami-device-plugin \ + --all-containers --tail=200 +``` + +根据错误日志定位原因: + +| 错误日志 | 原因 | +| --- | --- | +| `Incompatible strategy detected auto` | Device Plugin 容器没有获得 NVIDIA 驱动和设备,因而无法通过 NVML 发现 GPU。 | +| `invalid device discovery strategy` | Device Plugin 无法初始化 NVIDIA 设备发现,通常也是 runtime 未正确注入设备所致。 | +| `failed to locate libcuda.so` 或 `failed to locate libnvidia-ml.so` | HAMi 生成 CDI spec 时,无法从配置的驱动根目录找到驱动库。 | + +确认 runtime 和 CDI 配置: + +```bash +kubectl get clusterpolicy -o yaml | grep -A 5 'cdi:' +kubectl get runtimeclass nvidia + +kubectl get pods -n kube-system \ + -l app.kubernetes.io/component=hami-device-plugin \ + -o custom-columns=NAME:.metadata.name,RUNTIMECLASS:.spec.runtimeClassName +``` + +使用 GPU Operator 25.10+ 时,CDI 通常处于开启状态,集群中必须存在 `nvidia` RuntimeClass,HAMi Device Plugin 的 `RUNTIMECLASS` 列应显示 `nvidia`。 + +#### 解决方案 + +为 HAMi 配置 `nvidia` RuntimeClass,然后重启 Device Plugin: + +```bash +helm upgrade hami hami-charts/hami \ + --namespace kube-system \ + --reuse-values \ + --set devicePlugin.runtimeClassName=nvidia + +kubectl rollout restart daemonset/hami-device-plugin -n kube-system +kubectl rollout status daemonset/hami-device-plugin -n kube-system +``` + +如果 HAMi 已开启 CDI,且日志显示驱动库缺失,请使用 GPU Operator 对应的路径: + +```yaml +devicePlugin: + runtimeClassName: nvidia + deviceListStrategy: cdi-annotations + nvidiaDriverRoot: /run/nvidia/driver + nvidiaHookPath: /usr/local/nvidia/toolkit/nvidia-ctk +``` + +等待 NVIDIA 驱动和 Toolkit DaemonSet 就绪后,再重启 HAMi Device Plugin。如果驱动直接安装在宿主机,请将 `devicePlugin.nvidiaDriverRoot` 设为 `/`。 + +### 问题二:HAMi 调度的 Pod 无法启动 + +#### 定位原因 + +检查 Pod 事件和分配到的 RuntimeClass: + +```bash +kubectl describe pod -n +kubectl get pod -n \ + -o custom-columns=NAME:.metadata.name,RUNTIMECLASS:.spec.runtimeClassName +``` + +根据错误内容选择对应的排障路径: + +| Pod 事件中的错误 | 原因 | +| --- | --- | +| `libcuda.so.1: cannot open shared object file` | 容器启动时没有注入 NVIDIA 驱动库。 | +| `unresolvable CDI devices management.nvidia.com/gpu=GPU-...` | NVIDIA runtime 选择了 GPU 管理 CDI 设备,但无法生成或解析对应设备。 | +| `unresolvable CDI devices k8s.device-plugin.nvidia.com/gpu=GPU-...` | HAMi 返回了 CDI 设备,但容器运行时找不到匹配的 HAMi CDI spec。 | + +#### 解决方案 + +先确认 HAMi 使用的设备注入模式: + +```bash +helm get values hami -n kube-system | grep -A 5 'devicePlugin:' +``` + +- 默认的 `devicePlugin.deviceListStrategy=envvar` 模式:使用问题一中的 Helm 命令设置 `devicePlugin.runtimeClassName=nvidia`,让 NVIDIA runtime 处理通过 `NVIDIA_VISIBLE_DEVICES` 返回的 UUID。 +- `devicePlugin.deviceListStrategy=cdi-annotations` 模式:同时应用问题一列出的四个 CDI 参数,然后在节点上检查 `/var/run/cdi/k8s.device-plugin.nvidia.com-gpu.json`,确认其中包含分配到的 GPU UUID。 +- Container Toolkit 直接安装在宿主机:确认当前容器运行时配置中存在 `nvidia` runtime;修正配置后重启容器运行时。 + +不要混用 HAMi CDI annotation 与缺失或过期的 HAMi CDI spec。完整配置和验证步骤参见 [NVIDIA CDI 支持](../installation/configure-cdi.md)。 + +### 为什么会出现这个问题 + +从 [GPU Operator 25.10.0](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/25.10/cdi.html) 开始,CDI 默认开启,Operator 不再将 `nvidia` 设为默认 runtime。 + +在 25.10.0 之前,GPU Operator 通常会把 NVIDIA runtime 配置为默认 runtime。因此,每个 Pod 都会经过 NVIDIA runtime hook,并根据 `NVIDIA_VISIBLE_DEVICES` 注入指定的设备和驱动库。 + +从 25.10.0 开始,`runc` 保持为默认 runtime,普通 Device Plugin 工作负载由容器运行时通过原生 CDI 注入设备。通过 `NVIDIA_VISIBLE_DEVICES` 访问 GPU 的管理容器必须显式使用 `runtimeClassName: nvidia`,HAMi Device Plugin 就属于此类管理容器。 + +HAMi 支持两条设备注入路径: + +| HAMi 模式 | 分配结果 | runtime 要求 | +| --- | --- | --- | +| `envvar`(默认) | HAMi 将分配到的 GPU UUID 写入 `NVIDIA_VISIBLE_DEVICES`。 | 使用 GPU Operator 25.10+ 时,Pod 必须使用 `nvidia` RuntimeClass。 | +| `cdi-annotations` | HAMi 返回名为 `k8s.device-plugin.nvidia.com/gpu=GPU-...` 的 CDI 设备,并在节点上生成对应 CDI spec。 | 容器运行时必须开启 CDI,并能够读取当前的 HAMi CDI spec。 | + +HAMi Chart 会把 `devicePlugin.runtimeClassName` 同时应用到 Device Plugin,以及由 HAMi scheduler 修改的 NVIDIA 工作负载。因此,将其设为 `nvidia` 既能修复管理容器,也能保持工作负载的 runtime 路径一致。 + +对于新集群,推荐使用 GPU Operator,因为它提供统一的驱动、Container Toolkit 和监控组件配置与升级入口。如果这些组件已经安装在宿主机,并由你自行维护容器运行时配置,也可以不使用 GPU Operator;这种情况请参考[前置条件](../installation/prerequisites.md)。 + +:::warning + +使用 HAMi 时必须关闭 GPU Operator 自带的 Device Plugin。两者都会注册 `nvidia.com/gpu`,不能在同一节点同时运行。 + +```yaml +devicePlugin: + enabled: false +``` + +::: diff --git a/versioned_docs/version-v2.9.0/faq/faq.md b/versioned_docs/version-v2.9.0/faq/faq.md index fa4d842dc..d94b959ff 100644 --- a/versioned_docs/version-v2.9.0/faq/faq.md +++ b/versioned_docs/version-v2.9.0/faq/faq.md @@ -7,11 +7,11 @@ title: FAQ | **GPU Vendor** | **GPU Model** | **Granularity** | **Multi-GPU Support** | | --- | --- | --- | --- | | NVIDIA | Almost all mainstream consumer and data center GPUs | Core 1%, Memory 1M | Supported. Multi-GPU can still be split and shared using virtualization. | -| Ascend | 910A, 910B2, 910B3, 310P | Minimum granularity depends on the card type template. Refer to the [official templates](https://www.hiascend.com/document/detail/zh/mindx-dl/50rc1/AVI/cpaug/cpaug_0005.html). | Supported, but splitting is not supported when `npu > 1`. The entire card is exclusively allocated. | +| Huawei Ascend | 910A, 910B2, 910B3, 310P | Minimum granularity depends on the card type template. Refer to the [official templates](https://www.hiascend.com/document/detail/zh/mindx-dl/50rc1/AVI/cpaug/cpaug_0005.html). | Supported, but splitting is not supported when `npu > 1`. The entire card is exclusively allocated. | | Hygon | All | Core 1%, Memory 1M | Supported, but splitting is not supported when `dcu > 1`. The entire card is exclusively allocated. | | Cambricon | 370, 590 | Core 1%, Memory 256M | Supported, but splitting is not supported when `mlu > 1`. The entire card is exclusively allocated. | | Iluvatar | All | Core 1%, Memory 256M | Supported, but splitting is not supported when `gpu > 1`. The entire card is exclusively allocated. | -| Mthreads | MTT S4000 | Core 1 core group, Memory 512M | Supported, but splitting is not supported when `gpu > 1`. The entire card is exclusively allocated. | +| Moore Threads | MTT S4000 | Core 1 core group, Memory 512M | Supported, but splitting is not supported when `gpu > 1`. The entire card is exclusively allocated. | | MetaX | MXC500 | Does not support splitting, only whole card allocation is possible. | Supported, but all allocations are for whole cards. | ## What is vGPU? Why cannot I allocate two vGPUs on the same card despite seeing 10 vGPUs? @@ -157,7 +157,7 @@ Device Plugins can only report a single resource type. GPU memory and compute in ## Why don’t some domestic vendors require a runtime for installation? -Certain domestic vendors (e.g., Hygon, Cambricon) do not require a runtime because their DevicePlugin handles device discovery and mounting directly. In contrast, vendors like NVIDIA and Ascend rely on runtimes for environment configuration, device node mounting, and advanced functionality support. +Certain domestic vendors (e.g., Hygon, Cambricon) do not require a runtime because their DevicePlugin handles device discovery and mounting directly. In contrast, vendors like NVIDIA and Huawei Ascend rely on runtimes for environment configuration, device node mounting, and advanced functionality support. ### TL;DR @@ -173,12 +173,12 @@ HAMi's scheduler requires sufficient information from the Node to decode the cor If the official Device Plugin cannot provide the required information, HAMi develops its own. For example: -- Ascend’s official Device Plugin requires a separate plugin for each card type. HAMi abstracts these card templates into a unified plugin for easier integration with the scheduler. +- Huawei Ascend’s official Device Plugin requires a separate plugin for each card type. HAMi abstracts these card templates into a unified plugin for easier integration with the scheduler. - NVIDIA requires custom implementations to support advanced features like compute and memory limits, overcommitment, and NUMA awareness, necessitating HAMi’s custom Device Plugin. ## How does HAMi enforce GPU memory and compute limits? -HAMi injects `libvgpu.so` into containers via `/etc/ld.so.preload`. The library intercepts CUDA memory allocation calls and returns OOM when the `nvidia.com/gpumem` limit is exceeded; compute limits use a token-bucket throttle on kernel launch calls. Applications that bypass the CUDA library (Docker-in-Docker, direct driver API) are not covered. For the full interception flow, see [GPU Virtualization](./core-concepts/gpu-virtualization). +HAMi injects `libvgpu.so` into containers via `/etc/ld.so.preload`. The library intercepts CUDA memory allocation calls and returns OOM when the `nvidia.com/gpumem` limit is exceeded; compute limits use a token-bucket throttle on kernel launch calls. Applications that bypass the CUDA library (Docker-in-Docker, direct driver API) are not covered. For the full interception flow, see [GPU Virtualization](../core-concepts/gpu-virtualization). ## How does HAMi vGPU differ from NVIDIA MIG? When should I use each? @@ -194,11 +194,11 @@ HAMi vGPU is software-only with no hardware requirements. NVIDIA MIG is hardware | Dynamic reconfiguration | Yes, no node drain needed | Requires MIG profile reconfiguration | | Multi-tenant noise isolation | Best-effort | Strong | -Use HAMi vGPU when the GPU does not support MIG, workloads need flexible memory sizes, or dynamic repacking without node drains is needed. Use MIG when hard hardware isolation is a compliance or SLA requirement. HAMi also supports dynamic MIG via `mig-parted`; see [Dynamic MIG Support](./userguide/nvidia-device/dynamic-mig-support). +Use HAMi vGPU when the GPU does not support MIG, workloads need flexible memory sizes, or dynamic repacking without node drains is needed. Use MIG when hard hardware isolation is a compliance or SLA requirement. HAMi also supports dynamic MIG via `mig-parted`; see [Dynamic MIG Support](../userguide/nvidia-device/dynamic-mig-support). ## Why does nvidia-smi inside my container show less memory than on the host? -`libvgpu.so` intercepts `nvmlDeviceGetMemoryInfo` and related calls, returning the `nvidia.com/gpumem` limit instead of physical VRAM. This is intentional: workloads that size their allocations based on reported memory (such as vLLM) will use only their budget. The host’s `nvidia-smi` always shows physical memory. See [GPU Virtualization](./core-concepts/gpu-virtualization). +`libvgpu.so` intercepts `nvmlDeviceGetMemoryInfo` and related calls, returning the `nvidia.com/gpumem` limit instead of physical VRAM. This is intentional: workloads that size their allocations based on reported memory (such as vLLM) will use only their budget. The host’s `nvidia-smi` always shows physical memory. See [GPU Virtualization](../core-concepts/gpu-virtualization). ## Why is my nvidia.com/gpumem limit not enforced? {#why-is-my-nvidiagpumem-limit-not-enforced} @@ -206,7 +206,7 @@ The four most common causes: `CUDA_DISABLE_CONTROL=true` is set, the workload ru ## Does HAMi replace kube-scheduler or run alongside it? -HAMi runs alongside kube-scheduler as a [scheduler extender](https://github.com/kubernetes/design-proposals-archive/blob/main/scheduling/scheduler_extender.md) - it does not replace it. The MutatingWebhook sets `schedulerName: hami-scheduler` only on pods requesting HAMi resources; all other pods follow the default scheduler path unchanged. See [Architecture](./core-concepts/architecture). +HAMi runs alongside kube-scheduler as a [scheduler extender](https://github.com/kubernetes/design-proposals-archive/blob/main/scheduling/scheduler_extender.md) - it does not replace it. The MutatingWebhook sets `schedulerName: hami-scheduler` only on pods requesting HAMi resources; all other pods follow the default scheduler path unchanged. See [Architecture](../core-concepts/architecture). ## Does HAMi work with vLLM, and what are the known limitations for multi-GPU tensor parallelism? @@ -221,8 +221,10 @@ devicePlugin: enabled: false ``` -DCGM Exporter is not affected and continues to report physical-level counters normally. HAMi’s per-container virtual metrics are separate; see [GPU Utilization Metrics](./developers/gpu-utilization-metrics). +DCGM Exporter is not affected and continues to report physical-level counters normally. HAMi’s per-container virtual metrics are separate; see [GPU Utilization Metrics](../developers/gpu-utilization-metrics). + +If the HAMi Device Plugin or workloads fail after upgrading to GPU Operator 25.10+, see [NVIDIA containers fail with GPU Operator 25.10+](../troubleshooting/troubleshooting.md#nvidia-toolkit-gpu-operator-25-10). ## How do I set up Prometheus and Grafana monitoring for HAMi vGPU metrics? -The `hami-device-plugin` pod on each node exposes per-container vGPU metrics on port `31992` (configurable via `devicePlugin.monitorPort`). See [Grafana Dashboard](./userguide/monitoring/grafana-dashboard) for the full setup including Prometheus scrape config and dashboard import. +The `hami-device-plugin` pod on each node exposes per-container vGPU metrics on port `31992` (configurable via `devicePlugin.service.httpPort`). See [Grafana Dashboard](../userguide/monitoring/grafana-dashboard) for the full setup including Prometheus scrape config and dashboard import. diff --git a/versioned_docs/version-v2.9.0/troubleshooting/troubleshooting.md b/versioned_docs/version-v2.9.0/troubleshooting/troubleshooting.md index 115d5f38c..5bb7d2e9b 100644 --- a/versioned_docs/version-v2.9.0/troubleshooting/troubleshooting.md +++ b/versioned_docs/version-v2.9.0/troubleshooting/troubleshooting.md @@ -15,7 +15,7 @@ If a container exceeds its `nvidia.com/gpumem` limit, check the following causes containerd config dump | grep default_runtime_name ``` - The output must show `nvidia`. If not, follow the [Prerequisites](./installation/online-installation) guide. + The output must show `nvidia`. If not, follow the [Prerequisites](../installation/online-installation) guide. - If you don’t explicitly request vGPUs when using the device plugin with NVIDIA images, all GPUs on the host may be exposed to your container. - Currently, A100 MIG can be supported in only "none" and "mixed" modes. @@ -33,3 +33,130 @@ If a container exceeds its `nvidia.com/gpumem` limit, check the following causes ``` This will apply the fix automatically. + +## NVIDIA containers fail with GPU Operator 25.10+ {#nvidia-toolkit-gpu-operator-25-10} + +Use this section when the HAMi Device Plugin or a HAMi-scheduled NVIDIA workload stops starting after GPU Operator is installed or upgraded. + +### Problem 1: The HAMi Device Plugin fails to start + +#### Identify the cause + +Check the Device Plugin logs: + +```bash +kubectl logs -n kube-system \ + -l app.kubernetes.io/component=hami-device-plugin \ + --all-containers --tail=200 +``` + +Match the output to one of these errors: + +| Error in the log | Cause | +| --- | --- | +| `Incompatible strategy detected auto` | The Device Plugin cannot discover NVML devices because its container did not receive the NVIDIA driver and devices. | +| `invalid device discovery strategy` | The Device Plugin could not initialize NVIDIA device discovery, usually for the same runtime-injection reason. | +| `failed to locate libcuda.so` or `failed to locate libnvidia-ml.so` | HAMi cannot find the driver libraries under the configured driver root while generating a CDI specification. | + +Confirm the runtime and CDI configuration: + +```bash +kubectl get clusterpolicy -o yaml | grep -A 5 'cdi:' +kubectl get runtimeclass nvidia + +kubectl get pods -n kube-system \ + -l app.kubernetes.io/component=hami-device-plugin \ + -o custom-columns=NAME:.metadata.name,RUNTIMECLASS:.spec.runtimeClassName +``` + +With GPU Operator 25.10+, CDI is normally enabled, the `nvidia` RuntimeClass must exist, and the HAMi Device Plugin must show `nvidia` in the `RUNTIMECLASS` column. + +#### Solution + +Configure the `nvidia` RuntimeClass for HAMi and restart the Device Plugin: + +```bash +helm upgrade hami hami-charts/hami \ + --namespace kube-system \ + --reuse-values \ + --set devicePlugin.runtimeClassName=nvidia + +kubectl rollout restart daemonset/hami-device-plugin -n kube-system +kubectl rollout status daemonset/hami-device-plugin -n kube-system +``` + +If the logs report missing driver libraries while HAMi CDI is enabled, use the GPU Operator paths: + +```yaml +devicePlugin: + runtimeClassName: nvidia + deviceListStrategy: cdi-annotations + nvidiaDriverRoot: /run/nvidia/driver + nvidiaHookPath: /usr/local/nvidia/toolkit/nvidia-ctk +``` + +Wait until the NVIDIA driver and Toolkit DaemonSets are ready, then restart the HAMi Device Plugin. For host-installed drivers, set `devicePlugin.nvidiaDriverRoot` to `/` instead. + +### Problem 2: A HAMi-scheduled Pod fails to start + +#### Identify the cause + +Inspect the Pod events and its assigned RuntimeClass: + +```bash +kubectl describe pod -n +kubectl get pod -n \ + -o custom-columns=NAME:.metadata.name,RUNTIMECLASS:.spec.runtimeClassName +``` + +Use the error text to select the correct path: + +| Error in the Pod events | Cause | +| --- | --- | +| `libcuda.so.1: cannot open shared object file` | The container started without the NVIDIA driver libraries. | +| `unresolvable CDI devices management.nvidia.com/gpu=GPU-...` | The NVIDIA runtime selected a GPU management CDI device, but the corresponding device could not be generated or resolved. | +| `unresolvable CDI devices k8s.device-plugin.nvidia.com/gpu=GPU-...` | HAMi returned a CDI device, but the runtime cannot find a matching HAMi-generated CDI specification. | + +#### Solution + +First determine which HAMi injection mode is configured: + +```bash +helm get values hami -n kube-system | grep -A 5 'devicePlugin:' +``` + +- For the default `devicePlugin.deviceListStrategy=envvar` mode, set `devicePlugin.runtimeClassName=nvidia` by using the Helm command from Problem 1. This makes the NVIDIA runtime process the UUID returned through `NVIDIA_VISIBLE_DEVICES`. +- For `devicePlugin.deviceListStrategy=cdi-annotations`, apply all four CDI values shown in Problem 1. Then inspect `/var/run/cdi/k8s.device-plugin.nvidia.com-gpu.json` on the node and verify that it contains the allocated GPU UUID. +- For a host-installed Container Toolkit, confirm that the `nvidia` runtime is present in the active container runtime configuration. Restart the container runtime after correcting its configuration. + +Do not mix HAMi CDI annotations with a missing or stale HAMi CDI specification. See [NVIDIA CDI support](../installation/configure-cdi.md) for the complete setup and verification procedure. + +### Why this happens + +Starting with [GPU Operator 25.10.0](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/25.10/cdi.html), CDI is enabled by default and the Operator no longer makes the `nvidia` runtime the default runtime. + +Before 25.10.0, GPU Operator normally configured the NVIDIA runtime as the default. As a result, the NVIDIA runtime hook processed every Pod and injected the devices and driver libraries selected through `NVIDIA_VISIBLE_DEVICES`. + +With 25.10.0 and later, `runc` remains the default runtime and the container runtime uses native CDI for standard Device Plugin workloads. GPU management containers that access GPUs through `NVIDIA_VISIBLE_DEVICES`, including the HAMi Device Plugin, must explicitly use `runtimeClassName: nvidia`. + +HAMi supports two device-injection paths: + +| HAMi mode | Allocation result | Runtime requirement | +| --- | --- | --- | +| `envvar` (default) | HAMi writes the allocated GPU UUID to `NVIDIA_VISIBLE_DEVICES`. | On GPU Operator 25.10+, the Pod must use the `nvidia` RuntimeClass. | +| `cdi-annotations` | HAMi returns a CDI device named `k8s.device-plugin.nvidia.com/gpu=GPU-...` and generates its CDI specification on the node. | The container runtime must have CDI enabled and be able to read the current HAMi specification. | + +The HAMi chart applies `devicePlugin.runtimeClassName` both to the Device Plugin and to NVIDIA workloads mutated by the HAMi scheduler. This is why setting it to `nvidia` fixes the management container and keeps the workload runtime path consistent. + +For new clusters, GPU Operator is recommended because it provides one entry point for configuring and upgrading the driver, Container Toolkit, and monitoring components. If these components are already installed on the hosts and you maintain their runtime configuration yourself, GPU Operator is optional; follow [Prerequisites](../installation/prerequisites.md). + +:::warning + +Disable the GPU Operator Device Plugin when using HAMi. Both plugins advertise `nvidia.com/gpu` and must not run on the same nodes. + +```yaml +devicePlugin: + enabled: false +``` + +::: From d1fa2419220df2ebba766c607e15789c9f1498a1 Mon Sep 17 00:00:00 2001 From: Jifei Wang Date: Fri, 7 Aug 2026 13:27:33 +0800 Subject: [PATCH 4/7] fix links Signed-off-by: Jifei Wang --- docs/faq/faq.md | 12 ++++++------ docs/troubleshooting/troubleshooting.md | 2 +- .../current/faq/faq.md | 12 ++++++------ .../current/troubleshooting/troubleshooting.md | 2 +- .../version-v2.9.0/faq/faq.md | 12 ++++++------ .../troubleshooting/troubleshooting.md | 2 +- versioned_docs/version-v2.9.0/faq/faq.md | 12 ++++++------ .../troubleshooting/troubleshooting.md | 2 +- 8 files changed, 28 insertions(+), 28 deletions(-) diff --git a/docs/faq/faq.md b/docs/faq/faq.md index d94b959ff..69cf5654e 100644 --- a/docs/faq/faq.md +++ b/docs/faq/faq.md @@ -178,7 +178,7 @@ If the official Device Plugin cannot provide the required information, HAMi deve ## How does HAMi enforce GPU memory and compute limits? -HAMi injects `libvgpu.so` into containers via `/etc/ld.so.preload`. The library intercepts CUDA memory allocation calls and returns OOM when the `nvidia.com/gpumem` limit is exceeded; compute limits use a token-bucket throttle on kernel launch calls. Applications that bypass the CUDA library (Docker-in-Docker, direct driver API) are not covered. For the full interception flow, see [GPU Virtualization](../core-concepts/gpu-virtualization). +HAMi injects `libvgpu.so` into containers via `/etc/ld.so.preload`. The library intercepts CUDA memory allocation calls and returns OOM when the `nvidia.com/gpumem` limit is exceeded; compute limits use a token-bucket throttle on kernel launch calls. Applications that bypass the CUDA library (Docker-in-Docker, direct driver API) are not covered. For the full interception flow, see [GPU Virtualization](./core-concepts/gpu-virtualization). ## How does HAMi vGPU differ from NVIDIA MIG? When should I use each? @@ -194,11 +194,11 @@ HAMi vGPU is software-only with no hardware requirements. NVIDIA MIG is hardware | Dynamic reconfiguration | Yes, no node drain needed | Requires MIG profile reconfiguration | | Multi-tenant noise isolation | Best-effort | Strong | -Use HAMi vGPU when the GPU does not support MIG, workloads need flexible memory sizes, or dynamic repacking without node drains is needed. Use MIG when hard hardware isolation is a compliance or SLA requirement. HAMi also supports dynamic MIG via `mig-parted`; see [Dynamic MIG Support](../userguide/nvidia-device/dynamic-mig-support). +Use HAMi vGPU when the GPU does not support MIG, workloads need flexible memory sizes, or dynamic repacking without node drains is needed. Use MIG when hard hardware isolation is a compliance or SLA requirement. HAMi also supports dynamic MIG via `mig-parted`; see [Dynamic MIG Support](./userguide/nvidia-device/dynamic-mig-support). ## Why does nvidia-smi inside my container show less memory than on the host? -`libvgpu.so` intercepts `nvmlDeviceGetMemoryInfo` and related calls, returning the `nvidia.com/gpumem` limit instead of physical VRAM. This is intentional: workloads that size their allocations based on reported memory (such as vLLM) will use only their budget. The host’s `nvidia-smi` always shows physical memory. See [GPU Virtualization](../core-concepts/gpu-virtualization). +`libvgpu.so` intercepts `nvmlDeviceGetMemoryInfo` and related calls, returning the `nvidia.com/gpumem` limit instead of physical VRAM. This is intentional: workloads that size their allocations based on reported memory (such as vLLM) will use only their budget. The host’s `nvidia-smi` always shows physical memory. See [GPU Virtualization](./core-concepts/gpu-virtualization). ## Why is my nvidia.com/gpumem limit not enforced? {#why-is-my-nvidiagpumem-limit-not-enforced} @@ -206,7 +206,7 @@ The four most common causes: `CUDA_DISABLE_CONTROL=true` is set, the workload ru ## Does HAMi replace kube-scheduler or run alongside it? -HAMi runs alongside kube-scheduler as a [scheduler extender](https://github.com/kubernetes/design-proposals-archive/blob/main/scheduling/scheduler_extender.md) - it does not replace it. The MutatingWebhook sets `schedulerName: hami-scheduler` only on pods requesting HAMi resources; all other pods follow the default scheduler path unchanged. See [Architecture](../core-concepts/architecture). +HAMi runs alongside kube-scheduler as a [scheduler extender](https://github.com/kubernetes/design-proposals-archive/blob/main/scheduling/scheduler_extender.md) - it does not replace it. The MutatingWebhook sets `schedulerName: hami-scheduler` only on pods requesting HAMi resources; all other pods follow the default scheduler path unchanged. See [Architecture](./core-concepts/architecture). ## Does HAMi work with vLLM, and what are the known limitations for multi-GPU tensor parallelism? @@ -221,10 +221,10 @@ devicePlugin: enabled: false ``` -DCGM Exporter is not affected and continues to report physical-level counters normally. HAMi’s per-container virtual metrics are separate; see [GPU Utilization Metrics](../developers/gpu-utilization-metrics). +DCGM Exporter is not affected and continues to report physical-level counters normally. HAMi’s per-container virtual metrics are separate; see [GPU Utilization Metrics](./developers/gpu-utilization-metrics). If the HAMi Device Plugin or workloads fail after upgrading to GPU Operator 25.10+, see [NVIDIA containers fail with GPU Operator 25.10+](../troubleshooting/troubleshooting.md#nvidia-toolkit-gpu-operator-25-10). ## How do I set up Prometheus and Grafana monitoring for HAMi vGPU metrics? -The `hami-device-plugin` pod on each node exposes per-container vGPU metrics on port `31992` (configurable via `devicePlugin.service.httpPort`). See [Grafana Dashboard](../userguide/monitoring/grafana-dashboard) for the full setup including Prometheus scrape config and dashboard import. +The `hami-device-plugin` pod on each node exposes per-container vGPU metrics on port `31992` (configurable via `devicePlugin.service.httpPort`). See [Grafana Dashboard](./userguide/monitoring/grafana-dashboard) for the full setup including Prometheus scrape config and dashboard import. diff --git a/docs/troubleshooting/troubleshooting.md b/docs/troubleshooting/troubleshooting.md index 5bb7d2e9b..ec4851f41 100644 --- a/docs/troubleshooting/troubleshooting.md +++ b/docs/troubleshooting/troubleshooting.md @@ -15,7 +15,7 @@ If a container exceeds its `nvidia.com/gpumem` limit, check the following causes containerd config dump | grep default_runtime_name ``` - The output must show `nvidia`. If not, follow the [Prerequisites](../installation/online-installation) guide. + The output must show `nvidia`. If not, follow the [Prerequisites](./installation/online-installation) guide. - If you don’t explicitly request vGPUs when using the device plugin with NVIDIA images, all GPUs on the host may be exposed to your container. - Currently, A100 MIG can be supported in only "none" and "mixed" modes. diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/faq/faq.md b/i18n/zh/docusaurus-plugin-content-docs/current/faq/faq.md index 029c2fc13..c6435a0f6 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/faq/faq.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/faq/faq.md @@ -175,7 +175,7 @@ HAMi 的调度器需要从节点获取足够的 GPU 信息来完成资源调度 ## HAMi 如何强制执行 GPU 显存和算力限制? -HAMi 通过 `/etc/ld.so.preload` 将 `libvgpu.so` 注入容器。该库拦截 CUDA 显存分配调用,当超过 `nvidia.com/gpumem` 限制时返回 OOM;算力限制通过令牌桶算法对 kernel launch 调用进行节流。绕过 CUDA 库的应用(如 Docker-in-Docker、直接调用驱动 API)不受管控。完整的拦截流程参见 [GPU 虚拟化](../core-concepts/gpu-virtualization)。 +HAMi 通过 `/etc/ld.so.preload` 将 `libvgpu.so` 注入容器。该库拦截 CUDA 显存分配调用,当超过 `nvidia.com/gpumem` 限制时返回 OOM;算力限制通过令牌桶算法对 kernel launch 调用进行节流。绕过 CUDA 库的应用(如 Docker-in-Docker、直接调用驱动 API)不受管控。完整的拦截流程参见 [GPU 虚拟化](./core-concepts/gpu-virtualization)。 ## HAMi vGPU 与 NVIDIA MIG 有何区别?各适用于什么场景? @@ -191,11 +191,11 @@ HAMi vGPU 是纯软件方案,无硬件要求。NVIDIA MIG 是硬件分区, | 动态重配置 | 支持,无需排空节点 | 需要重新配置 MIG 配置文件 | | 多租户噪声隔离 | 尽力而为 | 强隔离 | -当 GPU 不支持 MIG、工作负载需要灵活的显存大小、或需要无需排空节点的动态重打包时,使用 HAMi vGPU。当硬隔离是合规或 SLA 要求时,使用 MIG。HAMi 也通过 `mig-parted` 支持动态 MIG;参见[动态 MIG 支持](../userguide/nvidia-device/dynamic-mig-support)。 +当 GPU 不支持 MIG、工作负载需要灵活的显存大小、或需要无需排空节点的动态重打包时,使用 HAMi vGPU。当硬隔离是合规或 SLA 要求时,使用 MIG。HAMi 也通过 `mig-parted` 支持动态 MIG;参见[动态 MIG 支持](./userguide/nvidia-device/dynamic-mig-support)。 ## 为什么容器内 nvidia-smi 显示的显存比宿主机少? -`libvgpu.so` 拦截了 `nvmlDeviceGetMemoryInfo` 及相关调用,返回 `nvidia.com/gpumem` 限制值而非物理显存。这是预期行为:根据上报显存大小进行分配的工作负载(如 vLLM)将只使用其配额。宿主机的 `nvidia-smi` 始终显示物理显存。参见 [GPU 虚拟化](../core-concepts/gpu-virtualization)。 +`libvgpu.so` 拦截了 `nvmlDeviceGetMemoryInfo` 及相关调用,返回 `nvidia.com/gpumem` 限制值而非物理显存。这是预期行为:根据上报显存大小进行分配的工作负载(如 vLLM)将只使用其配额。宿主机的 `nvidia-smi` 始终显示物理显存。参见 [GPU 虚拟化](./core-concepts/gpu-virtualization)。 ## 为什么 nvidia.com/gpumem 限制未生效? {#why-is-my-nvidiagpumem-limit-not-enforced} @@ -203,7 +203,7 @@ HAMi vGPU 是纯软件方案,无硬件要求。NVIDIA MIG 是硬件分区, ## HAMi 是替换 kube-scheduler 还是与其并行运行? -HAMi 作为 [scheduler extender](https://github.com/kubernetes/design-proposals-archive/blob/main/scheduling/scheduler_extender.md) 与 kube-scheduler 并行运行,不会替换它。MutatingWebhook 仅在请求 HAMi 资源的 Pod 上设置 `schedulerName: hami-scheduler`;所有其他 Pod 仍走默认调度器路径。参见[架构](../core-concepts/architecture)。 +HAMi 作为 [scheduler extender](https://github.com/kubernetes/design-proposals-archive/blob/main/scheduling/scheduler_extender.md) 与 kube-scheduler 并行运行,不会替换它。MutatingWebhook 仅在请求 HAMi 资源的 Pod 上设置 `schedulerName: hami-scheduler`;所有其他 Pod 仍走默认调度器路径。参见[架构](./core-concepts/architecture)。 ## HAMi 是否支持 vLLM?多 GPU 张量并行有哪些已知限制? @@ -218,10 +218,10 @@ devicePlugin: enabled: false ``` -DCGM Exporter 不受影响,继续正常上报物理级计数器。HAMi 的每容器虚拟指标是独立的;参见 [GPU 利用率指标](../developers/gpu-utilization-metrics)。 +DCGM Exporter 不受影响,继续正常上报物理级计数器。HAMi 的每容器虚拟指标是独立的;参见 [GPU 利用率指标](./developers/gpu-utilization-metrics)。 如果升级到 GPU Operator 25.10+ 后 HAMi Device Plugin 或工作负载启动失败,请参见[使用 GPU Operator 25.10+ 时 NVIDIA 容器启动失败](../troubleshooting/troubleshooting.md#nvidia-toolkit-gpu-operator-25-10)。 ## 如何为 HAMi vGPU 指标设置 Prometheus 和 Grafana 监控? -每个节点上的 `hami-device-plugin` Pod 在端口 `31992`(可通过 `devicePlugin.service.httpPort` 配置)上暴露每容器 vGPU 指标。完整的设置步骤(包括 Prometheus 采集配置和 Dashboard 导入)参见 [Grafana Dashboard](../userguide/monitoring/grafana-dashboard)。 +每个节点上的 `hami-device-plugin` Pod 在端口 `31992`(可通过 `devicePlugin.service.httpPort` 配置)上暴露每容器 vGPU 指标。完整的设置步骤(包括 Prometheus 采集配置和 Dashboard 导入)参见 [Grafana Dashboard](./userguide/monitoring/grafana-dashboard)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/troubleshooting/troubleshooting.md b/i18n/zh/docusaurus-plugin-content-docs/current/troubleshooting/troubleshooting.md index ebbd2efa8..6e997e10a 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/troubleshooting/troubleshooting.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/troubleshooting/troubleshooting.md @@ -16,7 +16,7 @@ translated: true containerd config dump | grep default_runtime_name ``` - 输出必须显示 `nvidia`。如未显示,请按照[前置条件](../installation/online-installation)指南操作。 + 输出必须显示 `nvidia`。如未显示,请按照[前置条件](./installation/online-installation)指南操作。 - 如果在使用 NVIDIA 镜像的设备插件时不请求 vGPU,机器上的所有 GPU 可能会在容器内暴露。 - 目前,A100 MIG 仅支持 "none" 和 "mixed" 模式。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/faq/faq.md b/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/faq/faq.md index 029c2fc13..c6435a0f6 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/faq/faq.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/faq/faq.md @@ -175,7 +175,7 @@ HAMi 的调度器需要从节点获取足够的 GPU 信息来完成资源调度 ## HAMi 如何强制执行 GPU 显存和算力限制? -HAMi 通过 `/etc/ld.so.preload` 将 `libvgpu.so` 注入容器。该库拦截 CUDA 显存分配调用,当超过 `nvidia.com/gpumem` 限制时返回 OOM;算力限制通过令牌桶算法对 kernel launch 调用进行节流。绕过 CUDA 库的应用(如 Docker-in-Docker、直接调用驱动 API)不受管控。完整的拦截流程参见 [GPU 虚拟化](../core-concepts/gpu-virtualization)。 +HAMi 通过 `/etc/ld.so.preload` 将 `libvgpu.so` 注入容器。该库拦截 CUDA 显存分配调用,当超过 `nvidia.com/gpumem` 限制时返回 OOM;算力限制通过令牌桶算法对 kernel launch 调用进行节流。绕过 CUDA 库的应用(如 Docker-in-Docker、直接调用驱动 API)不受管控。完整的拦截流程参见 [GPU 虚拟化](./core-concepts/gpu-virtualization)。 ## HAMi vGPU 与 NVIDIA MIG 有何区别?各适用于什么场景? @@ -191,11 +191,11 @@ HAMi vGPU 是纯软件方案,无硬件要求。NVIDIA MIG 是硬件分区, | 动态重配置 | 支持,无需排空节点 | 需要重新配置 MIG 配置文件 | | 多租户噪声隔离 | 尽力而为 | 强隔离 | -当 GPU 不支持 MIG、工作负载需要灵活的显存大小、或需要无需排空节点的动态重打包时,使用 HAMi vGPU。当硬隔离是合规或 SLA 要求时,使用 MIG。HAMi 也通过 `mig-parted` 支持动态 MIG;参见[动态 MIG 支持](../userguide/nvidia-device/dynamic-mig-support)。 +当 GPU 不支持 MIG、工作负载需要灵活的显存大小、或需要无需排空节点的动态重打包时,使用 HAMi vGPU。当硬隔离是合规或 SLA 要求时,使用 MIG。HAMi 也通过 `mig-parted` 支持动态 MIG;参见[动态 MIG 支持](./userguide/nvidia-device/dynamic-mig-support)。 ## 为什么容器内 nvidia-smi 显示的显存比宿主机少? -`libvgpu.so` 拦截了 `nvmlDeviceGetMemoryInfo` 及相关调用,返回 `nvidia.com/gpumem` 限制值而非物理显存。这是预期行为:根据上报显存大小进行分配的工作负载(如 vLLM)将只使用其配额。宿主机的 `nvidia-smi` 始终显示物理显存。参见 [GPU 虚拟化](../core-concepts/gpu-virtualization)。 +`libvgpu.so` 拦截了 `nvmlDeviceGetMemoryInfo` 及相关调用,返回 `nvidia.com/gpumem` 限制值而非物理显存。这是预期行为:根据上报显存大小进行分配的工作负载(如 vLLM)将只使用其配额。宿主机的 `nvidia-smi` 始终显示物理显存。参见 [GPU 虚拟化](./core-concepts/gpu-virtualization)。 ## 为什么 nvidia.com/gpumem 限制未生效? {#why-is-my-nvidiagpumem-limit-not-enforced} @@ -203,7 +203,7 @@ HAMi vGPU 是纯软件方案,无硬件要求。NVIDIA MIG 是硬件分区, ## HAMi 是替换 kube-scheduler 还是与其并行运行? -HAMi 作为 [scheduler extender](https://github.com/kubernetes/design-proposals-archive/blob/main/scheduling/scheduler_extender.md) 与 kube-scheduler 并行运行,不会替换它。MutatingWebhook 仅在请求 HAMi 资源的 Pod 上设置 `schedulerName: hami-scheduler`;所有其他 Pod 仍走默认调度器路径。参见[架构](../core-concepts/architecture)。 +HAMi 作为 [scheduler extender](https://github.com/kubernetes/design-proposals-archive/blob/main/scheduling/scheduler_extender.md) 与 kube-scheduler 并行运行,不会替换它。MutatingWebhook 仅在请求 HAMi 资源的 Pod 上设置 `schedulerName: hami-scheduler`;所有其他 Pod 仍走默认调度器路径。参见[架构](./core-concepts/architecture)。 ## HAMi 是否支持 vLLM?多 GPU 张量并行有哪些已知限制? @@ -218,10 +218,10 @@ devicePlugin: enabled: false ``` -DCGM Exporter 不受影响,继续正常上报物理级计数器。HAMi 的每容器虚拟指标是独立的;参见 [GPU 利用率指标](../developers/gpu-utilization-metrics)。 +DCGM Exporter 不受影响,继续正常上报物理级计数器。HAMi 的每容器虚拟指标是独立的;参见 [GPU 利用率指标](./developers/gpu-utilization-metrics)。 如果升级到 GPU Operator 25.10+ 后 HAMi Device Plugin 或工作负载启动失败,请参见[使用 GPU Operator 25.10+ 时 NVIDIA 容器启动失败](../troubleshooting/troubleshooting.md#nvidia-toolkit-gpu-operator-25-10)。 ## 如何为 HAMi vGPU 指标设置 Prometheus 和 Grafana 监控? -每个节点上的 `hami-device-plugin` Pod 在端口 `31992`(可通过 `devicePlugin.service.httpPort` 配置)上暴露每容器 vGPU 指标。完整的设置步骤(包括 Prometheus 采集配置和 Dashboard 导入)参见 [Grafana Dashboard](../userguide/monitoring/grafana-dashboard)。 +每个节点上的 `hami-device-plugin` Pod 在端口 `31992`(可通过 `devicePlugin.service.httpPort` 配置)上暴露每容器 vGPU 指标。完整的设置步骤(包括 Prometheus 采集配置和 Dashboard 导入)参见 [Grafana Dashboard](./userguide/monitoring/grafana-dashboard)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/troubleshooting/troubleshooting.md b/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/troubleshooting/troubleshooting.md index ebbd2efa8..6e997e10a 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/troubleshooting/troubleshooting.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/troubleshooting/troubleshooting.md @@ -16,7 +16,7 @@ translated: true containerd config dump | grep default_runtime_name ``` - 输出必须显示 `nvidia`。如未显示,请按照[前置条件](../installation/online-installation)指南操作。 + 输出必须显示 `nvidia`。如未显示,请按照[前置条件](./installation/online-installation)指南操作。 - 如果在使用 NVIDIA 镜像的设备插件时不请求 vGPU,机器上的所有 GPU 可能会在容器内暴露。 - 目前,A100 MIG 仅支持 "none" 和 "mixed" 模式。 diff --git a/versioned_docs/version-v2.9.0/faq/faq.md b/versioned_docs/version-v2.9.0/faq/faq.md index d94b959ff..69cf5654e 100644 --- a/versioned_docs/version-v2.9.0/faq/faq.md +++ b/versioned_docs/version-v2.9.0/faq/faq.md @@ -178,7 +178,7 @@ If the official Device Plugin cannot provide the required information, HAMi deve ## How does HAMi enforce GPU memory and compute limits? -HAMi injects `libvgpu.so` into containers via `/etc/ld.so.preload`. The library intercepts CUDA memory allocation calls and returns OOM when the `nvidia.com/gpumem` limit is exceeded; compute limits use a token-bucket throttle on kernel launch calls. Applications that bypass the CUDA library (Docker-in-Docker, direct driver API) are not covered. For the full interception flow, see [GPU Virtualization](../core-concepts/gpu-virtualization). +HAMi injects `libvgpu.so` into containers via `/etc/ld.so.preload`. The library intercepts CUDA memory allocation calls and returns OOM when the `nvidia.com/gpumem` limit is exceeded; compute limits use a token-bucket throttle on kernel launch calls. Applications that bypass the CUDA library (Docker-in-Docker, direct driver API) are not covered. For the full interception flow, see [GPU Virtualization](./core-concepts/gpu-virtualization). ## How does HAMi vGPU differ from NVIDIA MIG? When should I use each? @@ -194,11 +194,11 @@ HAMi vGPU is software-only with no hardware requirements. NVIDIA MIG is hardware | Dynamic reconfiguration | Yes, no node drain needed | Requires MIG profile reconfiguration | | Multi-tenant noise isolation | Best-effort | Strong | -Use HAMi vGPU when the GPU does not support MIG, workloads need flexible memory sizes, or dynamic repacking without node drains is needed. Use MIG when hard hardware isolation is a compliance or SLA requirement. HAMi also supports dynamic MIG via `mig-parted`; see [Dynamic MIG Support](../userguide/nvidia-device/dynamic-mig-support). +Use HAMi vGPU when the GPU does not support MIG, workloads need flexible memory sizes, or dynamic repacking without node drains is needed. Use MIG when hard hardware isolation is a compliance or SLA requirement. HAMi also supports dynamic MIG via `mig-parted`; see [Dynamic MIG Support](./userguide/nvidia-device/dynamic-mig-support). ## Why does nvidia-smi inside my container show less memory than on the host? -`libvgpu.so` intercepts `nvmlDeviceGetMemoryInfo` and related calls, returning the `nvidia.com/gpumem` limit instead of physical VRAM. This is intentional: workloads that size their allocations based on reported memory (such as vLLM) will use only their budget. The host’s `nvidia-smi` always shows physical memory. See [GPU Virtualization](../core-concepts/gpu-virtualization). +`libvgpu.so` intercepts `nvmlDeviceGetMemoryInfo` and related calls, returning the `nvidia.com/gpumem` limit instead of physical VRAM. This is intentional: workloads that size their allocations based on reported memory (such as vLLM) will use only their budget. The host’s `nvidia-smi` always shows physical memory. See [GPU Virtualization](./core-concepts/gpu-virtualization). ## Why is my nvidia.com/gpumem limit not enforced? {#why-is-my-nvidiagpumem-limit-not-enforced} @@ -206,7 +206,7 @@ The four most common causes: `CUDA_DISABLE_CONTROL=true` is set, the workload ru ## Does HAMi replace kube-scheduler or run alongside it? -HAMi runs alongside kube-scheduler as a [scheduler extender](https://github.com/kubernetes/design-proposals-archive/blob/main/scheduling/scheduler_extender.md) - it does not replace it. The MutatingWebhook sets `schedulerName: hami-scheduler` only on pods requesting HAMi resources; all other pods follow the default scheduler path unchanged. See [Architecture](../core-concepts/architecture). +HAMi runs alongside kube-scheduler as a [scheduler extender](https://github.com/kubernetes/design-proposals-archive/blob/main/scheduling/scheduler_extender.md) - it does not replace it. The MutatingWebhook sets `schedulerName: hami-scheduler` only on pods requesting HAMi resources; all other pods follow the default scheduler path unchanged. See [Architecture](./core-concepts/architecture). ## Does HAMi work with vLLM, and what are the known limitations for multi-GPU tensor parallelism? @@ -221,10 +221,10 @@ devicePlugin: enabled: false ``` -DCGM Exporter is not affected and continues to report physical-level counters normally. HAMi’s per-container virtual metrics are separate; see [GPU Utilization Metrics](../developers/gpu-utilization-metrics). +DCGM Exporter is not affected and continues to report physical-level counters normally. HAMi’s per-container virtual metrics are separate; see [GPU Utilization Metrics](./developers/gpu-utilization-metrics). If the HAMi Device Plugin or workloads fail after upgrading to GPU Operator 25.10+, see [NVIDIA containers fail with GPU Operator 25.10+](../troubleshooting/troubleshooting.md#nvidia-toolkit-gpu-operator-25-10). ## How do I set up Prometheus and Grafana monitoring for HAMi vGPU metrics? -The `hami-device-plugin` pod on each node exposes per-container vGPU metrics on port `31992` (configurable via `devicePlugin.service.httpPort`). See [Grafana Dashboard](../userguide/monitoring/grafana-dashboard) for the full setup including Prometheus scrape config and dashboard import. +The `hami-device-plugin` pod on each node exposes per-container vGPU metrics on port `31992` (configurable via `devicePlugin.service.httpPort`). See [Grafana Dashboard](./userguide/monitoring/grafana-dashboard) for the full setup including Prometheus scrape config and dashboard import. diff --git a/versioned_docs/version-v2.9.0/troubleshooting/troubleshooting.md b/versioned_docs/version-v2.9.0/troubleshooting/troubleshooting.md index 5bb7d2e9b..ec4851f41 100644 --- a/versioned_docs/version-v2.9.0/troubleshooting/troubleshooting.md +++ b/versioned_docs/version-v2.9.0/troubleshooting/troubleshooting.md @@ -15,7 +15,7 @@ If a container exceeds its `nvidia.com/gpumem` limit, check the following causes containerd config dump | grep default_runtime_name ``` - The output must show `nvidia`. If not, follow the [Prerequisites](../installation/online-installation) guide. + The output must show `nvidia`. If not, follow the [Prerequisites](./installation/online-installation) guide. - If you don’t explicitly request vGPUs when using the device plugin with NVIDIA images, all GPUs on the host may be exposed to your container. - Currently, A100 MIG can be supported in only "none" and "mixed" modes. From 7c0f0fd0e31efe5083159bb876a81f29a9f964a7 Mon Sep 17 00:00:00 2001 From: Jimmy Song Date: Tue, 11 Aug 2026 14:17:58 +0800 Subject: [PATCH 5/7] fix(tutorials): correct gpucores unit (30% not 30 cores) and clean up formatting (#753) * fix(nvml-mock): clarify resource limits for GPU scheduling in documentation Signed-off-by: Jimmy Song --- .../current/labs/local-fake-gpu.md | 2 -- .../current/labs/nvml-mock.md | 32 ++----------------- .../current/labs/online-install.md | 6 ---- .../current/labs/topology-aware-scheduling.md | 2 -- tutorials/labs/local-fake-gpu.md | 2 -- tutorials/labs/nvml-mock.md | 32 ++----------------- tutorials/labs/online-install.md | 6 ---- tutorials/labs/topology-aware-scheduling.md | 2 -- 8 files changed, 6 insertions(+), 78 deletions(-) diff --git a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/local-fake-gpu.md b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/local-fake-gpu.md index 8bb14b20e..aca0704fc 100644 --- a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/local-fake-gpu.md +++ b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/local-fake-gpu.md @@ -242,8 +242,6 @@ echo "NODE_NAME=${NODE_NAME}" | macOS | `orbstack` | | Linux | `hami-lab-control-plane` | ---- - :::info[步骤 2–7 在两个平台上完全相同] 从这里开始,所有命令在 macOS 和 Linux 上完全一致。你唯一会注意到的区别是示例输出中的节点名称。 diff --git a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/nvml-mock.md b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/nvml-mock.md index 88d2affe7..96e0e9908 100644 --- a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/nvml-mock.md +++ b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/nvml-mock.md @@ -138,8 +138,6 @@ Windows 用户请使用 [WSL2](https://learn.microsoft.com/zh-cn/windows/wsl/ins ::: ---- - ## 步骤 1:创建 kind 集群 ```bash @@ -159,8 +157,6 @@ echo "NODE_NAME=${NODE_NAME}" NODE_NAME=nvml-mock-test-control-plane ``` ---- - ## 步骤 2:构建并部署 nvml-mock nvml-mock 提供一个虚假的 `libnvidia-ml.so`、虚拟的 `/dev/nvidia*` 设备节点以及 PCI 拓扑条目,让 HAMi 的 device-plugin 在节点上看到 8 张 A100 GPU。 @@ -210,8 +206,6 @@ NAME GPU_PRESENT nvml-mock-test-control-plane true ``` ---- - ## 步骤 3:基于 `main` 分支构建 HAMi `main` 分支包含一个修复:当未启用 MIG 时,阻止调用 `nvidia-mig-parted`。从源码构建可确保该修复已包含在内,无需等待正式发布版本。 @@ -245,8 +239,6 @@ kind load docker-image hami:local --name nvml-mock-test 调度器和 device-plugin 二进制文件都打包在单个 `hami:local` 镜像中。 ---- - ## 步骤 4:部署 HAMi ### 4.1 通过 Helm 安装 @@ -332,8 +324,6 @@ hami-scheduler-7858c744cc-7pb79 2/2 Running 0 13m ::: ---- - ## 步骤 5:验证 GPU 资源 HAMi 将每张物理 GPU 切分成 10 个虚拟槽位。节点有 8 张物理 GPU,因此应该对外提供 **80** 个可分配的虚拟 GPU。 @@ -353,8 +343,6 @@ kubectl describe node ${NODE_NAME} | grep nvidia.com/gpu `Capacity` 和 `Allocatable` 都显示 `80`,确认 device-plugin 已注册全部虚拟 GPU 槽位。最后一行是 `Allocated resources` 表,当前为 `0`,因为还没有 Pod 申请 GPU。 ---- - ## 步骤 6:测试基础 GPU 调度 部署一个最小化的 Pod,申请一张 GPU。`CUDA_DISABLE_CONTROL=true` 阻止 HAMi 注入的 CUDA shim 尝试真实设备访问: @@ -406,8 +394,6 @@ hami.io/vgpu-devices-allocated: GPU-12345678-1234-1234-1234-123456780006,NVIDIA, > 注解格式为 `,<厂商>,<显存MiB>,<算力>`。A100 GPU 有 40960 MiB 显存,看到这个注解即确认调度器分配并记录了一个虚拟 GPU。 ---- - ## 步骤 7:测试 GPU 共享(时间片) 再部署三个 Pod,每个申请 1 张 GPU: @@ -457,8 +443,6 @@ gpu-test-4 1/1 Running 0 9s 四个 Pod 并发运行在 80 个虚拟 GPU 槽位的资源池上。调度器通过各自独立的 `vgpu-devices-allocated` 注解独立跟踪每次分配。 ---- - ## 步骤 8:测试显存和算力限制 ```bash @@ -485,7 +469,7 @@ EOF :::info -资源限制格式 `nvidia.com/gpumem` 接受**以 MiB 为单位的绝对值**:`"10"` 表示 10 MiB。`nvidia.com/gpucores: "30"` 表示在所选 GPU 上申请 30 个计算核心。 +资源限制格式 `nvidia.com/gpumem` 接受**以 MiB 为单位的绝对值**:`"10"` 表示 10 MiB。`nvidia.com/gpucores: "30"` 表示在所选 GPU 上申请 30% 的计算算力。 ::: @@ -501,9 +485,7 @@ kubectl describe pod gpu-limits | grep vgpu-devices-allocated hami.io/vgpu-devices-allocated: GPU-12345678-1234-1234-1234-123456780002,NVIDIA,10,30:; ``` -注解记录了 `10` MiB 和 `30` 个核心,正是所申请的值。 - ---- +注解记录了 `10` MiB 和 `30`(即 30% 的计算算力),正是所申请的值。 ## 步骤 9:测试百分比显存申请 @@ -566,8 +548,6 @@ GPU-12345678-1234-1234-1234-123456780003,NVIDIA,12288,100:; > 第三个字段显示 `12288` MiB(即 40960 MiB 的 30%),确认调度器正确地将百分比转换为本次分配的绝对显存预算。 ---- - ## 步骤 10:测试多 GPU 分配 ```bash @@ -637,8 +617,6 @@ kubectl get pod gpu-multi \ ::: ---- - ## 已验证功能总结 | 功能 | 测试 Pod | 如何验证 | @@ -646,7 +624,7 @@ kubectl get pod gpu-multi \ | 基础 GPU 调度 | `gpu-test-1` | 注解显示 1 个 vGPU UUID + 40960 MiB | | GPU 共享(时间片) | `gpu-test-1` 到 `gpu-test-4` | 4 个 Pod 并发运行 | | 显存限制(`gpumem`) | `gpu-limits` | 注解显示 `10` MiB | -| 算力限制(`gpucores`) | `gpu-limits` | 注解显示 `30` 个核心 | +| 算力限制(`gpucores`) | `gpu-limits` | 注解显示 `30`(30% 的计算算力) | | 百分比显存(`gpumem-percentage`) | `gpu-mem-30pct` | 注解显示 `12288` MiB(A100 的 30%) | | 多 GPU 分配 | `gpu-multi` | hami-scheduler 事件显示 `BindingSucceed` | @@ -659,8 +637,6 @@ kubectl get pod gpu-multi \ ::: ---- - ## 清理 删除所有测试 Pod: @@ -700,8 +676,6 @@ kind delete cluster --name nvml-mock-test ::: ---- - ## 下一步 - 切换到真实 GPU 集群(参见 [实验 1:在线安装 HAMi](/tutorials/labs/online-install)),用真实 CUDA 工作负载测试显存和算力隔离。 diff --git a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/online-install.md b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/online-install.md index 0065a2e00..0c7075a7c 100644 --- a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/online-install.md +++ b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/online-install.md @@ -54,8 +54,6 @@ flowchart LR > 费用提示:`n1-standard-4` + T4 虚拟机约 $0.55/小时。[实验 3](./gpu-partitioning.md) 和[实验 4](./hami-dra.md) 直接复用这套集群,一次开机即可完成全部三个实验。实验结束后请删除虚拟机。 ---- - ## 步骤 1: 创建 GCP 虚拟机 ### 目的 @@ -324,8 +322,6 @@ prometheus-prometheus-node-exporter-xxxxx 1/1 Running 0 > 如果安装失败,需要先卸载再重装:`helm uninstall -n monitoring prometheus` ---- - ## 步骤 5: 安装 GPU Operator ### 目的 @@ -396,8 +392,6 @@ kubectl -n gpu-operator exec -it $(kubectl get pods -n gpu-operator -l app=nvidi +-----------------------------------------------------------------------------------------+ ``` ---- - ## 步骤 6: 安装 HAMi ### 目的 diff --git a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/topology-aware-scheduling.md b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/topology-aware-scheduling.md index c08b3d95f..1753f404d 100644 --- a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/topology-aware-scheduling.md +++ b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/topology-aware-scheduling.md @@ -203,8 +203,6 @@ NODE_NAME=$(kubectl get nodes -o jsonpath='{.items[0].metadata.name}') echo "NODE_NAME=${NODE_NAME}" ``` ---- - :::info[后续大部分步骤在两个平台上是一致的] 从这里开始,大多数命令在 macOS 和 Linux 上是一致的。仍有少数步骤存在差异 —— 步骤 2.3 和步骤 3.1 中分别有一条仅 Linux 需要执行的额外 `kind load` 命令 —— 其余部分完全相同,只是示例输出中的节点名不同。 diff --git a/tutorials/labs/local-fake-gpu.md b/tutorials/labs/local-fake-gpu.md index 186a4724d..547110597 100644 --- a/tutorials/labs/local-fake-gpu.md +++ b/tutorials/labs/local-fake-gpu.md @@ -242,8 +242,6 @@ echo "NODE_NAME=${NODE_NAME}" | macOS | `orbstack` | | Linux | `hami-lab-control-plane` | ---- - :::info[Steps 2–7 are identical on both platforms] From here on, all commands work the same on macOS and Linux. The only difference you will notice is your node name in example outputs. diff --git a/tutorials/labs/nvml-mock.md b/tutorials/labs/nvml-mock.md index 5b5af24db..69f8ebc1f 100644 --- a/tutorials/labs/nvml-mock.md +++ b/tutorials/labs/nvml-mock.md @@ -138,8 +138,6 @@ Windows users Use [WSL2](https://learn.microsoft.com/en-us/windows/wsl/install) ::: ---- - ## Step 1: Create the kind Cluster ```bash @@ -159,8 +157,6 @@ Example output: NODE_NAME=nvml-mock-test-control-plane ``` ---- - ## Step 2: Build and Deploy nvml-mock nvml-mock provides a fake `libnvidia-ml.so`, virtual `/dev/nvidia*` device nodes, and PCI topology entries so HAMi's device-plugin sees 8 A100 GPUs on the node. @@ -210,8 +206,6 @@ NAME GPU_PRESENT nvml-mock-test-control-plane true ``` ---- - ## Step 3: Build HAMi from the `main` Branch The `main` branch contains a fix preventing `nvidia-mig-parted` from being called when MIG is not enabled. Building from source ensures the fix is present without waiting for a tagged release. @@ -245,8 +239,6 @@ kind load docker-image hami:local --name nvml-mock-test Both the scheduler and device-plugin binaries are packaged into the single `hami:local` image. ---- - ## Step 4: Deploy HAMi ### 4.1 Install via Helm @@ -332,8 +324,6 @@ The `vgpu-monitor` sidecar crashes because it requires real GPU monitoring infra ::: ---- - ## Step 5: Verify GPU Resources HAMi partitions each physical GPU into 10 virtual slots. With 8 physical GPUs the node should advertise **80** allocatable virtual GPUs. @@ -353,8 +343,6 @@ Expected output: Both `Capacity` and `Allocatable` showing `80` confirms the device-plugin registered all virtual GPU slots. The final line is the `Allocated resources` table — currently `0` because no Pods have claimed GPUs yet. ---- - ## Step 6: Test Basic GPU Scheduling Deploy a minimal Pod requesting one GPU. `CUDA_DISABLE_CONTROL=true` prevents HAMi's injected CUDA shim from attempting real device access: @@ -406,8 +394,6 @@ hami.io/vgpu-devices-allocated: GPU-12345678-1234-1234-1234-123456780006,NVIDIA, > The annotation format is `,,,`. A100 GPUs have 40960 MiB of VRAM — seeing this annotation confirms one virtual GPU was allocated and recorded by the scheduler. ---- - ## Step 7: Test GPU Sharing (Time-slicing) Deploy three more Pods each requesting 1 GPU: @@ -457,8 +443,6 @@ gpu-test-4 1/1 Running 0 9s All four Pods run concurrently against the pool of 80 virtual GPU slots. The scheduler independently tracks each allocation via its own `vgpu-devices-allocated` annotation. ---- - ## Step 8: Test Memory and Core Limits ```bash @@ -485,7 +469,7 @@ EOF :::info -Resource Limits Format `nvidia.com/gpumem` takes an **absolute value in MiB** — `"10"` means 10 MiB. `nvidia.com/gpucores: "30"` requests 30 compute cores on the selected GPU. +Resource Limits Format `nvidia.com/gpumem` takes an **absolute value in MiB** — `"10"` means 10 MiB. `nvidia.com/gpucores: "30"` requests 30% of the selected GPU's compute capacity. ::: @@ -501,9 +485,7 @@ Expected output: hami.io/vgpu-devices-allocated: GPU-12345678-1234-1234-1234-123456780002,NVIDIA,10,30:; ``` -The annotation records `10` MiB and `30` cores — exactly the values requested. - ---- +The annotation records `10` MiB and `30` (30% of the GPU's compute capacity) — exactly the values requested. ## Step 9: Test Percentage-Based Memory Request @@ -566,8 +548,6 @@ GPU-12345678-1234-1234-1234-123456780003,NVIDIA,12288,100:; > The third field shows `12288` MiB — 30% of 40960 MiB — confirming the scheduler correctly translated the percentage into an absolute memory budget for the allocation. ---- - ## Step 10: Test Multi-GPU Allocation ```bash @@ -637,8 +617,6 @@ You will see two semicolon-separated device entries, one per allocated vGPU slot ::: ---- - ## Summary of Verified Features | Feature | Test Pod | How It Is Verified | @@ -646,7 +624,7 @@ You will see two semicolon-separated device entries, one per allocated vGPU slot | Basic GPU scheduling | `gpu-test-1` | Annotation shows 1 vGPU UUID + 40960 MiB | | GPU sharing (time-slicing) | `gpu-test-1` through `gpu-test-4` | All 4 Pods run concurrently | | Memory limit (`gpumem`) | `gpu-limits` | Annotation shows `10` MiB | -| Core limit (`gpucores`) | `gpu-limits` | Annotation shows `30` cores | +| Compute limit (`gpucores`) | `gpu-limits` | Annotation shows `30` (30% of compute capacity) | | Percentage memory (`gpumem-percentage`) | `gpu-mem-30pct` | Annotation shows `12288` MiB (30% of A100) | | Multi-GPU allocation | `gpu-multi` | hami-scheduler events show `BindingSucceed` | @@ -659,8 +637,6 @@ You will see two semicolon-separated device entries, one per allocated vGPU slot ::: ---- - ## Cleanup Delete all test Pods: @@ -700,8 +676,6 @@ Skip the cluster deletion step if you want to keep the environment for further e ::: ---- - ## Next Steps - Move to a real GPU cluster (see [Lab 1: Online HAMi Installation](/tutorials/labs/online-install)) to test memory and core isolation with actual CUDA workloads. diff --git a/tutorials/labs/online-install.md b/tutorials/labs/online-install.md index de83a067d..7fe170a26 100644 --- a/tutorials/labs/online-install.md +++ b/tutorials/labs/online-install.md @@ -54,8 +54,6 @@ flowchart LR > Cost note: the `n1-standard-4` + T4 VM costs about $0.55 per hour. [Lab 3](./gpu-partitioning.md) and [Lab 4](./hami-dra.md) continue on this same cluster, so one session covers all three labs. Delete the VM when you finish. ---- - ## Step 1: Create a GCP Virtual Machine ### Purpose @@ -324,8 +322,6 @@ prometheus-prometheus-node-exporter-xxxxx 1/1 Running 0 > If the installation fails, uninstall first before retrying: `helm uninstall -n monitoring prometheus` ---- - ## Step 5: Install GPU Operator ### Purpose @@ -396,8 +392,6 @@ The expected output includes GPU information (driver version, CUDA version, GPU +-----------------------------------------------------------------------------------------+ ``` ---- - ## Step 6: Install HAMi ### Purpose diff --git a/tutorials/labs/topology-aware-scheduling.md b/tutorials/labs/topology-aware-scheduling.md index fdf09844d..e53704609 100644 --- a/tutorials/labs/topology-aware-scheduling.md +++ b/tutorials/labs/topology-aware-scheduling.md @@ -203,8 +203,6 @@ NODE_NAME=$(kubectl get nodes -o jsonpath='{.items[0].metadata.name}') echo "NODE_NAME=${NODE_NAME}" ``` ---- - :::info[Most remaining steps are the same on both platforms] From here on, most commands work the same on macOS and Linux. A couple of steps still differ - Step 2.3 and Step 3.1 each call out an extra `kind load` command needed only on Linux. Everything else is identical, aside from the node name you'll see in example outputs. From 55c060ca79e01c02308daee0201397ca81a0fb60 Mon Sep 17 00:00:00 2001 From: Reza Jelveh Date: Wed, 12 Aug 2026 00:46:50 +0900 Subject: [PATCH 6/7] add opensource summit korea deck Signed-off-by: Reza Jelveh --- .../2026-ossummit-korea/ossummit_korea.pdf | Bin 0 -> 132 bytes 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 static/resources/2026-ossummit-korea/ossummit_korea.pdf diff --git a/static/resources/2026-ossummit-korea/ossummit_korea.pdf b/static/resources/2026-ossummit-korea/ossummit_korea.pdf new file mode 100644 index 0000000000000000000000000000000000000000..59247705db70d7bb1f849f6ee8abf3d77f0ebfe4 GIT binary patch literal 132 zcmWN?%Mk)03;@tORnUM73Cd@?fg}hsDm#LEuzG!$cglPA@zQOrWA4V>`?@^p-2S&u z*=Rh?JcP^%YIN4KWxV;_r$lTEL~$Vl35z%kgasqyYjKd6Ie3Uzb1iIxMX8aHsU&bV Ns@cDyA%%!Os2^IKD31UD literal 0 HcmV?d00001 From c8d501d0190c79d91071d3f7edcd566eb073abfe Mon Sep 17 00:00:00 2001 From: Reza Jelveh Date: Wed, 12 Aug 2026 00:48:32 +0900 Subject: [PATCH 7/7] feat(events): add talk slides and community flyer resources for Korea event Signed-off-by: Reza Jelveh --- src/data/events.js | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/src/data/events.js b/src/data/events.js index ee0b78a2e..602628c65 100644 --- a/src/data/events.js +++ b/src/data/events.js @@ -162,6 +162,18 @@ const events = [ en: "Open-source agents like Hermes and OpenClaw can reason and use tools, but edge deployment is still hard - not the model, the compute underneath: limited memory, tight power budgets, no ops team. Democratizing agentic AI means fixing the compute layer. This deep dive covers GPU and memory slicing at the edge: carving one device's unified memory so multiple agents run concurrently on hardware as small as an NVIDIA Jetson, and time-sharing when demand exceeds memory. We compare Jetson-class GPUs with NPUs from Axelera and DeepX - higher performance per watt - and what it takes to make them schedulable. HAMi, the CNCF Incubation project for hardware-agnostic GPU virtualization, is that layer: one scheduling plane across heterogeneous accelerators, fine-grained memory slicing, open source. A blueprint for edge AI without a cloud budget.", zh: "开源智能体(如 Hermes 和 OpenClaw)能够推理并使用工具,但边缘部署仍然困难——问题不在模型,而在底层的算力:内存有限、功耗预算紧张、没有运维团队。让智能体 AI 普及化,关键在于修复算力层。本次深度解析涵盖边缘 GPU 与显存切片:将一块设备的统一内存切分,让多个智能体在 NVIDIA Jetson 这样的小型硬件上并发运行,并在内存不足时进行时间共享。我们对比 Jetson 级 GPU 与 Axelera、DeepX 的 NPU——每瓦性能更高——以及让它们可被调度的必要条件。HAMi 是 CNCF 孵化项目,提供与硬件无关的 GPU 虚拟化,正是这一层:跨异构加速器的统一调度平面、细粒度显存切片、开源。一份无需云预算的边缘 AI 蓝图。", }, + resources: { + communityFlyer: { + en: "Community Flyer", + zh: "社区宣传册", + url: "/resources/flyers/community-flyer.pdf", + }, + talkSlides: { + en: "Talk Slides", + zh: "演讲幻灯片", + url: "/resources/2026-ossummit-korea/ossummit_korea.pdf", + }, + }, cta: { discordUrl: "https://go.dynamia.ai/hami-chat-korea", githubUrl: "https://go.dynamia.ai/proj-hami-korea",