diff --git a/blog/authors.yml b/blog/authors.yml index 8050f773..aefe3899 100644 --- a/blog/authors.yml +++ b/blog/authors.yml @@ -15,3 +15,7 @@ archlitchi: name: Li Mengxuan title: HAMi Maintainer, Co-founder & CTO of Dynamia url: https://github.com/archlitchi + +rootsongjc: + name: Jimmy Song + url: https://github.com/rootsongjc diff --git a/blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md b/blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md new file mode 100644 index 00000000..62b1b04f --- /dev/null +++ b/blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md @@ -0,0 +1,261 @@ +--- +title: "GPU Memory Hard Isolation with KAI Scheduler and HAMi: How It Works and How to Verify It" +date: "2026-08-11" +description: "A concise explanation of the KAI Scheduler and HAMi-core isolation path, plus reproducible GKE evidence that two Pods sharing one NVIDIA T4 cannot exceed their individual GPU memory quotas." +authors: [rootsongjc] +tags: ["HAMi", "KAI Scheduler", "Hard Isolation", "GPU Sharing", "Kubernetes", "Cloud Native"] +--- + +The companion post [HAMi-core adopted by NVIDIA KAI Scheduler](/blog/hami-core-adopted-by-nvidia-kai-scheduler) already introduces KAI Scheduler and the collaboration behind this integration. This post skips that background and focuses on one question: **when KAI Scheduler places two Pods on one GPU, does HAMi-core actually enforce each Pod's memory quota?** + +We verified the currently documented combination—KAI Scheduler v0.17.0 and `kai-resource-isolator` 1.1.0-chart—on GKE 1.35/COS/CDI. Both Pods shared the same NVIDIA T4, each saw a 4147 MiB ceiling, a 3 GiB CUDA allocation succeeded, and a cumulative 5 GiB allocation failed. The optional monitor also exported live limit and usage metrics for both Pods. + +:::note About the captured output + +The UUID, memory ceiling, CUDA allocation results, and monitor metrics below came from the verified GKE run. Resource suffixes and addresses will differ in another cluster. + +::: + + + +## How the integration works + +The path has three responsibilities: + +- **KAI Scheduler** decides which GPU a Pod uses and injects the computed quota through `CUDA_DEVICE_MEMORY_LIMIT`. +- **`kai-resource-isolator`** distributes `libvgpu.so`; its webhook mounts the library and configures `ld.so.preload` in the workload Pod. Its optional monitor reads node-local HAMi caches and exposes `hami_*` metrics on `:9394`. +- **HAMi-core (`libvgpu.so`)** intercepts CUDA calls such as `cudaMalloc` and rejects allocations beyond the quota. + +```mermaid +%% title: KAI Scheduler and HAMi-core isolation path +graph TD + KAI["KAI Scheduler
computes the GPU memory quota"] + ENV["Inject CUDA_DEVICE_MEMORY_LIMIT"] + WEBHOOK["kai-resource-isolator webhook
mounts hostPath and ld.so.preload"] + LIB["libsync DaemonSet
provides libvgpu.so on each GPU node"] + RUN["HAMi-core intercepts CUDA calls"] + ENF["Reject over-quota allocations"] + MON["monitor DaemonSet
exports hami_* metrics on :9394"] + + KAI --> ENV --> WEBHOOK --> RUN --> ENF + LIB -. "provides libvgpu.so" .-> RUN + RUN -. "writes node-local cache" .-> MON + + style KAI fill:#d9f99d,stroke:#4f7d00,stroke-width:2px,color:#1f2937 + style WEBHOOK fill:#dbeafe,stroke:#1a5fb4,stroke-width:2px,color:#1f2937 + style LIB fill:#dbeafe,stroke:#1a5fb4,stroke-width:2px,color:#1f2937 + style RUN fill:#fef3c7,stroke:#b45309,stroke-width:2px,color:#1f2937 + style ENF fill:#dcfce7,stroke:#0b6b3c,stroke-width:2px,color:#1f2937 + style MON fill:#dbeafe,stroke:#1a5fb4,stroke-width:2px,color:#1f2937 +``` + +`CUDA_DEVICE_MEMORY_LIMIT` is the contract between scheduling and enforcement. KAI does not need to know how CUDA calls are intercepted, and HAMi-core does not need to know how the quota was calculated. KAI retains its own scheduling logic; it integrates with HAMi-core rather than replacing its scheduler with the full HAMi platform. + +At runtime, the dynamic linker loads `libvgpu.so` before the CUDA libraries. HAMi-core reads the injected quota, tracks the container's memory use, rewrites device queries so tools such as `nvidia-smi` show the quota, and returns an error when a new allocation would cross the limit. This is CUDA API-level enforcement, rather than an application voluntarily respecting a value. + +## Standard integration on an existing GPU cluster + +This section assumes that the Kubernetes cluster already has working NVIDIA GPUs: nodes advertise `nvidia.com/gpu`, and an ordinary whole-GPU Pod can run `nvidia-smi`. It shows the standard integration path without the GKE-specific workarounds from Lab 12. + +### 1. Install KAI Scheduler + +Enable GPU sharing and the `hamicore` binder plugin: + +```bash +helm install kai-scheduler \ + oci://ghcr.io/kai-scheduler/kai-scheduler/kai-scheduler \ + --namespace kai-scheduler --create-namespace \ + --version v0.17.0 \ + --set global.gpuSharing=true \ + --set binder.plugins.hamicore.enabled=true + +kubectl -n kai-scheduler wait --for=condition=available \ + --timeout=180s deploy --all +kubectl -n kai-scheduler wait --for=condition=Ready \ + --timeout=300s config/kai-config +kubectl get pods -n kai-scheduler +kubectl get queues +``` + +A healthy installation has all KAI components running and the default queue hierarchy available. Pod suffixes vary: + +```text +NAME READY STATUS +admission-... 1/1 Running +binder-... 1/1 Running +kai-operator-... 1/1 Running +kai-scheduler-default-... 1/1 Running +pod-grouper-... 1/1 Running +podgroup-controller-... 1/1 Running +queue-controller-... 1/1 Running + +NAME PARENT +default-parent-queue +default-queue default-parent-queue +``` + +### 2. Install kai-resource-isolator + +Install the HAMi-core library distributor, injection webhook, and optional monitor: + +```bash +helm install kai-resource-isolator \ + oci://docker.io/projecthami/kai-resource-isolator \ + --namespace kai-resource-isolator --create-namespace \ + --version 1.1.0-chart \ + --set monitor.enabled=true + +kubectl rollout status ds/kai-resource-isolator-libsync \ + -n kai-resource-isolator --timeout=300s +kubectl rollout status ds/kai-resource-isolator-monitor \ + -n kai-resource-isolator --timeout=300s +kubectl rollout status deploy/kai-resource-isolator-webhook \ + -n kai-resource-isolator --timeout=300s +kubectl get pods -n kai-resource-isolator +``` + +There should be one ready libsync Pod and one ready monitor Pod per GPU node, plus a ready webhook: + +```text +NAME READY STATUS +kai-resource-isolator-libsync-... 1/1 Running +kai-resource-isolator-monitor-... 1/1 Running +kai-resource-isolator-webhook-... 1/1 Running +``` + +### 3. Run a shared GPU Pod + +The `gpu-memory` annotation is an integer number of MiB without a suffix. The queue label and `schedulerName` send the Pod through KAI Scheduler: + +```bash +kubectl apply -f - <<'EOF' +apiVersion: v1 +kind: Pod +metadata: + name: kai-hami-check + labels: + kai.scheduler/queue: default-queue + annotations: + gpu-memory: "4096" +spec: + schedulerName: kai-scheduler + containers: + - name: cuda + image: nvidia/cuda:12.4.1-base-ubuntu22.04 + command: ["sleep", "infinity"] +EOF + +kubectl wait --for=condition=Ready pod/kai-hami-check --timeout=5m +kubectl get pod kai-hami-check -o wide +``` + +The Pod should be `Running` on a GPU node: + +```text +NAME READY STATUS NODE +kai-hami-check 1/1 Running gpu-node-1 +``` + +### 4. Check the scheduling-to-runtime handoff + +Inspect the quota from KAI, the preload file injected by the isolator, and the memory visible through HAMi-core: + +```bash +kubectl exec kai-hami-check -- sh -lc ' + test -n "$CUDA_DEVICE_MEMORY_LIMIT" + test -f /usr/local/vgpu/libvgpu.so + printf "limit=%s\n" "$CUDA_DEVICE_MEMORY_LIMIT" + cat /etc/ld.so.preload + nvidia-smi --query-gpu=uuid,memory.total --format=csv,noheader +' +``` + +For a 4096 MiB request on a 15360 MiB T4, the verified output was: + +```text +limit=4147m +/usr/local/vgpu/libvgpu.so +GPU-9acc8878-3967-5fb4-c534-43d6fd820fa6, 4147 MiB +``` + +These three lines prove that the integration chain is active: KAI supplied a quota, the isolator injected HAMi-core, and the container sees the resulting limit instead of the full card. The value is 4147 rather than exactly 4096 MiB because KAI converts the request to a two-decimal GPU fraction before calculating the enforced limit. + +### 5. Check the optional monitor + +Each monitor reads caches from its own node, so select the monitor Pod running on the workload node. Keep the port-forward command running in one terminal: + +```bash +export WORKLOAD_NODE=$(kubectl get pod kai-hami-check \ + -o jsonpath='{.spec.nodeName}') +export MONITOR_POD=$(kubectl get pods -n kai-resource-isolator \ + -l app.kubernetes.io/component=kai-vgpu-monitor \ + --field-selector="spec.nodeName=$WORKLOAD_NODE" \ + -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' | head -n 1) +test -n "$MONITOR_POD" || { + echo "No monitor Pod is running on $WORKLOAD_NODE" >&2 + exit 1 +} +kubectl port-forward -n kai-resource-isolator \ + "pod/$MONITOR_POD" 9394:9394 +``` + +From another terminal, confirm that the endpoint contains a series for the Pod: + +```bash +curl -s http://127.0.0.1:9394/metrics | grep \ + 'hami_vgpu_memory_limit_bytes.*pod="kai-hami-check"' +``` + +Expected shape: + +```text +hami_vgpu_memory_limit_bytes{...,pod="kai-hami-check",...} 4.348444672e+09 +``` + +This is enough for a basic integration check. To prove that the displayed ceiling is actually enforced, run a CUDA allocation test that succeeds below the quota and fails above it; the next section summarizes that result, and Lab 12 provides the complete program and procedure. + +Remove the basic test Pod when finished: + +```bash +kubectl delete pod kai-hami-check +``` + +## GKE verification: does the isolation hold? + +We ran the integration on a GKE 1.35/COS/CDI cluster with three `n1-standard-2` nodes, each carrying one NVIDIA T4. KAI Scheduler v0.17.0 handled shared scheduling, while `kai-resource-isolator` 1.1.0-chart injected HAMi-core. + +| Check | Captured result | What it proves | +| :-- | :-- | :-- | +| Node and GPU UUID | Both Pods ran on one single-GPU node and returned `GPU-9acc8878-...` | They shared the same physical T4 | +| Visible memory | Both Pods reported `4147 MiB`; the full card reported `15360 MiB` | HAMi-core exposed KAI's per-Pod quota | +| CUDA allocation | 3 GiB succeeded; a cumulative 5 GiB returned `out of memory` | The ceiling was enforced, not merely displayed | +| Concurrent isolation | Pod B still allocated its own 3 GiB while Pod A held 3 GiB | One Pod could not consume the other's quota | +| Monitor metrics | The same-node `:9394/metrics` endpoint reported both Pods' 4.348 GB limits and 3.328 GB live usage | The monitor exported non-empty per-Pod gauges from each container's cache | + +HAMi-core logged the over-quota allocation as: + +```text +Device 0 OOM 5475663872 / 4348444672 +allocate another 2 GiB: out of memory +PASS: in-quota allocation succeeded and over-quota allocation failed +``` + +Together, these checks connect scheduling onto one card, per-container visibility, actual CUDA allocation enforcement, and observability into one evidence chain. Because the monitor reads node-local caches, the lab queries the monitor instance on the workload node directly instead of using a Service that may select another node. + +:::note Isolation boundary + +This proves CUDA API-level memory enforcement, not a MIG-like hardware security boundary. The tested GKE compatibility path also used privileged workload containers, so it should not be treated as an untrusted multi-tenant security design. + +::: + +## Reproduce the result + +The standard KAI + HAMi-core installation is short. GKE 1.35/COS/CDI additionally requires environment-specific handling for the read-only root filesystem, RuntimeClass, NVML library paths, CDI device injection, and PriorityClass. Those operational details and their captured command output are maintained in: + +**[Lab 12: Verify KAI Scheduler and HAMi Memory Isolation on GKE](/tutorials/labs/kai-scheduler-hami-gke)** + +## Next steps + +- Background and collaboration history: [HAMi-core adopted by NVIDIA KAI Scheduler](/blog/hami-core-adopted-by-nvidia-kai-scheduler) +- User documentation: [How to use HAMi with KAI Scheduler](/docs/next/userguide/kai-scheduler/how-to-use-kai-scheduler) +- Repositories: [KAI-resource-isolator](https://github.com/Project-HAMi/KAI-resource-isolator), [HAMi-core](https://github.com/Project-HAMi/HAMi-core), and [KAI-Scheduler](https://github.com/kai-scheduler/KAI-Scheduler) diff --git a/i18n/zh/docusaurus-plugin-content-blog/authors.yml b/i18n/zh/docusaurus-plugin-content-blog/authors.yml index 2eb071bc..7a68f5d3 100644 --- a/i18n/zh/docusaurus-plugin-content-blog/authors.yml +++ b/i18n/zh/docusaurus-plugin-content-blog/authors.yml @@ -15,3 +15,7 @@ archlitchi: name: 李孟轩 title: HAMi 维护者，密瓜智能联合创始人兼 CTO url: https://github.com/archlitchi + +rootsongjc: + name: 宋净超（Jimmy Song） + url: https://github.com/rootsongjc diff --git a/i18n/zh/docusaurus-plugin-content-blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md b/i18n/zh/docusaurus-plugin-content-blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md new file mode 100644 index 00000000..35d4ec23 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md @@ -0,0 +1,261 @@ +--- +title: "KAI Scheduler 与 HAMi 的 GPU 显存硬隔离：运行原理与实践" +date: "2026-08-11" +description: "简要说明 KAI Scheduler 与 HAMi-core 的隔离链路，并通过可复现的 GKE 实测证明：共享同一张 NVIDIA T4 的两个 Pod 都无法越过各自的显存配额。" +authors: [rootsongjc] +tags: ["HAMi", "KAI Scheduler", "硬隔离", "GPU 共享", "Kubernetes", "云原生"] +--- + +上一篇 [《HAMi-core 被 NVIDIA KAI Scheduler 采用》](/zh/blog/hami-core-adopted-by-nvidia-kai-scheduler)已经介绍了 KAI Scheduler 和这项集成的协作背景。本文不再重复铺垫，只回答一个问题：**KAI Scheduler 把两个 Pod 调度到同一张 GPU 后，HAMi-core 是否真的能限制每个 Pod 的显存用量？** + +我们在 GKE 1.35/COS/CDI 上验证了当前文档支持的组合：KAI Scheduler v0.17.0 与 `kai-resource-isolator` 1.1.0-chart。两个 Pod 共享同一张 NVIDIA T4，各自看到 4147 MiB 上限；申请 3 GiB 成功，累计申请 5 GiB 失败。可选的 monitor 也导出了两个 Pod 的实时显存上限与用量。 + +:::note 关于实测输出 + +下文的 UUID、显存上限、CUDA 分配结果和 monitor 指标均来自该次 GKE 实测。资源后缀和地址在不同集群中会变化。 + +::: + + + +## 集成如何工作 + +整条链路只有三项职责： + +- **KAI Scheduler** 决定 Pod 使用哪张 GPU，并通过 `CUDA_DEVICE_MEMORY_LIMIT` 注入计算出的显存配额。 +- **`kai-resource-isolator`** 分发 `libvgpu.so`；其 webhook 为业务 Pod 挂载该库并配置 `ld.so.preload`。可选的 monitor 读取节点本地 HAMi 缓存，在 `:9394` 暴露 `hami_*` 指标。 +- **HAMi-core（`libvgpu.so`）** 拦截 `cudaMalloc` 等 CUDA 调用，拒绝超出配额的显存分配。 + +```mermaid +%% title: KAI Scheduler 与 HAMi-core 的隔离链路 +graph TD + KAI["KAI Scheduler
计算 GPU 显存配额"] + ENV["注入 CUDA_DEVICE_MEMORY_LIMIT"] + WEBHOOK["kai-resource-isolator webhook
挂载 hostPath 与 ld.so.preload"] + LIB["libsync DaemonSet
在各 GPU 节点提供 libvgpu.so"] + RUN["HAMi-core 拦截 CUDA 调用"] + ENF["拒绝超出配额的分配"] + MON["monitor DaemonSet
在 :9394 暴露 hami_* 指标"] + + KAI --> ENV --> WEBHOOK --> RUN --> ENF + LIB -. "提供 libvgpu.so" .-> RUN + RUN -. "写入节点本地缓存" .-> MON + + style KAI fill:#d9f99d,stroke:#4f7d00,stroke-width:2px,color:#1f2937 + style WEBHOOK fill:#dbeafe,stroke:#1a5fb4,stroke-width:2px,color:#1f2937 + style LIB fill:#dbeafe,stroke:#1a5fb4,stroke-width:2px,color:#1f2937 + style RUN fill:#fef3c7,stroke:#b45309,stroke-width:2px,color:#1f2937 + style ENF fill:#dcfce7,stroke:#0b6b3c,stroke-width:2px,color:#1f2937 + style MON fill:#dbeafe,stroke:#1a5fb4,stroke-width:2px,color:#1f2937 +``` + +`CUDA_DEVICE_MEMORY_LIMIT` 是调度层与隔离层之间的契约。KAI 不需要知道 CUDA 调用如何被拦截，HAMi-core 也不需要知道配额如何计算。KAI 保留自己的调度逻辑；它集成的是 HAMi-core，而不是用完整的 HAMi 平台替换自身调度器。 + +容器启动时，动态链接器会先于 CUDA 库加载 `libvgpu.so`。HAMi-core 读取注入的配额，跟踪容器显存用量，改写设备查询结果，使 `nvidia-smi` 等工具只显示该配额；新的分配一旦越界就返回错误。这是在 CUDA API 层强制执行，并非依赖应用自觉遵守一个数值。 + +## 在已有 GPU 集群上的标准集成步骤 + +下面默认 Kubernetes 集群中的 NVIDIA GPU 已经可用：节点能够上报 `nvidia.com/gpu`，普通整卡 Pod 也能正常执行 `nvidia-smi`。这里只写标准集成路径，不包含实验 12 中针对 GKE 的特殊适配。 + +### 1. 安装 KAI Scheduler + +启用 GPU 共享与 `hamicore` binder 插件： + +```bash +helm install kai-scheduler \ + oci://ghcr.io/kai-scheduler/kai-scheduler/kai-scheduler \ + --namespace kai-scheduler --create-namespace \ + --version v0.17.0 \ + --set global.gpuSharing=true \ + --set binder.plugins.hamicore.enabled=true + +kubectl -n kai-scheduler wait --for=condition=available \ + --timeout=180s deploy --all +kubectl -n kai-scheduler wait --for=condition=Ready \ + --timeout=300s config/kai-config +kubectl get pods -n kai-scheduler +kubectl get queues +``` + +正常情况下，KAI 的所有组件都处于运行状态，并且默认父子队列已经创建。Pod 后缀会因环境而异： + +```text +NAME READY STATUS +admission-... 1/1 Running +binder-... 1/1 Running +kai-operator-... 1/1 Running +kai-scheduler-default-... 1/1 Running +pod-grouper-... 1/1 Running +podgroup-controller-... 1/1 Running +queue-controller-... 1/1 Running + +NAME PARENT +default-parent-queue +default-queue default-parent-queue +``` + +### 2. 安装 kai-resource-isolator + +安装 HAMi-core 库分发组件、注入 webhook 和可选的 monitor： + +```bash +helm install kai-resource-isolator \ + oci://docker.io/projecthami/kai-resource-isolator \ + --namespace kai-resource-isolator --create-namespace \ + --version 1.1.0-chart \ + --set monitor.enabled=true + +kubectl rollout status ds/kai-resource-isolator-libsync \ + -n kai-resource-isolator --timeout=300s +kubectl rollout status ds/kai-resource-isolator-monitor \ + -n kai-resource-isolator --timeout=300s +kubectl rollout status deploy/kai-resource-isolator-webhook \ + -n kai-resource-isolator --timeout=300s +kubectl get pods -n kai-resource-isolator +``` + +每个 GPU 节点上都应该有一个就绪的 libsync Pod 和一个就绪的 monitor Pod，同时 webhook 也应处于就绪状态： + +```text +NAME READY STATUS +kai-resource-isolator-libsync-... 1/1 Running +kai-resource-isolator-monitor-... 1/1 Running +kai-resource-isolator-webhook-... 1/1 Running +``` + +### 3. 运行一个共享 GPU Pod + +`gpu-memory` 注解使用不带单位后缀的整数 MiB。队列标签与 `schedulerName` 会让 Pod 经 KAI Scheduler 调度： + +```bash +kubectl apply -f - <<'EOF' +apiVersion: v1 +kind: Pod +metadata: + name: kai-hami-check + labels: + kai.scheduler/queue: default-queue + annotations: + gpu-memory: "4096" +spec: + schedulerName: kai-scheduler + containers: + - name: cuda + image: nvidia/cuda:12.4.1-base-ubuntu22.04 + command: ["sleep", "infinity"] +EOF + +kubectl wait --for=condition=Ready pod/kai-hami-check --timeout=5m +kubectl get pod kai-hami-check -o wide +``` + +Pod 应该处于 `Running` 状态，并被调度到一个 GPU 节点： + +```text +NAME READY STATUS NODE +kai-hami-check 1/1 Running gpu-node-1 +``` + +### 4. 检查调度层到运行时的交接 + +检查 KAI 提供的配额、isolator 注入的 preload 文件，以及 HAMi-core 向容器暴露的显存： + +```bash +kubectl exec kai-hami-check -- sh -lc ' + test -n "$CUDA_DEVICE_MEMORY_LIMIT" + test -f /usr/local/vgpu/libvgpu.so + printf "limit=%s\n" "$CUDA_DEVICE_MEMORY_LIMIT" + cat /etc/ld.so.preload + nvidia-smi --query-gpu=uuid,memory.total --format=csv,noheader +' +``` + +在一张 15360 MiB T4 上申请 4096 MiB 时，实测输出为： + +```text +limit=4147m +/usr/local/vgpu/libvgpu.so +GPU-9acc8878-3967-5fb4-c534-43d6fd820fa6, 4147 MiB +``` + +这三行可以证明集成链路已经生效：KAI 提供了显存配额，isolator 注入了 HAMi-core，容器看到的是隔离后的上限而非整卡显存。结果是 4147 而不是刚好 4096 MiB，是因为 KAI 会先把请求换算成两位小数的 GPU fraction，再计算最终强制上限。 + +### 5. 检查可选的 monitor + +每个 monitor 只读取本节点上的缓存，因此要选择与业务 Pod 位于同一节点的 monitor。在一个终端中保持下面的端口转发命令运行： + +```bash +export WORKLOAD_NODE=$(kubectl get pod kai-hami-check \ + -o jsonpath='{.spec.nodeName}') +export MONITOR_POD=$(kubectl get pods -n kai-resource-isolator \ + -l app.kubernetes.io/component=kai-vgpu-monitor \ + --field-selector="spec.nodeName=$WORKLOAD_NODE" \ + -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' | head -n 1) +test -n "$MONITOR_POD" || { + echo "$WORKLOAD_NODE 上没有运行 monitor Pod" >&2 + exit 1 +} +kubectl port-forward -n kai-resource-isolator \ + "pod/$MONITOR_POD" 9394:9394 +``` + +在另一个终端确认端点中存在该 Pod 的指标： + +```bash +curl -s http://127.0.0.1:9394/metrics | grep \ + 'hami_vgpu_memory_limit_bytes.*pod="kai-hami-check"' +``` + +输出格式应类似： + +```text +hami_vgpu_memory_limit_bytes{...,pod="kai-hami-check",...} 4.348444672e+09 +``` + +以上检查足以确认基本集成链路正确。要证明显示出来的上限确实无法越过，还需要执行一次 CUDA 分配测试：配额内成功、超额失败。下一节汇总了这项实测结果，实验 12 则提供完整程序和操作过程。 + +完成基本检查后删除测试 Pod： + +```bash +kubectl delete pod kai-hami-check +``` + +## GKE 实测：隔离是否真的生效？ + +验证环境是 GKE 1.35/COS/CDI 集群，包含三个 `n1-standard-2` 节点，每个节点有一张 NVIDIA T4。KAI Scheduler v0.17.0 负责共享调度，`kai-resource-isolator` 1.1.0-chart 负责注入 HAMi-core。 + +| 检查项 | 实测结果 | 证明了什么 | +| :-- | :-- | :-- | +| 节点与 GPU UUID | 两个 Pod 位于同一单卡节点，并返回 `GPU-9acc8878-...` | 它们共享同一张物理 T4 | +| 可见显存 | 两个 Pod 均报告 `4147 MiB`，整卡报告 `15360 MiB` | HAMi-core 暴露了 KAI 分配给各 Pod 的配额 | +| CUDA 分配 | 3 GiB 成功，累计 5 GiB 返回 `out of memory` | 上限确实执行，而不只是显示值变化 | +| 并发隔离 | Pod A 持有 3 GiB 时，Pod B 仍能申请自己的 3 GiB | 一个 Pod 无法占用另一个 Pod 的配额 | +| Monitor 指标 | 同节点 `:9394/metrics` 返回两个 Pod 的 4.348 GB 上限和 3.328 GB 实时用量 | monitor 从容器缓存导出了非空的 Pod 级指标 | + +超额分配时，HAMi-core 记录： + +```text +Device 0 OOM 5475663872 / 4348444672 +allocate another 2 GiB: out of memory +PASS: in-quota allocation succeeded and over-quota allocation failed +``` + +这些结果把“调度到同一张卡”“容器内只见自身配额”“CUDA 分配确实越不过上限”和“monitor 能观测实时用量”连成了一条完整证据链。由于 monitor 读取节点本地缓存，实验会直接查询业务 Pod 所在节点的 monitor 实例，避免 Service 把请求转发到其他节点。 + +:::note 隔离边界 + +这里验证的是 CUDA API 层的显存限制，不是 MIG 一类硬件安全边界。本次 GKE 兼容路径还使用了特权业务容器，因此不应作为不可信多租户安全方案。 + +::: + +## 复现实测结果 + +标准 KAI + HAMi-core 安装链路并不长。GKE 1.35/COS/CDI 还需要针对只读根文件系统、RuntimeClass、NVML 库路径、CDI 设备注入和 PriorityClass 做环境适配。完整操作、实测命令输出与故障排查统一维护在： + +**[实验 12：在 GKE 上验证 KAI Scheduler 与 HAMi 显存隔离](/zh/tutorials/labs/kai-scheduler-hami-gke)** + +## 下一步 + +- 背景与协作过程：[《HAMi-core 被 NVIDIA KAI Scheduler 采用》](/zh/blog/hami-core-adopted-by-nvidia-kai-scheduler) +- 用户文档：[如何在 KAI Scheduler 中使用 HAMi](/zh/docs/next/userguide/kai-scheduler/how-to-use-kai-scheduler) +- 相关仓库：[KAI-resource-isolator](https://github.com/Project-HAMi/KAI-resource-isolator)、[HAMi-core](https://github.com/Project-HAMi/HAMi-core) 和 [KAI-Scheduler](https://github.com/kai-scheduler/KAI-Scheduler) diff --git a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/kai-scheduler-hami-gke.md b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/kai-scheduler-hami-gke.md new file mode 100644 index 00000000..d0fa92d2 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/kai-scheduler-hami-gke.md @@ -0,0 +1,516 @@ +--- +title: "实验 12: 在 GKE 上验证 KAI Scheduler 与 HAMi 显存隔离" +description: "在 GKE 上部署 KAI Scheduler 与 kai-resource-isolator，适配 COS/CDI 运行时，并通过 CUDA 分配验证 Pod 级 GPU 显存隔离。" +sidebar_label: "实验 12: GKE 上的 KAI + HAMi" +lab: + level: Advanced + duration: 约 90 分钟 + environment: GKE 1.35、COS、containerd CDI 与 NVIDIA Tesla T4 + cost: 需要付费的 GKE GPU 节点 + authors: + - rootsongjc + verified: "2026-08-12" +tags: + - kai-scheduler + - hami-core + - gke + - gpu-sharing +toc_max_heading_level: 2 +--- + +本实验在 GKE 上部署 KAI Scheduler v0.17.0 与 `kai-resource-isolator` 1.1.0-chart，并验证共享一张 Tesla T4 的两个 Pod 都无法越过各自的显存配额。实验还记录了验证环境中实际遇到的 GKE 1.35/COS/CDI 兼容问题。 + +:::warning 环境特有的 workaround + +本文的 RuntimeClass、Kyverno、hostPath、PriorityClass 和特权容器调整只适用于验证过的 GKE 1.35/COS/CDI 路径，不是 KAI + HAMi-core 的标准安装方式。只有确认出现对应症状时才应用相应修复。 + +::: + +## 你将学到什么 + +- 启用 KAI GPU sharing 与 `hamicore` binder 插件； +- 让 `kai-resource-isolator` 适配 GKE 的只读根文件系统与 CDI 设备注入； +- 证明两个 Pod 使用同一张 GPU，并分别只看到 4147 MiB； +- 用 `cudaMalloc` 验证配额内成功、越界失败和两个 Pod 互不影响。 + +## 实验概览 + +```mermaid +%% title: GKE 上的 KAI Scheduler 与 HAMi-core 实验流程 +flowchart LR + S1["步骤 1
验证 GKE GPU"] --> S2["步骤 2
补充 GPU 标签"] + S2 --> S3["步骤 3
安装 KAI"] + S3 --> S4["步骤 4
安装 isolator"] + S4 --> S5["步骤 5
应用 GKE 适配"] + S5 --> S6["步骤 6
共享一张 T4"] + S6 --> S7["步骤 7
执行 CUDA OOM 验证"] + S7 --> S8["步骤 8
验证 monitor 指标"] +``` + +## 前提条件 + +- 已启用 GKE 与 Compute Engine API 的 GCP 项目。 +- 一套使用 COS 节点的 GKE 1.35 集群，至少有一张 NVIDIA T4。验证集群有三个 `n1-standard-2` 节点，每节点一张 T4。 +- 使用 GKE 管理的 NVIDIA 驱动、device plugin 与 container toolkit。不要在 GKE 驱动之上重复安装 GPU Operator。 +- 具有集群管理员权限的 `gcloud`、与 GKE API Server 相差不超过一个次版本的 `kubectl`，以及 Helm 3 或 4。 +- [`tutorials/labs/examples/12-kai-scheduler-hami-gke/`](https://github.com/Project-HAMi/website/tree/master/tutorials/labs/examples/12-kai-scheduler-hami-gke) 下的实验文件。 + +验证集群的控制面与节点版本均为 GKE `1.35.6-gke.1250000`。GKE 补丁版本会逐渐下线，因此先从目标可用区选择当前可用的 1.35 版本，再创建集群： + +```bash +export GKE_VERSION=$(gcloud container get-server-config \ + --zone=asia-northeast1-a \ + --format='value(validMasterVersions)' | tr ';' '\n' | grep '^1\.35\.' | head -1) +test -n "$GKE_VERSION" + +gcloud container clusters create kai-hami-test --zone=asia-northeast1-a \ + --cluster-version="$GKE_VERSION" \ + --machine-type=n1-standard-2 --num-nodes=3 \ + --image-type=COS_CONTAINERD \ + --accelerator=type=nvidia-tesla-t4,count=1,gpu-driver-version=default +gcloud container clusters get-credentials kai-hami-test \ + --zone=asia-northeast1-a +``` + +GPU 节点会产生费用，完成实验后请执行清理步骤。 + +:::note 关于输出块 + +下文输出均采集自 2026-08-12 的验证运行。Pod 后缀、运行时长、IP 地址和节点名称因环境而异；复现时应重点比较组件名称、就绪状态、调度位置和测量值。 + +::: + +## 步骤 1: 验证 GKE GPU 栈 + +确认 GPU 节点已经上报 `nvidia.com/gpu`： + +```bash +kubectl get nodes \ + -o custom-columns="NAME:.metadata.name,GPU:.status.capacity.nvidia\.com/gpu,ACCEL:.metadata.labels.cloud\.google\.com/gke-accelerator" +``` + +验证集群包含三个 T4 节点： + +```plaintext +NAME GPU ACCEL +gke-kai-hami-test-default-pool-370c394b-fxh2 1 nvidia-tesla-t4 +gke-kai-hami-test-default-pool-370c394b-pm4j 1 nvidia-tesla-t4 +gke-kai-hami-test-default-pool-370c394b-r8n5 1 nvidia-tesla-t4 +``` + +如果 `GPU` 为空，且节点带有 `gke-no-default-nvidia-gpu-device-plugin=true`，启用 GKE device plugin： + +```bash +kubectl label node -l cloud.google.com/gke-accelerator=nvidia-tesla-t4 \ + gke-no-default-nvidia-gpu-device-plugin- +``` + +安装 KAI 前先运行普通整卡 Pod： + +```bash +kubectl apply -f - <<'EOF' +apiVersion: v1 +kind: Pod +metadata: + name: gpu-smi-test +spec: + restartPolicy: Never + containers: + - name: cuda + image: nvidia/cuda:12.4.1-base-ubuntu22.04 + command: ["nvidia-smi"] + resources: + limits: + nvidia.com/gpu: 1 +EOF +kubectl wait --for=jsonpath='{.status.phase}'=Succeeded \ + pod/gpu-smi-test --timeout=180s +kubectl logs gpu-smi-test +kubectl delete pod gpu-smi-test +``` + +实测 `nvidia-smi` 中与本实验相关的输出为： + +```plaintext +GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC + 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 +... 0MiB / 15360MiB +``` + +驱动补丁版本可能不同，但业务镜像必须与之兼容。记录这里报告的显存值，供步骤 2 使用。 + +## 步骤 2: 添加 KAI 所需的 GPU 标签 + +GKE 默认 device plugin 不提供完整的 GPU Feature Discovery 标签。KAI 在注册节点时读取 `nvidia.com/gpu.memory`，因此要在安装 KAI 前添加： + +```bash +kubectl label node -l cloud.google.com/gke-accelerator=nvidia-tesla-t4 \ + nvidia.com/gpu.memory=15360 \ + nvidia.com/gpu.product=NVIDIA-Tesla-T4 \ + nvidia.com/gpu.count=1 \ + nvidia.com/gpu.present=true --overwrite + +kubectl get nodes -o custom-columns=\ +'NAME:.metadata.name,GPU.MEMORY:.metadata.labels.nvidia\.com/gpu\.memory,GPU.PRODUCT:.metadata.labels.nvidia\.com/gpu\.product' +``` + +实测节点随后暴露了 KAI 读取的标签值： + +```plaintext +NAME GPU.MEMORY GPU.PRODUCT +gke-kai-hami-test-default-pool-370c394b-fxh2 15360 NVIDIA-Tesla-T4 +gke-kai-hami-test-default-pool-370c394b-pm4j 15360 NVIDIA-Tesla-T4 +gke-kai-hami-test-default-pool-370c394b-r8n5 15360 NVIDIA-Tesla-T4 +``` + +显存值应来自 `nvidia-smi`，不要使用 T4 标称的 16 GiB。如果 KAI 启动后才补标签，需要重启 `kai-scheduler` 刷新节点缓存。 + +## 步骤 3: 安装 KAI Scheduler 与默认队列 + +安装 KAI v0.17.0，启用 GPU sharing、HAMi-core 集成和 CDI： + +```bash +helm install kai-scheduler \ + oci://ghcr.io/kai-scheduler/kai-scheduler/kai-scheduler \ + --namespace kai-scheduler --create-namespace \ + --version v0.17.0 \ + --set global.gpuSharing=true \ + --set binder.plugins.hamicore.enabled=true \ + --set-string binder.plugins.gpusharing.arguments.cdiEnabled=true + +kubectl -n kai-scheduler wait --for=condition=available \ + --timeout=180s deploy --all +kubectl -n kai-scheduler wait --for=condition=Ready \ + --timeout=300s config/kai-config +kubectl get pods -n kai-scheduler +kubectl get queues +``` + +实测安装中的七个 KAI 控制面组件均处于运行状态。自动生成的 Pod 后缀会不同： + +```plaintext +NAME READY STATUS RESTARTS AGE +admission-759b9bb99c-... 1/1 Running 0 4m +binder-54665cc5d9-... 1/1 Running 0 4m +kai-operator-997c6886c-... 1/1 Running 0 4m +kai-scheduler-default-d85d7dbdf-... 1/1 Running 0 4m +pod-grouper-68f4fb47-... 1/1 Running 0 4m +podgroup-controller-5947b5b4dd-... 1/1 Running 0 4m +queue-controller-6cc8c844c8-... 1/1 Running 0 4m +``` + +KAI v0.17.0 还会自动创建默认的父子队列： + +```plaintext +NAME PARENT +default-parent-queue +default-queue default-parent-queue +``` + +:::important `cdiEnabled` 必须是字符串 + +这里必须使用 `--set-string`，不能使用普通的 `--set`。KAI v0.17.0 的 Helm values 虽然能接受两种写法，但生成的 `Config` CRD 字段类型是字符串。普通的 `--set ...=true` 会渲染成布尔值，使 `kai-config-deployer` hook 报错：`cdiEnabled ... must be of type string: "boolean"`。上面的正确命令已经过 Chart 渲染，并通过当前 GKE API Server 的 server-side dry-run。 + +::: + +## 步骤 4: 安装 kai-resource-isolator + +COS 的根文件系统只读，因此将 HAMi-core 写入 GKE 可写的 NVIDIA 目录： + +```bash +helm install kai-resource-isolator \ + oci://docker.io/projecthami/kai-resource-isolator \ + --namespace kai-resource-isolator --create-namespace \ + --version 1.1.0-chart \ + --set paths.containerVgpuMount=/home/kubernetes/bin/nvidia/vgpu \ + --set-string librarySync.priorityClassName= \ + --set-string monitor.priorityClassName= \ + --set monitor.enabled=true +``` + +`containerVgpuMount` 是 1.1.0-chart 中真正控制 libsync 目标目录、preload 文件、webhook 注入路径和 monitor 缓存的值。不要使用 `paths.hostInstallBase`：它虽然出现在该 Chart 的 values 中，但模板没有引用它。两个空 PriorityClass 值用于避免 GKE 拒绝系统命名空间之外使用 `system-node-critical` 的 Pod。 + +检查渲染后的路径并等待组件就绪： + +```bash +kubectl get cm kai-resource-isolator-ldpreload \ + -n kai-resource-isolator -o jsonpath='{.data.ld\.so\.preload}' +kubectl rollout status ds/kai-resource-isolator-libsync \ + -n kai-resource-isolator --timeout=300s +kubectl rollout status ds/kai-resource-isolator-monitor \ + -n kai-resource-isolator --timeout=300s +kubectl rollout status deploy/kai-resource-isolator-webhook \ + -n kai-resource-isolator --timeout=300s +kubectl get pods -n kai-resource-isolator +``` + +实测路径和组件状态如下： + +```plaintext +/home/kubernetes/bin/nvidia/vgpu/libvgpu.so + +NAME READY STATUS RESTARTS AGE +kai-resource-isolator-libsync-... 1/1 Running 0 2m +kai-resource-isolator-libsync-... 1/1 Running 0 2m +kai-resource-isolator-libsync-... 1/1 Running 0 2m +kai-resource-isolator-monitor-26bj8 1/1 Running 0 2m +kai-resource-isolator-monitor-hf67f 1/1 Running 0 2m +kai-resource-isolator-monitor-tnj9l 1/1 Running 0 2m +kai-resource-isolator-webhook-... 1/1 Running 0 2m +``` + +每个 GPU 节点都应有一个 libsync 和一个 monitor Pod，并且 webhook Pod 已就绪。 + +## 步骤 5: 适配 GKE CDI 设备路径 + +验证集群使用 CDI，没有注册 `nvidia` runtime handler，但 KAI reservation Pod 引用了 `runtimeClassName: nvidia`。只有 RuntimeClass 不存在时才创建兼容对象： + +```bash +kubectl get runtimeclass nvidia || kubectl apply \ + -f tutorials/labs/examples/12-kai-scheduler-hami-gke/02-runtimeclass.yaml +``` + +GKE device plugin 只向申请 `nvidia.com/gpu` 的 Pod 注入设备；KAI 共享 Pod 使用 `gpu-memory`，因此验证环境需要显式挂载设备与库。安装 Kyverno 并应用两条策略： + +```bash +helm repo add kyverno https://kyverno.github.io/kyverno/ +helm repo update +helm install kyverno kyverno/kyverno \ + --namespace kyverno --create-namespace +kubectl wait -n kyverno --for=condition=Ready pod \ + -l app.kubernetes.io/component=admission-controller --timeout=300s +kubectl apply \ + -f tutorials/labs/examples/12-kai-scheduler-hami-gke/03-gke-policies.yaml +kubectl wait --for=condition=Ready --timeout=180s \ + clusterpolicy/inject-nvidia-library-path \ + clusterpolicy/inject-gpu-devices +kubectl get runtimeclass nvidia +kubectl get pods -n kyverno +kubectl get clusterpolicy \ + inject-nvidia-library-path inject-gpu-devices +``` + +实测时兼容对象与 Kyverno 控制器均已就绪： + +```plaintext +NAME HANDLER AGE +nvidia runc 3m + +NAME READY STATUS RESTARTS AGE +kyverno-admission-controller-7cdf5b9c-... 1/1 Running 0 2m +kyverno-background-controller-7b54965bf9-... 1/1 Running 0 2m +kyverno-cleanup-controller-59c8fdfb66-... 1/1 Running 0 2m +kyverno-reports-controller-5c96886c9-... 1/1 Running 0 2m + +NAME ADMISSION BACKGROUND READY +inject-nvidia-library-path true true true +inject-gpu-devices true true true +``` + +第一条策略给 reservation Pod 添加 NVML 库路径；第二条给共享 Pod 挂载 `/dev/nvidia*`、`nvidia-smi` 与 NVIDIA 库。使用这条 workaround 时，共享 Pod 还需要 `privileged: true`。 + +:::caution 安全边界 + +本实验验证 CUDA API 层的显存限制，不是 MIG 一类硬件安全边界。GKE workaround 还使用了特权业务容器，不应将其作为不可信多租户安全方案。 + +::: + +## 步骤 6: 让两个 Pod 共享一张 T4 + +选择一台没有其他共享工作负载、且只有一张 T4 的节点。不要选择已经运行 `gpu-memory` 工作负载的节点：两个 4 GiB 实验 Pod 需要该卡至少还有 8 GiB 未被 KAI 分配。先列出候选节点与当前 Pod 分布： + +```bash +kubectl get nodes -l cloud.google.com/gke-accelerator=nvidia-tesla-t4 +kubectl get pods -A -o wide + +# 将下面的值替换为上面输出中的一台空闲单卡 T4 节点。 +export TEST_NODE= +test "$(kubectl get node "$TEST_NODE" \ + -o jsonpath='{.status.capacity.nvidia\.com/gpu}')" = "1" +kubectl label node "$TEST_NODE" hami.run/lab-12=true --overwrite + +kubectl create configmap kai-hami-lab12-source \ + --from-file=memory-limit.cu=tutorials/labs/examples/12-kai-scheduler-hami-gke/memory-limit.cu \ + --dry-run=client -o yaml | kubectl apply -f - +kubectl apply \ + -f tutorials/labs/examples/12-kai-scheduler-hami-gke/04-shared-pods.yaml +kubectl wait --for=condition=Ready \ + pod/kai-hami-lab12-a pod/kai-hami-lab12-b \ + --timeout=10m +``` + +检查节点位置，并从两个 Pod 的启动日志读取注入配额、物理 UUID 与可见显存： + +```bash +kubectl get pod kai-hami-lab12-a kai-hami-lab12-b -o wide +for pod in kai-hami-lab12-a kai-hami-lab12-b; do + echo "=== $pod ===" + kubectl logs "$pod" +done +``` + +实测时两个 Pod 在同一节点就绪： + +```plaintext +NAME READY STATUS RESTARTS IP NODE +kai-hami-lab12-a 1/1 Running 0 10.84.2.66 gke-kai-hami-test-default-pool-370c394b-pm4j +kai-hami-lab12-b 1/1 Running 0 10.84.2.67 gke-kai-hami-test-default-pool-370c394b-pm4j +``` + +两者的启动日志都返回： + +```plaintext +limit=4147m +GPU-9acc8878-3967-5fb4-c534-43d6fd820fa6, 4147 MiB +``` + +相同 UUID 证明两个 Pod 使用同一张 T4。4147 MiB 是 KAI 把 15360 MiB 卡上的 4096 MiB 请求换算为两位小数 fraction 后的舍入结果。 + +## 步骤 7: 验证 CUDA 显存上限 + +步骤 6 创建的源码 ConfigMap 已挂载到 `/lab-source`。每个 Pod 启动时会自动编译程序，成功申请 3 GiB 后保持 30 秒，再尝试追加 2 GiB。验证通过后，Pod 会再次启动程序并持续持有 3 GiB，供步骤 8 验证实时监控指标。两个阶段均成功启动后，readiness probe 才会成功。 + +查看两个 Pod 日志中的测试区间与结果： + +```bash +for pod in kai-hami-lab12-a kai-hami-lab12-b; do + kubectl logs "$pod" | grep -E \ + 'test_(start|end)=|allocate |PASS:' +done +``` + +实测时间区间互相重叠，并且每个 Pod 都返回 `PASS`： + +```plaintext +=== kai-hami-lab12-a === +test_start=2026-08-12T05:11:13Z +allocate 3 GiB: no error +allocate another 2 GiB: out of memory +PASS: in-quota allocation succeeded and over-quota allocation failed +test_end=2026-08-12T05:11:44Z + +=== kai-hami-lab12-b === +test_start=2026-08-12T05:11:11Z +allocate 3 GiB: no error +allocate another 2 GiB: out of memory +PASS: in-quota allocation succeeded and over-quota allocation failed +test_end=2026-08-12T05:11:43Z +``` + +比较 `test_start` 与 `test_end`，两个 30 秒区间必须重叠。实测时两个 Pod 在同时持有 3 GiB 的情况下都返回 `PASS`。各自累计申请 5 GiB 时，HAMi-core 都记录 `Device 0 OOM 5475663872 / 4348444672`。将验证程序作为容器启动命令执行，也避免了长时间 `kubectl exec` WebSocket 中断影响判断。 + +## 步骤 8: 验证 monitor 指标 + +monitor 是 DaemonSet：每个实例只读取本节点上的 HAMi 共享内存缓存。通过 Service 访问时可能被转发到其他节点的 monitor，从而看不到这两个 Pod 的序列。因此要直接查询工作负载所在节点的 monitor 实例。这里重新从运行中的 Pod 获取节点，不再依赖步骤 6 所在 shell 导出的临时变量。 + +创建一个临时 curl Pod，读取该 monitor 的 `:9394/metrics`： + +```bash +export TEST_NODE=$(kubectl get pod kai-hami-lab12-a \ + -o jsonpath='{.spec.nodeName}') +test -n "$TEST_NODE" + +export MONITOR_IP=$(kubectl get pods -n kai-resource-isolator \ + -l app.kubernetes.io/component=kai-vgpu-monitor \ + --field-selector="spec.nodeName=$TEST_NODE" \ + -o jsonpath='{range .items[*]}{.status.podIP}{"\n"}{end}' | head -n 1) + +if test -z "$MONITOR_IP"; then + echo "$TEST_NODE 上没有运行 monitor Pod" >&2 + kubectl get pods -n kai-resource-isolator \ + -l app.kubernetes.io/component=kai-vgpu-monitor -o wide + exit 1 +fi + +kubectl delete pod lab12-monitor-check --ignore-not-found +kubectl run lab12-monitor-check \ + --image=curlimages/curl:8.15.0 --restart=Never \ + --command -- sh -lc \ + "curl -fsS http://$MONITOR_IP:9394/metrics" +kubectl wait --for=jsonpath='{.status.phase}'=Succeeded \ + pod/lab12-monitor-check --timeout=180s +kubectl logs lab12-monitor-check | grep -E \ + '^hami_vgpu_memory_(used|limit)_bytes.*pod="kai-hami-lab12-[ab]"' +``` + +实测端点为每个 Pod 分别返回一条 `used` 和 `limit` 序列： + +```plaintext +hami_vgpu_memory_limit_bytes{...,pod="kai-hami-lab12-a",...} 4.348444672e+09 +hami_vgpu_memory_limit_bytes{...,pod="kai-hami-lab12-b",...} 4.348444672e+09 +hami_vgpu_memory_used_bytes{...,pod="kai-hami-lab12-a",...} 3.328180224e+09 +hami_vgpu_memory_used_bytes{...,pod="kai-hami-lab12-b",...} 3.328180224e+09 +``` + +limit 等于 4147 MiB 对应的字节数；used 略高于 3 GiB，因为其中包含 CUDA context 与分配器开销。这证明 monitor 确实发现了两个容器的缓存并导出了实时用量，而不是只提供一个没有业务序列的 Prometheus 端点。 + +## 故障排查 + +| 现象 | 验证环境中的原因 | 处理方式 | +| :-- | :-- | :-- | +| 共享 Pod Pending，提示 `didn't have enough resources: GPU memory` | KAI 缓存节点时缺少 `nvidia.com/gpu.memory` | 补标签并重启 `kai-scheduler` | +| Queue 被拒绝 | KAI admission webhook 尚未就绪 | 等待 KAI Deployment 后再创建队列 | +| `RuntimeClass "nvidia" not found` | GKE CDI 使用 `runc`，没有 NVIDIA handler | 应用 `02-runtimeclass.yaml` | +| reservation Pod 报 `ERROR_LIBRARY_NOT_FOUND` | NVML 位于 `/usr/local/nvidia/lib64`，但不在搜索路径 | 应用 Kyverno 库路径策略 | +| 共享 Pod 看不到 `/dev/nvidia*` | 它只申请 `gpu-memory`，GKE device plugin 不执行 Allocate | 应用 Kyverno 设备挂载策略 | +| libsync 报 `Read-only file system` | COS 根文件系统只读 | 设置 `paths.containerVgpuMount=/home/kubernetes/bin/nvidia/vgpu` | +| `libvgpu.so` 无法 preload | 实际挂载路径仍指向 `/usr/local/vgpu` | 使用验证过的 `containerVgpuMount` 值重新安装 | +| DaemonSet 因 `system-node-critical` 被拒绝 | GKE PriorityClass 配额限制用户命名空间 | 将 Chart 中两个 PriorityClass 值设为空字符串 | +| `CUDA driver version is insufficient` | CUDA 镜像版本超过节点驱动支持范围 | 使用实测的 CUDA 12.4.1 或其他兼容版本 | +| `kubectl exec` 出现 WebSocket EOF | 控制面 exec 流被重置，或客户端/服务端版本偏差不受支持 | 使用兼容的 `kubectl`；本实验在容器启动时执行验证，并通过 `kubectl logs` 读取 | +| monitor 查询不到 Pod 或没有实验序列 | `$TEST_NODE` 已失效，或工作负载节点上没有 monitor | 从 `kai-hami-lab12-a` 重新获取节点，再用 `kubectl get pods -n kai-resource-isolator -l app.kubernetes.io/component=kai-vgpu-monitor -o wide` 检查 DaemonSet | + +## 清理 + +删除业务 Pod 与测试标签： + +```bash +kubectl delete \ + -f tutorials/labs/examples/12-kai-scheduler-hami-gke/04-shared-pods.yaml +kubectl delete pod lab12-monitor-check --ignore-not-found +kubectl delete configmap kai-hami-lab12-source --ignore-not-found +kubectl label node "$TEST_NODE" hami.run/lab-12- --overwrite +``` + +如果集群只用于本实验，再删除其余组件： + +```bash +kubectl delete \ + -f tutorials/labs/examples/12-kai-scheduler-hami-gke/03-gke-policies.yaml +helm uninstall kyverno -n kyverno +helm uninstall kai-resource-isolator -n kai-resource-isolator +kubectl delete queues default-queue default-parent-queue --ignore-not-found +helm uninstall kai-scheduler -n kai-scheduler +``` + +这里显式删除 Queue 是有意为之：KAI 给默认队列添加了 `helm.sh/resource-policy: keep` 注解，因此 Helm 卸载时会保留它们。 + +只有步骤 5 创建了 RuntimeClass 时才删除 `02-runtimeclass.yaml`；如果该 RuntimeClass 原本就属于集群，应予以保留： + +```bash +kubectl delete \ + -f tutorials/labs/examples/12-kai-scheduler-hami-gke/02-runtimeclass.yaml +``` + +如果 GKE 集群专门为本实验创建，可将其删除： + +```bash +gcloud container clusters delete kai-hami-test \ + --zone=asia-northeast1-a +``` + +## 本实验验证了什么 + +| 结论 | 证据 | +| :-- | :-- | +| KAI 把两个分片工作负载调度到同一张 T4 | 两个 Pod 节点相同、GPU UUID 相同 | +| HAMi-core 改写每个 Pod 可见的显存上限 | 两个 Pod 均显示 4147 MiB，而非 15360 MiB | +| 上限被真正执行，而非只修改显示 | 3 GiB 成功，累计 5 GiB 返回 CUDA OOM | +| 一个 Pod 无法占用另一个 Pod 的配额 | Pod A 持有 3 GiB 时，Pod B 仍成功分配 3 GiB | +| 可选 monitor 能读取每容器缓存 | `:9394/metrics` 返回两个 Pod 的 4147 MiB 上限与实时 3 GiB 用量 | + +## 下一步 + +- 阅读[《KAI Scheduler 与 HAMi 的 GPU 显存硬隔离》](/zh/blog/kai-scheduler-hami-gpu-memory-hard-isolation)，了解架构与集成背景。 +- 对比[实验 3：HAMi GPU 切分](./gpu-partitioning.md)和[实验 7：k3s GPU 隔离](./hami-isolation-k3s.md)。 +- 关注 KAI 与 `kai-resource-isolator` 后续版本；当 GKE CDI 获得原生支持后，删除本实验中对应的 workaround。 diff --git a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/overview.md b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/overview.md index 32683f4b..80667767 100644 --- a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/overview.md @@ -19,4 +19,4 @@ import LabCardGridAuto from '@site/src/components/labs/LabCardGridAuto'; -每个实验都列出了各自的前提条件。实验 3 和 4 直接复用实验 1 搭建的集群，一次开机即可完成全部三个实验；实验 2 可在任意笔记本上运行，无需 GPU。实验 7 在租用的 GPU 虚拟机上自行搭建单节点 k3s 集群，不使用 GPU Operator。实验 8 需要已有的 Volcano GPU 集群，用于验证 Volcano vGPU、Gang 调度和队列级资源限制。实验 9 使用 Kueue 准入控制限制 HAMi vGPU 数量、显存和算力配额。实验 11 将从头搭建完整的 KServe Standard 推理环境，并通过 HAMi 原生 DRA Claim 让两个 vLLM 副本共享一张 GPU。 +每个实验都列出了各自的前提条件。实验 3 和 4 直接复用实验 1 搭建的集群，一次开机即可完成全部三个实验；实验 2 可在任意笔记本上运行，无需 GPU。实验 7 在租用的 GPU 虚拟机上自行搭建单节点 k3s 集群，不使用 GPU Operator。实验 8 需要已有的 Volcano GPU 集群，用于验证 Volcano vGPU、Gang 调度和队列级资源限制。实验 9 使用 Kueue 准入控制限制 HAMi vGPU 数量、显存和算力配额。实验 11 将从头搭建完整的 KServe Standard 推理环境，并通过 HAMi 原生 DRA Claim 让两个 vLLM 副本共享一张 GPU。实验 12 在 GKE 1.35/COS/CDI 上部署 KAI Scheduler 与 HAMi-core，并通过 CUDA 分配验证显存上限。 diff --git a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/tags.yml b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/tags.yml index 033e625a..eafe5c2a 100644 --- a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/tags.yml +++ b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/tags.yml @@ -10,15 +10,27 @@ "gang-scheduling": label: "gang-scheduling" permalink: "/gang-scheduling" +"gke": + label: "GKE" + permalink: "/gke" +"gpu-sharing": + label: "GPU 共享" + permalink: "/gpu-sharing" "hami": label: "hami" permalink: "/hami" +"hami-core": + label: "HAMi-core" + permalink: "/hami-core" "inference": label: "inference" permalink: "/inference" "k3s": label: "k3s" permalink: "/k-3-s" +"kai-scheduler": + label: "KAI Scheduler" + permalink: "/kai-scheduler" "kueue": label: "Kueue" permalink: "/kueue" diff --git a/sidebars-tutorials.js b/sidebars-tutorials.js index a8db41ff..9e28572c 100644 --- a/sidebars-tutorials.js +++ b/sidebars-tutorials.js @@ -66,6 +66,11 @@ module.exports = { id: "labs/kserve-hami-dra", customProps: { level: "Advanced", duration: "about 90 minutes" }, }, + { + type: "doc", + id: "labs/kai-scheduler-hami-gke", + customProps: { level: "Advanced", duration: "about 90 minutes" }, + }, ], }, ], diff --git a/tutorials/labs/examples/12-kai-scheduler-hami-gke/02-runtimeclass.yaml b/tutorials/labs/examples/12-kai-scheduler-hami-gke/02-runtimeclass.yaml new file mode 100644 index 00000000..5a36705d --- /dev/null +++ b/tutorials/labs/examples/12-kai-scheduler-hami-gke/02-runtimeclass.yaml @@ -0,0 +1,5 @@ +apiVersion: node.k8s.io/v1 +kind: RuntimeClass +metadata: + name: nvidia +handler: runc diff --git a/tutorials/labs/examples/12-kai-scheduler-hami-gke/03-gke-policies.yaml b/tutorials/labs/examples/12-kai-scheduler-hami-gke/03-gke-policies.yaml new file mode 100644 index 00000000..469af90d --- /dev/null +++ b/tutorials/labs/examples/12-kai-scheduler-hami-gke/03-gke-policies.yaml @@ -0,0 +1,88 @@ +apiVersion: kyverno.io/v1 +kind: ClusterPolicy +metadata: + name: inject-nvidia-library-path +spec: + rules: + - name: add-ld-library-path + match: + any: + - resources: + kinds: [Pod] + namespaces: [kai-resource-reservation] + mutate: + foreach: + - list: "request.object.spec.containers[]" + patchStrategicMerge: + spec: + containers: + - name: "{{ element.name }}" + env: + - name: LD_LIBRARY_PATH + value: /usr/local/nvidia/lib64 +--- +apiVersion: kyverno.io/v1 +kind: ClusterPolicy +metadata: + name: inject-gpu-devices +spec: + background: false + rules: + - name: add-gpu-volumes + match: + any: + - resources: + kinds: [Pod] + preconditions: + all: + - key: '{{ request.object.metadata.annotations."gpu-memory" || '''' }}' + operator: NotEquals + value: "" + mutate: + patchesJson6902: |- + - op: add + path: /spec/volumes/- + value: {name: nvidia-dev0, hostPath: {path: /dev/nvidia0, type: CharDevice}} + - op: add + path: /spec/volumes/- + value: {name: nvidia-ctl, hostPath: {path: /dev/nvidiactl, type: CharDevice}} + - op: add + path: /spec/volumes/- + value: {name: nvidia-uvm, hostPath: {path: /dev/nvidia-uvm, type: CharDevice}} + - op: add + path: /spec/volumes/- + value: {name: nvidia-modeset, hostPath: {path: /dev/nvidia-modeset, type: CharDevice}} + - op: add + path: /spec/volumes/- + value: {name: nvidia-caps, hostPath: {path: /dev/nvidia-caps, type: Directory}} + - op: add + path: /spec/volumes/- + value: {name: nvidia-bin, hostPath: {path: /home/kubernetes/bin/nvidia/bin, type: Directory}} + - op: add + path: /spec/volumes/- + value: {name: nvidia-lib, hostPath: {path: /home/kubernetes/bin/nvidia/lib64, type: Directory}} + - name: add-gpu-volume-mounts + match: + any: + - resources: + kinds: [Pod] + preconditions: + all: + - key: '{{ request.object.metadata.annotations."gpu-memory" || '''' }}' + operator: NotEquals + value: "" + mutate: + foreach: + - list: "request.object.spec.containers[]" + patchStrategicMerge: + spec: + containers: + - name: "{{ element.name }}" + volumeMounts: + - { name: nvidia-dev0, mountPath: /dev/nvidia0 } + - { name: nvidia-ctl, mountPath: /dev/nvidiactl } + - { name: nvidia-uvm, mountPath: /dev/nvidia-uvm } + - { name: nvidia-modeset, mountPath: /dev/nvidia-modeset } + - { name: nvidia-caps, mountPath: /dev/nvidia-caps } + - { name: nvidia-bin, mountPath: /usr/local/nvidia/bin } + - { name: nvidia-lib, mountPath: /usr/local/nvidia/lib64 } diff --git a/tutorials/labs/examples/12-kai-scheduler-hami-gke/04-shared-pods.yaml b/tutorials/labs/examples/12-kai-scheduler-hami-gke/04-shared-pods.yaml new file mode 100644 index 00000000..18d67829 --- /dev/null +++ b/tutorials/labs/examples/12-kai-scheduler-hami-gke/04-shared-pods.yaml @@ -0,0 +1,111 @@ +apiVersion: v1 +kind: Pod +metadata: + name: kai-hami-lab12-a + labels: + kai.scheduler/queue: default-queue + annotations: + gpu-memory: "4096" +spec: + schedulerName: kai-scheduler + restartPolicy: Never + nodeSelector: + hami.run/lab-12: "true" + containers: + - name: gpu-workload + image: nvidia/cuda:12.4.1-devel-ubuntu22.04 + command: ["/bin/bash", "-lc"] + args: + - | + nvcc -O2 /lab-source/memory-limit.cu -o /tmp/memory-limit-test + printf 'limit=%s\n' "$CUDA_DEVICE_MEMORY_LIMIT" + /usr/local/nvidia/bin/nvidia-smi \ + --query-gpu=uuid,memory.total --format=csv,noheader + echo "test_start=$(date -u +%FT%TZ)" + HOLD_SECONDS=30 /tmp/memory-limit-test + echo "test_end=$(date -u +%FT%TZ)" + (HOLD_SECONDS=3600 /tmp/memory-limit-test \ + > /tmp/monitor-hold.log 2>&1) & + for attempt in $(seq 1 30); do + grep -q 'allocate another 2 GiB: out of memory' \ + /tmp/monitor-hold.log && break + sleep 1 + done + grep -q 'allocate another 2 GiB: out of memory' \ + /tmp/monitor-hold.log + touch /tmp/memory-test-passed + exec sleep infinity + env: + - name: LD_LIBRARY_PATH + value: /usr/local/nvidia/lib64:/usr/local/cuda/lib64 + securityContext: + privileged: true + readinessProbe: + exec: + command: ["test", "-f", "/tmp/memory-test-passed"] + periodSeconds: 2 + failureThreshold: 150 + volumeMounts: + - name: lab-source + mountPath: /lab-source + readOnly: true + volumes: + - name: lab-source + configMap: + name: kai-hami-lab12-source +--- +apiVersion: v1 +kind: Pod +metadata: + name: kai-hami-lab12-b + labels: + kai.scheduler/queue: default-queue + annotations: + gpu-memory: "4096" +spec: + schedulerName: kai-scheduler + restartPolicy: Never + nodeSelector: + hami.run/lab-12: "true" + containers: + - name: gpu-workload + image: nvidia/cuda:12.4.1-devel-ubuntu22.04 + command: ["/bin/bash", "-lc"] + args: + - | + nvcc -O2 /lab-source/memory-limit.cu -o /tmp/memory-limit-test + printf 'limit=%s\n' "$CUDA_DEVICE_MEMORY_LIMIT" + /usr/local/nvidia/bin/nvidia-smi \ + --query-gpu=uuid,memory.total --format=csv,noheader + echo "test_start=$(date -u +%FT%TZ)" + HOLD_SECONDS=30 /tmp/memory-limit-test + echo "test_end=$(date -u +%FT%TZ)" + (HOLD_SECONDS=3600 /tmp/memory-limit-test \ + > /tmp/monitor-hold.log 2>&1) & + for attempt in $(seq 1 30); do + grep -q 'allocate another 2 GiB: out of memory' \ + /tmp/monitor-hold.log && break + sleep 1 + done + grep -q 'allocate another 2 GiB: out of memory' \ + /tmp/monitor-hold.log + touch /tmp/memory-test-passed + exec sleep infinity + env: + - name: LD_LIBRARY_PATH + value: /usr/local/nvidia/lib64:/usr/local/cuda/lib64 + securityContext: + privileged: true + readinessProbe: + exec: + command: ["test", "-f", "/tmp/memory-test-passed"] + periodSeconds: 2 + failureThreshold: 150 + volumeMounts: + - name: lab-source + mountPath: /lab-source + readOnly: true + volumes: + - name: lab-source + configMap: + name: kai-hami-lab12-source diff --git a/tutorials/labs/examples/12-kai-scheduler-hami-gke/memory-limit.cu b/tutorials/labs/examples/12-kai-scheduler-hami-gke/memory-limit.cu new file mode 100644 index 00000000..aea1e8c4 --- /dev/null +++ b/tutorials/labs/examples/12-kai-scheduler-hami-gke/memory-limit.cu @@ -0,0 +1,43 @@ +#include + +#include +#include +#include +#include + +int main() { + constexpr size_t GiB = 1024ULL * 1024ULL * 1024ULL; + void *within_limit = nullptr; + void *over_limit = nullptr; + + cudaError_t first = cudaMalloc(&within_limit, 3 * GiB); + std::cout << "allocate 3 GiB: " << cudaGetErrorString(first) << std::endl; + if (first != cudaSuccess) return 1; + + cudaError_t second = cudaMalloc(&over_limit, 2 * GiB); + std::cout << "allocate another 2 GiB: " << cudaGetErrorString(second) + << std::endl; + if (second == cudaSuccess) { + std::cerr << "ERROR: allocation exceeded the container quota" << std::endl; + cudaFree(over_limit); + cudaFree(within_limit); + return 2; + } + if (second != cudaErrorMemoryAllocation) { + std::cerr << "ERROR: expected out of memory, got " + << cudaGetErrorString(second) << std::endl; + cudaFree(within_limit); + return 3; + } + + const char *hold = std::getenv("HOLD_SECONDS"); + if (hold) { + std::this_thread::sleep_for(std::chrono::seconds(std::atoi(hold))); + } + + cudaFree(within_limit); + std::cout + << "PASS: in-quota allocation succeeded and over-quota allocation failed" + << std::endl; + return 0; +} diff --git a/tutorials/labs/kai-scheduler-hami-gke.md b/tutorials/labs/kai-scheduler-hami-gke.md new file mode 100644 index 00000000..acfd9ff2 --- /dev/null +++ b/tutorials/labs/kai-scheduler-hami-gke.md @@ -0,0 +1,516 @@ +--- +title: "Lab 12: Verify KAI Scheduler and HAMi Memory Isolation on GKE" +description: "Deploy KAI Scheduler and kai-resource-isolator on GKE, adapt the COS/CDI runtime path, and prove per-Pod GPU memory isolation with CUDA allocations." +sidebar_label: "Lab 12: KAI + HAMi on GKE" +lab: + level: Advanced + duration: about 90 minutes + environment: GKE 1.35 with COS, containerd CDI, and NVIDIA Tesla T4 GPUs + cost: requires billable GKE nodes with attached GPUs + authors: + - rootsongjc + verified: "2026-08-12" +tags: + - kai-scheduler + - hami-core + - gke + - gpu-sharing +toc_max_heading_level: 2 +--- + +This lab deploys KAI Scheduler v0.17.0 and `kai-resource-isolator` 1.1.0-chart on GKE, then proves that two Pods sharing one Tesla T4 cannot allocate beyond their individual memory quotas. It also documents the GKE 1.35/COS/CDI compatibility workarounds observed in the verified environment. + +:::warning Environment-specific workarounds + +The RuntimeClass, Kyverno, host path, PriorityClass, and privileged-container changes in this lab are specific to the verified GKE 1.35/COS/CDI path. They are not the standard KAI + HAMi-core installation. Apply each workaround only after confirming the matching symptom. + +::: + +## What You'll Learn + +- enable KAI GPU sharing and its `hamicore` binder plugin; +- adapt `kai-resource-isolator` to GKE's read-only root filesystem and CDI device injection; +- prove that two Pods use the same physical GPU and see separate 4147 MiB ceilings; and +- use `cudaMalloc` to prove in-quota success, over-quota failure, and cross-Pod independence. + +## Lab Overview + +```mermaid +%% title: KAI Scheduler and HAMi-core on GKE +flowchart LR + S1["Step 1
Verify GKE GPU"] --> S2["Step 2
Add GPU labels"] + S2 --> S3["Step 3
Install KAI"] + S3 --> S4["Step 4
Install isolator"] + S4 --> S5["Step 5
Apply GKE adaptations"] + S5 --> S6["Step 6
Share one T4"] + S6 --> S7["Step 7
Run CUDA OOM proof"] + S7 --> S8["Step 8
Verify monitor metrics"] +``` + +## Prerequisites + +- A GCP project with the GKE and Compute Engine APIs enabled. +- A GKE 1.35 cluster with COS nodes and at least one NVIDIA T4. The verified cluster had three `n1-standard-2` nodes, each with one T4. +- The GKE-managed NVIDIA driver, device plugin, and container toolkit. Do not install GPU Operator on top of the GKE-managed driver. +- `gcloud`, a `kubectl` version within one minor release of the GKE API server, and Helm 3 or 4 with cluster-admin permissions. +- The files under [`tutorials/labs/examples/12-kai-scheduler-hami-gke/`](https://github.com/Project-HAMi/website/tree/master/tutorials/labs/examples/12-kai-scheduler-hami-gke). + +The verified cluster ran GKE `1.35.6-gke.1250000` on both the control plane and nodes. Because GKE patch releases age out, first choose an available 1.35 version in your zone, then create the cluster with it: + +```bash +export GKE_VERSION=$(gcloud container get-server-config \ + --zone=asia-northeast1-a \ + --format='value(validMasterVersions)' | tr ';' '\n' | grep '^1\.35\.' | head -1) +test -n "$GKE_VERSION" + +gcloud container clusters create kai-hami-test --zone=asia-northeast1-a \ + --cluster-version="$GKE_VERSION" \ + --machine-type=n1-standard-2 --num-nodes=3 \ + --image-type=COS_CONTAINERD \ + --accelerator=type=nvidia-tesla-t4,count=1,gpu-driver-version=default +gcloud container clusters get-credentials kai-hami-test \ + --zone=asia-northeast1-a +``` + +GPU nodes are billable. Run the cleanup section when you finish. + +:::note About the output blocks + +The output blocks below were captured from the verified run on 2026-08-12. Pod suffixes, ages, IP addresses, and node names are environment-specific; compare the component names, readiness, placement, and measured values. + +::: + +## Step 1: Verify the GKE GPU Stack + +Confirm that each GPU node reports one extended resource: + +```bash +kubectl get nodes \ + -o custom-columns="NAME:.metadata.name,GPU:.status.capacity.nvidia\.com/gpu,ACCEL:.metadata.labels.cloud\.google\.com/gke-accelerator" +``` + +The verified cluster reported three T4 nodes: + +```plaintext +NAME GPU ACCEL +gke-kai-hami-test-default-pool-370c394b-fxh2 1 nvidia-tesla-t4 +gke-kai-hami-test-default-pool-370c394b-pm4j 1 nvidia-tesla-t4 +gke-kai-hami-test-default-pool-370c394b-r8n5 1 nvidia-tesla-t4 +``` + +If `GPU` is empty and the node has `gke-no-default-nvidia-gpu-device-plugin=true`, enable the GKE device plugin: + +```bash +kubectl label node -l cloud.google.com/gke-accelerator=nvidia-tesla-t4 \ + gke-no-default-nvidia-gpu-device-plugin- +``` + +Run an ordinary whole-GPU Pod before adding KAI: + +```bash +kubectl apply -f - <<'EOF' +apiVersion: v1 +kind: Pod +metadata: + name: gpu-smi-test +spec: + restartPolicy: Never + containers: + - name: cuda + image: nvidia/cuda:12.4.1-base-ubuntu22.04 + command: ["nvidia-smi"] + resources: + limits: + nvidia.com/gpu: 1 +EOF +kubectl wait --for=jsonpath='{.status.phase}'=Succeeded \ + pod/gpu-smi-test --timeout=180s +kubectl logs gpu-smi-test +kubectl delete pod gpu-smi-test +``` + +The relevant lines in the verified `nvidia-smi` output were: + +```plaintext +GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC + 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 +... 0MiB / 15360MiB +``` + +The driver patch may differ, but the workload image must be compatible with it. Record the reported memory value for Step 2. + +## Step 2: Add the GPU Labels KAI Reads + +GKE's default device plugin does not provide all GPU Feature Discovery labels. KAI reads `nvidia.com/gpu.memory` when it registers the node, so add the labels before installing KAI: + +```bash +kubectl label node -l cloud.google.com/gke-accelerator=nvidia-tesla-t4 \ + nvidia.com/gpu.memory=15360 \ + nvidia.com/gpu.product=NVIDIA-Tesla-T4 \ + nvidia.com/gpu.count=1 \ + nvidia.com/gpu.present=true --overwrite + +kubectl get nodes -o custom-columns=\ +'NAME:.metadata.name,GPU.MEMORY:.metadata.labels.nvidia\.com/gpu\.memory,GPU.PRODUCT:.metadata.labels.nvidia\.com/gpu\.product' +``` + +The verified nodes then exposed the values KAI reads: + +```plaintext +NAME GPU.MEMORY GPU.PRODUCT +gke-kai-hami-test-default-pool-370c394b-fxh2 15360 NVIDIA-Tesla-T4 +gke-kai-hami-test-default-pool-370c394b-pm4j 15360 NVIDIA-Tesla-T4 +gke-kai-hami-test-default-pool-370c394b-r8n5 15360 NVIDIA-Tesla-T4 +``` + +Use the value reported by `nvidia-smi`, not the T4's marketed 16 GiB. If you add the labels after KAI starts, restart `kai-scheduler` so it refreshes its node cache. + +## Step 3: Install KAI Scheduler and Its Default Queues + +Install KAI v0.17.0 with GPU sharing, HAMi-core integration, and CDI enabled: + +```bash +helm install kai-scheduler \ + oci://ghcr.io/kai-scheduler/kai-scheduler/kai-scheduler \ + --namespace kai-scheduler --create-namespace \ + --version v0.17.0 \ + --set global.gpuSharing=true \ + --set binder.plugins.hamicore.enabled=true \ + --set-string binder.plugins.gpusharing.arguments.cdiEnabled=true + +kubectl -n kai-scheduler wait --for=condition=available \ + --timeout=180s deploy --all +kubectl -n kai-scheduler wait --for=condition=Ready \ + --timeout=300s config/kai-config +kubectl get pods -n kai-scheduler +kubectl get queues +``` + +All seven KAI control-plane components were running in the verified installation. Generated Pod suffixes will differ: + +```plaintext +NAME READY STATUS RESTARTS AGE +admission-759b9bb99c-... 1/1 Running 0 4m +binder-54665cc5d9-... 1/1 Running 0 4m +kai-operator-997c6886c-... 1/1 Running 0 4m +kai-scheduler-default-d85d7dbdf-... 1/1 Running 0 4m +pod-grouper-68f4fb47-... 1/1 Running 0 4m +podgroup-controller-5947b5b4dd-... 1/1 Running 0 4m +queue-controller-6cc8c844c8-... 1/1 Running 0 4m +``` + +KAI v0.17.0 also created its default parent and child queues automatically: + +```plaintext +NAME PARENT +default-parent-queue +default-queue default-parent-queue +``` + +:::important `cdiEnabled` must be a string + +Use `--set-string`, not `--set`. KAI v0.17.0's Helm values accept either representation, but the generated `Config` CRD field is a string. Plain `--set ...=true` renders a boolean and makes the `kai-config-deployer` hook fail with `cdiEnabled ... must be of type string: "boolean"`. The command above was rendered and accepted by the current GKE API server with server-side dry-run. + +::: + +## Step 4: Install kai-resource-isolator + +COS mounts the root filesystem read-only, so write HAMi-core under GKE's writable NVIDIA directory: + +```bash +helm install kai-resource-isolator \ + oci://docker.io/projecthami/kai-resource-isolator \ + --namespace kai-resource-isolator --create-namespace \ + --version 1.1.0-chart \ + --set paths.containerVgpuMount=/home/kubernetes/bin/nvidia/vgpu \ + --set-string librarySync.priorityClassName= \ + --set-string monitor.priorityClassName= \ + --set monitor.enabled=true +``` + +`containerVgpuMount` is the effective Chart value for the libsync destination, preload file, webhook injection path, and monitor cache. Do not use `paths.hostInstallBase`: it is declared in 1.1.0-chart's values but is not referenced by that Chart's templates. The two empty PriorityClass values prevent GKE from rejecting `system-node-critical` Pods outside a system namespace. + +Verify the rendered paths and wait for the components: + +```bash +kubectl get cm kai-resource-isolator-ldpreload \ + -n kai-resource-isolator -o jsonpath='{.data.ld\.so\.preload}' +kubectl rollout status ds/kai-resource-isolator-libsync \ + -n kai-resource-isolator --timeout=300s +kubectl rollout status ds/kai-resource-isolator-monitor \ + -n kai-resource-isolator --timeout=300s +kubectl rollout status deploy/kai-resource-isolator-webhook \ + -n kai-resource-isolator --timeout=300s +kubectl get pods -n kai-resource-isolator +``` + +The path and component status in the verified run were: + +```plaintext +/home/kubernetes/bin/nvidia/vgpu/libvgpu.so + +NAME READY STATUS RESTARTS AGE +kai-resource-isolator-libsync-... 1/1 Running 0 2m +kai-resource-isolator-libsync-... 1/1 Running 0 2m +kai-resource-isolator-libsync-... 1/1 Running 0 2m +kai-resource-isolator-monitor-26bj8 1/1 Running 0 2m +kai-resource-isolator-monitor-hf67f 1/1 Running 0 2m +kai-resource-isolator-monitor-tnj9l 1/1 Running 0 2m +kai-resource-isolator-webhook-... 1/1 Running 0 2m +``` + +There must be one libsync and one monitor Pod per GPU node, plus a ready webhook Pod. + +## Step 5: Adapt the GKE CDI Device Path + +The verified GKE 1.35 nodes used CDI and did not register an `nvidia` runtime handler, while KAI reservation Pods referenced `runtimeClassName: nvidia`. Create the compatibility RuntimeClass only if it is absent: + +```bash +kubectl get runtimeclass nvidia || kubectl apply \ + -f tutorials/labs/examples/12-kai-scheduler-hami-gke/02-runtimeclass.yaml +``` + +GKE's device plugin injects devices only into Pods requesting `nvidia.com/gpu`. KAI shared Pods request `gpu-memory` instead, so the verified environment needed explicit device and library mounts. Install Kyverno and apply the two policies: + +```bash +helm repo add kyverno https://kyverno.github.io/kyverno/ +helm repo update +helm install kyverno kyverno/kyverno \ + --namespace kyverno --create-namespace +kubectl wait -n kyverno --for=condition=Ready pod \ + -l app.kubernetes.io/component=admission-controller --timeout=300s +kubectl apply \ + -f tutorials/labs/examples/12-kai-scheduler-hami-gke/03-gke-policies.yaml +kubectl wait --for=condition=Ready --timeout=180s \ + clusterpolicy/inject-nvidia-library-path \ + clusterpolicy/inject-gpu-devices +kubectl get runtimeclass nvidia +kubectl get pods -n kyverno +kubectl get clusterpolicy \ + inject-nvidia-library-path inject-gpu-devices +``` + +The compatibility objects and Kyverno controllers were ready in the verified run: + +```plaintext +NAME HANDLER AGE +nvidia runc 3m + +NAME READY STATUS RESTARTS AGE +kyverno-admission-controller-7cdf5b9c-... 1/1 Running 0 2m +kyverno-background-controller-7b54965bf9-... 1/1 Running 0 2m +kyverno-cleanup-controller-59c8fdfb66-... 1/1 Running 0 2m +kyverno-reports-controller-5c96886c9-... 1/1 Running 0 2m + +NAME ADMISSION BACKGROUND READY +inject-nvidia-library-path true true true +inject-gpu-devices true true true +``` + +The first policy adds the NVML library path to reservation Pods. The second mounts `/dev/nvidia*`, `nvidia-smi`, and NVIDIA libraries into shared Pods. In this workaround path, shared Pods also require `privileged: true`. + +:::caution Security boundary + +This lab proves CUDA API-level memory enforcement. It is not a MIG-like hardware security boundary, and the GKE workaround uses privileged workload containers. Do not treat it as an untrusted multi-tenant security design. + +::: + +## Step 6: Place Two Pods on One T4 + +Choose an otherwise idle T4 node with exactly one GPU. Do not select a node that already runs a `gpu-memory` workload: two 4 GiB Lab Pods need at least 8 GiB of unallocated KAI memory on that card. List the candidates and current Pod placement first: + +```bash +kubectl get nodes -l cloud.google.com/gke-accelerator=nvidia-tesla-t4 +kubectl get pods -A -o wide + +# Replace this value with one idle, single-T4 node from the output above. +export TEST_NODE= +test "$(kubectl get node "$TEST_NODE" \ + -o jsonpath='{.status.capacity.nvidia\.com/gpu}')" = "1" +kubectl label node "$TEST_NODE" hami.run/lab-12=true --overwrite + +kubectl create configmap kai-hami-lab12-source \ + --from-file=memory-limit.cu=tutorials/labs/examples/12-kai-scheduler-hami-gke/memory-limit.cu \ + --dry-run=client -o yaml | kubectl apply -f - +kubectl apply \ + -f tutorials/labs/examples/12-kai-scheduler-hami-gke/04-shared-pods.yaml +kubectl wait --for=condition=Ready \ + pod/kai-hami-lab12-a pod/kai-hami-lab12-b \ + --timeout=10m +``` + +Verify the node placement, then read the injected quota, physical UUID, and visible memory from each Pod's startup log: + +```bash +kubectl get pod kai-hami-lab12-a kai-hami-lab12-b -o wide +for pod in kai-hami-lab12-a kai-hami-lab12-b; do + echo "=== $pod ===" + kubectl logs "$pod" +done +``` + +Both Pods were ready on the same node in the verified run: + +```plaintext +NAME READY STATUS RESTARTS IP NODE +kai-hami-lab12-a 1/1 Running 0 10.84.2.66 gke-kai-hami-test-default-pool-370c394b-pm4j +kai-hami-lab12-b 1/1 Running 0 10.84.2.67 gke-kai-hami-test-default-pool-370c394b-pm4j +``` + +Their startup logs both reported: + +```plaintext +limit=4147m +GPU-9acc8878-3967-5fb4-c534-43d6fd820fa6, 4147 MiB +``` + +The matching UUID proves both Pods use the same T4; the 4147 MiB value is KAI's two-decimal fraction rounding of a 4096 MiB request on a 15360 MiB card. + +## Step 7: Prove the CUDA Memory Ceiling + +The source ConfigMap created in Step 6 is mounted at `/lab-source`. On startup, each Pod compiles the program, holds a successful 3 GiB allocation for 30 seconds, and attempts to allocate another 2 GiB. After that proof passes, it starts the same program again and keeps 3 GiB allocated so Step 8 can verify the live monitor gauges. The readiness probe succeeds only after both phases start successfully. + +Inspect the test interval and result recorded in each Pod log: + +```bash +for pod in kai-hami-lab12-a kai-hami-lab12-b; do + kubectl logs "$pod" | grep -E \ + 'test_(start|end)=|allocate |PASS:' +done +``` + +The captured intervals overlapped, and each Pod returned `PASS`: + +```plaintext +=== kai-hami-lab12-a === +test_start=2026-08-12T05:11:13Z +allocate 3 GiB: no error +allocate another 2 GiB: out of memory +PASS: in-quota allocation succeeded and over-quota allocation failed +test_end=2026-08-12T05:11:44Z + +=== kai-hami-lab12-b === +test_start=2026-08-12T05:11:11Z +allocate 3 GiB: no error +allocate another 2 GiB: out of memory +PASS: in-quota allocation succeeded and over-quota allocation failed +test_end=2026-08-12T05:11:43Z +``` + +Compare `test_start` and `test_end`: the two 30-second intervals must overlap. In the verified run, both Pods returned `PASS` while holding 3 GiB concurrently. HAMi-core logged `Device 0 OOM 5475663872 / 4348444672` for each Pod's 5 GiB cumulative request. Running the proof as the container startup command also avoids making the result depend on a long-lived `kubectl exec` WebSocket. + +## Step 8: Verify the Monitor Metrics + +The monitor is a DaemonSet: each instance reads the HAMi shared-memory cache on its own node. A Service can forward to a monitor on a different node and return no series for these Pods, so query the monitor instance on the workload node directly. Derive that node again from the running Pod instead of relying on the shell variable exported in Step 6. + +Create a short-lived curl Pod that reads that monitor's `:9394/metrics` endpoint: + +```bash +export TEST_NODE=$(kubectl get pod kai-hami-lab12-a \ + -o jsonpath='{.spec.nodeName}') +test -n "$TEST_NODE" + +export MONITOR_IP=$(kubectl get pods -n kai-resource-isolator \ + -l app.kubernetes.io/component=kai-vgpu-monitor \ + --field-selector="spec.nodeName=$TEST_NODE" \ + -o jsonpath='{range .items[*]}{.status.podIP}{"\n"}{end}' | head -n 1) + +if test -z "$MONITOR_IP"; then + echo "No monitor Pod is running on $TEST_NODE" >&2 + kubectl get pods -n kai-resource-isolator \ + -l app.kubernetes.io/component=kai-vgpu-monitor -o wide + exit 1 +fi + +kubectl delete pod lab12-monitor-check --ignore-not-found +kubectl run lab12-monitor-check \ + --image=curlimages/curl:8.15.0 --restart=Never \ + --command -- sh -lc \ + "curl -fsS http://$MONITOR_IP:9394/metrics" +kubectl wait --for=jsonpath='{.status.phase}'=Succeeded \ + pod/lab12-monitor-check --timeout=180s +kubectl logs lab12-monitor-check | grep -E \ + '^hami_vgpu_memory_(used|limit)_bytes.*pod="kai-hami-lab12-[ab]"' +``` + +The verified endpoint returned one `used` and one `limit` series for each Pod: + +```plaintext +hami_vgpu_memory_limit_bytes{...,pod="kai-hami-lab12-a",...} 4.348444672e+09 +hami_vgpu_memory_limit_bytes{...,pod="kai-hami-lab12-b",...} 4.348444672e+09 +hami_vgpu_memory_used_bytes{...,pod="kai-hami-lab12-a",...} 3.328180224e+09 +hami_vgpu_memory_used_bytes{...,pod="kai-hami-lab12-b",...} 3.328180224e+09 +``` + +The limit equals 4147 MiB in bytes. The used value is slightly above 3 GiB because it includes CUDA context and allocator overhead. This proves the monitor found both per-container caches and exported live usage rather than only serving an empty Prometheus endpoint. + +## Troubleshooting + +| Symptom | Cause in the verified environment | Action | +| :-- | :-- | :-- | +| Shared Pod stays Pending with `didn't have enough resources: GPU memory` | `nvidia.com/gpu.memory` was missing when KAI cached the node | Add the label and restart `kai-scheduler` | +| Queue object is rejected | KAI admission webhook is not ready | Wait for KAI Deployments, then apply the queues | +| `RuntimeClass "nvidia" not found` | GKE CDI uses `runc` and has no NVIDIA handler | Apply `02-runtimeclass.yaml` | +| Reservation Pod reports `ERROR_LIBRARY_NOT_FOUND` | NVML exists under `/usr/local/nvidia/lib64` but is not in the search path | Apply the Kyverno library-path policy | +| Shared Pod cannot see `/dev/nvidia*` | It requests `gpu-memory`, so the GKE device plugin does not run Allocate | Apply the Kyverno device-mount policy | +| libsync reports `Read-only file system` | COS root filesystem is read-only | Set `paths.containerVgpuMount=/home/kubernetes/bin/nvidia/vgpu` | +| `libvgpu.so` cannot be preloaded | The effective mount path still points to `/usr/local/vgpu` | Reinstall with the verified `containerVgpuMount` value | +| DaemonSet rejected for `system-node-critical` | GKE PriorityClass quota blocks user namespaces | Set both Chart PriorityClass values to an empty string | +| `CUDA driver version is insufficient` | CUDA image is newer than the node driver supports | Use the verified CUDA 12.4.1 image or another compatible version | +| `kubectl exec` ends with WebSocket EOF | Control-plane exec stream reset or unsupported client/server skew | Use a compatible `kubectl`; the Lab proof runs at container startup and is read with `kubectl logs` | +| Monitor lookup returns no Pod or no series | `$TEST_NODE` was stale, or the monitor is absent from the workload node | Derive the node from `kai-hami-lab12-a`, then inspect the monitor DaemonSet with `kubectl get pods -n kai-resource-isolator -l app.kubernetes.io/component=kai-vgpu-monitor -o wide` | + +## Cleanup + +Remove the workload and test label: + +```bash +kubectl delete \ + -f tutorials/labs/examples/12-kai-scheduler-hami-gke/04-shared-pods.yaml +kubectl delete pod lab12-monitor-check --ignore-not-found +kubectl delete configmap kai-hami-lab12-source --ignore-not-found +kubectl label node "$TEST_NODE" hami.run/lab-12- --overwrite +``` + +If this cluster is dedicated to the lab, remove the remaining components: + +```bash +kubectl delete \ + -f tutorials/labs/examples/12-kai-scheduler-hami-gke/03-gke-policies.yaml +helm uninstall kyverno -n kyverno +helm uninstall kai-resource-isolator -n kai-resource-isolator +kubectl delete queues default-queue default-parent-queue --ignore-not-found +helm uninstall kai-scheduler -n kai-scheduler +``` + +The explicit Queue deletion is intentional: KAI annotates its default queues with `helm.sh/resource-policy: keep`, so Helm preserves them during uninstall. + +Delete `02-runtimeclass.yaml` only if Step 5 created it; preserve a RuntimeClass that already belonged to the cluster: + +```bash +kubectl delete \ + -f tutorials/labs/examples/12-kai-scheduler-hami-gke/02-runtimeclass.yaml +``` + +Delete the GKE cluster if you created it only for this exercise: + +```bash +gcloud container clusters delete kai-hami-test \ + --zone=asia-northeast1-a +``` + +## What This Lab Proved + +| Claim | Evidence | +| :-- | :-- | +| KAI schedules two fractional workloads onto one T4 | Same node and same GPU UUID in both Pods | +| HAMi-core changes the visible per-Pod memory ceiling | Both Pods report 4147 MiB instead of 15360 MiB | +| The ceiling is enforced, not only displayed | 3 GiB succeeds; the cumulative 5 GiB request returns CUDA OOM | +| One Pod cannot consume the other's quota | Pod B succeeds while Pod A holds 3 GiB | +| The optional monitor reads per-container caches | `:9394/metrics` reports both Pods' 4147 MiB limits and live 3 GiB usage | + +## Next Steps + +- Read [GPU Memory Hard Isolation with KAI Scheduler and HAMi](/blog/kai-scheduler-hami-gpu-memory-hard-isolation) for the architecture and integration background. +- Compare this path with [Lab 3: GPU Partitioning](./gpu-partitioning.md) and [Lab 7: k3s Isolation](./hami-isolation-k3s.md). +- Track upstream KAI and `kai-resource-isolator` releases; once GKE CDI support is native, remove the matching workaround from this lab. diff --git a/tutorials/overview.md b/tutorials/overview.md index 010b9373..5cfd995f 100644 --- a/tutorials/overview.md +++ b/tutorials/overview.md @@ -17,4 +17,4 @@ Background knowledge that the labs build on. ## Labs - Each lab lists its own prerequisites. Labs 3 and 4 continue from the cluster Lab 1 builds, so a single session covers all three; Lab 2 runs on any laptop with no GPU required. Lab 7 brings up its own single-node k3s cluster on a rented GPU VM, without the GPU Operator. Lab 8 requires an existing Volcano GPU cluster and validates Volcano vGPU, Gang scheduling, and queue-level limits. Lab 9 uses Kueue admission control to enforce HAMi vGPU count, memory, and compute quotas. Lab 11 builds a complete KServe Standard inference stack and runs two vLLM replicas on one GPU through native HAMi DRA claims. + Each lab lists its own prerequisites. Labs 3 and 4 continue from the cluster Lab 1 builds, so a single session covers all three; Lab 2 runs on any laptop with no GPU required. Lab 7 brings up its own single-node k3s cluster on a rented GPU VM, without the GPU Operator. Lab 8 requires an existing Volcano GPU cluster and validates Volcano vGPU, Gang scheduling, and queue-level limits. Lab 9 uses Kueue admission control to enforce HAMi vGPU count, memory, and compute quotas. Lab 11 builds a complete KServe Standard inference stack and runs two vLLM replicas on one GPU through native HAMi DRA claims. Lab 12 deploys KAI Scheduler and HAMi-core on GKE 1.35/COS/CDI and proves the memory ceiling with CUDA allocations.