From 875fdc1bf7c4cb46fb129863442aba504f081a41 Mon Sep 17 00:00:00 2001 From: Jimmy Song Date: Wed, 12 Aug 2026 10:32:15 +0800 Subject: [PATCH 1/6] feat(blog): add KAI Scheduler + HAMi GPU memory hard isolation post Bilingual (en/zh) post on KAI Scheduler v0.17.0 + kai-resource-isolator CUDA-level memory isolation. Signed-off-by: Jimmy Song --- blog/authors.yml | 4 + .../index.md | 833 ++++++++++++++++++ .../authors.yml | 4 + .../index.md | 824 +++++++++++++++++ 4 files changed, 1665 insertions(+) create mode 100644 blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md create mode 100644 i18n/zh/docusaurus-plugin-content-blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md diff --git a/blog/authors.yml b/blog/authors.yml index 8050f7739..aefe38997 100644 --- a/blog/authors.yml +++ b/blog/authors.yml @@ -15,3 +15,7 @@ archlitchi: name: Li Mengxuan title: HAMi Maintainer, Co-founder & CTO of Dynamia url: https://github.com/archlitchi + +rootsongjc: + name: Jimmy Song + url: https://github.com/rootsongjc diff --git a/blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md b/blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md new file mode 100644 index 000000000..6657f188d --- /dev/null +++ b/blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md @@ -0,0 +1,833 @@ +--- +title: "GPU Memory Hard Isolation with KAI Scheduler and HAMi: How It Works and How to Verify It" +date: "2026-08-11" +description: "From the CUDA interception principle to a reproducible verification on GKE: use KAI Scheduler v0.17.0 for shared GPU scheduling and kai-resource-isolator 1.1.0-chart (built on HAMi-core) for CUDA level memory isolation, so two pods on a single NVIDIA T4 each see only their own memory quota. Explains the CUDA_DEVICE_MEMORY_LIMIT contract between the scheduling layer and the isolation layer, plus the node label, queue, and RuntimeClass pitfalls." +authors: [rootsongjc] +tags: + [ + "HAMi", + "KAI Scheduler", + "GPU Memory Hard Isolation", + "GPU Sharing", + "Kubernetes", + "Cloud Native", + ] +--- + +GPU sharing has been discussed in the Kubernetes ecosystem for years, but the scheduling layer and the isolation layer have long operated in isolation from each other. The scheduler places several pods onto the same card, yet once a container touches the GPU it still sees the full device memory. Whichever container calls `cudaMalloc` first can occupy everything, so the isolation is effectively absent. So called "sharing" is really just "grabbing", with no resource guarantee at all. + +Solving this requires the scheduling layer (deciding "who uses which GPU, and how much") and the isolation layer (guaranteeing "once a quota is set, it cannot be exceeded") to cooperate. **HAMi-core** is exactly such an isolation engine, reusable by multiple schedulers. Before KAI Scheduler, it already supported the Kubernetes native scheduler (via HAMi's own `hami-scheduler` extender), [Kueue](/docs/userguide/kueue/how-to-use-kueue), [Volcano](/docs/installation/how-to-use-volcano-vgpu), and more (see the full landscape in [Ecosystem Integrations](/docs/next/core-concepts/ecosystem-integrations)). + +**Starting with KAI Scheduler v0.16.4, NVIDIA's KAI Scheduler officially joined this list**, building in HAMi-core as the isolation engine for its GPU sharing (the current version at the time of writing is v0.17.0). That means when you schedule GPU workloads with KAI Scheduler, you no longer get only "cooperative sharing"; you get true hard isolation. This post covers two things: + +- **Principles**: what each of KAI Scheduler, `kai-resource-isolator`, and HAMi-core is responsible for, how they connect, and how the `CUDA_DEVICE_MEMORY_LIMIT` contract ties the scheduling layer to the isolation layer. +- **Practice**: a fully reproducible verification example on GKE (a single NVIDIA T4 card, two pods sharing it, each seeing only its own memory quota), with every step carrying its config, its expected output, and the "why" behind it. + +The background story and collaboration timeline are in the companion post [HAMi-core adopted by NVIDIA KAI Scheduler: GPU sharing enters the hard isolation era](/blog/hami-core-adopted-by-nvidia-kai-scheduler). + +:::note About the output in this post + +The commands in the second half are steps you can reproduce end to end on GKE. The shown command output and metrics are expected examples; actual values depend on your cluster. + +::: + + + +## Background: why sharing is not isolation + +| Layer | Owner | What it solves | What it does not solve | +| --- | --- | --- | --- | +| Scheduling | KAI Scheduler, Volcano, Kueue, ... | Multiple pods can land on the same GPU | The container still sees all the memory | +| Runtime | HAMi-core (`libvgpu.so`) | Intercepts CUDA calls, enforces a memory quota | On its own, does not know how much each pod should get | + +Real GPU sharing needs both layers, and they must cooperate: the scheduling layer decides "who uses which GPU, and how much", and the isolation layer guarantees "the agreed amount is all you get". The catch is that **the isolation layer needs to know "how much", a number it cannot compute on its own, because that number comes from the scheduling layer**. + +HAMi-core, refined by the HAMi community over years (a CNCF incubating project), is exactly that isolation layer, and it is **decoupled from any specific scheduler**: well before KAI Scheduler, HAMi-core already worked with the Kubernetes native scheduler (via HAMi's own `hami-scheduler` extender), [Kueue](/docs/userguide/kueue/how-to-use-kueue), [Volcano](/docs/installation/how-to-use-volcano-vgpu), Koordinator, and others (see the full landscape in [Ecosystem Integrations](/docs/next/core-concepts/ecosystem-integrations)). + +**KAI Scheduler joined this list in v0.16.4**, building HAMi-core in as the isolation engine for its GPU sharing. What this post explains is the role of each of the three components on the KAI Scheduler integration path, and how they connect. + +## How it works: one contract, three steps + +### What each of the three components is and does + +The three components have similar-sounding names, so getting their roles straight is the prerequisite for understanding the whole chain: + +- **HAMi-core (`libvgpu.so`)**: HAMi's CUDA interception library (CNCF incubating), and **the isolation engine itself**. It intercepts CUDA calls (like `cudaMalloc`) inside the container via `LD_PRELOAD` and enforces a memory quota. It does not care who provided the quota: any scheduler that hands in the quota by convention gets isolation for free. Before KAI, it was already reused by HAMi's own device-plugin/webhook, Volcano's `volcano-vgpu-device-plugin`, and others. + +- **KAI Scheduler**: NVIDIA's open source Kubernetes scheduler for AI workloads (descended from Run:ai, CNCF Sandbox). It only owns the **scheduling layer**, deciding which node a pod lands on, which GPU it uses, and how much memory it gets. Since v0.16.4, its `hamicore` plugin writes the computed memory quota into the container's environment variable at bind time. + +- **`kai-resource-isolator`**: a companion component **provided by the HAMi project specifically for the KAI Scheduler integration path**. It ships HAMi-core's `libvgpu.so` to every GPU node and uses a MutatingWebhook to rewrite pods, injecting the library and `ld.so.preload`. In other words, it is the bridge that turns KAI's scheduling decision into isolation HAMi-core can actually enforce. + +```mermaid +%% title: Responsibilities of the three components and how they connect +graph LR + subgraph SCH["Scheduling layer"] + KAI["KAI Scheduler
(hamicore plugin)

Role: compute the quota
bind Pod to a node
inject CUDA_DEVICE_MEMORY_LIMIT"] + end + subgraph BRIDGE["Bridge layer (provided by the HAMi project)"] + ISO["kai-resource-isolator

Role: ship libvgpu.so
webhook injects library + ld.so.preload"] + end + subgraph ISO2["Isolation layer"] + HAMI["HAMi-core / libvgpu.so
(CNCF incubating)

Role: intercept cudaMalloc
enforce the memory quota"] + end + + KAI -->|"CUDA_DEVICE_MEMORY_LIMIT
+ Pod bind result"| ISO + ISO -->|"libvgpu.so in place
ld.so.preload points at it"| HAMI + HAMI -.->|"same engine reused
(also by K8s / Volcano / Kueue / ...)"| SCH + + style KAI fill:#d9f99d,stroke:#4f7d00,stroke-width:2px,color:#1f2937 + style ISO fill:#dbeafe,stroke:#1a5fb4,stroke-width:2px,color:#1f2937 + style HAMI fill:#fef3c7,stroke:#b45309,stroke-width:2px,color:#1f2937 + style SCH fill:#f7fee7,stroke:#4f7d00,stroke-width:1px,color:#1f2937 + style BRIDGE fill:#eff6ff,stroke:#1a5fb4,stroke-width:1px,color:#1f2937 + style ISO2 fill:#fffbeb,stroke:#b45309,stroke-width:1px,color:#1f2937 +``` + +In one sentence: **KAI Scheduler computes the quota, `kai-resource-isolator` puts the isolation library in place, and HAMi-core actually enforces the isolation at runtime.** + +:::note Why HAMi-core, not the full HAMi platform + +KAI Scheduler integrates with **HAMi-core itself**, not the full HAMi platform. KAI keeps its own scheduling capability (it is not replaced by `hami-scheduler`) and only brings in HAMi-core for GPU memory isolation. This mirrors how Volcano works (using `volcano-vgpu-device-plugin` + HAMi-core): each scheduler keeps its own, and they share HAMi-core as the common isolation engine. + +::: + +### The contract: `CUDA_DEVICE_MEMORY_LIMIT` + +The whole chain works because the scheduling layer and the isolation layer agreed on a minimal hand off point: the environment variable **`CUDA_DEVICE_MEMORY_LIMIT`**. + +- **KAI Scheduler (scheduling layer)** computes "how much memory this pod may use" and writes it into the container's environment variables when it binds the pod to a node. +- **HAMi-core (isolation layer)** reads that variable and, at runtime, actually keeps memory usage under that ceiling. + +This contract matters because it **fully decouples the two sides**: KAI does not need to know how CUDA is intercepted, and HAMi-core does not need to know how the share was computed. As long as both honor that one variable, any scheduler can reuse the same isolation engine. That is exactly why HAMi-core can support multiple schedulers at once (see "What this means" at the end). + +### The three steps + +```mermaid +%% title: Three step cooperation between KAI Scheduler and kai-resource-isolator +graph TD + KAI["KAI Scheduler
hamicore plugin
reads gpu-memory annotation"] + ENV["1. Inject at scheduling time
CUDA_DEVICE_MEMORY_LIMIT env var"] + ISO["2. kai-resource-isolator
Mutating Webhook rewrites Pod"] + LIB["libsync DaemonSet
distributes libvgpu.so to /usr/local/vgpu"] + MON["monitor DaemonSet
exposes hami_* metrics on :9394"] + RUN["3. Container starts
libvgpu.so intercepts cudaMalloc via LD_PRELOAD"] + ENF["Rejects over limit allocation
nvidia-smi shows only the quota"] + + KAI --> ENV --> ISO --> RUN --> ENF + LIB -. "inject hostPath and ld.so.preload" .-> ISO + MON -. "collects per container memory" .-> RUN + + style KAI fill:#d9f99d,stroke:#4f7d00,stroke-width:2px,color:#1f2937 + style ENV fill:#d9f99d,stroke:#4f7d00,stroke-width:2px,color:#1f2937 + style ISO fill:#dbeafe,stroke:#1a5fb4,stroke-width:2px,color:#1f2937 + style LIB fill:#dbeafe,stroke:#1a5fb4,stroke-width:2px,color:#1f2937 + style MON fill:#dbeafe,stroke:#1a5fb4,stroke-width:2px,color:#1f2937 + style RUN fill:#fef3c7,stroke:#b45309,stroke-width:2px,color:#1f2937 + style ENF fill:#dcfce7,stroke:#0b6b3c,stroke-width:2px,color:#1f2937 +``` + +In one sentence: **KAI says you may only use this much, and the isolator makes sure you really can only use this much.** + +The three components divide the work as follows (see the [KAI Scheduler docs on HAMi resource isolation](https://github.com/kai-scheduler/KAI-Scheduler/blob/main/docs/gpu-sharing/hami/README.md)): + +- **libsync DaemonSet** copies `libvgpu.so` to `/usr/local/vgpu` on every GPU node. +- **mutating webhook** injects the hostPath volume mount into the pod, points `/etc/ld.so.preload` at `libvgpu.so`, and writes the `POD_UID`, `CONTAINER_NAME`, and `CONTAINER_VGPU_MOUNT` environment variables. +- **monitor DaemonSet** (optional) reads each container shared memory cache and exposes metrics on `:9394`. + +### How the CUDA interception works + +The "last mile" of isolation happens inside the container process. The chain is: + +1. **KAI injects `CUDA_DEVICE_MEMORY_LIMIT` at scheduling time** (carrying the pod's requested memory quota, in MiB). +2. **The kai-resource-isolator webhook rewrites the pod**: it mounts the host's `libvgpu.so` and points `/etc/ld.so.preload` at it. `ld.so.preload` is a dynamic linker mechanism: any shared library listed there is loaded before all other libraries. +3. **Once the container process starts**, every call into the CUDA runtime (`libcudart`) or driver API passes through `libvgpu.so` first. It intercepts memory allocation calls like `cudaMalloc`, reads the quota from `CUDA_DEVICE_MEMORY_LIMIT`, accumulates the container's memory usage, and rejects any allocation that would exceed the quota. +4. **The visible effect**: `nvidia-smi` shows only the quota memory (HAMi-core rewrites the device query responses), and no matter how hard the container calls `cudaMalloc`, it cannot cross that line. + +That is what "hard isolation" means: it is not left to application discipline, but enforced at the CUDA call layer. + +:::tip How this integration came together + +This integration path is the result of more than a year of work between the HAMi community and the NVIDIA KAI Scheduler team. The split is clean: KAI injects the environment variable, HAMi provides the resource isolation components. The full timeline and contributors are in the companion post [HAMi-core adopted by NVIDIA KAI Scheduler](/blog/hami-core-adopted-by-nvidia-kai-scheduler). + +::: + +## What these two versions bring + +### KAI Scheduler (HAMi-core hard isolation built in since v0.16.4) + +KAI Scheduler's support for HAMi-core has been a built-in capability **since v0.16.4** (this post uses the current version, v0.17.0). The key piece is the `hamicore` plugin: once enabled, when KAI binds a shared GPU pod to a node, it injects the `CUDA_DEVICE_MEMORY_LIMIT` environment variable into the container based on the `gpu-memory` (or `gpu-fraction`) annotation, exactly the quota that HAMi-core needs to enforce isolation, per the contract above. + +Other GPU related changes in v0.17.0 include: fixing invalid volume names caused by `/` in shared pod names; correcting the allocation math for `MinNodeGPUMemoryMiB` and fractional `gpu-memory`; and using the largest GPU profile in the cluster for overLimit decisions. It also adds preemption-delay (a time window for Cluster Autoscaler to bring up nodes), NUMA aware scoring, and GitOps and ArgoCD installation support. + +### kai-resource-isolator 1.1.0-chart + +This is the isolator shipped alongside HAMi. It receives the quota injected by KAI and, before the container actually starts, puts the HAMi-core `libvgpu.so` in place. Compared with the first release, 1.1.0 fills in a set of improvements that make it production ready: + +- **New `kai-vgpu-monitor`**: runs as a DaemonSet, exposes HAMi compatible metrics on `:9394` (`hami_vgpu_memory_used_bytes`, `hami_vgpu_memory_limit_bytes`, `hami_container_device_utilization_ratio`), supports ServiceMonitor, and can be scraped directly by Prometheus. +- **Multi container injection fix**: when a pod has multiple containers, the webhook now handles them correctly and no longer skips any. +- **Security tightening**: the webhook now uses a namespaced Issuer (instead of a ClusterIssuer), and the ClusterRole no longer reads Secrets. +- **Global image repository** precedence cleaned up, and `hamicore` installation parameters corrected. + +## Practice: verifying hard isolation on GKE + +The example runs on an existing GKE cluster: **3 `n1-standard-2` nodes, each with 1 NVIDIA T4 (`nvidia-smi` reports 15360 MiB of VRAM)**. Two containers each request about 4 GiB and share one of those T4s. Inside `nvidia-smi` each sees only its own quota, with no interference. Each step below comes with its config, its expected output, and the why behind it. + +The diagram below is an overview of the whole exercise: the order of the eight steps, and the two prerequisites that are easiest to get wrong (the GFD labels must be in place before installing KAI; the queue must exist before KAI will schedule). Note: this post **does not install the NVIDIA GPU Operator**; GKE's default device plugin + driver + container toolkit are enough. The preparation below is just two steps: "enable the default device plugin" and "add a few GFD labels". + +```mermaid +%% title: GKE exercise flow (Step 1 to Step 8) +graph TD + S1["Step 1
Confirm cluster + GPU hardware
n1-standard-2 + T4"] + S2["Step 2
Enable GKE default device plugin
verify nvidia-smi"] + S3["Step 3
Add GFD labels
nvidia.com/gpu.memory etc."] + S4["Step 4
Install KAI Scheduler v0.17.0
Create the queue"] + S5["Step 5
Install kai-resource-isolator
1.1.0-chart"] + S6["Step 6
Single pod, verify isolation
nvidia-smi shows quota"] + S7["Step 7
Two pods share one T4
each sees its own quota"] + S8["Step 8 optional
Look at monitor metrics"] + + S1 --> S2 --> S3 --> S4 --> S5 --> S6 --> S7 --> S8 + + S3 -. "GFD labels must be set BEFORE KAI
else shared pods stuck Pending" .-> S4 + S4 -. "Queue must exist
else pods are never scheduled" .-> S6 + + style S1 fill:#dbeafe,stroke:#1a5fb4,stroke-width:2px,color:#1f2937 + style S2 fill:#fef3c7,stroke:#b45309,stroke-width:2px,color:#1f2937 + style S3 fill:#fef3c7,stroke:#b45309,stroke-width:2px,color:#1f2937 + style S4 fill:#d9f99d,stroke:#4f7d00,stroke-width:2px,color:#1f2937 + style S5 fill:#d9f99d,stroke:#4f7d00,stroke-width:2px,color:#1f2937 + style S6 fill:#dcfce7,stroke:#0b6b3c,stroke-width:2px,color:#1f2937 + style S7 fill:#dcfce7,stroke:#0b6b3c,stroke-width:2px,color:#1f2937 + style S8 fill:#eff6ff,stroke:#1a5fb4,stroke-width:1px,color:#1f2937 +``` + +### Prerequisites + +- A GCP project with GKE and Compute Engine APIs enabled. +- An existing GKE cluster with NVIDIA T4 GPUs in its node pool (this example uses 3 `n1-standard-2` nodes, each with 1× T4). The cluster for this example was created with the node label `gke-no-default-nvidia-gpu-device-plugin=true`, which **disables GKE's default NVIDIA device plugin**, while the driver is still installed by GKE automatically. This is why Step 1 below shows "hardware present, but `nvidia.com/gpu` resource empty"; Step 2 re-enables the device plugin. If you do not have a cluster yet, you can create an equivalent one with: + + ```bash + gcloud container clusters create test-cluster --zone=asia-northeast1-a \ + --machine-type=n1-standard-2 --num-nodes=3 \ + --accelerator=type=nvidia-tesla-t4,count=1,gpu-driver-version=default + gcloud container clusters get-credentials test-cluster --zone=asia-northeast1-a + ``` + + The `n1` machine family does not include a GPU by itself, so you **must** pass `--accelerator` to attach the T4; `gpu-driver-version=default` lets GKE install a matching NVIDIA driver and container toolkit. With this, the device plugin is enabled by default (you can skip the label-removal in Step 2). + +- `gcloud`, `kubectl`, and `helm` (>= 3) authenticated, with `kubectl` already pointing at the cluster. +- The KAI queue created in Step 4 below. + +### Step 1: confirm the cluster and GPU hardware are ready (initial state) + +With the cluster already in place, first take a look at its initial state. This step is read-only. + +```bash +kubectl get nodes -o custom-columns="NAME:.metadata.name,GPU:.status.capacity.nvidia\.com/gpu,ACCEL:.metadata.labels.cloud\.google\.com/gke-accelerator" +``` + +For the cluster in this example, the initial output looks like this: 3 nodes, all with `nvidia-tesla-t4` in the `ACCEL` column (T4 hardware present), but `GPU` is ``. + +``` +NAME GPU ACCEL +gke-test-cluster-default-pool-xxxxxxxx nvidia-tesla-t4 +gke-test-cluster-default-pool-yyyyyyyy nvidia-tesla-t4 +gke-test-cluster-default-pool-zzzzzzzz nvidia-tesla-t4 +``` + +**How to read this**: the `ACCEL` value `nvidia-tesla-t4` confirms the T4 hardware is attached and GKE has already installed the NVIDIA kernel driver. But the `GPU` column is ``, which means **the device plugin has not reported the `nvidia.com/gpu` resource yet**. The reason is that this cluster was created with the node label `gke-no-default-nvidia-gpu-device-plugin=true`, which disables GKE's default NVIDIA device plugin. The next step re-enables it. + +If your cluster already shows `1` in the `GPU` column (the default device plugin was not disabled at creation), you can skip Step 2 and jump straight to Step 3 to add the GFD labels. + +### Step 2: enable GKE's default device plugin and verify the GPU works + +Remove the `gke-no-default-nvidia-gpu-device-plugin` label from the nodes, and GKE's built-in device plugin DaemonSet (whose scheduling rule is "only schedule when this label is not `true`") will land on the nodes automatically. + +```bash +kubectl label node -l cloud.google.com/gke-accelerator=nvidia-tesla-t4 \ + gke-no-default-nvidia-gpu-device-plugin- +``` + +The trailing `-` on the label name means "delete this label". After about 20 seconds, the device plugin pods come up: + +```bash +kubectl get pod -n kube-system -o wide | grep nvidia-gpu-device-plugin | grep -v "0/0" +``` + +Expected output (example): 3 `nvidia-gpu-device-plugin-*-ubuntu` pods, one per node, all `3/3 Running`. + +``` +NAME READY STATUS NODE +nvidia-gpu-device-plugin-small-ubuntu-xxx 3/3 Running gke-test-cluster-default-pool-xxxxxxxx +nvidia-gpu-device-plugin-small-ubuntu-yyy 3/3 Running gke-test-cluster-default-pool-yyyyyyyy +nvidia-gpu-device-plugin-small-ubuntu-zzz 3/3 Running gke-test-cluster-default-pool-zzzzzzzz +``` + +Once the device plugin is up, the `nvidia.com/gpu` resource appears: + +```bash +kubectl get nodes -o custom-columns="NAME:.metadata.name,GPU:.status.capacity.nvidia\.com/gpu" +``` + +Expected output (example): the `GPU` column on each node goes from `` to `1`. + +``` +NAME GPU +gke-test-cluster-default-pool-xxxxxxxx 1 +gke-test-cluster-default-pool-yyyyyyyy 1 +gke-test-cluster-default-pool-zzzzzzzz 1 +``` + +**Why no GPU Operator is needed**: GKE's default GPU stack already sets up three things: the NVIDIA kernel driver (installed by GKE), the device plugin (just enabled), and the container toolkit / nvidia runtime (configured automatically by GKE when `gpu-driver-version=default`). The GPU Operator is a tool that bundles these together for automatic management, but GKE is already managing them; installing the Operator on top actually conflicts with GKE's driver (in practice: the Operator's driver container cannot unload the kernel module GKE already loaded). So for running this experiment on GKE, the default stack is enough. + +Finally, verify the toolkit works too (a container can run `nvidia-smi`) with a real GPU pod: + +```bash +cat <<'EOF' | kubectl apply -f - && \ +kubectl wait --for=condition=Ready pod/gpu-smi-test --timeout=120s && \ +kubectl logs gpu-smi-test && \ +kubectl delete pod gpu-smi-test +apiVersion: v1 +kind: Pod +metadata: + name: gpu-smi-test +spec: + restartPolicy: Never + containers: + - name: cuda + image: nvidia/cuda:12.4.1-base-ubuntu22.04 + command: ["nvidia-smi"] + resources: + limits: + nvidia.com/gpu: 1 +EOF +``` + +Expected output (real): you see the T4 and the driver version; note the memory is **15360 MiB** (the T4's actual addressable memory, which Step 3 uses for the label). + +``` ++-----------------------------------------------------------------------------------------+ +| NVIDIA-SMI 580.159.04 Driver Version: 580.159.04 CUDA Version: 13.0 | ++-----------------------------------------+------------------------+----------------------+ +| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 | +| N/A 45C P8 17W / 70W | 0MiB / 15360MiB | 0% Default | ++-----------------------------------------+------------------------+----------------------+ +``` + +Seeing this table means the driver, device plugin, and toolkit are all in place, and the container can actually use the GPU. + +### Step 3: add the GFD labels (needed by KAI scheduling) + +KAI's hamicore plugin relies on the node label `nvidia.com/gpu.memory` to know each card's memory, so it can convert a `gpu-memory` request into a fraction. GKE's default device plugin sets `nvidia.com/gpu.present=true`, but **does not set `gpu.memory` / `gpu.count` / `gpu.product`** (those usually come from GPU Feature Discovery, which GKE's default stack does not run in full). So you add them manually. + +Use the memory value **15360** (MiB) you saw from `nvidia-smi` in Step 2: + +```bash +kubectl label node -l cloud.google.com/gke-accelerator=nvidia-tesla-t4 \ + nvidia.com/gpu.memory=15360 \ + nvidia.com/gpu.product=NVIDIA-Tesla-T4 \ + nvidia.com/gpu.count=1 \ + nvidia.com/gpu.present=true --overwrite +``` + +:::warning Critical step: labels must be in place before installing KAI + +KAI's hamicore plugin caches per GPU memory **at node registration time** by reading `node.labels["nvidia.com/gpu.memory"]`. Without this label it errors out, shared pods stay `Pending` forever, and the logs show `didn't have enough resources: GPU memory`. So this step must be done **before** installing KAI in Step 4. If you only add the label after KAI is installed, restart `kai-scheduler` so it re-caches. + +::: + +Confirm the labels are in place: + +```bash +kubectl get nodes -o custom-columns="NAME:.metadata.name,GPUMEM:.metadata.labels.nvidia\.com/gpu\.memory,GPUPROD:.metadata.labels.nvidia\.com/gpu\.product,GPUPRESENT:.metadata.labels.nvidia\.com/gpu\.present" +``` + +Expected output (example): each node shows `15360` under `GPUMEM`, `NVIDIA-Tesla-T4` under `GPUPROD`, and `true` under `GPUPRESENT`. + +``` +NAME GPUMEM GPUPROD GPUPRESENT +gke-test-cluster-default-pool-xxxxxxxx 15360 NVIDIA-Tesla-T4 true +gke-test-cluster-default-pool-yyyyyyyy 15360 NVIDIA-Tesla-T4 true +gke-test-cluster-default-pool-zzzzzzzz 15360 NVIDIA-Tesla-T4 true +``` + +**Why `15360` and not `16384`**: the T4 is nominally 16 GiB, but `nvidia-smi` reports the actual addressable memory as 15360 MiB (some memory is reserved by the driver / ECC). KAI uses this number to convert `gpu-memory` requests into fractions, so the label value must match the `nvidia-smi` number, otherwise the ceiling the pod sees will not match your expectation. + +### Step 4: install KAI Scheduler v0.17.0 and create the queue + +```bash +helm install kai-scheduler oci://ghcr.io/kai-scheduler/kai-scheduler/kai-scheduler \ + --set global.gpuSharing=true \ + --set binder.plugins.hamicore.enabled=true \ + --set binder.plugins.gpusharing.arguments.cdiEnabled=true \ + --namespace kai-scheduler --create-namespace \ + --version v0.17.0 + +# Wait for the admission webhook to be ready (signing the certificate takes a +# moment) before creating the queue, otherwise Queue creation is rejected. +kubectl -n kai-scheduler wait --for=condition=available --timeout=180s deploy --all +``` + +**Why these switches**: + +- `global.gpuSharing=true` turns on shared scheduling, allowing KAI to place multiple pods on the same GPU. +- `binder.plugins.hamicore.enabled=true` makes KAI inject `CUDA_DEVICE_MEMORY_LIMIT` at bind time. This is the contract that connects the scheduling layer to the isolation layer. Without it, the isolation layer never gets the quota and effectively is not installed. +- `binder.plugins.gpusharing.arguments.cdiEnabled=true`: **required on GKE 1.33+**. KAI's gpusharing plugin defaults to `cdiEnabled=false` (it assumes the GPU is injected via the NVIDIA runtime), but GKE 1.33+ containerd uses the CDI mode (`enable_cdi=true`) to inject GPUs. Without this flag, the GPU reservation pod KAI creates cannot find NVML (it fails with `ERROR_LIBRARY_NOT_FOUND`), and shared pods stay `Pending`. In the KAI source, CDI is auto-detected from the GPU Operator's ClusterPolicy, but we did not install the GPU Operator (see Step 2), so it must be enabled explicitly. (See the [KAI binder docs](https://github.com/kai-scheduler/KAI-Scheduler/blob/main/docs/developer/binder.md).) + +Then create the queue. **The queue must exist before KAI will schedule anything.** A pod points at its queue via the `kai.scheduler/queue: default-queue` label; if the queue (CRD `scheduling.run.ai/v2`) does not exist, the pod stays `Pending`. The manifest below creates two layers, `default` (parent) and `default-queue` (child), with all quotas open (`-1`) to keep the demo simple: + +```yaml +# queue.yaml +apiVersion: scheduling.run.ai/v2 +kind: Queue +metadata: + name: default +spec: + resources: + cpu: { quota: -1, limit: -1, overQuotaWeight: 1 } + memory: { quota: -1, limit: -1, overQuotaWeight: 1 } + gpu: { quota: -1, limit: -1, overQuotaWeight: 1 } +--- +apiVersion: scheduling.run.ai/v2 +kind: Queue +metadata: + name: default-queue +spec: + parentQueue: default + resources: + cpu: { quota: -1, limit: -1, overQuotaWeight: 1 } + memory: { quota: -1, limit: -1, overQuotaWeight: 1 } + gpu: { quota: -1, limit: -1, overQuotaWeight: 1 } +``` + +```bash +kubectl apply -f queue.yaml +kubectl get queues +``` + +Expected output (example): + +``` +NAME PARENT AGE +default 5s +default-queue default 5s +``` + +Both queues present and the child correctly pointing at the parent: then you can move on. + +### Step 5: install kai-resource-isolator 1.1.0-chart + +```bash +helm install kai-resource-isolator oci://docker.io/projecthami/kai-resource-isolator \ + --namespace kai-resource-isolator --create-namespace \ + --set monitor.enabled=true \ + --set monitor.serviceMonitor.enabled=true \ + --set paths.hostInstallBase=/home/kubernetes/bin/nvidia \ + --version 1.1.0-chart +``` + +**Why the version has a `-chart` suffix**: the chart version is **`1.1.0-chart`** (with the `-chart` suffix). The matching git tag is `v1.1.0`; do not confuse the two, since writing `1.1.0` or `v1.1.0` will not find the package. If your node NVML requires the NVIDIA runtime, also add `--set monitor.runtimeClassName=nvidia` (GKE GPU nodes already have this). + +**Why change `paths.hostInstallBase`**: the isolator's libsync DaemonSet writes `libvgpu.so` to `{hostInstallBase}/vgpu/` on the host, which defaults to `/usr/local`. But **GKE nodes have a read-only root filesystem** (ext2 mounted `ro`), so `/usr/local` is not writable and libsync fails with `Read-only file system`. `/home/kubernetes/bin/nvidia` is a **writable** directory on GKE (the standard path where GKE installs the NVIDIA driver), so the library goes there. On self-built clusters where the root filesystem is writable, you can omit this flag (use the default `/usr/local`). + +Confirm all three components are up: + +```bash +kubectl -n kai-resource-isolator get pods +``` + +Expected output (example): + +``` +NAME READY STATUS +kai-resource-isolator-xxxxxxxx-xxxxx 1/1 Running # webhook +kai-libsync-xxxxx 1/1 Running # distributes libvgpu.so +kai-vgpu-monitor-xxxxx 1/1 Running # :9394 metrics +``` + +All three pods `Running` means the webhook, libsync, and monitor are in place. On GKE, however, you still need a few extra adaptations before the isolation chain works end to end (see Step 5b below). On a self built cluster with a writable root filesystem, a `nvidia` RuntimeClass present, and NVML libraries in the standard paths, you can skip that section. + +### Step 5b: GKE environment adaptation (required on GKE 1.33+) + +The following four adaptations are confirmed GKE-specific requirements, each solving a concrete problem: + +**5b-1. Create the `nvidia` RuntimeClass (absent on GKE 1.35)** + +GKE 1.33+ injects GPUs in CDI mode, so no `nvidia` runtime handler is registered on the node and there is no `nvidia` RuntimeClass. But KAI's reservation pod checks for it at creation time. Create a RuntimeClass pointing at the default `runc` handler (CDI injects the device at the containerd layer, so the nvidia runtime is not needed): + +```bash +cat <<'EOF' | kubectl apply -f - +apiVersion: node.k8s.io/v1 +kind: RuntimeClass +metadata: + name: nvidia +handler: runc +EOF +``` + +**5b-2. Patch the isolator DaemonSets to remove `system-node-critical` priority** + +GKE forbids pods in user namespaces from using the `system-node-critical` PriorityClass (`insufficient quota to match these scopes`). The isolator's libsync and monitor DaemonSets use it by default, so it has to be removed: + +```bash +for ds in kai-resource-isolator-libsync kai-resource-isolator-monitor; do + kubectl patch ds "$ds" -n kai-resource-isolator \ + --type=json -p='[{"op":"replace","path":"/spec/template/spec/priorityClassName","value":null}]' +done +``` + +**5b-3. Install Kyverno and inject `LD_LIBRARY_PATH`** + +KAI's reservation pod uses NVML to query GPU information, but its container image does not ship `libnvidia-ml.so`. GKE injects the NVML library into the container at `/usr/local/nvidia/lib64/`, but it does not set `LD_LIBRARY_PATH` (the `nvidia/cuda` image carries this variable, while the reservation image does not). Use [Kyverno](https://kyverno.io/)'s MutatingWebhook to automatically inject this environment variable into pods in the reservation namespace: + +```bash +# Install Kyverno (ships its own TLS certificate management, no cert-manager needed) +helm install kyverno kyverno/kyverno -n kyverno --create-namespace +kubectl wait --for=condition=ready pod -l app.kubernetes.io/component=admission-controller \ + -n kyverno --timeout=300s + +# Create the policy: inject LD_LIBRARY_PATH into all pods in the kai-resource-reservation namespace +cat <<'EOF' | kubectl apply -f - +apiVersion: kyverno.io/v1 +kind: ClusterPolicy +metadata: + name: inject-nvidia-library-path +spec: + rules: + - name: add-ld-library-path + match: + any: + - resources: + kinds: [Pod] + namespaces: [kai-resource-reservation] + mutate: + foreach: + - list: "request.object.spec.containers[]" + patchStrategicMerge: + metadata: {} + spec: + containers: + - name: "{{ element.name }}" + env: + - name: LD_LIBRARY_PATH + value: /usr/local/nvidia/lib64 +EOF +``` + +The `inject-nvidia-library-path` policy above only solves the NVML query problem for the reservation pod. **Shared pods need a second policy, `inject-gpu-devices`**, which mounts the GPU device nodes and NVIDIA tool directories into the shared pods. The reason: under GKE CDI mode, the device plugin only runs `Allocate` and injects devices for pods that request the `nvidia.com/gpu` extended resource; KAI's shared pods use the `gpu-memory` annotation (no extended resource request), so the device plugin does not inject `/dev/nvidia*`, the `nvidia-smi` binary, or the NVML library into them. Without this policy, `nvidia-smi` inside a shared pod fails because it cannot find the device. The policy below matches any pod carrying the `gpu-memory` annotation (any namespace) and auto-mounts `/dev/nvidia0`, `/dev/nvidiactl`, `/dev/nvidia-uvm`, `/dev/nvidia-modeset`, `/dev/nvidia-caps`, plus the host's `nvidia-smi` binary directory and the `libnvidia-ml.so` library directory: + +```bash +# Second policy: inject GPU device nodes + NVIDIA library directories into all +# shared pods carrying the gpu-memory annotation +cat <<'EOF' | kubectl apply -f - +apiVersion: kyverno.io/v1 +kind: ClusterPolicy +metadata: + name: inject-gpu-devices +spec: + background: false + rules: + - name: add-gpu-volumes + match: + any: + - resources: + kinds: [Pod] + preconditions: + all: + - key: "{{ request.object.metadata.annotations.\"gpu-memory\" || '' }}" + operator: NotEquals + value: "" + mutate: + patchesJson6902: |- + - op: add + path: /spec/volumes/- + value: + name: nvidia-dev0 + hostPath: {path: /dev/nvidia0, type: CharDevice} + - op: add + path: /spec/volumes/- + value: + name: nvidia-ctl + hostPath: {path: /dev/nvidiactl, type: CharDevice} + - op: add + path: /spec/volumes/- + value: + name: nvidia-uvm + hostPath: {path: /dev/nvidia-uvm, type: CharDevice} + - op: add + path: /spec/volumes/- + value: + name: nvidia-modeset + hostPath: {path: /dev/nvidia-modeset, type: CharDevice} + - op: add + path: /spec/volumes/- + value: + name: nvidia-caps + hostPath: {path: /dev/nvidia-caps, type: Directory} + - op: add + path: /spec/volumes/- + value: + name: nvidia-bin + hostPath: {path: /home/kubernetes/bin/nvidia/bin, type: Directory} + - op: add + path: /spec/volumes/- + value: + name: nvidia-lib + hostPath: {path: /home/kubernetes/bin/nvidia/lib64, type: Directory} + - name: add-gpu-volumemounts + match: + any: + - resources: + kinds: [Pod] + preconditions: + all: + - key: "{{ request.object.metadata.annotations.\"gpu-memory\" || '' }}" + operator: NotEquals + value: "" + mutate: + foreach: + - list: "request.object.spec.containers[]" + patchStrategicMerge: + spec: + containers: + - name: "{{ element.name }}" + volumeMounts: + - {name: nvidia-dev0, mountPath: /dev/nvidia0} + - {name: nvidia-ctl, mountPath: /dev/nvidiactl} + - {name: nvidia-uvm, mountPath: /dev/nvidia-uvm} + - {name: nvidia-modeset, mountPath: /dev/nvidia-modeset} + - {name: nvidia-caps, mountPath: /dev/nvidia-caps} + - {name: nvidia-bin, mountPath: /usr/local/nvidia/bin} + - {name: nvidia-lib, mountPath: /usr/local/nvidia/lib64} +EOF +``` + +With both policies in place, shared pods get the device nodes (`/dev/nvidia*`), the `nvidia-smi` binary (`/usr/local/nvidia/bin`), and the NVML library (`/usr/local/nvidia/lib64`). Note: device nodes alone are not enough; the pod must also set `securityContext.privileged: true` (see Step 6 and Step 7), because HAMi-core's `libvgpu.so` requires privileged mode to initialize NVML (confirmed by testing). All three pieces (device nodes + library directories + privileged mode) are required; once they are all in place, the shared pod can successfully run `nvidia-smi` and see the isolated memory. + +**5b-4. Fix the `ld.so.preload` path written by libsync** + +The isolator's libsync writes `libvgpu.so` and `ld.so.preload` to `/home/kubernetes/bin/nvidia/vgpu/` (the `hostInstallBase` from Step 5), but the chart's `ld.so.preload` ConfigMap may still carry the old `/usr/local/vgpu/libvgpu.so` path. Verify and correct it to point at the right path: + +```bash +# Inspect the ConfigMap content +kubectl get cm kai-resource-isolator-ldpreload -n kai-resource-isolator \ + -o jsonpath='{.data.ld\.so\.preload}' +# If it is not /home/kubernetes/bin/nvidia/vgpu/libvgpu.so, fix it +kubectl create cm kai-resource-isolator-ldpreload -n kai-resource-isolator \ + --from-literal=ld.so.preload="/home/kubernetes/bin/nvidia/vgpu/libvgpu.so" \ + --dry-run=client -o yaml | kubectl apply -f - + +# Restart the libsync pods so they rewrite with the corrected ConfigMap +kubectl delete pod -n kai-resource-isolator \ + $(kubectl get pod -n kai-resource-isolator -l app.kubernetes.io/component=kai-libsync -o jsonpath='{.items[*].metadata.name}') +``` + +Additionally, the monitor DaemonSet has a hostPath volume pointing to the old `/usr/local/vgpu/containers` (read-only rootfs), which gets stuck in `ContainerCreating`. Patch it to the writable area too: + +```bash +kubectl patch ds kai-resource-isolator-monitor -n kai-resource-isolator --type=json \ + -p='[{"op":"replace","path":"/spec/template/spec/volumes/0","value":{"hostPath":{"path":"/home/kubernetes/bin/nvidia/vgpu/containers","type":"DirectoryOrCreate"},"name":"containers"}}]' +``` + +After these four steps, the reservation pod can successfully initialize NVML and hold the GPU, and shared pods can be scheduled and load `libvgpu.so`. Verify: + +```bash +# The reservation pod should be Running (no longer CrashLoopBackOff) +kubectl get pod -n kai-resource-reservation +# Inside a shared pod, libvgpu.so should preload successfully (no "cannot be preloaded" error) +kubectl exec -- cat /etc/ld.so.preload +# The memory limit KAI injected +kubectl exec -- env | grep CUDA_DEVICE_MEMORY_LIMIT +``` + +:::tip Verified + +Verified by testing: with Kyverno auto-mounting GPU device nodes + NVIDIA library directories + privileged mode into shared pods, shared pods can successfully run `nvidia-smi` and see only their memory quota (4147 MiB instead of the full 15360 MiB), confirming isolation works. The exact setup is the `inject-gpu-devices` policy in Step 5b-3 above, combined with `securityContext.privileged: true` in the Step 6 and Step 7 pod templates. + +::: + +### Step 6: a single pod, memory locked down + +First verify isolation works on a single pod, before moving to the sharing scenario; this makes problems easier to localize. + +```yaml +# single-pod.yaml +apiVersion: v1 +kind: Pod +metadata: + name: gpu-isolation-demo + labels: + kai.scheduler/queue: default-queue + annotations: + gpu-memory: "4096" # integer MiB, no unit suffix +spec: + schedulerName: kai-scheduler + restartPolicy: Never + containers: + - name: gpu-workload + image: nvidia/cuda:12.9.2-base-ubuntu24.04 + command: ["sleep", "infinity"] + securityContext: + privileged: true +``` + +**Why it is written this way** (matching the [KAI docs](https://github.com/kai-scheduler/KAI-Scheduler/blob/main/docs/gpu-sharing/hami/README.md) pod template): + +- `schedulerName: kai-scheduler` routes the pod through KAI (the default is `default-scheduler`). +- `kai.scheduler/queue: default-queue` attaches the pod to the queue from Step 4; a label that does not match a queue means the pod is never scheduled. +- `gpu-memory: "4096"` is the pod's memory quota, an integer MiB with no unit suffix. KAI uses it to inject `CUDA_DEVICE_MEMORY_LIMIT=4096` (MiB), which HAMi-core then enforces. +- **Do not add `resources.limits.nvidia.com/gpu`**. KAI's admission rejects pods that request both a whole card and a memory quota (`cannot request both GPU and GPU memory`); memory sharing is driven entirely by the `gpu-memory` annotation, and the whole-card device-plugin resource is allocated by KAI behind the scenes. + +```bash +kubectl apply -f single-pod.yaml +kubectl wait --for=condition=Ready pod/gpu-isolation-demo --timeout=300s +kubectl exec gpu-isolation-demo -- nvidia-smi +``` + +Expected output (real output, verified): + +``` ++-----------------------------------------------------------------------------+ +| NVIDIA-SMI ... Driver Version: ... CUDA Version: 12.9 | +|-------------------------------+----------------------+----------------------+ +| 0 NVIDIA T4 ... | 4147MiB / 4147MiB ... 0% Default | ++-------------------------------+----------------------+----------------------+ +``` + +The full T4 has 15360 MiB, but inside the container only about 4147 MiB is visible (the actual ceiling after the quota 4096 is converted to a fraction), which means HAMi-core rewrote the device query response, so isolation is working. If you still see 15360 MiB here, the isolator webhook is most likely not taking effect (check that the Step 5 pods are Running and that the pod is covered by the webhook's namespace selector). + +### Step 7: two pods sharing one T4 + +With isolation verified on a single pod, verify genuine sharing: two pods landing on the same card at the same time, each seeing only its own quota. + +```yaml +# shared-t4.yaml +--- +apiVersion: v1 +kind: Pod +metadata: + name: gpu-share-a + labels: + kai.scheduler/queue: default-queue + annotations: + gpu-memory: "4096" +spec: + schedulerName: kai-scheduler + restartPolicy: Never + containers: + - name: gpu-workload + image: nvidia/cuda:12.9.2-base-ubuntu24.04 + command: ["sleep", "infinity"] + securityContext: + privileged: true +--- +apiVersion: v1 +kind: Pod +metadata: + name: gpu-share-b + labels: + kai.scheduler/queue: default-queue + annotations: + gpu-memory: "4096" +spec: + schedulerName: kai-scheduler + restartPolicy: Never + containers: + - name: gpu-workload + image: nvidia/cuda:12.9.2-base-ubuntu24.04 + command: ["sleep", "infinity"] + securityContext: + privileged: true +``` + +Each pod requests `4096` MiB; together that is 8 GiB, well under a single 15360 MiB T4, so KAI schedules both onto the same card (rather than one card each). + +```bash +kubectl apply -f shared-t4.yaml +kubectl wait --for=condition=Ready pod/gpu-share-a pod/gpu-share-b --timeout=300s +kubectl exec gpu-share-a -- nvidia-smi --query-gpu=memory.total --format=csv,noheader +kubectl exec gpu-share-b -- nvidia-smi --query-gpu=memory.total --format=csv,noheader +``` + +Expected output (example): both pods return about 4147 MiB (the actual ceiling after the quota 4096 is converted to a fraction). + +``` +4147 MiB +4147 MiB +``` + +Then confirm they really share the same card rather than each taking their own: + +```bash +kubectl get pod -o wide | grep gpu-share +``` + +Expected output (example): both pods show the same `NODE` (landed on the same node), and that node only has 1 T4, proving they share the same card. + +``` +NAME READY STATUS RESTARTS AGE IP NODE +gpu-share-a 1/1 Running 0 2m 10.x.x.x gke-test-cluster-default-pool-xxxxxxxx +gpu-share-b 1/1 Running 0 2m 10.x.x.x gke-test-cluster-default-pool-xxxxxxxx +``` + +**Why `4096` and not more**: the T4 has 15360 MiB of actual memory, and two pods each requesting 4096 MiB (8 GiB total) leaves nearly half the card free, which makes scheduling more robust. `gpu-memory` is an integer MiB. The scheduler converts it to a two-decimal GPU fraction and then lands it as `CUDA_DEVICE_MEMORY_LIMIT`. `4096 / 15360 ≈ 0.27`, so the actual ceiling you see is about 4147 MiB rather than exactly 4096 (see pitfall #4 for the rounding details). A 15 GiB card is cleanly split into two slices that do not bleed into each other, and that is hard isolation in action. + +### Step 8 (optional): look at monitor metrics + +```bash +kubectl -n kai-resource-isolator port-forward daemonset/kai-vgpu-monitor 9394:9394 & +curl -s localhost:9394/metrics | grep hami_vgpu_memory_limit_bytes +``` + +Expected (example): + +``` +hami_vgpu_memory_limit_bytes{...} 4.147e+06 +``` + +**Why this metric**: `hami_vgpu_memory_limit_bytes` directly reflects the memory ceiling HAMi-core set for each container, and it matches the `nvidia-smi` number, making it the most direct observation point for verifying isolation. With Prometheus and Grafana, per container memory usage, limits, and utilization become ready made charts. + +## Pitfalls worth knowing up front + +1. **The `nvidia.com/gpu.memory` node label must appear first** (most important). KAI hamicore caches per card memory at node registration; without the label, shared pods get stuck `Pending` with `didn't have enough resources: GPU memory`. GKE's default device plugin **does not set this label** (it only sets `nvidia.com/gpu.present`), so this post adds it manually in Step 3. **The label must be added before installing KAI**; if you add it only after KAI is installed, restart `kai-scheduler` so it re-caches. Use the memory `nvidia-smi` reports (15360 for the T4 here, not the nominal 16384). +2. **Create the queue manually.** A pod points at its queue via the `kai.scheduler/queue` label, the CRD is `scheduling.run.ai/v2` `Queue`; without the queue, pods are never scheduled. Step 4 provides the manifest, and it must be applied after the KAI admission webhook is ready. +3. **RuntimeClass `nvidia`.** KAI shared reservation pods carry `runtimeClassName: nvidia` by default. GKE GPU nodes ship it (COS plus containerd registers the `nvidia` handler, performing real injection), so you usually do nothing. On a self built cluster that lacks it, create one pointing at the runtime that actually performs NVIDIA injection on your nodes. +4. **Memory precision is rounded.** `gpu-memory` is an integer MiB (no unit suffix). The scheduler converts it to a two decimal GPU fraction and then lands it as `CUDA_DEVICE_MEMORY_LIMIT`. In Step 6 and Step 7 of this post, requesting `4096` on a 15360 MiB T4 gives `4096 / 15360 ≈ 0.27`, so the actual ceiling you see is about 4147 MiB rather than exactly 4096 (this is exactly the T4 example in the official docs, see [Memory value precision](https://github.com/kai-scheduler/KAI-Scheduler/blob/main/docs/gpu-sharing/hami/README.md#memory-value-precision)). When you need finer control, reverse the math from the fraction. +5. **The chart version has a `-chart` suffix.** The OCI chart tag for `kai-resource-isolator` is `1.1.0-chart`, not `1.1.0` and not `v1.1.0`. +6. **Opting out of isolation is supported.** A single pod can set the annotation `kai-resource-isolator.io/inject: "false"`; an entire namespace can set the label `kai-resource-isolator.io/webhook=ignore`. Useful for gradual rollout. +7. **Monitor runtime and nodeSelector.** If your node NVML requires the NVIDIA runtime (some self built clusters), set `monitor.runtimeClassName=nvidia`; GKE GPU nodes satisfy this by default. The monitor DaemonSet also selects nodes via the `nvidia.com/gpu.present=true` nodeSelector; **this post adds `gpu.present=true` manually in Step 3, so the monitor pod schedules normally**. If you forget this label, the monitor pod will not come up; add it or change the nodeSelector. +8. **The driver is a prerequisite.** The isolation stack depends on the host NVIDIA driver. This post uses the GKE default stack (create the cluster with `gpu-driver-version=default`, and GKE installs the driver + device plugin + container toolkit automatically); on a self built cluster, get `nvidia-smi` working on the node first with the NVIDIA driver and container runtime. +9. **The isolator DaemonSets may fail to start on GKE (PriorityClass quota).** The kai-resource-isolator libsync and monitor DaemonSets default to the `system-node-critical` priority class, which GKE forbids for pods in user namespaces (`insufficient quota to match these scopes: [{PriorityClass In [system-node-critical ...]}]`). If you hit this, remove the `priorityClassName` from both DaemonSets: + + ```bash + for ds in kai-resource-isolator-libsync kai-resource-isolator-monitor; do + kubectl patch ds $ds -n kai-resource-isolator \ + --type=json -p='[{"op":"replace","path":"/spec/template/spec/priorityClassName","value":null}]' + done + ``` + + This triggers a rolling update and the pods reschedule. Note: `helm upgrade` reverts this patch, so re-apply it after each isolator upgrade (or, for a more durable fix, add a ResourceQuota that allows critical pods in the isolator namespace). + +10. **Do not put `resources.limits.nvidia.com/gpu` on shared pods.** KAI's admission rejects pods that request both a whole card and a memory quota (`cannot request both GPU and GPU memory`). Memory sharing is driven entirely by the `gpu-memory` annotation; follow the [KAI official pod template](https://github.com/kai-scheduler/KAI-Scheduler/blob/main/docs/gpu-sharing/hami/README.md) and omit the `resources` section. +11. **KAI's config-deployer hook may fail on GKE.** The `helm install kai-scheduler` post-install hook (the `kai-config-deployer` Job) can stall on GKE 1.35 due to an API server authentication issue (OpenAPI schema download fails with `the server has asked for the client to provide credentials`), so helm reports `INSTALLATION FAILED`. However, the KAI CRDs and the `kai-operator` are already installed. The fix is to apply the Config CR (the very resource the hook tries to deploy) manually: render the chart with `helm template ... > rendered.yaml`, extract `kai-config.yaml` from the `kai-config-manifest` ConfigMap (note that `cdiEnabled` must be a **quoted string** `"true"`, the CRD rejects a boolean), then `kubectl apply --validate=false -f kai-config.yaml`. Once kai-operator sees the Config CR, it brings up binder/admission/scheduler and the rest automatically. If the release status is stuck at `failed`, use `helm upgrade ... --no-hooks` to correct the state (it skips the hook and idempotently reconciles the resources). +12. **GKE nodes have a read-only root filesystem, so libsync cannot write to `/usr/local/vgpu`.** GKE nodes mount `/` as `ext2 ro`; the isolator's libsync writes `libvgpu.so` to `/usr/local/vgpu` by default and crashes with `Read-only file system`. The `--set paths.hostInstallBase=/home/kubernetes/bin/nvidia` flag in Step 5 fixes this by placing the library in GKE's writable area. Note: `paths.hostInstallBase` only changes libsync's write path and the webhook's hostPath; the chart's `ld.so.preload` ConfigMap may not be updated to match, so it needs manual verification/fixing (see Step 5b-4). +13. **GKE 1.35 has no `nvidia` RuntimeClass.** GKE 1.33+ uses CDI mode, so no nvidia runtime handler is registered on the node. KAI needs this RuntimeClass to exist when creating shared pods, otherwise the pod is rejected (`RuntimeClass "nvidia" not found`). Create one pointing at the default `runc` handler (see Step 5b-1). On a self built cluster that already registers a real nvidia runtime handler, this is not needed. +14. **The reservation pod cannot find the NVML library on GKE CDI.** GKE injects the NVML library into the container at `/usr/local/nvidia/lib64/`, but does not set `LD_LIBRARY_PATH`. The `nvidia/cuda` image carries this variable so it runs, but KAI's reservation image does not. The fix is to use a Kyverno MutatingWebhook to inject it automatically (see Step 5b-3). Related KAI issue: [#1449](https://github.com/kai-scheduler/kai-scheduler/issues/1449). +15. **Shared pods on GKE CDI need extra GPU device node mounts (solved).** The GKE device plugin only injects devices into pods that request `nvidia.com/gpu`; shared pods use the `gpu-memory` annotation instead. The fix: use the Kyverno `inject-gpu-devices` policy to auto-mount them (see Step 5b-3) and set `privileged: true` on the pod (HAMi-core's `libvgpu.so` needs privileged mode to initialize NVML, confirmed by testing). With both in place, shared pods can run `nvidia-smi` and see only their memory quota (4147 MiB instead of the full 15360 MiB). + +## What this means + +HAMi-core was never positioned as "the isolation feature of some particular scheduler"; it is a **scheduler-decoupled, reusable isolation base**. Well before KAI Scheduler, it already powered the Kubernetes native scheduler, [Kueue](/docs/userguide/kueue/how-to-use-kueue), [Volcano](/docs/installation/how-to-use-volcano-vgpu), Koordinator, and other paths (see [Ecosystem Integrations](/docs/next/core-concepts/ecosystem-integrations)). KAI Scheduler v0.16.4 joining the list extends this ecosystem to NVIDIA's official AI scheduler: + +- For **KAI users**: GPU sharing finally has a matching runtime hard isolation, so sharing no longer means running without guarantees. +- For **HAMi users**: there is now a path that does not lock you to a specific virtual device plugin and goes directly through NVIDIA's official scheduler, while keeping the metric surface (`hami_*`) compatible. +- For the **community**: the contract between the scheduling layer and the isolation layer (`CUDA_DEVICE_MEMORY_LIMIT`) is validated once more; any future scheduler that honors it can reuse the same isolation engine. + +Behind this is more than a year of careful alignment between the KAI Scheduler team (Run:ai) and the HAMi maintainers. The `LD_PRELOAD`, the webhook, the metrics port, and the opt-out switch were each settled by both sides; the security tightening in `kai-resource-isolator` 1.1.0 (namespaced Issuer, tightened ClusterRole) was polished through community review line by line. + +## Next steps + +- The background story: [HAMi-core adopted by NVIDIA KAI Scheduler](/blog/hami-core-adopted-by-nvidia-kai-scheduler) +- User docs: [How to use HAMi with KAI Scheduler](/docs/next/userguide/kai-scheduler/how-to-use-kai-scheduler) +- Related repos: [Project-HAMi/KAI-resource-isolator](https://github.com/Project-HAMi/KAI-resource-isolator), [Project-HAMi/HAMi-core](https://github.com/Project-HAMi/HAMi-core) (the CNCF incubating CUDA interception library), [kai-scheduler/KAI-Scheduler](https://github.com/kai-scheduler/KAI-Scheduler), [KAI Scheduler HAMi resource isolation docs](https://github.com/kai-scheduler/KAI-Scheduler/blob/main/docs/gpu-sharing/hami/README.md) +- Run it on your own GKE, AWS, or self built cluster, and share real results in an issue or the community group. If HAMi-core lacks support for a certain card or CUDA version, open an issue at [Project-HAMi/HAMi](https://github.com/Project-HAMi/HAMi). This is the feedback the community values most. diff --git a/i18n/zh/docusaurus-plugin-content-blog/authors.yml b/i18n/zh/docusaurus-plugin-content-blog/authors.yml index 2eb071bc0..7a68f5d33 100644 --- a/i18n/zh/docusaurus-plugin-content-blog/authors.yml +++ b/i18n/zh/docusaurus-plugin-content-blog/authors.yml @@ -15,3 +15,7 @@ archlitchi: name: 李孟轩 title: HAMi 维护者，密瓜智能联合创始人兼 CTO url: https://github.com/archlitchi + +rootsongjc: + name: 宋净超（Jimmy Song） + url: https://github.com/rootsongjc diff --git a/i18n/zh/docusaurus-plugin-content-blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md b/i18n/zh/docusaurus-plugin-content-blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md new file mode 100644 index 000000000..8b7565861 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md @@ -0,0 +1,824 @@ +--- +title: "KAI Scheduler 与 HAMi 的 GPU 显存硬隔离：运行原理与实践" +date: "2026-08-11" +description: "从 CUDA 拦截原理到 GKE 上的可复现验证：用 KAI Scheduler v0.17.0 做 GPU 共享调度、kai-resource-isolator 1.1.0-chart（基于 HAMi-core）做 CUDA 级显存隔离，在单张 NVIDIA T4 上让两个 Pod 各自只见自己的显存配额。讲清 CUDA_DEVICE_MEMORY_LIMIT 这一调度层与隔离层的契约，以及节点标签、队列、RuntimeClass 等关键注意事项。" +authors: [rootsongjc] +tags: ["HAMi", "KAI Scheduler", "硬隔离", "GPU 共享", "Kubernetes", "云原生"] +--- + +GPU 共享在 Kubernetes 生态里讨论了很多年，但调度层与隔离层长期各自为政：调度器把几个 Pod 分到同一张卡，容器进入 GPU 后却依然能看到整张卡的显存，谁先发起 `cudaMalloc` 谁就占满，隔离形同虚设。所谓「共享」其实只是「抢」，没有任何资源保障可言。 + +要解决这个问题，需要调度层（决定「谁能用哪张卡、用多少」）和隔离层（保证「说好用多少就只能用多少」）协同。**HAMi-core** 正是这样一个可被多种调度器复用的隔离底座。在 KAI Scheduler 之前，它已经支持 Kubernetes 原生调度器（经 HAMi 自带的 `hami-scheduler` 扩展）、[Kueue](/zh/docs/userguide/kueue/how-to-use-kueue)、[Volcano](/zh/docs/installation/how-to-use-volcano-vgpu) 等（完整生态见 [HAMi 生态集成](/zh/docs/next/core-concepts/ecosystem-integrations)）。 + +**自 KAI Scheduler v0.16.4 起，NVIDIA 的 KAI Scheduler 也正式加入这一行列**，把 HAMi-core 作为其 GPU 共享的隔离引擎内置支持（当前版本为 v0.17.0）。这意味着用 KAI Scheduler 调度 GPU 工作负载时，不再只有「协作式共享」，而是有了真正的硬隔离保障。本文讲清两件事： + +- **原理**：KAI Scheduler、`kai-resource-isolator`、HAMi-core 三者各自负责什么、如何衔接，以及 `CUDA_DEVICE_MEMORY_LIMIT` 这一契约如何把调度层与隔离层连起来。 +- **实践**：一套可在 GKE 上完整复现的验证示例（单张 NVIDIA T4 卡，两个 Pod 共享，各自只见自己的显存配额），每一步都附配置文件、预期输出，以及「为什么这样做」的解释。 + +背景故事与协作时间线见 [《HAMi-core 被 NVIDIA KAI Scheduler 采用：GPU 共享正式迈入硬隔离时代》](/zh/blog/hami-core-adopted-by-nvidia-kai-scheduler)。 + +:::note 关于本文的输出 + +下半部分的命令为可在 GKE 上完整复现的步骤，所附命令输出与指标为预期示意，实际数值以你的集群环境为准。 + +::: + + + +## 背景：为什么「共享」不等于「隔离」 + +| 层次 | 负责方 | 解决了什么 | 没有解决什么 | +| --- | --- | --- | --- | +| 调度层 | KAI Scheduler、Volcano、Kueue 等 | 多个 Pod 能被分到同一张 GPU | 容器内仍可见全部显存 | +| 运行时层 | HAMi-core（`libvgpu.so`） | 拦截 CUDA 调用，按配额限制显存 | 单独使用时，不知道每个 Pod 该分多少 | + +要实现真正的 GPU 共享，这两层缺一不可，而且必须协同：调度层决定「谁能用哪张卡、用多少」，隔离层保证「说好用多少就只能用多少」。问题是，**隔离层需要知道「到底用多少」这个数字，而它本身是算不出来的，这个数字来自调度层**。 + +HAMi 社区多年打磨的 HAMi-core（CNCF 孵化项目）正是这样的隔离层，而且它是**与调度器解耦**的：在 KAI Scheduler 之前，HAMi-core 已经通过不同的方式与 Kubernetes 原生调度器（经 HAMi 自带的 `hami-scheduler` 扩展）、[Kueue](/zh/docs/userguide/kueue/how-to-use-kueue)、[Volcano](/zh/docs/installation/how-to-use-volcano-vgpu)、Koordinator 等协同工作（完整生态见 [HAMi 生态集成](/zh/docs/next/core-concepts/ecosystem-integrations)）。 + +**KAI Scheduler 在 v0.16.4 正式加入这一行列**，把 HAMi-core 作为其 GPU 共享的隔离引擎内置支持。本文要讲清的，就是 KAI Scheduler 这条集成路径上三个组件各自的角色，以及它们之间如何衔接。 + +## 运行原理：一个契约、三步协作 + +### 三个组件各是什么、各做什么 + +先把三个名字容易混淆的组件讲清楚，这是理解整条链路的前提： + +- **HAMi-core（`libvgpu.so`）**：HAMi 项目的 CUDA 拦截库（CNCF 孵化），是**隔离引擎本身**。它通过 `LD_PRELOAD` 拦截容器里的 CUDA 调用（如 `cudaMalloc`），按一个显存配额强制限制。它不关心配额是谁给的：任何调度器只要按约定把配额传进来，它都能执行隔离。在 KAI 之前，它已经被 HAMi 自带的 device-plugin/webhook、Volcano 的 `volcano-vgpu-device-plugin` 等复用。 + +- **KAI Scheduler**：NVIDIA 开源的 Kubernetes AI 工作负载调度器（源自 Run:ai，CNCF Sandbox）。它只负责**调度层**，即决定 Pod 落到哪个节点、用哪张 GPU、分多少显存。自 v0.16.4 起，它的 `hamicore` 插件在绑定 Pod 时，会把算好的显存配额写进容器的环境变量。 + +- **`kai-resource-isolator`**：HAMi 项目侧为 KAI Scheduler 这条集成路径**专门提供的配套组件**。它把 HAMi-core 的 `libvgpu.so` 分发到每个 GPU 节点，并用 MutatingWebhook 改写 Pod，把库和 `ld.so.preload` 注入进去。换言之，它是「把 KAI 的调度决策落地成 HAMi-core 能执行的隔离」的桥梁。 + +```mermaid +%% title: 三个组件的职责与衔接 +graph LR + subgraph SCH["调度层"] + KAI["KAI Scheduler
（hamicore 插件）

职责：算配额
把 Pod 绑定到节点
注入 CUDA_DEVICE_MEMORY_LIMIT"] + end + subgraph BRIDGE["衔接层（HAMi 项目提供）"] + ISO["kai-resource-isolator

职责：分发 libvgpu.so
webhook 注入库与 ld.so.preload"] + end + subgraph ISO2["隔离层"] + HAMI["HAMi-core / libvgpu.so
（CNCF 孵化）

职责：拦截 cudaMalloc
按配额强制限制显存"] + end + + KAI -->|"CUDA_DEVICE_MEMORY_LIMIT
+ Pod 绑定结果"| ISO + ISO -->|"libvgpu.so 就位
ld.so.preload 指向它"| HAMI + HAMI -.->|"复用同一引擎
（也被 K8s/Volcano/Kueue 等复用）"| SCH + + style KAI fill:#d9f99d,stroke:#4f7d00,stroke-width:2px,color:#1f2937 + style ISO fill:#dbeafe,stroke:#1a5fb4,stroke-width:2px,color:#1f2937 + style HAMI fill:#fef3c7,stroke:#b45309,stroke-width:2px,color:#1f2937 + style SCH fill:#f7fee7,stroke:#4f7d00,stroke-width:1px,color:#1f2937 + style BRIDGE fill:#eff6ff,stroke:#1a5fb4,stroke-width:1px,color:#1f2937 + style ISO2 fill:#fffbeb,stroke:#b45309,stroke-width:1px,color:#1f2937 +``` + +一句话：**KAI Scheduler 算配额，`kai-resource-isolator` 把隔离库就位，HAMi-core 在运行时真正执行隔离。** + +:::note 为什么是 HAMi-core，不是完整 HAMi 平台 + +KAI Scheduler 的集成目标是 **HAMi-core 本身**，而不是完整的 HAMi 平台。KAI 保留自己的调度能力（不替换成 `hami-scheduler`），只引入 HAMi-core 来做 GPU 显存隔离。这与 Volcano（用 `volcano-vgpu-device-plugin` + HAMi-core）的分工模式是同一思路：调度器各用各的，隔离引擎共用 HAMi-core。 + +::: + +### 契约：`CUDA_DEVICE_MEMORY_LIMIT` + +整条链路之所以能成立，是因为调度层和隔离层约定了一个极简的交接点：环境变量 **`CUDA_DEVICE_MEMORY_LIMIT`**。 + +- **KAI Scheduler（调度层）** 负责算出「这个 Pod 能用多少显存」，并在绑定节点时把它写进容器的环境变量。 +- **HAMi-core（隔离层）** 负责读这个环境变量，并在运行时真正把显存用量卡在这个上限以内。 + +这个契约之所以重要，是因为它**把两件事彻底解耦**：KAI 不需要知道 CUDA 怎么被拦截，HAMi-core 不需要知道份额是怎么算出来的。两边只要都遵守 `CUDA_DEVICE_MEMORY_LIMIT` 这一个变量，任何调度器都能复用同一套隔离引擎。这正是 HAMi-core 能同时支持多个调度器的根本原因（见文末「这意味着什么」）。 + +### 三步协作 + +```mermaid +%% title: KAI Scheduler 与 kai-resource-isolator 的三步协作 +graph TD + KAI["KAI Scheduler
hamicore 插件
读取 gpu-memory 注解"] + ENV["1. 调度时注入
CUDA_DEVICE_MEMORY_LIMIT 环境变量"] + ISO["2. kai-resource-isolator
Mutating Webhook 改写 Pod"] + LIB["libsync DaemonSet
分发 libvgpu.so 到 /usr/local/vgpu"] + MON["monitor DaemonSet
:9394 暴露 hami_* 指标"] + RUN["3. 容器启动
libvgpu.so 经 LD_PRELOAD 拦截 cudaMalloc"] + ENF["按 LIMIT 拒绝超额分配
nvidia-smi 仅显示配额内显存"] + + KAI --> ENV --> ISO --> RUN --> ENF + LIB -. "注入 hostPath 与 ld.so.preload" .-> ISO + MON -. "采集各容器显存" .-> RUN + + style KAI fill:#d9f99d,stroke:#4f7d00,stroke-width:2px,color:#1f2937 + style ENV fill:#d9f99d,stroke:#4f7d00,stroke-width:2px,color:#1f2937 + style ISO fill:#dbeafe,stroke:#1a5fb4,stroke-width:2px,color:#1f2937 + style LIB fill:#dbeafe,stroke:#1a5fb4,stroke-width:2px,color:#1f2937 + style MON fill:#dbeafe,stroke:#1a5fb4,stroke-width:2px,color:#1f2937 + style RUN fill:#fef3c7,stroke:#b45309,stroke-width:2px,color:#1f2937 + style ENF fill:#dcfce7,stroke:#0b6b3c,stroke-width:2px,color:#1f2937 +``` + +一句话概括：**KAI 说「你只能用这么多」，isolator 负责「让你真的只能用这么多」。** + +三个组件的分工如下（详见 [KAI Scheduler 官方文档「HAMi 资源隔离」](https://github.com/kai-scheduler/KAI-Scheduler/blob/main/docs/gpu-sharing/hami/README.md)）： + +- **libsync DaemonSet**：把 `libvgpu.so` 复制到每个 GPU 节点的 `/usr/local/vgpu`。 +- **mutating webhook**：给 Pod 注入 hostPath 卷挂载，把 `/etc/ld.so.preload` 指向 `libvgpu.so`，并写入 `POD_UID`、`CONTAINER_NAME`、`CONTAINER_VGPU_MOUNT` 环境变量。 +- **monitor DaemonSet**（可选）：读取各容器的共享内存缓存，在 `:9394` 暴露指标。 + +### CUDA 拦截是如何生效的 + +隔离的「最后一公里」发生在容器进程里，链路是这样的： + +1. **KAI 在调度时注入 `CUDA_DEVICE_MEMORY_LIMIT`**（带上 Pod 申请的显存配额，单位 MiB）。 +2. **kai-resource-isolator 的 webhook 改写 Pod**：挂载宿主机上的 `libvgpu.so`，并把 `/etc/ld.so.preload` 指向它。`ld.so.preload` 是动态链接器的机制，被列在里面的共享库会在所有其他库之前加载。 +3. **容器进程启动后**，任何对 CUDA 运行时（`libcudart`）或驱动 API 的调用，都会先经过 `libvgpu.so`。后者拦截 `cudaMalloc` 之类的显存分配调用，从 `CUDA_DEVICE_MEMORY_LIMIT` 读出配额，累计该容器的显存用量；一旦超额就拒绝分配。 +4. **对外可见的效果**：`nvidia-smi` 只显示配额内的显存（HAMi-core 会改写设备查询的返回值），容器再怎么 `cudaMalloc` 也越不过这条线。 + +这就是「硬隔离」的含义：不是靠应用自觉，而是在 CUDA 调用这一层强制执行。 + +:::tip 集成的来龙去脉 + +这条集成路径是 HAMi 社区与 NVIDIA KAI Scheduler 团队一年多协作的结果，分工很清晰：KAI 负责注入环境变量，HAMi 负责资源隔离组件。完整的时间线与参与人员见姊妹篇 [《HAMi-core 被 NVIDIA KAI Scheduler 采用》](/zh/blog/hami-core-adopted-by-nvidia-kai-scheduler)。 + +::: + +## 这两个版本带来了什么 + +### KAI Scheduler（自 v0.16.4 起内置 HAMi-core 硬隔离） + +KAI Scheduler 对 HAMi-core 的支持自 **v0.16.4** 起成为内置能力（本文示例使用当前版本 v0.17.0）。关键在于 `hamicore` 插件：启用后，KAI 在把共享 GPU 的 Pod 绑定到节点时，会按 `gpu-memory`（或 `gpu-fraction`）注解向容器注入 `CUDA_DEVICE_MEMORY_LIMIT` 环境变量，这正是上一节契约里 HAMi-core 执行隔离所需的配额。 + +v0.17.0 中其他与 GPU 相关的改动还包括：修正共享 Pod 名称带「`/`」造成的非法卷名问题；修正 `MinNodeGPUMemoryMiB` 与 fractional `gpu-memory` 的分配计算；overLimit 判定改用集群最大 GPU 规格。此外还有 preemption-delay（为 Cluster Autoscaler 留出拉起节点的时间窗）、NUMA 感知打分、GitOps 与 ArgoCD 安装支持等。 + +### kai-resource-isolator 1.1.0-chart + +这是与 HAMi 配套发布的隔离器。它接收 KAI 注入的配额，在容器真正运行起来之前，把 HAMi-core 的 `libvgpu.so` 注入到位。相对首个版本，1.1.0 补齐了一组让它具备生产可用性的改进： + +- **新增 `kai-vgpu-monitor`**：以 DaemonSet 形式运行，在 `:9394` 暴露 HAMi 兼容指标（`hami_vgpu_memory_used_bytes`、`hami_vgpu_memory_limit_bytes`、`hami_container_device_utilization_ratio`），支持 ServiceMonitor，可被 Prometheus 直接抓取。 +- **多容器注入修复**：一个 Pod 内有多个容器时，webhook 现在能正确处理，不再漏注。 +- **安全收紧**：webhook 改用命名空间内 Issuer（不再使用 ClusterIssuer），ClusterRole 收回读 Secret 的权限。 +- **全局镜像仓库**优先级理顺，`hamicore` 安装参数修正。 + +## 实践：在 GKE 上验证硬隔离 + +示例运行在一个已有的 GKE 集群上：**3 个 `n1-standard-2` 节点，每节点挂载 1 张 NVIDIA T4（`nvidia-smi` 实测显存 15360 MiB）**。两个容器各申请约 4 GiB，共享其中一张 T4；`nvidia-smi` 里各自只见自己的配额，互不干扰。下面每一步都给出配置、预期输出，以及为什么这样做。 + +下图是整个实验的总览：8 个步骤的顺序，以及其中两个最容易踩坑的前置依赖（GFD 标签必须在 KAI 之前打好；队列必须存在才会调度）。注意：本文**不需要安装 NVIDIA GPU Operator**，GKE 默认的 device plugin + 驱动 + container toolkit 已经够用，下面只要「启用默认 device plugin」和「补几个 GFD 标签」两步准备即可。 + +```mermaid +%% title: GKE 实验流程（Step 1 到 Step 8） +graph TD + S1["Step 1
确认集群与 GPU 硬件
n1-standard-2 + T4"] + S2["Step 2
启用 GKE 默认 device plugin
验证 nvidia-smi"] + S3["Step 3
打 GFD 标签
nvidia.com/gpu.memory 等"] + S4["Step 4
安装 KAI Scheduler v0.17.0
创建队列"] + S5["Step 5
安装 kai-resource-isolator
1.1.0-chart"] + S6["Step 6
单 Pod 验证隔离
nvidia-smi 见配额内显存"] + S7["Step 7
两 Pod 共享一张 T4
各见自己的配额"] + S8["Step 8 可选
查看 monitor 指标"] + + S1 --> S2 --> S3 --> S4 --> S5 --> S6 --> S7 --> S8 + + S3 -. "⚠️ GFD 标签必须在 KAI 之前打
否则共享 Pod 卡 Pending" .-> S4 + S4 -. "⚠️ 队列必须存在
否则 Pod 不被调度" .-> S6 + + style S1 fill:#dbeafe,stroke:#1a5fb4,stroke-width:2px,color:#1f2937 + style S2 fill:#fef3c7,stroke:#b45309,stroke-width:2px,color:#1f2937 + style S3 fill:#fef3c7,stroke:#b45309,stroke-width:2px,color:#1f2937 + style S4 fill:#d9f99d,stroke:#4f7d00,stroke-width:2px,color:#1f2937 + style S5 fill:#d9f99d,stroke:#4f7d00,stroke-width:2px,color:#1f2937 + style S6 fill:#dcfce7,stroke:#0b6b3c,stroke-width:2px,color:#1f2937 + style S7 fill:#dcfce7,stroke:#0b6b3c,stroke-width:2px,color:#1f2937 + style S8 fill:#eff6ff,stroke:#1a5fb4,stroke-width:1px,color:#1f2937 +``` + +### 前置条件 + +- 一个 GCP 项目，已启用 GKE 与 Compute Engine API。 +- 一个已有的 GKE 集群，节点池里带 NVIDIA T4 GPU（本文示例为 3 个 `n1-standard-2` 节点，每节点 1× T4）。本例这个集群建的时候带了节点标签 `gke-no-default-nvidia-gpu-device-plugin=true`，即**禁用了 GKE 默认的 NVIDIA device plugin**，驱动则由 GKE 自动安装。这正是后面 Step 1 会看到的「硬件在、但 `nvidia.com/gpu` 资源为空」这一初始状态的由来，Step 2 会把 device plugin 重新启用。如果你还没建好，可以用下面的命令建一个同等规格的集群： + + ```bash + gcloud container clusters create test-cluster --zone=asia-northeast1-a \ + --machine-type=n1-standard-2 --num-nodes=3 \ + --accelerator=type=nvidia-tesla-t4,count=1,gpu-driver-version=default + gcloud container clusters get-credentials test-cluster --zone=asia-northeast1-a + ``` + + `n1` 机型本身不含 GPU，所以**必须**用 `--accelerator` 显式指定 T4；`gpu-driver-version=default` 让 GKE 自动安装匹配的 NVIDIA 驱动和 container toolkit。这样建出来的集群，device plugin 默认就是启用的（Step 2 可以跳过删标签那步）。 + +- `gcloud`、`kubectl`、`helm`（≥ 3）已登录，`kubectl` 已能访问该集群。 +- KAI 队列由下文 Step 4 创建。 + +### Step 1：确认集群与 GPU 硬件就绪（初始状态） + +集群已经建好后，先看一眼它的初始状态。这一步只看、不改。 + +```bash +kubectl get nodes -o custom-columns="NAME:.metadata.name,GPU:.status.capacity.nvidia\.com/gpu,ACCEL:.metadata.labels.cloud\.google\.com/gke-accelerator" +``` + +本例这个集群的初始输出是这样的：3 个节点，`ACCEL` 列都是 `nvidia-tesla-t4`（T4 硬件在），但 `GPU` 列是 ``。 + +``` +NAME GPU ACCEL +gke-test-cluster-default-pool-xxxxxxxx nvidia-tesla-t4 +gke-test-cluster-default-pool-yyyyyyyy nvidia-tesla-t4 +gke-test-cluster-default-pool-zzzzzzzz nvidia-tesla-t4 +``` + +**怎么读这个输出**：`ACCEL` 列的 `nvidia-tesla-t4` 确认 T4 硬件已经挂载到节点上，GKE 也已经把 NVIDIA 内核驱动装好了。但 `GPU` 列是 ``，说明 **device plugin 还没上报 `nvidia.com/gpu` 资源**。原因是这个集群建的时候带了节点标签 `gke-no-default-nvidia-gpu-device-plugin=true`，把 GKE 默认的 NVIDIA device plugin 禁用了。下一启用它。 + +如果你的集群 `GPU` 列已经直接是 `1`（建集群时没禁用默认 device plugin），可以跳过 Step 2，直接到 Step 3 打 GFD 标签。 + +### Step 2：启用 GKE 默认 device plugin，并验证 GPU 可用 + +把节点上的 `gke-no-default-nvidia-gpu-device-plugin` 标签删掉，GKE 自带的 device plugin DaemonSet（它的调度规则是「这个标签不为 `true` 才上」）就会自动调度到节点上。 + +```bash +kubectl label node -l cloud.google.com/gke-accelerator=nvidia-tesla-t4 \ + gke-no-default-nvidia-gpu-device-plugin- +``` + +标签名末尾的 `-` 表示「删除这个标签」。执行后等大约 20 秒，device plugin Pod 就会起来： + +```bash +kubectl get pod -n kube-system -o wide | grep nvidia-gpu-device-plugin | grep -v "0/0" +``` + +预期输出（示意）：3 个 `nvidia-gpu-device-plugin-*-ubuntu` Pod，每个节点一个，都是 `3/3 Running`。 + +``` +NAME READY STATUS NODE +nvidia-gpu-device-plugin-small-ubuntu-xxx 3/3 Running gke-test-cluster-default-pool-xxxxxxxx +nvidia-gpu-device-plugin-small-ubuntu-yyy 3/3 Running gke-test-cluster-default-pool-yyyyyyyy +nvidia-gpu-device-plugin-small-ubuntu-zzz 3/3 Running gke-test-cluster-default-pool-zzzzzzzz +``` + +device plugin 起来后，`nvidia.com/gpu` 资源就有了： + +```bash +kubectl get nodes -o custom-columns="NAME:.metadata.name,GPU:.status.capacity.nvidia\.com/gpu" +``` + +预期输出（示意）：每个节点的 `GPU` 列从 `` 变成 `1`。 + +``` +NAME GPU +gke-test-cluster-default-pool-xxxxxxxx 1 +gke-test-cluster-default-pool-yyyyyyyy 1 +gke-test-cluster-default-pool-zzzzzzzz 1 +``` + +**为什么不需要装 GPU Operator**：GKE 默认的 GPU 栈已经把三样东西都配好了：NVIDIA 内核驱动（GKE 自动装）、device plugin（刚启用）、container toolkit / nvidia runtime（`gpu-driver-version=default` 时 GKE 自动配）。GPU Operator 是把这些打包在一起自动管理的工具，但 GKE 自己已经在管了，再装反而会和 GKE 的驱动冲突（实测：GPU Operator 的驱动容器无法卸载 GKE 已加载的内核模块）。所以 GKE 上跑通实验，靠 GKE 默认栈就够了。 + +最后用一个真正的 GPU Pod 验证 toolkit 也工作（容器能跑 `nvidia-smi`）： + +```bash +cat <<'EOF' | kubectl apply -f - && \ +kubectl wait --for=condition=Ready pod/gpu-smi-test --timeout=120s && \ +kubectl logs gpu-smi-test && \ +kubectl delete pod gpu-smi-test +apiVersion: v1 +kind: Pod +metadata: + name: gpu-smi-test +spec: + restartPolicy: Never + containers: + - name: cuda + image: nvidia/cuda:12.4.1-base-ubuntu22.04 + command: ["nvidia-smi"] + resources: + limits: + nvidia.com/gpu: 1 +EOF +``` + +预期输出（实测）：能看到 T4 和驱动版本，注意显存是 **15360 MiB**（这是 T4 实际可寻址显存，下面 Step 3 打标签要用这个数）。 + +``` ++-----------------------------------------------------------------------------------------+ +| NVIDIA-SMI 580.159.04 Driver Version: 580.159.04 CUDA Version: 13.0 | ++-----------------------------------------+------------------------+----------------------+ +| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 | +| N/A 45C P8 17W / 70W | 0MiB / 15360MiB | 0% Default | ++-----------------------------------------+------------------------+----------------------+ +``` + +看到这张表，说明驱动、device plugin、toolkit 全部就位，容器已经能真正使用 GPU 了。 + +### Step 3：打 GFD 标签（KAI 调度需要） + +KAI 的 hamicore 插件要靠节点标签 `nvidia.com/gpu.memory` 来知道每张卡的显存，才能把 `gpu-memory` 请求换算成分数。GKE 默认 device plugin 会打 `nvidia.com/gpu.present=true`，但**不打 `gpu.memory` / `gpu.count` / `gpu.product` 这几个**（那些通常由 GPU Feature Discovery 打，而 GKE 默认栈不跑完整 GFD）。所以需要手动补上。 + +用 Step 2 里 `nvidia-smi` 看到的显存值 **15360**（MiB）： + +```bash +kubectl label node -l cloud.google.com/gke-accelerator=nvidia-tesla-t4 \ + nvidia.com/gpu.memory=15360 \ + nvidia.com/gpu.product=NVIDIA-Tesla-T4 \ + nvidia.com/gpu.count=1 \ + nvidia.com/gpu.present=true --overwrite +``` + +:::warning 关键一步：标签必须在安装 KAI 之前打好 + +KAI 的 hamicore 插件在**节点注册时**缓存每张卡的显存（读取 `node.labels["nvidia.com/gpu.memory"]`）。缺这个标签它会直接报错，共享 Pod 永远停在 `Pending`，日志中出现 `didn't have enough resources: GPU memory`。所以这一步必须在 Step 4 装 KAI **之前**完成。万一装完 KAI 才补标签，重启 `kai-scheduler` 让它重新缓存即可。 + +::: + +确认标签到位： + +```bash +kubectl get nodes -o custom-columns="NAME:.metadata.name,GPUMEM:.metadata.labels.nvidia\.com/gpu\.memory,GPUPROD:.metadata.labels.nvidia\.com/gpu\.product,GPUPRESENT:.metadata.labels.nvidia\.com/gpu\.present" +``` + +预期输出（示意）：每个节点的 `GPUMEM` 为 `15360`，`GPUPROD` 为 `NVIDIA-Tesla-T4`，`GPUPRESENT` 为 `true`。 + +``` +NAME GPUMEM GPUPROD GPUPRESENT +gke-test-cluster-default-pool-xxxxxxxx 15360 NVIDIA-Tesla-T4 true +gke-test-cluster-default-pool-yyyyyyyy 15360 NVIDIA-Tesla-T4 true +gke-test-cluster-default-pool-zzzzzzzz 15360 NVIDIA-Tesla-T4 true +``` + +**为什么是 `15360` 而不是 `16384`**：T4 标称 16 GiB，但 `nvidia-smi` 报告的实际可寻址显存是 15360 MiB（部分显存被驱动/ECC 预留）。KAI 用这个数把 `gpu-memory` 请求换算成分数，所以标签值必须和 `nvidia-smi` 的数字一致，否则 Pod 看到的上限会和你预期不符。 + +### Step 4：安装 KAI Scheduler v0.17.0，并创建队列 + +```bash +helm install kai-scheduler oci://ghcr.io/kai-scheduler/kai-scheduler/kai-scheduler \ + --set global.gpuSharing=true \ + --set binder.plugins.hamicore.enabled=true \ + --set binder.plugins.gpusharing.arguments.cdiEnabled=true \ + --namespace kai-scheduler --create-namespace \ + --version v0.17.0 + +# 等 admission webhook 就绪（签发证书需要一点时间），再创建队列， +# 否则 Queue 的创建会被拒绝。 +kubectl -n kai-scheduler wait --for=condition=available --timeout=180s deploy --all +``` + +**为什么是这几个开关**： + +- `global.gpuSharing=true` 打开共享调度，允许 KAI 把多个 Pod 放到同一张 GPU 上。 +- `binder.plugins.hamicore.enabled=true` 让 KAI 在绑定时注入 `CUDA_DEVICE_MEMORY_LIMIT`，这是把调度层和隔离层连起来的那个契约。没开它，隔离层就拿不到配额，等于没装。 +- `binder.plugins.gpusharing.arguments.cdiEnabled=true`：**GKE 1.33+ 必加**。KAI 的 gpusharing 插件默认 `cdiEnabled=false`（假设 GPU 通过 NVIDIA runtime 注入），但 GKE 1.33+ 的 containerd 用 CDI 模式（`enable_cdi=true`）注入 GPU。不开这个开关，KAI 创建的 GPU reservation Pod 会找不到 NVML（报 `ERROR_LIBRARY_NOT_FOUND`），共享 Pod 卡在 `Pending`。KAI 源码里 CDI 优先靠 GPU Operator 的 ClusterPolicy 自动检测，但我们没装 GPU Operator（见 Step 2），所以必须显式打开。（参考 [KAI binder 文档](https://github.com/kai-scheduler/KAI-Scheduler/blob/main/docs/developer/binder.md)。） + +随后创建队列。**队列必须存在，KAI 才会调度。** Pod 靠 `kai.scheduler/queue: default-queue` 标签指向队列；队列（CRD `scheduling.run.ai/v2`）不存在，Pod 就一直 `Pending`。下面这份清单建立了 `default`（父）与 `default-queue`（子）两层，配额全部放开（`-1`），方便演示： + +```yaml +# queue.yaml +apiVersion: scheduling.run.ai/v2 +kind: Queue +metadata: + name: default +spec: + resources: + cpu: { quota: -1, limit: -1, overQuotaWeight: 1 } + memory: { quota: -1, limit: -1, overQuotaWeight: 1 } + gpu: { quota: -1, limit: -1, overQuotaWeight: 1 } +--- +apiVersion: scheduling.run.ai/v2 +kind: Queue +metadata: + name: default-queue +spec: + parentQueue: default + resources: + cpu: { quota: -1, limit: -1, overQuotaWeight: 1 } + memory: { quota: -1, limit: -1, overQuotaWeight: 1 } + gpu: { quota: -1, limit: -1, overQuotaWeight: 1 } +``` + +```bash +kubectl apply -f queue.yaml +kubectl get queues +``` + +预期输出（示意）： + +``` +NAME PARENT AGE +default 5s +default-queue default 5s +``` + +两个队列都在、子队列正确指向父队列，才能进入下一步。 + +### Step 5：安装 kai-resource-isolator 1.1.0-chart + +```bash +helm install kai-resource-isolator oci://docker.io/projecthami/kai-resource-isolator \ + --namespace kai-resource-isolator --create-namespace \ + --set monitor.enabled=true \ + --set monitor.serviceMonitor.enabled=true \ + --set paths.hostInstallBase=/home/kubernetes/bin/nvidia \ + --version 1.1.0-chart +``` + +**为什么版本号带 `-chart` 后缀**：chart 版本是 **`1.1.0-chart`**（带 `-chart` 后缀），对应的 git tag 是 `v1.1.0`，两者不要混淆，写成 `1.1.0` 或 `v1.1.0` 会拉不到包。若节点的 NVML 必须走 NVIDIA runtime，再补一个 `--set monitor.runtimeClassName=nvidia`（GKE GPU 节点默认已具备）。 + +**为什么改 `paths.hostInstallBase`**：isolator 的 libsync DaemonSet 要把 `libvgpu.so` 写到宿主机的 `{hostInstallBase}/vgpu/`，默认是 `/usr/local`。但 **GKE 节点的根文件系统是只读的**（ext2 `ro` 挂载），`/usr/local` 不可写，libsync 会报 `Read-only file system` 起不来。`/home/kubernetes/bin/nvidia` 是 GKE 上 NVIDIA 驱动所在的**可写**目录（也是 GKE 装 driver 的标准路径），所以把库放这里。非 GKE 的自建集群根文件系统可写时，这个参数可以不加（用默认 `/usr/local`）。 + +确认三个组件都已就绪： + +```bash +kubectl -n kai-resource-isolator get pods +``` + +预期输出（示意）： + +``` +NAME READY STATUS +kai-resource-isolator-xxxxxxxx-xxxxx 1/1 Running # webhook +kai-libsync-xxxxx 1/1 Running # 分发 libvgpu.so +kai-vgpu-monitor-xxxxx 1/1 Running # :9394 指标 +``` + +三个 Pod 都 `Running`，意味着 webhook、libsync、monitor 全部就位。但在 GKE 上，还需要做几步额外适配才能让隔离链路完整工作（见下文）。自建集群如果根文件系统可写、有 `nvidia` RuntimeClass、节点 NVML 库在标准路径，可以跳过本节。 + +### Step 5b：GKE 环境适配（GKE 1.33+ 必做） + +以下四个适配是实测确认的 GKE 特有要求，每一步都解决一个具体问题： + +**5b-1. 创建 `nvidia` RuntimeClass（GKE 1.35 没有）** + +GKE 1.33+ 用 CDI 模式注入 GPU，节点上没有注册 `nvidia` runtime handler，因此也没有 `nvidia` RuntimeClass。但 KAI 的 reservation Pod 在创建时会检查它。创建一个指向默认 `runc` handler 的 RuntimeClass（CDI 在 containerd 层注入设备，不需要 nvidia runtime）： + +```bash +cat <<'EOF' | kubectl apply -f - +apiVersion: node.k8s.io/v1 +kind: RuntimeClass +metadata: + name: nvidia +handler: runc +EOF +``` + +**5b-2. patch isolator DaemonSet 去掉 `system-node-critical` 优先级** + +GKE 不允许用户 namespace 的 Pod 使用 `system-node-critical` PriorityClass（报 `insufficient quota to match these scopes`）。isolator 的 libsync 和 monitor DaemonSet 默认用了它，需要去掉： + +```bash +for ds in kai-resource-isolator-libsync kai-resource-isolator-monitor; do + kubectl patch ds "$ds" -n kai-resource-isolator \ + --type=json -p='[{"op":"replace","path":"/spec/template/spec/priorityClassName","value":null}]' +done +``` + +**5b-3. 安装 Kyverno 并注入 `LD_LIBRARY_PATH`** + +KAI 的 reservation Pod 用 NVML 查询 GPU 信息，但它的容器镜像里没有 `libnvidia-ml.so`。GKE 会把 NVML 库注入到容器的 `/usr/local/nvidia/lib64/`，但不会设 `LD_LIBRARY_PATH`（nvidia/cuda 镜像自带这个变量，而 reservation 镜像没有）。用 [Kyverno](https://kyverno.io/) 的 MutatingWebhook 自动给 reservation namespace 的 Pod 注入这个环境变量： + +```bash +# 安装 Kyverno（自带 TLS 证书管理，不需要 cert-manager） +helm install kyverno kyverno/kyverno -n kyverno --create-namespace +kubectl wait --for=condition=ready pod -l app.kubernetes.io/component=admission-controller \ + -n kyverno --timeout=300s + +# 创建策略：给 kai-resource-reservation namespace 的所有 Pod 注入 LD_LIBRARY_PATH +cat <<'EOF' | kubectl apply -f - +apiVersion: kyverno.io/v1 +kind: ClusterPolicy +metadata: + name: inject-nvidia-library-path +spec: + rules: + - name: add-ld-library-path + match: + any: + - resources: + kinds: [Pod] + namespaces: [kai-resource-reservation] + mutate: + foreach: + - list: "request.object.spec.containers[]" + patchStrategicMerge: + metadata: {} + spec: + containers: + - name: "{{ element.name }}" + env: + - name: LD_LIBRARY_PATH + value: /usr/local/nvidia/lib64 +EOF +``` + +上面这条 `inject-nvidia-library-path` 策略只解决了 reservation Pod 的 NVML 查询问题。**共享 Pod 还需要第二条策略 `inject-gpu-devices`**，作用是把 GPU 设备节点和 NVIDIA 工具目录挂进共享 Pod。原因是：GKE CDI 模式下，device plugin 只对 request 了 `nvidia.com/gpu` 扩展资源的 Pod 执行 `Allocate` 并注入设备；而 KAI 的共享 Pod 用 `gpu-memory` 注解（不 request 扩展资源），所以 device plugin 不会给它们注入 `/dev/nvidia*`、`nvidia-smi` 二进制和 NVML 库。没有这条策略，共享 Pod 里 `nvidia-smi` 会因为找不到设备而无法运行。下面这份策略匹配所有带 `gpu-memory` 注解的 Pod（任意 namespace），自动挂载 `/dev/nvidia0`、`/dev/nvidiactl`、`/dev/nvidia-uvm`、`/dev/nvidia-modeset`、`/dev/nvidia-caps` 以及宿主机上的 `nvidia-smi` 二进制目录和 `libnvidia-ml.so` 库目录： + +```bash +# 第二条策略：给所有带 gpu-memory 注解的共享 Pod 注入 GPU 设备节点和 NVIDIA 库目录 +cat <<'EOF' | kubectl apply -f - +apiVersion: kyverno.io/v1 +kind: ClusterPolicy +metadata: + name: inject-gpu-devices +spec: + background: false + rules: + - name: add-gpu-volumes + match: + any: + - resources: + kinds: [Pod] + preconditions: + all: + - key: "{{ request.object.metadata.annotations.\"gpu-memory\" || '' }}" + operator: NotEquals + value: "" + mutate: + patchesJson6902: |- + - op: add + path: /spec/volumes/- + value: + name: nvidia-dev0 + hostPath: {path: /dev/nvidia0, type: CharDevice} + - op: add + path: /spec/volumes/- + value: + name: nvidia-ctl + hostPath: {path: /dev/nvidiactl, type: CharDevice} + - op: add + path: /spec/volumes/- + value: + name: nvidia-uvm + hostPath: {path: /dev/nvidia-uvm, type: CharDevice} + - op: add + path: /spec/volumes/- + value: + name: nvidia-modeset + hostPath: {path: /dev/nvidia-modeset, type: CharDevice} + - op: add + path: /spec/volumes/- + value: + name: nvidia-caps + hostPath: {path: /dev/nvidia-caps, type: Directory} + - op: add + path: /spec/volumes/- + value: + name: nvidia-bin + hostPath: {path: /home/kubernetes/bin/nvidia/bin, type: Directory} + - op: add + path: /spec/volumes/- + value: + name: nvidia-lib + hostPath: {path: /home/kubernetes/bin/nvidia/lib64, type: Directory} + - name: add-gpu-volumemounts + match: + any: + - resources: + kinds: [Pod] + preconditions: + all: + - key: "{{ request.object.metadata.annotations.\"gpu-memory\" || '' }}" + operator: NotEquals + value: "" + mutate: + foreach: + - list: "request.object.spec.containers[]" + patchStrategicMerge: + spec: + containers: + - name: "{{ element.name }}" + volumeMounts: + - {name: nvidia-dev0, mountPath: /dev/nvidia0} + - {name: nvidia-ctl, mountPath: /dev/nvidiactl} + - {name: nvidia-uvm, mountPath: /dev/nvidia-uvm} + - {name: nvidia-modeset, mountPath: /dev/nvidia-modeset} + - {name: nvidia-caps, mountPath: /dev/nvidia-caps} + - {name: nvidia-bin, mountPath: /usr/local/nvidia/bin} + - {name: nvidia-lib, mountPath: /usr/local/nvidia/lib64} +EOF +``` + +两条策略一起用，共享 Pod 就同时有了设备节点（`/dev/nvidia*`）、`nvidia-smi` 二进制（`/usr/local/nvidia/bin`）和 NVML 库（`/usr/local/nvidia/lib64`）。注意：仅有设备节点还不够，Pod 还必须开 `securityContext.privileged: true`（见 Step 6、Step 7），因为 HAMi-core 的 `libvgpu.so` 需要特权模式才能初始化 NVML（实测确认）。这三件事（设备节点 + 库目录 + privileged）缺一不可，配齐后共享 Pod 才能成功跑通 `nvidia-smi` 并看到隔离后的显存。 + +**5b-4. 修正 libsync 写入的 `ld.so.preload` 路径 + monitor hostPath** + +isolator 的 libsync 把 `libvgpu.so` 和 `ld.so.preload` 写到 `/home/kubernetes/bin/nvidia/vgpu/`（Step 5 的 `hostInstallBase`），但 chart 的 `ld.so.preload` ConfigMap 里可能还是旧的 `/usr/local/vgpu/libvgpu.so` 路径。确认并修正它指向正确路径： + +```bash +# 确认 ConfigMap 内容 +kubectl get cm kai-resource-isolator-ldpreload -n kai-resource-isolator \ + -o jsonpath='{.data.ld\.so\.preload}' +# 如果不是 /home/kubernetes/bin/nvidia/vgpu/libvgpu.so，修正它 +kubectl create cm kai-resource-isolator-ldpreload -n kai-resource-isolator \ + --from-literal=ld.so.preload="/home/kubernetes/bin/nvidia/vgpu/libvgpu.so" \ + --dry-run=client -o yaml | kubectl apply -f - + +# 重启 libsync Pod 让它用修正后的 ConfigMap 重新写入 +kubectl delete pod -n kai-resource-isolator \ + $(kubectl get pod -n kai-resource-isolator -l app.kubernetes.io/component=kai-libsync -o jsonpath='{.items[*].metadata.name}') +``` + +另外，monitor DaemonSet 也有一个 hostPath volume 指向旧的 `/usr/local/vgpu/containers`（只读 rootfs），会卡在 `ContainerCreating`。同样改成可写区： + +```bash +kubectl patch ds kai-resource-isolator-monitor -n kai-resource-isolator --type=json \ + -p='[{"op":"replace","path":"/spec/template/spec/volumes/0","value":{"hostPath":{"path":"/home/kubernetes/bin/nvidia/vgpu/containers","type":"DirectoryOrCreate"},"name":"containers"}}]' +``` + +完成以上四步后，reservation Pod 能成功初始化 NVML 并持有 GPU，共享 Pod 能成功调度并加载 `libvgpu.so`。验证： + +```bash +# reservation Pod 应该 Running（不再 CrashLoopBackOff） +kubectl get pod -n kai-resource-reservation +# 共享 Pod 里 libvgpu.so 应该成功 preload（没有 "cannot be preloaded" 错误） +kubectl exec -- cat /etc/ld.so.preload +# KAI 注入的显存限制 +kubectl exec -- env | grep CUDA_DEVICE_MEMORY_LIMIT +``` + +:::tip 实测验证 + +经实测验证，通过 Kyverno 自动给共享 Pod 挂载 GPU 设备节点 + NVIDIA 库目录 + privileged 模式，共享 Pod 可以成功运行 `nvidia-smi`，看到配额内的显存（4147 MiB 而非整卡 15360 MiB），隔离生效。具体做法见上面 Step 5b-3 的 `inject-gpu-devices` 策略与 Step 6、Step 7 Pod 模板里的 `securityContext.privileged: true`。 + +::: + +### Step 6：单 Pod，显存被「锁」住 + +先用一个 Pod 验证隔离本身能生效，再上共享场景，便于定位问题。 + +```yaml +# single-pod.yaml +apiVersion: v1 +kind: Pod +metadata: + name: gpu-isolation-demo + labels: + kai.scheduler/queue: default-queue + annotations: + gpu-memory: "4096" # 整数 MiB，不带后缀 +spec: + schedulerName: kai-scheduler + restartPolicy: Never + containers: + - name: gpu-workload + image: nvidia/cuda:12.9.2-base-ubuntu24.04 + command: ["sleep", "infinity"] + securityContext: + privileged: true +``` + +**为什么这样写**（对照 [KAI 官方文档](https://github.com/kai-scheduler/KAI-Scheduler/blob/main/docs/gpu-sharing/hami/README.md)的 Pod 模板）： + +- `schedulerName: kai-scheduler` 让 Pod 走 KAI 调度（默认是 `default-scheduler`）。 +- `kai.scheduler/queue: default-queue` 把 Pod 挂到 Step 4 建的队列上；标签对应不上的队列，Pod 不会被调度。 +- `gpu-memory: "4096"` 注解是这个 Pod 的显存配额，整数 MiB、不带单位后缀。KAI 会据此注入 `CUDA_DEVICE_MEMORY_LIMIT=4096`（MiB），HAMi-core 据此拦截。 +- **不要写 `resources.limits.nvidia.com/gpu`**。KAI 的 admission 会拒绝同时申请「整卡」和「显存配额」的 Pod，报 `cannot request both GPU and GPU memory`；显存共享完全由 `gpu-memory` 注解控制，device plugin 的整卡资源由 KAI 在背后分配。 + +```bash +kubectl apply -f single-pod.yaml +kubectl wait --for=condition=Ready pod/gpu-isolation-demo --timeout=300s +kubectl exec gpu-isolation-demo -- nvidia-smi +``` + +预期输出（实测）： + +``` ++-----------------------------------------------------------------------------+ +| NVIDIA-SMI ... Driver Version: ... CUDA Version: 12.9 | +|-------------------------------+----------------------+----------------------+ +| 0 NVIDIA T4 ... | 4147MiB / 4147MiB ... 0% Default | ++-------------------------------+----------------------+----------------------+ +``` + +整张 T4 本是 15360 MiB，容器里只看到约 4147 MiB（配额 4096 经 fraction 折算后的实际上限），说明 HAMi-core 改写了设备查询的返回值，隔离生效。如果这里仍看到 15360 MiB，多半是 isolator webhook 没生效（检查 Step 5 的 Pod 是否 Running、Pod 是否被 webhook 命名空间选择器覆盖）。 + +### Step 7：两个 Pod 共享一张 T4 + +隔离单 Pod 验证通过后，验证真正的共享：两个 Pod 同时落在同一张卡上，各自只见自己的配额。 + +```yaml +# shared-t4.yaml +--- +apiVersion: v1 +kind: Pod +metadata: + name: gpu-share-a + labels: + kai.scheduler/queue: default-queue + annotations: + gpu-memory: "4096" +spec: + schedulerName: kai-scheduler + restartPolicy: Never + containers: + - name: gpu-workload + image: nvidia/cuda:12.9.2-base-ubuntu24.04 + command: ["sleep", "infinity"] + securityContext: + privileged: true +--- +apiVersion: v1 +kind: Pod +metadata: + name: gpu-share-b + labels: + kai.scheduler/queue: default-queue + annotations: + gpu-memory: "4096" +spec: + schedulerName: kai-scheduler + restartPolicy: Never + containers: + - name: gpu-workload + image: nvidia/cuda:12.9.2-base-ubuntu24.04 + command: ["sleep", "infinity"] + securityContext: + privileged: true +``` + +两个 Pod 各申请 `4096` MiB，加起来 8 GiB，远小于单张 T4 的 15360 MiB，因此 KAI 会把它们调度到同一张卡上（而不是各占一卡）。 + +```bash +kubectl apply -f shared-t4.yaml +kubectl wait --for=condition=Ready pod/gpu-share-a pod/gpu-share-b --timeout=300s +kubectl exec gpu-share-a -- nvidia-smi --query-gpu=memory.total --format=csv,noheader +kubectl exec gpu-share-b -- nvidia-smi --query-gpu=memory.total --format=csv,noheader +``` + +预期输出（示意）：两个 Pod 都返回约 4147 MiB（配额 4096 经 fraction 折算后的上限）。 + +``` +4147 MiB +4147 MiB +``` + +再确认它们确实共享同一张卡，而不是各占一卡： + +```bash +kubectl get pod -o wide | grep gpu-share +``` + +预期输出（示意）：两个 Pod 的 `NODE` 列相同（落在同一个节点上），且这个节点只有 1 张 T4，证明它们共享的是同一张卡。 + +``` +NAME READY STATUS RESTARTS AGE IP NODE +gpu-share-a 1/1 Running 0 2m 10.x.x.x gke-test-cluster-default-pool-xxxxxxxx +gpu-share-b 1/1 Running 0 2m 10.x.x.x gke-test-cluster-default-pool-xxxxxxxx +``` + +**为什么是 `4096` 而不是更大**：T4 实际显存 15360 MiB，两个 Pod 各要 4096 MiB（共 8 GiB），留出近一半余量，KAI 调度起来更稳妥。`gpu-memory` 是整数 MiB，调度器会把它折算成两位小数的 GPU fraction 再落地为 `CUDA_DEVICE_MEMORY_LIMIT`。`4096 / 15360 ≈ 0.27`，实际看到的上限约为 4147 MiB 而非精确 4096（折算细节见坑 #4）。一张 15 GiB 的卡被干净地切成两份，互不越界，这就是硬隔离的效果。 + +### Step 8（可选）：查看 monitor 指标 + +```bash +kubectl -n kai-resource-isolator port-forward daemonset/kai-vgpu-monitor 9394:9394 & +curl -s localhost:9394/metrics | grep hami_vgpu_memory_limit_bytes +``` + +预期（示意）： + +``` +hami_vgpu_memory_limit_bytes{...} 4.147e+06 +``` + +**为什么用这个指标**：`hami_vgpu_memory_limit_bytes` 直接反映 HAMi-core 给每个容器设的显存上限，和 `nvidia-smi` 的数字对得上，是验证隔离最直接的观测点。配合 Prometheus 与 Grafana 后，每个容器的显存用量、上限与利用率都是现成的曲线。 + +## 几个值得提前知道的坑 + +1. **`nvidia.com/gpu.memory` 节点标签必须先出现**（最重要）。KAI hamicore 在节点注册时缓存每卡显存，缺标签会导致共享 Pod 卡在 `Pending` 并报 `didn't have enough resources: GPU memory`。GKE 默认 device plugin **不打这个标签**（它只打 `nvidia.com/gpu.present`），所以本文 Step 3 手动补上。**补标签必须在装 KAI 之前完成**；万一装完 KAI 才补，重启 `kai-scheduler` 让它重新缓存即可。标签值要用 `nvidia-smi` 报的显存（本例 T4 是 15360，不是标称的 16384）。 +2. **队列必须手动创建**。Pod 靠 `kai.scheduler/queue` 标签指向队列，CRD 是 `scheduling.run.ai/v2` 的 `Queue`；队列不存在 Pod 就不会被调度。本文 Step 4 已提供清单，且要等 KAI admission webhook 就绪后再创建。 +3. **RuntimeClass `nvidia`**。KAI 的共享 reservation Pod 默认带 `runtimeClassName: nvidia`。GKE GPU 节点自带（COS 加 containerd 注册了 `nvidia` handler，真正注入 GPU），一般无需干预；自建集群若缺失，创建一个指向节点上真正执行 NVIDIA 注入的 runtime。 +4. **显存精度会四舍五入**。`gpu-memory` 是整数 MiB（无单位后缀），调度器会把它折算成两位小数的 GPU fraction 再落地为 `CUDA_DEVICE_MEMORY_LIMIT`。本文 Step 6、Step 7 在显存 15360 MiB 的 T4 上申请 `4096`，`4096 / 15360 ≈ 0.27`，实际看到的上限约为 4147 MiB 而非精确 4096（这正好是官方文档举的 T4 例子，见 [Memory value precision](https://github.com/kai-scheduler/KAI-Scheduler/blob/main/docs/gpu-sharing/hami/README.md#memory-value-precision)）。需要更精细的控制时，按 fraction 反推即可。 +5. **Chart 版本带 `-chart` 后缀**。`kai-resource-isolator` 的 OCI chart 标签是 `1.1.0-chart`，不是 `1.1.0` 也不是 `v1.1.0`。 +6. **想退出隔离也支持**：单个 Pod 加注解 `kai-resource-isolator.io/inject: "false"`；整个命名空间打标签 `kai-resource-isolator.io/webhook=ignore`，便于灰度迁移。 +7. **monitor 的 runtime 与 nodeSelector**。如果节点 NVML 必须走 NVIDIA runtime（部分自建集群），记得设置 `monitor.runtimeClassName=nvidia`；GKE GPU 节点默认满足。另外 monitor DaemonSet 默认靠 `nvidia.com/gpu.present=true` 这个 nodeSelector 选择节点；**本文 Step 3 手动补了 `gpu.present=true`，monitor Pod 能正常调度**。如果忘了打这个标签，monitor Pod 不会上来，补打或修改 nodeSelector 即可。 +8. **驱动是前置条件**。隔离栈依赖宿主机 NVIDIA 驱动。本文用 GKE 默认栈（建集群带 `gpu-driver-version=default`，GKE 自动装驱动 + device plugin + container toolkit）；自建集群请先用 NVIDIA Driver 加 container runtime 在节点上跑通 `nvidia-smi`。 +9. **GKE 上 isolator 的 DaemonSet 可能起不来（PriorityClass 配额）**。kai-resource-isolator 的 libsync 和 monitor DaemonSet 默认用 `system-node-critical` 优先级，而 GKE 不允许用户命名空间的 Pod 用这个 PriorityClass（报 `insufficient quota to match these scopes: [{PriorityClass In [system-node-critical ...]}]`）。如果遇到，把两个 DaemonSet 的 `priorityClassName` 去掉即可： + + ```bash + for ds in kai-resource-isolator-libsync kai-resource-isolator-monitor; do + kubectl patch ds $ds -n kai-resource-isolator \ + --type=json -p='[{"op":"replace","path":"/spec/template/spec/priorityClassName","value":null}]' + done + ``` + + patch 后会触发滚动更新，Pod 重新调度上来。注意：`helm upgrade` 会把这个 patch 覆盖回去，所以每次升级 isolator 后要重新 patch（或给 isolator namespace 加一个允许 critical pods 的 ResourceQuota 作为更持久的解法）。 + +10. **共享 Pod 模板不要写 `resources.limits.nvidia.com/gpu`**。KAI 的 admission 会拒绝同时申请整卡和显存配额的 Pod（报 `cannot request both GPU and GPU memory`）。显存共享完全由 `gpu-memory` 注解控制，按 [KAI 官方 Pod 模板](https://github.com/kai-scheduler/KAI-Scheduler/blob/main/docs/gpu-sharing/hami/README.md)写即可，不加 `resources` 段。 +11. **GKE 上 KAI 的 config-deployer hook 可能失败**。`helm install kai-scheduler` 的 post-install hook（`kai-config-deployer` Job）在 GKE 1.35 上可能因 API server 认证问题（OpenAPI schema 下载报 `the server has asked for the client to provide credentials`）一直跑不完，导致 helm 报 `INSTALLATION FAILED`。但 KAI 的 CRD 和 `kai-operator` 其实已经装好了。最快的修复：手动 apply Config CR（用 `helm template` 渲染，注意 `cdiEnabled` 必须是带引号字符串 `"true"`，CRD 不接受 boolean），再 `kubectl apply --validate=false -f kai-config.yaml`；kai-operator 看到 Config CR 后会自动拉起所有组件。如果 release 状态卡在 `failed`，用 `helm upgrade ... --no-hooks` 修正状态（跳过 hook，幂等地重新确认资源）。 +12. **GKE 节点根文件系统只读，libsync 写不进 `/usr/local/vgpu`**。GKE 节点的 `/` 是 `ext2 ro` 挂载，isolator 的 libsync 默认把 `libvgpu.so` 写到 `/usr/local/vgpu`，会报 `Read-only file system` 而 CrashLoop。Step 5 的 `--set paths.hostInstallBase=/home/kubernetes/bin/nvidia` 就是解决这个（把库写到 GKE 的可写区）。注意：`paths.hostInstallBase` 只改了 libsync 的写入路径和 webhook 的 hostPath，但 chart 的 `ld.so.preload` ConfigMap 可能没跟着改，需要手动确认/修正（见 Step 5b-4）。 +13. **GKE 1.35 没有 `nvidia` RuntimeClass**。GKE 1.33+ 用 CDI 模式，节点没有注册 nvidia runtime handler。KAI 创建共享 Pod 时需要这个 RuntimeClass 存在，否则 Pod 被拒绝（`RuntimeClass "nvidia" not found`）。创建一个指向默认 `runc` handler 的即可（见 Step 5b-1）。自建集群如果已注册了真正的 nvidia runtime handler，则不需要。 +14. **reservation Pod 在 GKE CDI 模式下找不到 NVML 库**。GKE 把 NVML 库注入到容器的 `/usr/local/nvidia/lib64/`，但不设 `LD_LIBRARY_PATH`。`nvidia/cuda` 镜像自带这个变量所以能跑，但 KAI 的 reservation 镜像没有。解法是用 Kyverno MutatingWebhook 自动注入（见 Step 5b-3）。对应的 KAI issue：[#1449](https://github.com/kai-scheduler/kai-scheduler/issues/1449)。 +15. **GKE CDI 模式下共享 Pod 需要额外挂载 GPU 设备节点（已解决）**。GKE device plugin 只对 request 了 `nvidia.com/gpu` 的 Pod 注入设备；共享 Pod 用 `gpu-memory` 注解（无 resource request），所以 device plugin 不响应。解法：用 Kyverno `inject-gpu-devices` 策略自动挂载（见 Step 5b-3）+ Pod 加 `privileged: true`（HAMi-core 的 `libvgpu.so` 需要特权模式初始化 NVML，实测确认）。配齐这两项后，共享 Pod 能成功跑通 `nvidia-smi` 并看到配额内的显存（4147 MiB 而非整卡 15360 MiB）。 + +## 这意味着什么 + +HAMi-core 的定位从来不是「某个调度器的附属隔离功能」，而是一个**与调度器解耦、可被各路调度器复用的隔离底座**。在 KAI Scheduler 之前，它已经支撑了 Kubernetes 原生调度器、[Kueue](/zh/docs/userguide/kueue/how-to-use-kueue)、[Volcano](/zh/docs/installation/how-to-use-volcano-vgpu)、Koordinator 等多条路径（见 [HAMi 生态集成](/zh/docs/next/core-concepts/ecosystem-integrations)）。KAI Scheduler v0.16.4 的加入，是把这个生态又扩到了 NVIDIA 官方的 AI 调度器： + +- 对 **KAI 用户**：GPU 共享终于有了与之匹配的运行时硬隔离，共享不再等于缺少保障。 +- 对 **HAMi 用户**：多了一条不绑死特定虚拟化设备插件、直接走 NVIDIA 官方调度器的集成路径，指标体系（`hami_*`）也保持兼容。 +- 对 **社区**：调度层与隔离层之间的契约（`CUDA_DEVICE_MEMORY_LIMIT`）被又一次验证可行，后续任何调度器只要遵守它，就能复用同一套隔离引擎。 + +这背后是 KAI Scheduler 团队（Run:ai）与 HAMi 维护者一年多反复对齐的结果。`LD_PRELOAD`、webhook、指标端口、退出开关，每一个都是两边共同敲定的；`kai-resource-isolator` 1.1.0 里那批安全收紧（命名空间内 Issuer、收紧 ClusterRole），也都是社区评审一条条打磨出来的。 + +## 下一步 + +- 背景故事：[《HAMi-core 被 NVIDIA KAI Scheduler 采用》](/zh/blog/hami-core-adopted-by-nvidia-kai-scheduler) +- 用户文档：[如何在 KAI Scheduler 中使用 HAMi](/zh/docs/next/userguide/kai-scheduler/how-to-use-kai-scheduler) +- 相关仓库：[Project-HAMi/KAI-resource-isolator](https://github.com/Project-HAMi/KAI-resource-isolator) · [Project-HAMi/HAMi-core](https://github.com/Project-HAMi/HAMi-core)（CNCF 孵化的 CUDA 拦截库） · [kai-scheduler/KAI-Scheduler](https://github.com/kai-scheduler/KAI-Scheduler) · [KAI Scheduler HAMi 资源隔离文档](https://github.com/kai-scheduler/KAI-Scheduler/blob/main/docs/gpu-sharing/hami/README.md) +- 欢迎在你自己的 GKE、AWS 或自建集群上跑一遍，并把真实结果反馈到 issue 或社区群。若缺 HAMi-core 在某种卡或某种 CUDA 版本上的支持，直接到 [Project-HAMi/HAMi](https://github.com/Project-HAMi/HAMi) 提 issue，这是社区最看重的反馈。 From b39e7605734dfffd89946de0583b209bc91838f8 Mon Sep 17 00:00:00 2001 From: Jimmy Song Date: Wed, 12 Aug 2026 17:26:42 +0800 Subject: [PATCH 2/6] feat(tutorials): Add Lab 12 for KAI Scheduler and HAMi memory isolation on GKE - Introduced Lab 12: Verify KAI Scheduler and HAMi Memory Isolation on GKE, detailing deployment and memory quota validation. - Updated overview.md to include Lab 12 in the tutorial summary. - Added new YAML files for queue configuration, runtime class, and GKE policies to support the lab. - Created shared Pods and memory limit CUDA example to demonstrate GPU memory isolation. - Enhanced tags.yml with new tags for GKE and GPU sharing. - Updated sidebars to include the new lab for navigation. Signed-off-by: Jimmy Song --- .../index.md | 669 +---------------- .../index.md | 694 ++---------------- .../current/labs/kai-scheduler-hami-gke.md | 418 +++++++++++ .../current/overview.md | 2 +- .../current/tags.yml | 12 + sidebars-tutorials.js | 5 + .../12-kai-scheduler-hami-gke/01-queues.yaml | 20 + .../02-runtimeclass.yaml | 5 + .../03-gke-policies.yaml | 88 +++ .../04-shared-pods.yaml | 111 +++ .../12-kai-scheduler-hami-gke/memory-limit.cu | 37 + tutorials/labs/kai-scheduler-hami-gke.md | 418 +++++++++++ tutorials/overview.md | 2 +- 13 files changed, 1191 insertions(+), 1290 deletions(-) create mode 100644 i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/kai-scheduler-hami-gke.md create mode 100644 tutorials/labs/examples/12-kai-scheduler-hami-gke/01-queues.yaml create mode 100644 tutorials/labs/examples/12-kai-scheduler-hami-gke/02-runtimeclass.yaml create mode 100644 tutorials/labs/examples/12-kai-scheduler-hami-gke/03-gke-policies.yaml create mode 100644 tutorials/labs/examples/12-kai-scheduler-hami-gke/04-shared-pods.yaml create mode 100644 tutorials/labs/examples/12-kai-scheduler-hami-gke/memory-limit.cu create mode 100644 tutorials/labs/kai-scheduler-hami-gke.md diff --git a/blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md b/blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md index 6657f188d..fe37541a6 100644 --- a/blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md +++ b/blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md @@ -18,16 +18,16 @@ GPU sharing has been discussed in the Kubernetes ecosystem for years, but the sc Solving this requires the scheduling layer (deciding "who uses which GPU, and how much") and the isolation layer (guaranteeing "once a quota is set, it cannot be exceeded") to cooperate. **HAMi-core** is exactly such an isolation engine, reusable by multiple schedulers. Before KAI Scheduler, it already supported the Kubernetes native scheduler (via HAMi's own `hami-scheduler` extender), [Kueue](/docs/userguide/kueue/how-to-use-kueue), [Volcano](/docs/installation/how-to-use-volcano-vgpu), and more (see the full landscape in [Ecosystem Integrations](/docs/next/core-concepts/ecosystem-integrations)). -**Starting with KAI Scheduler v0.16.4, NVIDIA's KAI Scheduler officially joined this list**, building in HAMi-core as the isolation engine for its GPU sharing (the current version at the time of writing is v0.17.0). That means when you schedule GPU workloads with KAI Scheduler, you no longer get only "cooperative sharing"; you get true hard isolation. This post covers two things: +**Starting with KAI Scheduler v0.16.4, NVIDIA's KAI Scheduler officially joined this list**, building in HAMi-core as the isolation engine for its GPU sharing. This post verifies the currently documented supported combination, KAI Scheduler v0.17.0 plus `kai-resource-isolator` 1.1.0-chart. That means when you schedule GPU workloads with this integration enabled, you no longer get only "cooperative sharing"; you get CUDA API-level memory enforcement. This post covers two things: - **Principles**: what each of KAI Scheduler, `kai-resource-isolator`, and HAMi-core is responsible for, how they connect, and how the `CUDA_DEVICE_MEMORY_LIMIT` contract ties the scheduling layer to the isolation layer. -- **Practice**: a fully reproducible verification example on GKE (a single NVIDIA T4 card, two pods sharing it, each seeing only its own memory quota), with every step carrying its config, its expected output, and the "why" behind it. +- **Practice**: an end-to-end GKE verification on one NVIDIA T4, including a direct `cudaMalloc` proof that the per-Pod memory quota cannot be exceeded. The complete procedure is in Lab 12. The background story and collaboration timeline are in the companion post [HAMi-core adopted by NVIDIA KAI Scheduler: GPU sharing enters the hard isolation era](/blog/hami-core-adopted-by-nvidia-kai-scheduler). :::note About the output in this post -The commands in the second half are steps you can reproduce end to end on GKE. The shown command output and metrics are expected examples; actual values depend on your cluster. +The GKE UUID, memory ceiling, error message, and CUDA allocation results marked as captured came from the verified GKE 1.35/COS/CDI run. Resource names are shortened where necessary; values in another cluster will differ. ::: @@ -36,13 +36,13 @@ The commands in the second half are steps you can reproduce end to end on GKE. T ## Background: why sharing is not isolation | Layer | Owner | What it solves | What it does not solve | -| --- | --- | --- | --- | +| :-- | :-- | :-- | :-- | | Scheduling | KAI Scheduler, Volcano, Kueue, ... | Multiple pods can land on the same GPU | The container still sees all the memory | | Runtime | HAMi-core (`libvgpu.so`) | Intercepts CUDA calls, enforces a memory quota | On its own, does not know how much each pod should get | Real GPU sharing needs both layers, and they must cooperate: the scheduling layer decides "who uses which GPU, and how much", and the isolation layer guarantees "the agreed amount is all you get". The catch is that **the isolation layer needs to know "how much", a number it cannot compute on its own, because that number comes from the scheduling layer**. -HAMi-core, refined by the HAMi community over years (a CNCF incubating project), is exactly that isolation layer, and it is **decoupled from any specific scheduler**: well before KAI Scheduler, HAMi-core already worked with the Kubernetes native scheduler (via HAMi's own `hami-scheduler` extender), [Kueue](/docs/userguide/kueue/how-to-use-kueue), [Volcano](/docs/installation/how-to-use-volcano-vgpu), Koordinator, and others (see the full landscape in [Ecosystem Integrations](/docs/next/core-concepts/ecosystem-integrations)). +HAMi-core, refined by the HAMi community over years, is exactly that isolation layer, and it is **decoupled from any specific scheduler**: well before KAI Scheduler, HAMi-core already worked with the Kubernetes native scheduler (via HAMi's own `hami-scheduler` extender), [Kueue](/docs/userguide/kueue/how-to-use-kueue), [Volcano](/docs/installation/how-to-use-volcano-vgpu), Koordinator, and others (see the full landscape in [Ecosystem Integrations](/docs/next/core-concepts/ecosystem-integrations)). **KAI Scheduler joined this list in v0.16.4**, building HAMi-core in as the isolation engine for its GPU sharing. What this post explains is the role of each of the three components on the KAI Scheduler integration path, and how they connect. @@ -54,7 +54,7 @@ The three components have similar-sounding names, so getting their roles straigh - **HAMi-core (`libvgpu.so`)**: HAMi's CUDA interception library (CNCF incubating), and **the isolation engine itself**. It intercepts CUDA calls (like `cudaMalloc`) inside the container via `LD_PRELOAD` and enforces a memory quota. It does not care who provided the quota: any scheduler that hands in the quota by convention gets isolation for free. Before KAI, it was already reused by HAMi's own device-plugin/webhook, Volcano's `volcano-vgpu-device-plugin`, and others. -- **KAI Scheduler**: NVIDIA's open source Kubernetes scheduler for AI workloads (descended from Run:ai, CNCF Sandbox). It only owns the **scheduling layer**, deciding which node a pod lands on, which GPU it uses, and how much memory it gets. Since v0.16.4, its `hamicore` plugin writes the computed memory quota into the container's environment variable at bind time. +- **KAI Scheduler**: NVIDIA's open source Kubernetes scheduler for AI workloads. It only owns the **scheduling layer**, deciding which node a pod lands on, which GPU it uses, and how much memory it gets. Since v0.16.4, its `hamicore` plugin writes the computed memory quota into the container's environment variable at bind time. - **`kai-resource-isolator`**: a companion component **provided by the HAMi project specifically for the KAI Scheduler integration path**. It ships HAMi-core's `libvgpu.so` to every GPU node and uses a MutatingWebhook to rewrite pods, injecting the library and `ld.so.preload`. In other words, it is the bridge that turns KAI's scheduling decision into isolation HAMi-core can actually enforce. @@ -155,665 +155,58 @@ This integration path is the result of more than a year of work between the HAMi ### KAI Scheduler (HAMi-core hard isolation built in since v0.16.4) -KAI Scheduler's support for HAMi-core has been a built-in capability **since v0.16.4** (this post uses the current version, v0.17.0). The key piece is the `hamicore` plugin: once enabled, when KAI binds a shared GPU pod to a node, it injects the `CUDA_DEVICE_MEMORY_LIMIT` environment variable into the container based on the `gpu-memory` (or `gpu-fraction`) annotation, exactly the quota that HAMi-core needs to enforce isolation, per the contract above. +KAI Scheduler's support for HAMi-core first appeared in **v0.16.4**. The current integration documentation requires KAI Scheduler v0.17.0 or later, and this post tests v0.17.0. The key piece is the `hamicore` plugin: once enabled, when KAI binds a shared GPU pod to a node, it injects the `CUDA_DEVICE_MEMORY_LIMIT` environment variable into the container based on the `gpu-memory` (or `gpu-fraction`) annotation, exactly the quota that HAMi-core needs to enforce isolation, per the contract above. Other GPU related changes in v0.17.0 include: fixing invalid volume names caused by `/` in shared pod names; correcting the allocation math for `MinNodeGPUMemoryMiB` and fractional `gpu-memory`; and using the largest GPU profile in the cluster for overLimit decisions. It also adds preemption-delay (a time window for Cluster Autoscaler to bring up nodes), NUMA aware scoring, and GitOps and ArgoCD installation support. ### kai-resource-isolator 1.1.0-chart -This is the isolator shipped alongside HAMi. It receives the quota injected by KAI and, before the container actually starts, puts the HAMi-core `libvgpu.so` in place. Compared with the first release, 1.1.0 fills in a set of improvements that make it production ready: +This is the isolator shipped alongside HAMi. It receives the quota injected by KAI and, before the container actually starts, puts the HAMi-core `libvgpu.so` in place. Compared with the first release, 1.1.0 adds several operational improvements: - **New `kai-vgpu-monitor`**: runs as a DaemonSet, exposes HAMi compatible metrics on `:9394` (`hami_vgpu_memory_used_bytes`, `hami_vgpu_memory_limit_bytes`, `hami_container_device_utilization_ratio`), supports ServiceMonitor, and can be scraped directly by Prometheus. - **Multi container injection fix**: when a pod has multiple containers, the webhook now handles them correctly and no longer skips any. - **Security tightening**: the webhook now uses a namespaced Issuer (instead of a ClusterIssuer), and the ClusterRole no longer reads Secrets. - **Global image repository** precedence cleaned up, and `hamicore` installation parameters corrected. -## Practice: verifying hard isolation on GKE +## GKE Verification: Does the Isolation Actually Hold? -The example runs on an existing GKE cluster: **3 `n1-standard-2` nodes, each with 1 NVIDIA T4 (`nvidia-smi` reports 15360 MiB of VRAM)**. Two containers each request about 4 GiB and share one of those T4s. Inside `nvidia-smi` each sees only its own quota, with no interference. Each step below comes with its config, its expected output, and the why behind it. +We completed an end-to-end test on a GKE 1.35/COS/CDI cluster with three `n1-standard-2` nodes, each carrying one NVIDIA T4. KAI Scheduler v0.17.0 handled shared scheduling, while `kai-resource-isolator` 1.1.0-chart injected HAMi-core. -The diagram below is an overview of the whole exercise: the order of the eight steps, and the two prerequisites that are easiest to get wrong (the GFD labels must be in place before installing KAI; the queue must exist before KAI will schedule). Note: this post **does not install the NVIDIA GPU Operator**; GKE's default device plugin + driver + container toolkit are enough. The preparation below is just two steps: "enable the default device plugin" and "add a few GFD labels". +The verification went beyond `nvidia-smi`: -```mermaid -%% title: GKE exercise flow (Step 1 to Step 8) -graph TD - S1["Step 1
Confirm cluster + GPU hardware
n1-standard-2 + T4"] - S2["Step 2
Enable GKE default device plugin
verify nvidia-smi"] - S3["Step 3
Add GFD labels
nvidia.com/gpu.memory etc."] - S4["Step 4
Install KAI Scheduler v0.17.0
Create the queue"] - S5["Step 5
Install kai-resource-isolator
1.1.0-chart"] - S6["Step 6
Single pod, verify isolation
nvidia-smi shows quota"] - S7["Step 7
Two pods share one T4
each sees its own quota"] - S8["Step 8 optional
Look at monitor metrics"] - - S1 --> S2 --> S3 --> S4 --> S5 --> S6 --> S7 --> S8 - - S3 -. "GFD labels must be set BEFORE KAI
else shared pods stuck Pending" .-> S4 - S4 -. "Queue must exist
else pods are never scheduled" .-> S6 - - style S1 fill:#dbeafe,stroke:#1a5fb4,stroke-width:2px,color:#1f2937 - style S2 fill:#fef3c7,stroke:#b45309,stroke-width:2px,color:#1f2937 - style S3 fill:#fef3c7,stroke:#b45309,stroke-width:2px,color:#1f2937 - style S4 fill:#d9f99d,stroke:#4f7d00,stroke-width:2px,color:#1f2937 - style S5 fill:#d9f99d,stroke:#4f7d00,stroke-width:2px,color:#1f2937 - style S6 fill:#dcfce7,stroke:#0b6b3c,stroke-width:2px,color:#1f2937 - style S7 fill:#dcfce7,stroke:#0b6b3c,stroke-width:2px,color:#1f2937 - style S8 fill:#eff6ff,stroke:#1a5fb4,stroke-width:1px,color:#1f2937 -``` - -### Prerequisites - -- A GCP project with GKE and Compute Engine APIs enabled. -- An existing GKE cluster with NVIDIA T4 GPUs in its node pool (this example uses 3 `n1-standard-2` nodes, each with 1× T4). The cluster for this example was created with the node label `gke-no-default-nvidia-gpu-device-plugin=true`, which **disables GKE's default NVIDIA device plugin**, while the driver is still installed by GKE automatically. This is why Step 1 below shows "hardware present, but `nvidia.com/gpu` resource empty"; Step 2 re-enables the device plugin. If you do not have a cluster yet, you can create an equivalent one with: - - ```bash - gcloud container clusters create test-cluster --zone=asia-northeast1-a \ - --machine-type=n1-standard-2 --num-nodes=3 \ - --accelerator=type=nvidia-tesla-t4,count=1,gpu-driver-version=default - gcloud container clusters get-credentials test-cluster --zone=asia-northeast1-a - ``` - - The `n1` machine family does not include a GPU by itself, so you **must** pass `--accelerator` to attach the T4; `gpu-driver-version=default` lets GKE install a matching NVIDIA driver and container toolkit. With this, the device plugin is enabled by default (you can skip the label-removal in Step 2). - -- `gcloud`, `kubectl`, and `helm` (>= 3) authenticated, with `kubectl` already pointing at the cluster. -- The KAI queue created in Step 4 below. - -### Step 1: confirm the cluster and GPU hardware are ready (initial state) - -With the cluster already in place, first take a look at its initial state. This step is read-only. - -```bash -kubectl get nodes -o custom-columns="NAME:.metadata.name,GPU:.status.capacity.nvidia\.com/gpu,ACCEL:.metadata.labels.cloud\.google\.com/gke-accelerator" -``` - -For the cluster in this example, the initial output looks like this: 3 nodes, all with `nvidia-tesla-t4` in the `ACCEL` column (T4 hardware present), but `GPU` is ``. - -``` -NAME GPU ACCEL -gke-test-cluster-default-pool-xxxxxxxx nvidia-tesla-t4 -gke-test-cluster-default-pool-yyyyyyyy nvidia-tesla-t4 -gke-test-cluster-default-pool-zzzzzzzz nvidia-tesla-t4 -``` - -**How to read this**: the `ACCEL` value `nvidia-tesla-t4` confirms the T4 hardware is attached and GKE has already installed the NVIDIA kernel driver. But the `GPU` column is ``, which means **the device plugin has not reported the `nvidia.com/gpu` resource yet**. The reason is that this cluster was created with the node label `gke-no-default-nvidia-gpu-device-plugin=true`, which disables GKE's default NVIDIA device plugin. The next step re-enables it. - -If your cluster already shows `1` in the `GPU` column (the default device plugin was not disabled at creation), you can skip Step 2 and jump straight to Step 3 to add the GFD labels. - -### Step 2: enable GKE's default device plugin and verify the GPU works - -Remove the `gke-no-default-nvidia-gpu-device-plugin` label from the nodes, and GKE's built-in device plugin DaemonSet (whose scheduling rule is "only schedule when this label is not `true`") will land on the nodes automatically. - -```bash -kubectl label node -l cloud.google.com/gke-accelerator=nvidia-tesla-t4 \ - gke-no-default-nvidia-gpu-device-plugin- -``` - -The trailing `-` on the label name means "delete this label". After about 20 seconds, the device plugin pods come up: - -```bash -kubectl get pod -n kube-system -o wide | grep nvidia-gpu-device-plugin | grep -v "0/0" -``` - -Expected output (example): 3 `nvidia-gpu-device-plugin-*-ubuntu` pods, one per node, all `3/3 Running`. - -``` -NAME READY STATUS NODE -nvidia-gpu-device-plugin-small-ubuntu-xxx 3/3 Running gke-test-cluster-default-pool-xxxxxxxx -nvidia-gpu-device-plugin-small-ubuntu-yyy 3/3 Running gke-test-cluster-default-pool-yyyyyyyy -nvidia-gpu-device-plugin-small-ubuntu-zzz 3/3 Running gke-test-cluster-default-pool-zzzzzzzz -``` - -Once the device plugin is up, the `nvidia.com/gpu` resource appears: - -```bash -kubectl get nodes -o custom-columns="NAME:.metadata.name,GPU:.status.capacity.nvidia\.com/gpu" -``` - -Expected output (example): the `GPU` column on each node goes from `` to `1`. - -``` -NAME GPU -gke-test-cluster-default-pool-xxxxxxxx 1 -gke-test-cluster-default-pool-yyyyyyyy 1 -gke-test-cluster-default-pool-zzzzzzzz 1 -``` - -**Why no GPU Operator is needed**: GKE's default GPU stack already sets up three things: the NVIDIA kernel driver (installed by GKE), the device plugin (just enabled), and the container toolkit / nvidia runtime (configured automatically by GKE when `gpu-driver-version=default`). The GPU Operator is a tool that bundles these together for automatic management, but GKE is already managing them; installing the Operator on top actually conflicts with GKE's driver (in practice: the Operator's driver container cannot unload the kernel module GKE already loaded). So for running this experiment on GKE, the default stack is enough. - -Finally, verify the toolkit works too (a container can run `nvidia-smi`) with a real GPU pod: - -```bash -cat <<'EOF' | kubectl apply -f - && \ -kubectl wait --for=condition=Ready pod/gpu-smi-test --timeout=120s && \ -kubectl logs gpu-smi-test && \ -kubectl delete pod gpu-smi-test -apiVersion: v1 -kind: Pod -metadata: - name: gpu-smi-test -spec: - restartPolicy: Never - containers: - - name: cuda - image: nvidia/cuda:12.4.1-base-ubuntu22.04 - command: ["nvidia-smi"] - resources: - limits: - nvidia.com/gpu: 1 -EOF -``` - -Expected output (real): you see the T4 and the driver version; note the memory is **15360 MiB** (the T4's actual addressable memory, which Step 3 uses for the label). - -``` -+-----------------------------------------------------------------------------------------+ -| NVIDIA-SMI 580.159.04 Driver Version: 580.159.04 CUDA Version: 13.0 | -+-----------------------------------------+------------------------+----------------------+ -| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 | -| N/A 45C P8 17W / 70W | 0MiB / 15360MiB | 0% Default | -+-----------------------------------------+------------------------+----------------------+ -``` - -Seeing this table means the driver, device plugin, and toolkit are all in place, and the container can actually use the GPU. - -### Step 3: add the GFD labels (needed by KAI scheduling) - -KAI's hamicore plugin relies on the node label `nvidia.com/gpu.memory` to know each card's memory, so it can convert a `gpu-memory` request into a fraction. GKE's default device plugin sets `nvidia.com/gpu.present=true`, but **does not set `gpu.memory` / `gpu.count` / `gpu.product`** (those usually come from GPU Feature Discovery, which GKE's default stack does not run in full). So you add them manually. - -Use the memory value **15360** (MiB) you saw from `nvidia-smi` in Step 2: - -```bash -kubectl label node -l cloud.google.com/gke-accelerator=nvidia-tesla-t4 \ - nvidia.com/gpu.memory=15360 \ - nvidia.com/gpu.product=NVIDIA-Tesla-T4 \ - nvidia.com/gpu.count=1 \ - nvidia.com/gpu.present=true --overwrite -``` - -:::warning Critical step: labels must be in place before installing KAI - -KAI's hamicore plugin caches per GPU memory **at node registration time** by reading `node.labels["nvidia.com/gpu.memory"]`. Without this label it errors out, shared pods stay `Pending` forever, and the logs show `didn't have enough resources: GPU memory`. So this step must be done **before** installing KAI in Step 4. If you only add the label after KAI is installed, restart `kai-scheduler` so it re-caches. - -::: - -Confirm the labels are in place: - -```bash -kubectl get nodes -o custom-columns="NAME:.metadata.name,GPUMEM:.metadata.labels.nvidia\.com/gpu\.memory,GPUPROD:.metadata.labels.nvidia\.com/gpu\.product,GPUPRESENT:.metadata.labels.nvidia\.com/gpu\.present" -``` - -Expected output (example): each node shows `15360` under `GPUMEM`, `NVIDIA-Tesla-T4` under `GPUPROD`, and `true` under `GPUPRESENT`. - -``` -NAME GPUMEM GPUPROD GPUPRESENT -gke-test-cluster-default-pool-xxxxxxxx 15360 NVIDIA-Tesla-T4 true -gke-test-cluster-default-pool-yyyyyyyy 15360 NVIDIA-Tesla-T4 true -gke-test-cluster-default-pool-zzzzzzzz 15360 NVIDIA-Tesla-T4 true -``` - -**Why `15360` and not `16384`**: the T4 is nominally 16 GiB, but `nvidia-smi` reports the actual addressable memory as 15360 MiB (some memory is reserved by the driver / ECC). KAI uses this number to convert `gpu-memory` requests into fractions, so the label value must match the `nvidia-smi` number, otherwise the ceiling the pod sees will not match your expectation. - -### Step 4: install KAI Scheduler v0.17.0 and create the queue - -```bash -helm install kai-scheduler oci://ghcr.io/kai-scheduler/kai-scheduler/kai-scheduler \ - --set global.gpuSharing=true \ - --set binder.plugins.hamicore.enabled=true \ - --set binder.plugins.gpusharing.arguments.cdiEnabled=true \ - --namespace kai-scheduler --create-namespace \ - --version v0.17.0 - -# Wait for the admission webhook to be ready (signing the certificate takes a -# moment) before creating the queue, otherwise Queue creation is rejected. -kubectl -n kai-scheduler wait --for=condition=available --timeout=180s deploy --all -``` - -**Why these switches**: - -- `global.gpuSharing=true` turns on shared scheduling, allowing KAI to place multiple pods on the same GPU. -- `binder.plugins.hamicore.enabled=true` makes KAI inject `CUDA_DEVICE_MEMORY_LIMIT` at bind time. This is the contract that connects the scheduling layer to the isolation layer. Without it, the isolation layer never gets the quota and effectively is not installed. -- `binder.plugins.gpusharing.arguments.cdiEnabled=true`: **required on GKE 1.33+**. KAI's gpusharing plugin defaults to `cdiEnabled=false` (it assumes the GPU is injected via the NVIDIA runtime), but GKE 1.33+ containerd uses the CDI mode (`enable_cdi=true`) to inject GPUs. Without this flag, the GPU reservation pod KAI creates cannot find NVML (it fails with `ERROR_LIBRARY_NOT_FOUND`), and shared pods stay `Pending`. In the KAI source, CDI is auto-detected from the GPU Operator's ClusterPolicy, but we did not install the GPU Operator (see Step 2), so it must be enabled explicitly. (See the [KAI binder docs](https://github.com/kai-scheduler/KAI-Scheduler/blob/main/docs/developer/binder.md).) - -Then create the queue. **The queue must exist before KAI will schedule anything.** A pod points at its queue via the `kai.scheduler/queue: default-queue` label; if the queue (CRD `scheduling.run.ai/v2`) does not exist, the pod stays `Pending`. The manifest below creates two layers, `default` (parent) and `default-queue` (child), with all quotas open (`-1`) to keep the demo simple: - -```yaml -# queue.yaml -apiVersion: scheduling.run.ai/v2 -kind: Queue -metadata: - name: default -spec: - resources: - cpu: { quota: -1, limit: -1, overQuotaWeight: 1 } - memory: { quota: -1, limit: -1, overQuotaWeight: 1 } - gpu: { quota: -1, limit: -1, overQuotaWeight: 1 } ---- -apiVersion: scheduling.run.ai/v2 -kind: Queue -metadata: - name: default-queue -spec: - parentQueue: default - resources: - cpu: { quota: -1, limit: -1, overQuotaWeight: 1 } - memory: { quota: -1, limit: -1, overQuotaWeight: 1 } - gpu: { quota: -1, limit: -1, overQuotaWeight: 1 } -``` - -```bash -kubectl apply -f queue.yaml -kubectl get queues -``` - -Expected output (example): - -``` -NAME PARENT AGE -default 5s -default-queue default 5s -``` - -Both queues present and the child correctly pointing at the parent: then you can move on. - -### Step 5: install kai-resource-isolator 1.1.0-chart - -```bash -helm install kai-resource-isolator oci://docker.io/projecthami/kai-resource-isolator \ - --namespace kai-resource-isolator --create-namespace \ - --set monitor.enabled=true \ - --set monitor.serviceMonitor.enabled=true \ - --set paths.hostInstallBase=/home/kubernetes/bin/nvidia \ - --version 1.1.0-chart -``` - -**Why the version has a `-chart` suffix**: the chart version is **`1.1.0-chart`** (with the `-chart` suffix). The matching git tag is `v1.1.0`; do not confuse the two, since writing `1.1.0` or `v1.1.0` will not find the package. If your node NVML requires the NVIDIA runtime, also add `--set monitor.runtimeClassName=nvidia` (GKE GPU nodes already have this). - -**Why change `paths.hostInstallBase`**: the isolator's libsync DaemonSet writes `libvgpu.so` to `{hostInstallBase}/vgpu/` on the host, which defaults to `/usr/local`. But **GKE nodes have a read-only root filesystem** (ext2 mounted `ro`), so `/usr/local` is not writable and libsync fails with `Read-only file system`. `/home/kubernetes/bin/nvidia` is a **writable** directory on GKE (the standard path where GKE installs the NVIDIA driver), so the library goes there. On self-built clusters where the root filesystem is writable, you can omit this flag (use the default `/usr/local`). - -Confirm all three components are up: - -```bash -kubectl -n kai-resource-isolator get pods -``` - -Expected output (example): - -``` -NAME READY STATUS -kai-resource-isolator-xxxxxxxx-xxxxx 1/1 Running # webhook -kai-libsync-xxxxx 1/1 Running # distributes libvgpu.so -kai-vgpu-monitor-xxxxx 1/1 Running # :9394 metrics -``` - -All three pods `Running` means the webhook, libsync, and monitor are in place. On GKE, however, you still need a few extra adaptations before the isolation chain works end to end (see Step 5b below). On a self built cluster with a writable root filesystem, a `nvidia` RuntimeClass present, and NVML libraries in the standard paths, you can skip that section. - -### Step 5b: GKE environment adaptation (required on GKE 1.33+) - -The following four adaptations are confirmed GKE-specific requirements, each solving a concrete problem: - -**5b-1. Create the `nvidia` RuntimeClass (absent on GKE 1.35)** +| Check | Captured result | What it proves | +| :-- | :-- | :-- | +| Node and GPU UUID | Both Pods ran on one single-GPU node and returned `GPU-9acc8878-...` | They shared the same physical T4 | +| Visible memory | Both Pods reported `4147 MiB`; the full card reported `15360 MiB` | HAMi-core exposed KAI's per-Pod quota | +| CUDA allocation | 3 GiB succeeded; a cumulative 5 GiB returned `out of memory` | The ceiling was enforced, not merely displayed | +| Concurrent isolation | Pod B still allocated its own 3 GiB while Pod A held 3 GiB | One Pod could not consume the other's quota | +| Monitor metrics | The same-node `:9394/metrics` endpoint reported both Pods' 4.348 GB limits and 3.328 GB live usage | The monitor read each container's shared-memory cache and exported non-empty per-Pod gauges | -GKE 1.33+ injects GPUs in CDI mode, so no `nvidia` runtime handler is registered on the node and there is no `nvidia` RuntimeClass. But KAI's reservation pod checks for it at creation time. Create a RuntimeClass pointing at the default `runc` handler (CDI injects the device at the containerd layer, so the nvidia runtime is not needed): +HAMi-core logged the over-quota allocation as: -```bash -cat <<'EOF' | kubectl apply -f - -apiVersion: node.k8s.io/v1 -kind: RuntimeClass -metadata: - name: nvidia -handler: runc -EOF +```text +Device 0 OOM 5475663872 / 4348444672 +allocate another 2 GiB: out of memory +PASS: in-quota allocation succeeded and over-quota allocation failed ``` -**5b-2. Patch the isolator DaemonSets to remove `system-node-critical` priority** - -GKE forbids pods in user namespaces from using the `system-node-critical` PriorityClass (`insufficient quota to match these scopes`). The isolator's libsync and monitor DaemonSets use it by default, so it has to be removed: - -```bash -for ds in kai-resource-isolator-libsync kai-resource-isolator-monitor; do - kubectl patch ds "$ds" -n kai-resource-isolator \ - --type=json -p='[{"op":"replace","path":"/spec/template/spec/priorityClassName","value":null}]' -done -``` - -**5b-3. Install Kyverno and inject `LD_LIBRARY_PATH`** - -KAI's reservation pod uses NVML to query GPU information, but its container image does not ship `libnvidia-ml.so`. GKE injects the NVML library into the container at `/usr/local/nvidia/lib64/`, but it does not set `LD_LIBRARY_PATH` (the `nvidia/cuda` image carries this variable, while the reservation image does not). Use [Kyverno](https://kyverno.io/)'s MutatingWebhook to automatically inject this environment variable into pods in the reservation namespace: - -```bash -# Install Kyverno (ships its own TLS certificate management, no cert-manager needed) -helm install kyverno kyverno/kyverno -n kyverno --create-namespace -kubectl wait --for=condition=ready pod -l app.kubernetes.io/component=admission-controller \ - -n kyverno --timeout=300s - -# Create the policy: inject LD_LIBRARY_PATH into all pods in the kai-resource-reservation namespace -cat <<'EOF' | kubectl apply -f - -apiVersion: kyverno.io/v1 -kind: ClusterPolicy -metadata: - name: inject-nvidia-library-path -spec: - rules: - - name: add-ld-library-path - match: - any: - - resources: - kinds: [Pod] - namespaces: [kai-resource-reservation] - mutate: - foreach: - - list: "request.object.spec.containers[]" - patchStrategicMerge: - metadata: {} - spec: - containers: - - name: "{{ element.name }}" - env: - - name: LD_LIBRARY_PATH - value: /usr/local/nvidia/lib64 -EOF -``` - -The `inject-nvidia-library-path` policy above only solves the NVML query problem for the reservation pod. **Shared pods need a second policy, `inject-gpu-devices`**, which mounts the GPU device nodes and NVIDIA tool directories into the shared pods. The reason: under GKE CDI mode, the device plugin only runs `Allocate` and injects devices for pods that request the `nvidia.com/gpu` extended resource; KAI's shared pods use the `gpu-memory` annotation (no extended resource request), so the device plugin does not inject `/dev/nvidia*`, the `nvidia-smi` binary, or the NVML library into them. Without this policy, `nvidia-smi` inside a shared pod fails because it cannot find the device. The policy below matches any pod carrying the `gpu-memory` annotation (any namespace) and auto-mounts `/dev/nvidia0`, `/dev/nvidiactl`, `/dev/nvidia-uvm`, `/dev/nvidia-modeset`, `/dev/nvidia-caps`, plus the host's `nvidia-smi` binary directory and the `libnvidia-ml.so` library directory: - -```bash -# Second policy: inject GPU device nodes + NVIDIA library directories into all -# shared pods carrying the gpu-memory annotation -cat <<'EOF' | kubectl apply -f - -apiVersion: kyverno.io/v1 -kind: ClusterPolicy -metadata: - name: inject-gpu-devices -spec: - background: false - rules: - - name: add-gpu-volumes - match: - any: - - resources: - kinds: [Pod] - preconditions: - all: - - key: "{{ request.object.metadata.annotations.\"gpu-memory\" || '' }}" - operator: NotEquals - value: "" - mutate: - patchesJson6902: |- - - op: add - path: /spec/volumes/- - value: - name: nvidia-dev0 - hostPath: {path: /dev/nvidia0, type: CharDevice} - - op: add - path: /spec/volumes/- - value: - name: nvidia-ctl - hostPath: {path: /dev/nvidiactl, type: CharDevice} - - op: add - path: /spec/volumes/- - value: - name: nvidia-uvm - hostPath: {path: /dev/nvidia-uvm, type: CharDevice} - - op: add - path: /spec/volumes/- - value: - name: nvidia-modeset - hostPath: {path: /dev/nvidia-modeset, type: CharDevice} - - op: add - path: /spec/volumes/- - value: - name: nvidia-caps - hostPath: {path: /dev/nvidia-caps, type: Directory} - - op: add - path: /spec/volumes/- - value: - name: nvidia-bin - hostPath: {path: /home/kubernetes/bin/nvidia/bin, type: Directory} - - op: add - path: /spec/volumes/- - value: - name: nvidia-lib - hostPath: {path: /home/kubernetes/bin/nvidia/lib64, type: Directory} - - name: add-gpu-volumemounts - match: - any: - - resources: - kinds: [Pod] - preconditions: - all: - - key: "{{ request.object.metadata.annotations.\"gpu-memory\" || '' }}" - operator: NotEquals - value: "" - mutate: - foreach: - - list: "request.object.spec.containers[]" - patchStrategicMerge: - spec: - containers: - - name: "{{ element.name }}" - volumeMounts: - - {name: nvidia-dev0, mountPath: /dev/nvidia0} - - {name: nvidia-ctl, mountPath: /dev/nvidiactl} - - {name: nvidia-uvm, mountPath: /dev/nvidia-uvm} - - {name: nvidia-modeset, mountPath: /dev/nvidia-modeset} - - {name: nvidia-caps, mountPath: /dev/nvidia-caps} - - {name: nvidia-bin, mountPath: /usr/local/nvidia/bin} - - {name: nvidia-lib, mountPath: /usr/local/nvidia/lib64} -EOF -``` - -With both policies in place, shared pods get the device nodes (`/dev/nvidia*`), the `nvidia-smi` binary (`/usr/local/nvidia/bin`), and the NVML library (`/usr/local/nvidia/lib64`). Note: device nodes alone are not enough; the pod must also set `securityContext.privileged: true` (see Step 6 and Step 7), because HAMi-core's `libvgpu.so` requires privileged mode to initialize NVML (confirmed by testing). All three pieces (device nodes + library directories + privileged mode) are required; once they are all in place, the shared pod can successfully run `nvidia-smi` and see the isolated memory. - -**5b-4. Fix the `ld.so.preload` path written by libsync** - -The isolator's libsync writes `libvgpu.so` and `ld.so.preload` to `/home/kubernetes/bin/nvidia/vgpu/` (the `hostInstallBase` from Step 5), but the chart's `ld.so.preload` ConfigMap may still carry the old `/usr/local/vgpu/libvgpu.so` path. Verify and correct it to point at the right path: - -```bash -# Inspect the ConfigMap content -kubectl get cm kai-resource-isolator-ldpreload -n kai-resource-isolator \ - -o jsonpath='{.data.ld\.so\.preload}' -# If it is not /home/kubernetes/bin/nvidia/vgpu/libvgpu.so, fix it -kubectl create cm kai-resource-isolator-ldpreload -n kai-resource-isolator \ - --from-literal=ld.so.preload="/home/kubernetes/bin/nvidia/vgpu/libvgpu.so" \ - --dry-run=client -o yaml | kubectl apply -f - - -# Restart the libsync pods so they rewrite with the corrected ConfigMap -kubectl delete pod -n kai-resource-isolator \ - $(kubectl get pod -n kai-resource-isolator -l app.kubernetes.io/component=kai-libsync -o jsonpath='{.items[*].metadata.name}') -``` - -Additionally, the monitor DaemonSet has a hostPath volume pointing to the old `/usr/local/vgpu/containers` (read-only rootfs), which gets stuck in `ContainerCreating`. Patch it to the writable area too: - -```bash -kubectl patch ds kai-resource-isolator-monitor -n kai-resource-isolator --type=json \ - -p='[{"op":"replace","path":"/spec/template/spec/volumes/0","value":{"hostPath":{"path":"/home/kubernetes/bin/nvidia/vgpu/containers","type":"DirectoryOrCreate"},"name":"containers"}}]' -``` - -After these four steps, the reservation pod can successfully initialize NVML and hold the GPU, and shared pods can be scheduled and load `libvgpu.so`. Verify: - -```bash -# The reservation pod should be Running (no longer CrashLoopBackOff) -kubectl get pod -n kai-resource-reservation -# Inside a shared pod, libvgpu.so should preload successfully (no "cannot be preloaded" error) -kubectl exec -- cat /etc/ld.so.preload -# The memory limit KAI injected -kubectl exec -- env | grep CUDA_DEVICE_MEMORY_LIMIT -``` +Together, these checks connect scheduling onto one card, per-container visibility, actual CUDA allocation enforcement, and observability into one evidence chain. Because the monitor runs as a DaemonSet and reads node-local caches, Lab 12 queries the instance on the workload node directly instead of relying on a Service that may select another node. -:::tip Verified +:::note Isolation boundary -Verified by testing: with Kyverno auto-mounting GPU device nodes + NVIDIA library directories + privileged mode into shared pods, shared pods can successfully run `nvidia-smi` and see only their memory quota (4147 MiB instead of the full 15360 MiB), confirming isolation works. The exact setup is the `inject-gpu-devices` policy in Step 5b-3 above, combined with `securityContext.privileged: true` in the Step 6 and Step 7 pod templates. +This proves CUDA API-level memory enforcement, not a MIG-like hardware security boundary. The tested GKE compatibility path also used privileged workload containers, so it should not be treated as an untrusted multi-tenant security design. ::: -### Step 6: a single pod, memory locked down - -First verify isolation works on a single pod, before moving to the sharing scenario; this makes problems easier to localize. - -```yaml -# single-pod.yaml -apiVersion: v1 -kind: Pod -metadata: - name: gpu-isolation-demo - labels: - kai.scheduler/queue: default-queue - annotations: - gpu-memory: "4096" # integer MiB, no unit suffix -spec: - schedulerName: kai-scheduler - restartPolicy: Never - containers: - - name: gpu-workload - image: nvidia/cuda:12.9.2-base-ubuntu24.04 - command: ["sleep", "infinity"] - securityContext: - privileged: true -``` - -**Why it is written this way** (matching the [KAI docs](https://github.com/kai-scheduler/KAI-Scheduler/blob/main/docs/gpu-sharing/hami/README.md) pod template): - -- `schedulerName: kai-scheduler` routes the pod through KAI (the default is `default-scheduler`). -- `kai.scheduler/queue: default-queue` attaches the pod to the queue from Step 4; a label that does not match a queue means the pod is never scheduled. -- `gpu-memory: "4096"` is the pod's memory quota, an integer MiB with no unit suffix. KAI uses it to inject `CUDA_DEVICE_MEMORY_LIMIT=4096` (MiB), which HAMi-core then enforces. -- **Do not add `resources.limits.nvidia.com/gpu`**. KAI's admission rejects pods that request both a whole card and a memory quota (`cannot request both GPU and GPU memory`); memory sharing is driven entirely by the `gpu-memory` annotation, and the whole-card device-plugin resource is allocated by KAI behind the scenes. - -```bash -kubectl apply -f single-pod.yaml -kubectl wait --for=condition=Ready pod/gpu-isolation-demo --timeout=300s -kubectl exec gpu-isolation-demo -- nvidia-smi -``` - -Expected output (real output, verified): - -``` -+-----------------------------------------------------------------------------+ -| NVIDIA-SMI ... Driver Version: ... CUDA Version: 12.9 | -|-------------------------------+----------------------+----------------------+ -| 0 NVIDIA T4 ... | 4147MiB / 4147MiB ... 0% Default | -+-------------------------------+----------------------+----------------------+ -``` - -The full T4 has 15360 MiB, but inside the container only about 4147 MiB is visible (the actual ceiling after the quota 4096 is converted to a fraction), which means HAMi-core rewrote the device query response, so isolation is working. If you still see 15360 MiB here, the isolator webhook is most likely not taking effect (check that the Step 5 pods are Running and that the pod is covered by the webhook's namespace selector). - -### Step 7: two pods sharing one T4 - -With isolation verified on a single pod, verify genuine sharing: two pods landing on the same card at the same time, each seeing only its own quota. - -```yaml -# shared-t4.yaml ---- -apiVersion: v1 -kind: Pod -metadata: - name: gpu-share-a - labels: - kai.scheduler/queue: default-queue - annotations: - gpu-memory: "4096" -spec: - schedulerName: kai-scheduler - restartPolicy: Never - containers: - - name: gpu-workload - image: nvidia/cuda:12.9.2-base-ubuntu24.04 - command: ["sleep", "infinity"] - securityContext: - privileged: true ---- -apiVersion: v1 -kind: Pod -metadata: - name: gpu-share-b - labels: - kai.scheduler/queue: default-queue - annotations: - gpu-memory: "4096" -spec: - schedulerName: kai-scheduler - restartPolicy: Never - containers: - - name: gpu-workload - image: nvidia/cuda:12.9.2-base-ubuntu24.04 - command: ["sleep", "infinity"] - securityContext: - privileged: true -``` - -Each pod requests `4096` MiB; together that is 8 GiB, well under a single 15360 MiB T4, so KAI schedules both onto the same card (rather than one card each). +### Why the full procedure is not in this post -```bash -kubectl apply -f shared-t4.yaml -kubectl wait --for=condition=Ready pod/gpu-share-a pod/gpu-share-b --timeout=300s -kubectl exec gpu-share-a -- nvidia-smi --query-gpu=memory.total --format=csv,noheader -kubectl exec gpu-share-b -- nvidia-smi --query-gpu=memory.total --format=csv,noheader -``` - -Expected output (example): both pods return about 4147 MiB (the actual ceiling after the quota 4096 is converted to a fraction). - -``` -4147 MiB -4147 MiB -``` - -Then confirm they really share the same card rather than each taking their own: - -```bash -kubectl get pod -o wide | grep gpu-share -``` - -Expected output (example): both pods show the same `NODE` (landed on the same node), and that node only has 1 T4, proving they share the same card. +The standard KAI + HAMi-core path is short. GKE 1.35/COS/CDI adds version-specific concerns around the read-only root filesystem, RuntimeClass, NVML library paths, CDI device injection, PriorityClass, and `kubectl exec` WebSocket resets. Those steps need independent maintenance and fit a reproducible lab better than the narrative of this post. -``` -NAME READY STATUS RESTARTS AGE IP NODE -gpu-share-a 1/1 Running 0 2m 10.x.x.x gke-test-cluster-default-pool-xxxxxxxx -gpu-share-b 1/1 Running 0 2m 10.x.x.x gke-test-cluster-default-pool-xxxxxxxx -``` - -**Why `4096` and not more**: the T4 has 15360 MiB of actual memory, and two pods each requesting 4096 MiB (8 GiB total) leaves nearly half the card free, which makes scheduling more robust. `gpu-memory` is an integer MiB. The scheduler converts it to a two-decimal GPU fraction and then lands it as `CUDA_DEVICE_MEMORY_LIMIT`. `4096 / 15360 ≈ 0.27`, so the actual ceiling you see is about 4147 MiB rather than exactly 4096 (see pitfall #4 for the rounding details). A 15 GiB card is cleanly split into two slices that do not bleed into each other, and that is hard isolation in action. +The complete cluster setup, manifests, Kyverno policies, CUDA program, monitor verification, captured outputs, troubleshooting table, and cleanup commands are in: -### Step 8 (optional): look at monitor metrics - -```bash -kubectl -n kai-resource-isolator port-forward daemonset/kai-vgpu-monitor 9394:9394 & -curl -s localhost:9394/metrics | grep hami_vgpu_memory_limit_bytes -``` - -Expected (example): - -``` -hami_vgpu_memory_limit_bytes{...} 4.147e+06 -``` +**[Lab 12: Verify KAI Scheduler and HAMi Memory Isolation on GKE](/tutorials/labs/kai-scheduler-hami-gke)** -**Why this metric**: `hami_vgpu_memory_limit_bytes` directly reflects the memory ceiling HAMi-core set for each container, and it matches the `nvidia-smi` number, making it the most direct observation point for verifying isolation. With Prometheus and Grafana, per container memory usage, limits, and utilization become ready made charts. - -## Pitfalls worth knowing up front - -1. **The `nvidia.com/gpu.memory` node label must appear first** (most important). KAI hamicore caches per card memory at node registration; without the label, shared pods get stuck `Pending` with `didn't have enough resources: GPU memory`. GKE's default device plugin **does not set this label** (it only sets `nvidia.com/gpu.present`), so this post adds it manually in Step 3. **The label must be added before installing KAI**; if you add it only after KAI is installed, restart `kai-scheduler` so it re-caches. Use the memory `nvidia-smi` reports (15360 for the T4 here, not the nominal 16384). -2. **Create the queue manually.** A pod points at its queue via the `kai.scheduler/queue` label, the CRD is `scheduling.run.ai/v2` `Queue`; without the queue, pods are never scheduled. Step 4 provides the manifest, and it must be applied after the KAI admission webhook is ready. -3. **RuntimeClass `nvidia`.** KAI shared reservation pods carry `runtimeClassName: nvidia` by default. GKE GPU nodes ship it (COS plus containerd registers the `nvidia` handler, performing real injection), so you usually do nothing. On a self built cluster that lacks it, create one pointing at the runtime that actually performs NVIDIA injection on your nodes. -4. **Memory precision is rounded.** `gpu-memory` is an integer MiB (no unit suffix). The scheduler converts it to a two decimal GPU fraction and then lands it as `CUDA_DEVICE_MEMORY_LIMIT`. In Step 6 and Step 7 of this post, requesting `4096` on a 15360 MiB T4 gives `4096 / 15360 ≈ 0.27`, so the actual ceiling you see is about 4147 MiB rather than exactly 4096 (this is exactly the T4 example in the official docs, see [Memory value precision](https://github.com/kai-scheduler/KAI-Scheduler/blob/main/docs/gpu-sharing/hami/README.md#memory-value-precision)). When you need finer control, reverse the math from the fraction. -5. **The chart version has a `-chart` suffix.** The OCI chart tag for `kai-resource-isolator` is `1.1.0-chart`, not `1.1.0` and not `v1.1.0`. -6. **Opting out of isolation is supported.** A single pod can set the annotation `kai-resource-isolator.io/inject: "false"`; an entire namespace can set the label `kai-resource-isolator.io/webhook=ignore`. Useful for gradual rollout. -7. **Monitor runtime and nodeSelector.** If your node NVML requires the NVIDIA runtime (some self built clusters), set `monitor.runtimeClassName=nvidia`; GKE GPU nodes satisfy this by default. The monitor DaemonSet also selects nodes via the `nvidia.com/gpu.present=true` nodeSelector; **this post adds `gpu.present=true` manually in Step 3, so the monitor pod schedules normally**. If you forget this label, the monitor pod will not come up; add it or change the nodeSelector. -8. **The driver is a prerequisite.** The isolation stack depends on the host NVIDIA driver. This post uses the GKE default stack (create the cluster with `gpu-driver-version=default`, and GKE installs the driver + device plugin + container toolkit automatically); on a self built cluster, get `nvidia-smi` working on the node first with the NVIDIA driver and container runtime. -9. **The isolator DaemonSets may fail to start on GKE (PriorityClass quota).** The kai-resource-isolator libsync and monitor DaemonSets default to the `system-node-critical` priority class, which GKE forbids for pods in user namespaces (`insufficient quota to match these scopes: [{PriorityClass In [system-node-critical ...]}]`). If you hit this, remove the `priorityClassName` from both DaemonSets: - - ```bash - for ds in kai-resource-isolator-libsync kai-resource-isolator-monitor; do - kubectl patch ds $ds -n kai-resource-isolator \ - --type=json -p='[{"op":"replace","path":"/spec/template/spec/priorityClassName","value":null}]' - done - ``` - - This triggers a rolling update and the pods reschedule. Note: `helm upgrade` reverts this patch, so re-apply it after each isolator upgrade (or, for a more durable fix, add a ResourceQuota that allows critical pods in the isolator namespace). - -10. **Do not put `resources.limits.nvidia.com/gpu` on shared pods.** KAI's admission rejects pods that request both a whole card and a memory quota (`cannot request both GPU and GPU memory`). Memory sharing is driven entirely by the `gpu-memory` annotation; follow the [KAI official pod template](https://github.com/kai-scheduler/KAI-Scheduler/blob/main/docs/gpu-sharing/hami/README.md) and omit the `resources` section. -11. **KAI's config-deployer hook may fail on GKE.** The `helm install kai-scheduler` post-install hook (the `kai-config-deployer` Job) can stall on GKE 1.35 due to an API server authentication issue (OpenAPI schema download fails with `the server has asked for the client to provide credentials`), so helm reports `INSTALLATION FAILED`. However, the KAI CRDs and the `kai-operator` are already installed. The fix is to apply the Config CR (the very resource the hook tries to deploy) manually: render the chart with `helm template ... > rendered.yaml`, extract `kai-config.yaml` from the `kai-config-manifest` ConfigMap (note that `cdiEnabled` must be a **quoted string** `"true"`, the CRD rejects a boolean), then `kubectl apply --validate=false -f kai-config.yaml`. Once kai-operator sees the Config CR, it brings up binder/admission/scheduler and the rest automatically. If the release status is stuck at `failed`, use `helm upgrade ... --no-hooks` to correct the state (it skips the hook and idempotently reconciles the resources). -12. **GKE nodes have a read-only root filesystem, so libsync cannot write to `/usr/local/vgpu`.** GKE nodes mount `/` as `ext2 ro`; the isolator's libsync writes `libvgpu.so` to `/usr/local/vgpu` by default and crashes with `Read-only file system`. The `--set paths.hostInstallBase=/home/kubernetes/bin/nvidia` flag in Step 5 fixes this by placing the library in GKE's writable area. Note: `paths.hostInstallBase` only changes libsync's write path and the webhook's hostPath; the chart's `ld.so.preload` ConfigMap may not be updated to match, so it needs manual verification/fixing (see Step 5b-4). -13. **GKE 1.35 has no `nvidia` RuntimeClass.** GKE 1.33+ uses CDI mode, so no nvidia runtime handler is registered on the node. KAI needs this RuntimeClass to exist when creating shared pods, otherwise the pod is rejected (`RuntimeClass "nvidia" not found`). Create one pointing at the default `runc` handler (see Step 5b-1). On a self built cluster that already registers a real nvidia runtime handler, this is not needed. -14. **The reservation pod cannot find the NVML library on GKE CDI.** GKE injects the NVML library into the container at `/usr/local/nvidia/lib64/`, but does not set `LD_LIBRARY_PATH`. The `nvidia/cuda` image carries this variable so it runs, but KAI's reservation image does not. The fix is to use a Kyverno MutatingWebhook to inject it automatically (see Step 5b-3). Related KAI issue: [#1449](https://github.com/kai-scheduler/kai-scheduler/issues/1449). -15. **Shared pods on GKE CDI need extra GPU device node mounts (solved).** The GKE device plugin only injects devices into pods that request `nvidia.com/gpu`; shared pods use the `gpu-memory` annotation instead. The fix: use the Kyverno `inject-gpu-devices` policy to auto-mount them (see Step 5b-3) and set `privileged: true` on the pod (HAMi-core's `libvgpu.so` needs privileged mode to initialize NVML, confirmed by testing). With both in place, shared pods can run `nvidia-smi` and see only their memory quota (4147 MiB instead of the full 15360 MiB). +If you only need the architecture, stop here. If you want to reproduce it on GKE, continue with Lab 12. ## What this means diff --git a/i18n/zh/docusaurus-plugin-content-blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md b/i18n/zh/docusaurus-plugin-content-blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md index 8b7565861..b7ed8a197 100644 --- a/i18n/zh/docusaurus-plugin-content-blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md +++ b/i18n/zh/docusaurus-plugin-content-blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md @@ -6,33 +6,33 @@ authors: [rootsongjc] tags: ["HAMi", "KAI Scheduler", "硬隔离", "GPU 共享", "Kubernetes", "云原生"] --- -GPU 共享在 Kubernetes 生态里讨论了很多年，但调度层与隔离层长期各自为政：调度器把几个 Pod 分到同一张卡，容器进入 GPU 后却依然能看到整张卡的显存，谁先发起 `cudaMalloc` 谁就占满，隔离形同虚设。所谓「共享」其实只是「抢」，没有任何资源保障可言。 +GPU 共享在 Kubernetes 生态里讨论了很多年，但调度层与隔离层长期各自为政：调度器把几个 Pod 分到同一张卡，容器进入 GPU 后却依然能看到整张卡的显存，谁先发起 `cudaMalloc` 谁就占满，隔离形同虚设。所谓“共享”其实只是“抢”，没有任何资源保障可言。 -要解决这个问题，需要调度层（决定「谁能用哪张卡、用多少」）和隔离层（保证「说好用多少就只能用多少」）协同。**HAMi-core** 正是这样一个可被多种调度器复用的隔离底座。在 KAI Scheduler 之前，它已经支持 Kubernetes 原生调度器（经 HAMi 自带的 `hami-scheduler` 扩展）、[Kueue](/zh/docs/userguide/kueue/how-to-use-kueue)、[Volcano](/zh/docs/installation/how-to-use-volcano-vgpu) 等（完整生态见 [HAMi 生态集成](/zh/docs/next/core-concepts/ecosystem-integrations)）。 +要解决这个问题，需要调度层（决定“谁能用哪张卡、用多少”）和隔离层（保证“说好用多少就只能用多少”）协同。**HAMi-core** 正是这样一个可被多种调度器复用的隔离底座。在 KAI Scheduler 之前，它已经支持 Kubernetes 原生调度器（经 HAMi 自带的 `hami-scheduler` 扩展）、[Kueue](/zh/docs/userguide/kueue/how-to-use-kueue)、[Volcano](/zh/docs/installation/how-to-use-volcano-vgpu) 等（完整生态见 [HAMi 生态集成](/zh/docs/next/core-concepts/ecosystem-integrations)）。 -**自 KAI Scheduler v0.16.4 起，NVIDIA 的 KAI Scheduler 也正式加入这一行列**，把 HAMi-core 作为其 GPU 共享的隔离引擎内置支持（当前版本为 v0.17.0）。这意味着用 KAI Scheduler 调度 GPU 工作负载时，不再只有「协作式共享」，而是有了真正的硬隔离保障。本文讲清两件事： +**自 KAI Scheduler v0.16.4 起，NVIDIA 的 KAI Scheduler 也正式加入这一行列**，把 HAMi-core 作为其 GPU 共享的隔离引擎内置支持。本文验证当前文档支持的组合：KAI Scheduler v0.17.0 与 `kai-resource-isolator` 1.1.0-chart。启用这条集成后，GPU 共享不再只是“协作式共享”，而是在 CUDA API 层执行显存上限。本文讲清两件事： - **原理**：KAI Scheduler、`kai-resource-isolator`、HAMi-core 三者各自负责什么、如何衔接，以及 `CUDA_DEVICE_MEMORY_LIMIT` 这一契约如何把调度层与隔离层连起来。 -- **实践**：一套可在 GKE 上完整复现的验证示例（单张 NVIDIA T4 卡，两个 Pod 共享，各自只见自己的显存配额），每一步都附配置文件、预期输出，以及「为什么这样做」的解释。 +- **实践**：一套在 GKE 上完成的端到端验证（单张 NVIDIA T4 卡，两个 Pod 共享），并通过 `cudaMalloc` 证明显存配额无法被越过。完整复现步骤见实验 12。 背景故事与协作时间线见 [《HAMi-core 被 NVIDIA KAI Scheduler 采用：GPU 共享正式迈入硬隔离时代》](/zh/blog/hami-core-adopted-by-nvidia-kai-scheduler)。 :::note 关于本文的输出 -下半部分的命令为可在 GKE 上完整复现的步骤，所附命令输出与指标为预期示意，实际数值以你的集群环境为准。 +下半部分的命令已在本文的 GKE 1.35/COS/CDI 集群上实际执行。标为“实测”的 UUID、显存上限、错误信息和 CUDA 分配结果来自该次运行；标为“示意”的资源名称与地址经过脱敏，实际值以你的集群环境为准。 ::: -## 背景：为什么「共享」不等于「隔离」 +## 背景：为什么“共享”不等于“隔离” | 层次 | 负责方 | 解决了什么 | 没有解决什么 | -| --- | --- | --- | --- | +| :-- | :-- | :-- | :-- | | 调度层 | KAI Scheduler、Volcano、Kueue 等 | 多个 Pod 能被分到同一张 GPU | 容器内仍可见全部显存 | | 运行时层 | HAMi-core（`libvgpu.so`） | 拦截 CUDA 调用，按配额限制显存 | 单独使用时，不知道每个 Pod 该分多少 | -要实现真正的 GPU 共享，这两层缺一不可，而且必须协同：调度层决定「谁能用哪张卡、用多少」，隔离层保证「说好用多少就只能用多少」。问题是，**隔离层需要知道「到底用多少」这个数字，而它本身是算不出来的，这个数字来自调度层**。 +要实现真正的 GPU 共享，这两层缺一不可，而且必须协同：调度层决定“谁能用哪张卡、用多少”，隔离层保证“说好用多少就只能用多少”。问题是，**隔离层需要知道“到底用多少”这个数字，而它本身是算不出来的，这个数字来自调度层**。 HAMi 社区多年打磨的 HAMi-core（CNCF 孵化项目）正是这样的隔离层，而且它是**与调度器解耦**的：在 KAI Scheduler 之前，HAMi-core 已经通过不同的方式与 Kubernetes 原生调度器（经 HAMi 自带的 `hami-scheduler` 扩展）、[Kueue](/zh/docs/userguide/kueue/how-to-use-kueue)、[Volcano](/zh/docs/installation/how-to-use-volcano-vgpu)、Koordinator 等协同工作（完整生态见 [HAMi 生态集成](/zh/docs/next/core-concepts/ecosystem-integrations)）。 @@ -44,11 +44,11 @@ HAMi 社区多年打磨的 HAMi-core（CNCF 孵化项目）正是这样的隔离 先把三个名字容易混淆的组件讲清楚，这是理解整条链路的前提： -- **HAMi-core（`libvgpu.so`）**：HAMi 项目的 CUDA 拦截库（CNCF 孵化），是**隔离引擎本身**。它通过 `LD_PRELOAD` 拦截容器里的 CUDA 调用（如 `cudaMalloc`），按一个显存配额强制限制。它不关心配额是谁给的：任何调度器只要按约定把配额传进来，它都能执行隔离。在 KAI 之前，它已经被 HAMi 自带的 device-plugin/webhook、Volcano 的 `volcano-vgpu-device-plugin` 等复用。 +- **HAMi-core（`libvgpu.so`）**：HAMi 项目的 CUDA 拦截库，是**隔离引擎本身**。它通过 `LD_PRELOAD` 拦截容器里的 CUDA 调用（如 `cudaMalloc`），按一个显存配额强制限制。它不关心配额是谁给的：任何调度器只要按约定把配额传进来，它都能执行隔离。在 KAI 之前，它已经被 HAMi 自带的 device-plugin/webhook、Volcano 的 `volcano-vgpu-device-plugin` 等复用。 -- **KAI Scheduler**：NVIDIA 开源的 Kubernetes AI 工作负载调度器（源自 Run:ai，CNCF Sandbox）。它只负责**调度层**，即决定 Pod 落到哪个节点、用哪张 GPU、分多少显存。自 v0.16.4 起，它的 `hamicore` 插件在绑定 Pod 时，会把算好的显存配额写进容器的环境变量。 +- **KAI Scheduler**：NVIDIA 开源的 Kubernetes AI 工作负载调度器。它只负责**调度层**，即决定 Pod 落到哪个节点、用哪张 GPU、分多少显存。自 v0.16.4 起，它的 `hamicore` 插件在绑定 Pod 时，会把算好的显存配额写进容器的环境变量。 -- **`kai-resource-isolator`**：HAMi 项目侧为 KAI Scheduler 这条集成路径**专门提供的配套组件**。它把 HAMi-core 的 `libvgpu.so` 分发到每个 GPU 节点，并用 MutatingWebhook 改写 Pod，把库和 `ld.so.preload` 注入进去。换言之，它是「把 KAI 的调度决策落地成 HAMi-core 能执行的隔离」的桥梁。 +- **`kai-resource-isolator`**：HAMi 项目侧为 KAI Scheduler 这条集成路径**专门提供的配套组件**。它把 HAMi-core 的 `libvgpu.so` 分发到每个 GPU 节点，并用 MutatingWebhook 改写 Pod，把库和 `ld.so.preload` 注入进去。换言之，它是“把 KAI 的调度决策落地成 HAMi-core 能执行的隔离”的桥梁。 ```mermaid %% title: 三个组件的职责与衔接 @@ -87,10 +87,10 @@ KAI Scheduler 的集成目标是 **HAMi-core 本身**，而不是完整的 HAMi 整条链路之所以能成立，是因为调度层和隔离层约定了一个极简的交接点：环境变量 **`CUDA_DEVICE_MEMORY_LIMIT`**。 -- **KAI Scheduler（调度层）** 负责算出「这个 Pod 能用多少显存」，并在绑定节点时把它写进容器的环境变量。 +- **KAI Scheduler（调度层）** 负责算出“这个 Pod 能用多少显存”，并在绑定节点时把它写进容器的环境变量。 - **HAMi-core（隔离层）** 负责读这个环境变量，并在运行时真正把显存用量卡在这个上限以内。 -这个契约之所以重要，是因为它**把两件事彻底解耦**：KAI 不需要知道 CUDA 怎么被拦截，HAMi-core 不需要知道份额是怎么算出来的。两边只要都遵守 `CUDA_DEVICE_MEMORY_LIMIT` 这一个变量，任何调度器都能复用同一套隔离引擎。这正是 HAMi-core 能同时支持多个调度器的根本原因（见文末「这意味着什么」）。 +这个契约之所以重要，是因为它**把两件事彻底解耦**：KAI 不需要知道 CUDA 怎么被拦截，HAMi-core 不需要知道份额是怎么算出来的。两边只要都遵守 `CUDA_DEVICE_MEMORY_LIMIT` 这一个变量，任何调度器都能复用同一套隔离引擎。这正是 HAMi-core 能同时支持多个调度器的根本原因（见文末“这意味着什么”）。 ### 三步协作 @@ -118,9 +118,9 @@ graph TD style ENF fill:#dcfce7,stroke:#0b6b3c,stroke-width:2px,color:#1f2937 ``` -一句话概括：**KAI 说「你只能用这么多」，isolator 负责「让你真的只能用这么多」。** +一句话概括：**KAI 说“你只能用这么多”，isolator 负责“让你真的只能用这么多”。** -三个组件的分工如下（详见 [KAI Scheduler 官方文档「HAMi 资源隔离」](https://github.com/kai-scheduler/KAI-Scheduler/blob/main/docs/gpu-sharing/hami/README.md)）： +三个组件的分工如下（详见 [KAI Scheduler 官方文档“HAMi 资源隔离”](https://github.com/kai-scheduler/KAI-Scheduler/blob/main/docs/gpu-sharing/hami/README.md)）： - **libsync DaemonSet**：把 `libvgpu.so` 复制到每个 GPU 节点的 `/usr/local/vgpu`。 - **mutating webhook**：给 Pod 注入 hostPath 卷挂载，把 `/etc/ld.so.preload` 指向 `libvgpu.so`，并写入 `POD_UID`、`CONTAINER_NAME`、`CONTAINER_VGPU_MOUNT` 环境变量。 @@ -128,14 +128,14 @@ graph TD ### CUDA 拦截是如何生效的 -隔离的「最后一公里」发生在容器进程里，链路是这样的： +隔离的“最后一公里”发生在容器进程里，链路是这样的： 1. **KAI 在调度时注入 `CUDA_DEVICE_MEMORY_LIMIT`**（带上 Pod 申请的显存配额，单位 MiB）。 2. **kai-resource-isolator 的 webhook 改写 Pod**：挂载宿主机上的 `libvgpu.so`，并把 `/etc/ld.so.preload` 指向它。`ld.so.preload` 是动态链接器的机制，被列在里面的共享库会在所有其他库之前加载。 3. **容器进程启动后**，任何对 CUDA 运行时（`libcudart`）或驱动 API 的调用，都会先经过 `libvgpu.so`。后者拦截 `cudaMalloc` 之类的显存分配调用，从 `CUDA_DEVICE_MEMORY_LIMIT` 读出配额，累计该容器的显存用量；一旦超额就拒绝分配。 4. **对外可见的效果**：`nvidia-smi` 只显示配额内的显存（HAMi-core 会改写设备查询的返回值），容器再怎么 `cudaMalloc` 也越不过这条线。 -这就是「硬隔离」的含义：不是靠应用自觉，而是在 CUDA 调用这一层强制执行。 +这就是“硬隔离”的含义：不是靠应用自觉，而是在 CUDA 调用这一层强制执行。 :::tip 集成的来龙去脉 @@ -147,668 +147,62 @@ graph TD ### KAI Scheduler（自 v0.16.4 起内置 HAMi-core 硬隔离） -KAI Scheduler 对 HAMi-core 的支持自 **v0.16.4** 起成为内置能力（本文示例使用当前版本 v0.17.0）。关键在于 `hamicore` 插件：启用后，KAI 在把共享 GPU 的 Pod 绑定到节点时，会按 `gpu-memory`（或 `gpu-fraction`）注解向容器注入 `CUDA_DEVICE_MEMORY_LIMIT` 环境变量，这正是上一节契约里 HAMi-core 执行隔离所需的配额。 +KAI Scheduler 对 HAMi-core 的支持最早出现在 **v0.16.4**。当前集成文档要求 KAI Scheduler v0.17.0 或更高版本，本文实测 v0.17.0。关键在于 `hamicore` 插件：启用后，KAI 在把共享 GPU 的 Pod 绑定到节点时，会按 `gpu-memory`（或 `gpu-fraction`）注解向容器注入 `CUDA_DEVICE_MEMORY_LIMIT` 环境变量，这正是上一节契约里 HAMi-core 执行隔离所需的配额。 -v0.17.0 中其他与 GPU 相关的改动还包括：修正共享 Pod 名称带「`/`」造成的非法卷名问题；修正 `MinNodeGPUMemoryMiB` 与 fractional `gpu-memory` 的分配计算；overLimit 判定改用集群最大 GPU 规格。此外还有 preemption-delay（为 Cluster Autoscaler 留出拉起节点的时间窗）、NUMA 感知打分、GitOps 与 ArgoCD 安装支持等。 +v0.17.0 中其他与 GPU 相关的改动还包括：修正共享 Pod 名称带“`/`”造成的非法卷名问题；修正 `MinNodeGPUMemoryMiB` 与 fractional `gpu-memory` 的分配计算；overLimit 判定改用集群最大 GPU 规格。此外还有 preemption-delay（为 Cluster Autoscaler 留出拉起节点的时间窗）、NUMA 感知打分、GitOps 与 ArgoCD 安装支持等。 ### kai-resource-isolator 1.1.0-chart -这是与 HAMi 配套发布的隔离器。它接收 KAI 注入的配额，在容器真正运行起来之前，把 HAMi-core 的 `libvgpu.so` 注入到位。相对首个版本，1.1.0 补齐了一组让它具备生产可用性的改进： +这是与 HAMi 配套发布的隔离器。它接收 KAI 注入的配额，在容器真正运行起来之前，把 HAMi-core 的 `libvgpu.so` 注入到位。相对首个版本，1.1.0 增加了一组运维改进： - **新增 `kai-vgpu-monitor`**：以 DaemonSet 形式运行，在 `:9394` 暴露 HAMi 兼容指标（`hami_vgpu_memory_used_bytes`、`hami_vgpu_memory_limit_bytes`、`hami_container_device_utilization_ratio`），支持 ServiceMonitor，可被 Prometheus 直接抓取。 - **多容器注入修复**：一个 Pod 内有多个容器时，webhook 现在能正确处理，不再漏注。 - **安全收紧**：webhook 改用命名空间内 Issuer（不再使用 ClusterIssuer），ClusterRole 收回读 Secret 的权限。 - **全局镜像仓库**优先级理顺，`hamicore` 安装参数修正。 -## 实践：在 GKE 上验证硬隔离 +## GKE 实测：隔离是否真的生效 -示例运行在一个已有的 GKE 集群上：**3 个 `n1-standard-2` 节点，每节点挂载 1 张 NVIDIA T4（`nvidia-smi` 实测显存 15360 MiB）**。两个容器各申请约 4 GiB，共享其中一张 T4；`nvidia-smi` 里各自只见自己的配额，互不干扰。下面每一步都给出配置、预期输出，以及为什么这样做。 +我们在一套 GKE 1.35/COS/CDI 集群上完成了端到端验证：3 个 `n1-standard-2` 节点，每节点一张 NVIDIA T4；KAI Scheduler v0.17.0 负责共享调度，`kai-resource-isolator` 1.1.0-chart 负责注入 HAMi-core。 -下图是整个实验的总览：8 个步骤的顺序，以及其中两个最容易踩坑的前置依赖（GFD 标签必须在 KAI 之前打好；队列必须存在才会调度）。注意：本文**不需要安装 NVIDIA GPU Operator**，GKE 默认的 device plugin + 驱动 + container toolkit 已经够用，下面只要「启用默认 device plugin」和「补几个 GFD 标签」两步准备即可。 +验证没有止步于 `nvidia-smi`。两个 Pod 被固定到一个只有一张 T4 的节点上，并完成了三层检查： -```mermaid -%% title: GKE 实验流程（Step 1 到 Step 8） -graph TD - S1["Step 1
确认集群与 GPU 硬件
n1-standard-2 + T4"] - S2["Step 2
启用 GKE 默认 device plugin
验证 nvidia-smi"] - S3["Step 3
打 GFD 标签
nvidia.com/gpu.memory 等"] - S4["Step 4
安装 KAI Scheduler v0.17.0
创建队列"] - S5["Step 5
安装 kai-resource-isolator
1.1.0-chart"] - S6["Step 6
单 Pod 验证隔离
nvidia-smi 见配额内显存"] - S7["Step 7
两 Pod 共享一张 T4
各见自己的配额"] - S8["Step 8 可选
查看 monitor 指标"] - - S1 --> S2 --> S3 --> S4 --> S5 --> S6 --> S7 --> S8 - - S3 -. "⚠️ GFD 标签必须在 KAI 之前打
否则共享 Pod 卡 Pending" .-> S4 - S4 -. "⚠️ 队列必须存在
否则 Pod 不被调度" .-> S6 - - style S1 fill:#dbeafe,stroke:#1a5fb4,stroke-width:2px,color:#1f2937 - style S2 fill:#fef3c7,stroke:#b45309,stroke-width:2px,color:#1f2937 - style S3 fill:#fef3c7,stroke:#b45309,stroke-width:2px,color:#1f2937 - style S4 fill:#d9f99d,stroke:#4f7d00,stroke-width:2px,color:#1f2937 - style S5 fill:#d9f99d,stroke:#4f7d00,stroke-width:2px,color:#1f2937 - style S6 fill:#dcfce7,stroke:#0b6b3c,stroke-width:2px,color:#1f2937 - style S7 fill:#dcfce7,stroke:#0b6b3c,stroke-width:2px,color:#1f2937 - style S8 fill:#eff6ff,stroke:#1a5fb4,stroke-width:1px,color:#1f2937 -``` - -### 前置条件 - -- 一个 GCP 项目，已启用 GKE 与 Compute Engine API。 -- 一个已有的 GKE 集群，节点池里带 NVIDIA T4 GPU（本文示例为 3 个 `n1-standard-2` 节点，每节点 1× T4）。本例这个集群建的时候带了节点标签 `gke-no-default-nvidia-gpu-device-plugin=true`，即**禁用了 GKE 默认的 NVIDIA device plugin**，驱动则由 GKE 自动安装。这正是后面 Step 1 会看到的「硬件在、但 `nvidia.com/gpu` 资源为空」这一初始状态的由来，Step 2 会把 device plugin 重新启用。如果你还没建好，可以用下面的命令建一个同等规格的集群： - - ```bash - gcloud container clusters create test-cluster --zone=asia-northeast1-a \ - --machine-type=n1-standard-2 --num-nodes=3 \ - --accelerator=type=nvidia-tesla-t4,count=1,gpu-driver-version=default - gcloud container clusters get-credentials test-cluster --zone=asia-northeast1-a - ``` - - `n1` 机型本身不含 GPU，所以**必须**用 `--accelerator` 显式指定 T4；`gpu-driver-version=default` 让 GKE 自动安装匹配的 NVIDIA 驱动和 container toolkit。这样建出来的集群，device plugin 默认就是启用的（Step 2 可以跳过删标签那步）。 - -- `gcloud`、`kubectl`、`helm`（≥ 3）已登录，`kubectl` 已能访问该集群。 -- KAI 队列由下文 Step 4 创建。 - -### Step 1：确认集群与 GPU 硬件就绪（初始状态） - -集群已经建好后，先看一眼它的初始状态。这一步只看、不改。 - -```bash -kubectl get nodes -o custom-columns="NAME:.metadata.name,GPU:.status.capacity.nvidia\.com/gpu,ACCEL:.metadata.labels.cloud\.google\.com/gke-accelerator" -``` - -本例这个集群的初始输出是这样的：3 个节点，`ACCEL` 列都是 `nvidia-tesla-t4`（T4 硬件在），但 `GPU` 列是 ``。 - -``` -NAME GPU ACCEL -gke-test-cluster-default-pool-xxxxxxxx nvidia-tesla-t4 -gke-test-cluster-default-pool-yyyyyyyy nvidia-tesla-t4 -gke-test-cluster-default-pool-zzzzzzzz nvidia-tesla-t4 -``` - -**怎么读这个输出**：`ACCEL` 列的 `nvidia-tesla-t4` 确认 T4 硬件已经挂载到节点上，GKE 也已经把 NVIDIA 内核驱动装好了。但 `GPU` 列是 ``，说明 **device plugin 还没上报 `nvidia.com/gpu` 资源**。原因是这个集群建的时候带了节点标签 `gke-no-default-nvidia-gpu-device-plugin=true`，把 GKE 默认的 NVIDIA device plugin 禁用了。下一启用它。 - -如果你的集群 `GPU` 列已经直接是 `1`（建集群时没禁用默认 device plugin），可以跳过 Step 2，直接到 Step 3 打 GFD 标签。 - -### Step 2：启用 GKE 默认 device plugin，并验证 GPU 可用 +| 检查 | 实测结果 | 证明了什么 | +| :-- | :-- | :-- | +| 节点与 GPU UUID | 两个 Pod 位于同一节点，UUID 均为 `GPU-9acc8878-...` | 两个 Pod 确实共享同一张物理卡 | +| 可见显存 | 两个 Pod 均为 `4147 MiB`，整卡为 `15360 MiB` | KAI 注入的配额已由 HAMi-core 呈现到容器 | +| CUDA 分配 | 3 GiB 成功，累计申请 5 GiB 返回 `out of memory` | 显存上限被实际执行，不只是修改显示 | +| 并发隔离 | Pod A 持有 3 GiB 时，Pod B 仍成功获得自己的 3 GiB | 一个 Pod 无法占用另一个 Pod 的配额 | +| monitor 指标 | 同节点 `:9394/metrics` 返回两个 Pod 各自 4.348 GB 上限与 3.328 GB 实时用量 | monitor 读取了每个容器的共享内存缓存，并导出非空的 Pod 级指标 | -把节点上的 `gke-no-default-nvidia-gpu-device-plugin` 标签删掉，GKE 自带的 device plugin DaemonSet（它的调度规则是「这个标签不为 `true` 才上」）就会自动调度到节点上。 +HAMi-core 在越界时记录了： -```bash -kubectl label node -l cloud.google.com/gke-accelerator=nvidia-tesla-t4 \ - gke-no-default-nvidia-gpu-device-plugin- +```text +Device 0 OOM 5475663872 / 4348444672 +allocate another 2 GiB: out of memory +PASS: in-quota allocation succeeded and over-quota allocation failed ``` -标签名末尾的 `-` 表示「删除这个标签」。执行后等大约 20 秒，device plugin Pod 就会起来： - -```bash -kubectl get pod -n kube-system -o wide | grep nvidia-gpu-device-plugin | grep -v "0/0" -``` - -预期输出（示意）：3 个 `nvidia-gpu-device-plugin-*-ubuntu` Pod，每个节点一个，都是 `3/3 Running`。 - -``` -NAME READY STATUS NODE -nvidia-gpu-device-plugin-small-ubuntu-xxx 3/3 Running gke-test-cluster-default-pool-xxxxxxxx -nvidia-gpu-device-plugin-small-ubuntu-yyy 3/3 Running gke-test-cluster-default-pool-yyyyyyyy -nvidia-gpu-device-plugin-small-ubuntu-zzz 3/3 Running gke-test-cluster-default-pool-zzzzzzzz -``` - -device plugin 起来后，`nvidia.com/gpu` 资源就有了： - -```bash -kubectl get nodes -o custom-columns="NAME:.metadata.name,GPU:.status.capacity.nvidia\.com/gpu" -``` - -预期输出（示意）：每个节点的 `GPU` 列从 `` 变成 `1`。 - -``` -NAME GPU -gke-test-cluster-default-pool-xxxxxxxx 1 -gke-test-cluster-default-pool-yyyyyyyy 1 -gke-test-cluster-default-pool-zzzzzzzz 1 -``` - -**为什么不需要装 GPU Operator**：GKE 默认的 GPU 栈已经把三样东西都配好了：NVIDIA 内核驱动（GKE 自动装）、device plugin（刚启用）、container toolkit / nvidia runtime（`gpu-driver-version=default` 时 GKE 自动配）。GPU Operator 是把这些打包在一起自动管理的工具，但 GKE 自己已经在管了，再装反而会和 GKE 的驱动冲突（实测：GPU Operator 的驱动容器无法卸载 GKE 已加载的内核模块）。所以 GKE 上跑通实验，靠 GKE 默认栈就够了。 - -最后用一个真正的 GPU Pod 验证 toolkit 也工作（容器能跑 `nvidia-smi`）： - -```bash -cat <<'EOF' | kubectl apply -f - && \ -kubectl wait --for=condition=Ready pod/gpu-smi-test --timeout=120s && \ -kubectl logs gpu-smi-test && \ -kubectl delete pod gpu-smi-test -apiVersion: v1 -kind: Pod -metadata: - name: gpu-smi-test -spec: - restartPolicy: Never - containers: - - name: cuda - image: nvidia/cuda:12.4.1-base-ubuntu22.04 - command: ["nvidia-smi"] - resources: - limits: - nvidia.com/gpu: 1 -EOF -``` - -预期输出（实测）：能看到 T4 和驱动版本，注意显存是 **15360 MiB**（这是 T4 实际可寻址显存，下面 Step 3 打标签要用这个数）。 - -``` -+-----------------------------------------------------------------------------------------+ -| NVIDIA-SMI 580.159.04 Driver Version: 580.159.04 CUDA Version: 13.0 | -+-----------------------------------------+------------------------+----------------------+ -| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 | -| N/A 45C P8 17W / 70W | 0MiB / 15360MiB | 0% Default | -+-----------------------------------------+------------------------+----------------------+ -``` +这组结果把“调度到同一张卡”“容器只看到自己的配额”“CUDA 无法越过配额”和“指标可观测”连成了完整证据链。monitor 以 DaemonSet 运行并读取节点本地缓存，因此实验 12 会直接查询工作负载所在节点的实例，而不是依赖可能选中其他节点的 Service。 -看到这张表，说明驱动、device plugin、toolkit 全部就位，容器已经能真正使用 GPU 了。 +:::note 隔离边界 -### Step 3：打 GFD 标签（KAI 调度需要） - -KAI 的 hamicore 插件要靠节点标签 `nvidia.com/gpu.memory` 来知道每张卡的显存，才能把 `gpu-memory` 请求换算成分数。GKE 默认 device plugin 会打 `nvidia.com/gpu.present=true`，但**不打 `gpu.memory` / `gpu.count` / `gpu.product` 这几个**（那些通常由 GPU Feature Discovery 打，而 GKE 默认栈不跑完整 GFD）。所以需要手动补上。 - -用 Step 2 里 `nvidia-smi` 看到的显存值 **15360**（MiB）： - -```bash -kubectl label node -l cloud.google.com/gke-accelerator=nvidia-tesla-t4 \ - nvidia.com/gpu.memory=15360 \ - nvidia.com/gpu.product=NVIDIA-Tesla-T4 \ - nvidia.com/gpu.count=1 \ - nvidia.com/gpu.present=true --overwrite -``` - -:::warning 关键一步：标签必须在安装 KAI 之前打好 - -KAI 的 hamicore 插件在**节点注册时**缓存每张卡的显存（读取 `node.labels["nvidia.com/gpu.memory"]`）。缺这个标签它会直接报错，共享 Pod 永远停在 `Pending`，日志中出现 `didn't have enough resources: GPU memory`。所以这一步必须在 Step 4 装 KAI **之前**完成。万一装完 KAI 才补标签，重启 `kai-scheduler` 让它重新缓存即可。 +这里验证的是 HAMi-core 在 CUDA API 层执行的显存配额，不是 MIG 一类硬件安全边界。本次 GKE 兼容路径还使用了特权业务容器，因此不应直接解读为面向不可信多租户的安全隔离方案。 ::: -确认标签到位： - -```bash -kubectl get nodes -o custom-columns="NAME:.metadata.name,GPUMEM:.metadata.labels.nvidia\.com/gpu\.memory,GPUPROD:.metadata.labels.nvidia\.com/gpu\.product,GPUPRESENT:.metadata.labels.nvidia\.com/gpu\.present" -``` - -预期输出（示意）：每个节点的 `GPUMEM` 为 `15360`，`GPUPROD` 为 `NVIDIA-Tesla-T4`，`GPUPRESENT` 为 `true`。 - -``` -NAME GPUMEM GPUPROD GPUPRESENT -gke-test-cluster-default-pool-xxxxxxxx 15360 NVIDIA-Tesla-T4 true -gke-test-cluster-default-pool-yyyyyyyy 15360 NVIDIA-Tesla-T4 true -gke-test-cluster-default-pool-zzzzzzzz 15360 NVIDIA-Tesla-T4 true -``` - -**为什么是 `15360` 而不是 `16384`**：T4 标称 16 GiB，但 `nvidia-smi` 报告的实际可寻址显存是 15360 MiB（部分显存被驱动/ECC 预留）。KAI 用这个数把 `gpu-memory` 请求换算成分数，所以标签值必须和 `nvidia-smi` 的数字一致，否则 Pod 看到的上限会和你预期不符。 - -### Step 4：安装 KAI Scheduler v0.17.0，并创建队列 - -```bash -helm install kai-scheduler oci://ghcr.io/kai-scheduler/kai-scheduler/kai-scheduler \ - --set global.gpuSharing=true \ - --set binder.plugins.hamicore.enabled=true \ - --set binder.plugins.gpusharing.arguments.cdiEnabled=true \ - --namespace kai-scheduler --create-namespace \ - --version v0.17.0 - -# 等 admission webhook 就绪（签发证书需要一点时间），再创建队列， -# 否则 Queue 的创建会被拒绝。 -kubectl -n kai-scheduler wait --for=condition=available --timeout=180s deploy --all -``` - -**为什么是这几个开关**： - -- `global.gpuSharing=true` 打开共享调度，允许 KAI 把多个 Pod 放到同一张 GPU 上。 -- `binder.plugins.hamicore.enabled=true` 让 KAI 在绑定时注入 `CUDA_DEVICE_MEMORY_LIMIT`，这是把调度层和隔离层连起来的那个契约。没开它，隔离层就拿不到配额，等于没装。 -- `binder.plugins.gpusharing.arguments.cdiEnabled=true`：**GKE 1.33+ 必加**。KAI 的 gpusharing 插件默认 `cdiEnabled=false`（假设 GPU 通过 NVIDIA runtime 注入），但 GKE 1.33+ 的 containerd 用 CDI 模式（`enable_cdi=true`）注入 GPU。不开这个开关，KAI 创建的 GPU reservation Pod 会找不到 NVML（报 `ERROR_LIBRARY_NOT_FOUND`），共享 Pod 卡在 `Pending`。KAI 源码里 CDI 优先靠 GPU Operator 的 ClusterPolicy 自动检测，但我们没装 GPU Operator（见 Step 2），所以必须显式打开。（参考 [KAI binder 文档](https://github.com/kai-scheduler/KAI-Scheduler/blob/main/docs/developer/binder.md)。） - -随后创建队列。**队列必须存在，KAI 才会调度。** Pod 靠 `kai.scheduler/queue: default-queue` 标签指向队列；队列（CRD `scheduling.run.ai/v2`）不存在，Pod 就一直 `Pending`。下面这份清单建立了 `default`（父）与 `default-queue`（子）两层，配额全部放开（`-1`），方便演示： - -```yaml -# queue.yaml -apiVersion: scheduling.run.ai/v2 -kind: Queue -metadata: - name: default -spec: - resources: - cpu: { quota: -1, limit: -1, overQuotaWeight: 1 } - memory: { quota: -1, limit: -1, overQuotaWeight: 1 } - gpu: { quota: -1, limit: -1, overQuotaWeight: 1 } ---- -apiVersion: scheduling.run.ai/v2 -kind: Queue -metadata: - name: default-queue -spec: - parentQueue: default - resources: - cpu: { quota: -1, limit: -1, overQuotaWeight: 1 } - memory: { quota: -1, limit: -1, overQuotaWeight: 1 } - gpu: { quota: -1, limit: -1, overQuotaWeight: 1 } -``` - -```bash -kubectl apply -f queue.yaml -kubectl get queues -``` - -预期输出（示意）： - -``` -NAME PARENT AGE -default 5s -default-queue default 5s -``` - -两个队列都在、子队列正确指向父队列，才能进入下一步。 - -### Step 5：安装 kai-resource-isolator 1.1.0-chart - -```bash -helm install kai-resource-isolator oci://docker.io/projecthami/kai-resource-isolator \ - --namespace kai-resource-isolator --create-namespace \ - --set monitor.enabled=true \ - --set monitor.serviceMonitor.enabled=true \ - --set paths.hostInstallBase=/home/kubernetes/bin/nvidia \ - --version 1.1.0-chart -``` - -**为什么版本号带 `-chart` 后缀**：chart 版本是 **`1.1.0-chart`**（带 `-chart` 后缀），对应的 git tag 是 `v1.1.0`，两者不要混淆，写成 `1.1.0` 或 `v1.1.0` 会拉不到包。若节点的 NVML 必须走 NVIDIA runtime，再补一个 `--set monitor.runtimeClassName=nvidia`（GKE GPU 节点默认已具备）。 - -**为什么改 `paths.hostInstallBase`**：isolator 的 libsync DaemonSet 要把 `libvgpu.so` 写到宿主机的 `{hostInstallBase}/vgpu/`，默认是 `/usr/local`。但 **GKE 节点的根文件系统是只读的**（ext2 `ro` 挂载），`/usr/local` 不可写，libsync 会报 `Read-only file system` 起不来。`/home/kubernetes/bin/nvidia` 是 GKE 上 NVIDIA 驱动所在的**可写**目录（也是 GKE 装 driver 的标准路径），所以把库放这里。非 GKE 的自建集群根文件系统可写时，这个参数可以不加（用默认 `/usr/local`）。 - -确认三个组件都已就绪： - -```bash -kubectl -n kai-resource-isolator get pods -``` - -预期输出（示意）： - -``` -NAME READY STATUS -kai-resource-isolator-xxxxxxxx-xxxxx 1/1 Running # webhook -kai-libsync-xxxxx 1/1 Running # 分发 libvgpu.so -kai-vgpu-monitor-xxxxx 1/1 Running # :9394 指标 -``` - -三个 Pod 都 `Running`，意味着 webhook、libsync、monitor 全部就位。但在 GKE 上，还需要做几步额外适配才能让隔离链路完整工作（见下文）。自建集群如果根文件系统可写、有 `nvidia` RuntimeClass、节点 NVML 库在标准路径，可以跳过本节。 - -### Step 5b：GKE 环境适配（GKE 1.33+ 必做） - -以下四个适配是实测确认的 GKE 特有要求，每一步都解决一个具体问题： - -**5b-1. 创建 `nvidia` RuntimeClass（GKE 1.35 没有）** - -GKE 1.33+ 用 CDI 模式注入 GPU，节点上没有注册 `nvidia` runtime handler，因此也没有 `nvidia` RuntimeClass。但 KAI 的 reservation Pod 在创建时会检查它。创建一个指向默认 `runc` handler 的 RuntimeClass（CDI 在 containerd 层注入设备，不需要 nvidia runtime）： - -```bash -cat <<'EOF' | kubectl apply -f - -apiVersion: node.k8s.io/v1 -kind: RuntimeClass -metadata: - name: nvidia -handler: runc -EOF -``` - -**5b-2. patch isolator DaemonSet 去掉 `system-node-critical` 优先级** - -GKE 不允许用户 namespace 的 Pod 使用 `system-node-critical` PriorityClass（报 `insufficient quota to match these scopes`）。isolator 的 libsync 和 monitor DaemonSet 默认用了它，需要去掉： - -```bash -for ds in kai-resource-isolator-libsync kai-resource-isolator-monitor; do - kubectl patch ds "$ds" -n kai-resource-isolator \ - --type=json -p='[{"op":"replace","path":"/spec/template/spec/priorityClassName","value":null}]' -done -``` - -**5b-3. 安装 Kyverno 并注入 `LD_LIBRARY_PATH`** - -KAI 的 reservation Pod 用 NVML 查询 GPU 信息，但它的容器镜像里没有 `libnvidia-ml.so`。GKE 会把 NVML 库注入到容器的 `/usr/local/nvidia/lib64/`，但不会设 `LD_LIBRARY_PATH`（nvidia/cuda 镜像自带这个变量，而 reservation 镜像没有）。用 [Kyverno](https://kyverno.io/) 的 MutatingWebhook 自动给 reservation namespace 的 Pod 注入这个环境变量： - -```bash -# 安装 Kyverno（自带 TLS 证书管理，不需要 cert-manager） -helm install kyverno kyverno/kyverno -n kyverno --create-namespace -kubectl wait --for=condition=ready pod -l app.kubernetes.io/component=admission-controller \ - -n kyverno --timeout=300s - -# 创建策略：给 kai-resource-reservation namespace 的所有 Pod 注入 LD_LIBRARY_PATH -cat <<'EOF' | kubectl apply -f - -apiVersion: kyverno.io/v1 -kind: ClusterPolicy -metadata: - name: inject-nvidia-library-path -spec: - rules: - - name: add-ld-library-path - match: - any: - - resources: - kinds: [Pod] - namespaces: [kai-resource-reservation] - mutate: - foreach: - - list: "request.object.spec.containers[]" - patchStrategicMerge: - metadata: {} - spec: - containers: - - name: "{{ element.name }}" - env: - - name: LD_LIBRARY_PATH - value: /usr/local/nvidia/lib64 -EOF -``` - -上面这条 `inject-nvidia-library-path` 策略只解决了 reservation Pod 的 NVML 查询问题。**共享 Pod 还需要第二条策略 `inject-gpu-devices`**，作用是把 GPU 设备节点和 NVIDIA 工具目录挂进共享 Pod。原因是：GKE CDI 模式下，device plugin 只对 request 了 `nvidia.com/gpu` 扩展资源的 Pod 执行 `Allocate` 并注入设备；而 KAI 的共享 Pod 用 `gpu-memory` 注解（不 request 扩展资源），所以 device plugin 不会给它们注入 `/dev/nvidia*`、`nvidia-smi` 二进制和 NVML 库。没有这条策略，共享 Pod 里 `nvidia-smi` 会因为找不到设备而无法运行。下面这份策略匹配所有带 `gpu-memory` 注解的 Pod（任意 namespace），自动挂载 `/dev/nvidia0`、`/dev/nvidiactl`、`/dev/nvidia-uvm`、`/dev/nvidia-modeset`、`/dev/nvidia-caps` 以及宿主机上的 `nvidia-smi` 二进制目录和 `libnvidia-ml.so` 库目录： - -```bash -# 第二条策略：给所有带 gpu-memory 注解的共享 Pod 注入 GPU 设备节点和 NVIDIA 库目录 -cat <<'EOF' | kubectl apply -f - -apiVersion: kyverno.io/v1 -kind: ClusterPolicy -metadata: - name: inject-gpu-devices -spec: - background: false - rules: - - name: add-gpu-volumes - match: - any: - - resources: - kinds: [Pod] - preconditions: - all: - - key: "{{ request.object.metadata.annotations.\"gpu-memory\" || '' }}" - operator: NotEquals - value: "" - mutate: - patchesJson6902: |- - - op: add - path: /spec/volumes/- - value: - name: nvidia-dev0 - hostPath: {path: /dev/nvidia0, type: CharDevice} - - op: add - path: /spec/volumes/- - value: - name: nvidia-ctl - hostPath: {path: /dev/nvidiactl, type: CharDevice} - - op: add - path: /spec/volumes/- - value: - name: nvidia-uvm - hostPath: {path: /dev/nvidia-uvm, type: CharDevice} - - op: add - path: /spec/volumes/- - value: - name: nvidia-modeset - hostPath: {path: /dev/nvidia-modeset, type: CharDevice} - - op: add - path: /spec/volumes/- - value: - name: nvidia-caps - hostPath: {path: /dev/nvidia-caps, type: Directory} - - op: add - path: /spec/volumes/- - value: - name: nvidia-bin - hostPath: {path: /home/kubernetes/bin/nvidia/bin, type: Directory} - - op: add - path: /spec/volumes/- - value: - name: nvidia-lib - hostPath: {path: /home/kubernetes/bin/nvidia/lib64, type: Directory} - - name: add-gpu-volumemounts - match: - any: - - resources: - kinds: [Pod] - preconditions: - all: - - key: "{{ request.object.metadata.annotations.\"gpu-memory\" || '' }}" - operator: NotEquals - value: "" - mutate: - foreach: - - list: "request.object.spec.containers[]" - patchStrategicMerge: - spec: - containers: - - name: "{{ element.name }}" - volumeMounts: - - {name: nvidia-dev0, mountPath: /dev/nvidia0} - - {name: nvidia-ctl, mountPath: /dev/nvidiactl} - - {name: nvidia-uvm, mountPath: /dev/nvidia-uvm} - - {name: nvidia-modeset, mountPath: /dev/nvidia-modeset} - - {name: nvidia-caps, mountPath: /dev/nvidia-caps} - - {name: nvidia-bin, mountPath: /usr/local/nvidia/bin} - - {name: nvidia-lib, mountPath: /usr/local/nvidia/lib64} -EOF -``` - -两条策略一起用，共享 Pod 就同时有了设备节点（`/dev/nvidia*`）、`nvidia-smi` 二进制（`/usr/local/nvidia/bin`）和 NVML 库（`/usr/local/nvidia/lib64`）。注意：仅有设备节点还不够，Pod 还必须开 `securityContext.privileged: true`（见 Step 6、Step 7），因为 HAMi-core 的 `libvgpu.so` 需要特权模式才能初始化 NVML（实测确认）。这三件事（设备节点 + 库目录 + privileged）缺一不可，配齐后共享 Pod 才能成功跑通 `nvidia-smi` 并看到隔离后的显存。 - -**5b-4. 修正 libsync 写入的 `ld.so.preload` 路径 + monitor hostPath** - -isolator 的 libsync 把 `libvgpu.so` 和 `ld.so.preload` 写到 `/home/kubernetes/bin/nvidia/vgpu/`（Step 5 的 `hostInstallBase`），但 chart 的 `ld.so.preload` ConfigMap 里可能还是旧的 `/usr/local/vgpu/libvgpu.so` 路径。确认并修正它指向正确路径： - -```bash -# 确认 ConfigMap 内容 -kubectl get cm kai-resource-isolator-ldpreload -n kai-resource-isolator \ - -o jsonpath='{.data.ld\.so\.preload}' -# 如果不是 /home/kubernetes/bin/nvidia/vgpu/libvgpu.so，修正它 -kubectl create cm kai-resource-isolator-ldpreload -n kai-resource-isolator \ - --from-literal=ld.so.preload="/home/kubernetes/bin/nvidia/vgpu/libvgpu.so" \ - --dry-run=client -o yaml | kubectl apply -f - - -# 重启 libsync Pod 让它用修正后的 ConfigMap 重新写入 -kubectl delete pod -n kai-resource-isolator \ - $(kubectl get pod -n kai-resource-isolator -l app.kubernetes.io/component=kai-libsync -o jsonpath='{.items[*].metadata.name}') -``` - -另外，monitor DaemonSet 也有一个 hostPath volume 指向旧的 `/usr/local/vgpu/containers`（只读 rootfs），会卡在 `ContainerCreating`。同样改成可写区： - -```bash -kubectl patch ds kai-resource-isolator-monitor -n kai-resource-isolator --type=json \ - -p='[{"op":"replace","path":"/spec/template/spec/volumes/0","value":{"hostPath":{"path":"/home/kubernetes/bin/nvidia/vgpu/containers","type":"DirectoryOrCreate"},"name":"containers"}}]' -``` - -完成以上四步后，reservation Pod 能成功初始化 NVML 并持有 GPU，共享 Pod 能成功调度并加载 `libvgpu.so`。验证： - -```bash -# reservation Pod 应该 Running（不再 CrashLoopBackOff） -kubectl get pod -n kai-resource-reservation -# 共享 Pod 里 libvgpu.so 应该成功 preload（没有 "cannot be preloaded" 错误） -kubectl exec -- cat /etc/ld.so.preload -# KAI 注入的显存限制 -kubectl exec -- env | grep CUDA_DEVICE_MEMORY_LIMIT -``` - -:::tip 实测验证 - -经实测验证，通过 Kyverno 自动给共享 Pod 挂载 GPU 设备节点 + NVIDIA 库目录 + privileged 模式，共享 Pod 可以成功运行 `nvidia-smi`，看到配额内的显存（4147 MiB 而非整卡 15360 MiB），隔离生效。具体做法见上面 Step 5b-3 的 `inject-gpu-devices` 策略与 Step 6、Step 7 Pod 模板里的 `securityContext.privileged: true`。 - -::: - -### Step 6：单 Pod，显存被「锁」住 - -先用一个 Pod 验证隔离本身能生效，再上共享场景，便于定位问题。 - -```yaml -# single-pod.yaml -apiVersion: v1 -kind: Pod -metadata: - name: gpu-isolation-demo - labels: - kai.scheduler/queue: default-queue - annotations: - gpu-memory: "4096" # 整数 MiB，不带后缀 -spec: - schedulerName: kai-scheduler - restartPolicy: Never - containers: - - name: gpu-workload - image: nvidia/cuda:12.9.2-base-ubuntu24.04 - command: ["sleep", "infinity"] - securityContext: - privileged: true -``` - -**为什么这样写**（对照 [KAI 官方文档](https://github.com/kai-scheduler/KAI-Scheduler/blob/main/docs/gpu-sharing/hami/README.md)的 Pod 模板）： - -- `schedulerName: kai-scheduler` 让 Pod 走 KAI 调度（默认是 `default-scheduler`）。 -- `kai.scheduler/queue: default-queue` 把 Pod 挂到 Step 4 建的队列上；标签对应不上的队列，Pod 不会被调度。 -- `gpu-memory: "4096"` 注解是这个 Pod 的显存配额，整数 MiB、不带单位后缀。KAI 会据此注入 `CUDA_DEVICE_MEMORY_LIMIT=4096`（MiB），HAMi-core 据此拦截。 -- **不要写 `resources.limits.nvidia.com/gpu`**。KAI 的 admission 会拒绝同时申请「整卡」和「显存配额」的 Pod，报 `cannot request both GPU and GPU memory`；显存共享完全由 `gpu-memory` 注解控制，device plugin 的整卡资源由 KAI 在背后分配。 - -```bash -kubectl apply -f single-pod.yaml -kubectl wait --for=condition=Ready pod/gpu-isolation-demo --timeout=300s -kubectl exec gpu-isolation-demo -- nvidia-smi -``` - -预期输出（实测）： - -``` -+-----------------------------------------------------------------------------+ -| NVIDIA-SMI ... Driver Version: ... CUDA Version: 12.9 | -|-------------------------------+----------------------+----------------------+ -| 0 NVIDIA T4 ... | 4147MiB / 4147MiB ... 0% Default | -+-------------------------------+----------------------+----------------------+ -``` - -整张 T4 本是 15360 MiB，容器里只看到约 4147 MiB（配额 4096 经 fraction 折算后的实际上限），说明 HAMi-core 改写了设备查询的返回值，隔离生效。如果这里仍看到 15360 MiB，多半是 isolator webhook 没生效（检查 Step 5 的 Pod 是否 Running、Pod 是否被 webhook 命名空间选择器覆盖）。 - -### Step 7：两个 Pod 共享一张 T4 - -隔离单 Pod 验证通过后，验证真正的共享：两个 Pod 同时落在同一张卡上，各自只见自己的配额。 - -```yaml -# shared-t4.yaml ---- -apiVersion: v1 -kind: Pod -metadata: - name: gpu-share-a - labels: - kai.scheduler/queue: default-queue - annotations: - gpu-memory: "4096" -spec: - schedulerName: kai-scheduler - restartPolicy: Never - containers: - - name: gpu-workload - image: nvidia/cuda:12.9.2-base-ubuntu24.04 - command: ["sleep", "infinity"] - securityContext: - privileged: true ---- -apiVersion: v1 -kind: Pod -metadata: - name: gpu-share-b - labels: - kai.scheduler/queue: default-queue - annotations: - gpu-memory: "4096" -spec: - schedulerName: kai-scheduler - restartPolicy: Never - containers: - - name: gpu-workload - image: nvidia/cuda:12.9.2-base-ubuntu24.04 - command: ["sleep", "infinity"] - securityContext: - privileged: true -``` - -两个 Pod 各申请 `4096` MiB，加起来 8 GiB，远小于单张 T4 的 15360 MiB，因此 KAI 会把它们调度到同一张卡上（而不是各占一卡）。 - -```bash -kubectl apply -f shared-t4.yaml -kubectl wait --for=condition=Ready pod/gpu-share-a pod/gpu-share-b --timeout=300s -kubectl exec gpu-share-a -- nvidia-smi --query-gpu=memory.total --format=csv,noheader -kubectl exec gpu-share-b -- nvidia-smi --query-gpu=memory.total --format=csv,noheader -``` - -预期输出（示意）：两个 Pod 都返回约 4147 MiB（配额 4096 经 fraction 折算后的上限）。 - -``` -4147 MiB -4147 MiB -``` - -再确认它们确实共享同一张卡，而不是各占一卡： - -```bash -kubectl get pod -o wide | grep gpu-share -``` - -预期输出（示意）：两个 Pod 的 `NODE` 列相同（落在同一个节点上），且这个节点只有 1 张 T4，证明它们共享的是同一张卡。 - -``` -NAME READY STATUS RESTARTS AGE IP NODE -gpu-share-a 1/1 Running 0 2m 10.x.x.x gke-test-cluster-default-pool-xxxxxxxx -gpu-share-b 1/1 Running 0 2m 10.x.x.x gke-test-cluster-default-pool-xxxxxxxx -``` - -**为什么是 `4096` 而不是更大**：T4 实际显存 15360 MiB，两个 Pod 各要 4096 MiB（共 8 GiB），留出近一半余量，KAI 调度起来更稳妥。`gpu-memory` 是整数 MiB，调度器会把它折算成两位小数的 GPU fraction 再落地为 `CUDA_DEVICE_MEMORY_LIMIT`。`4096 / 15360 ≈ 0.27`，实际看到的上限约为 4147 MiB 而非精确 4096（折算细节见坑 #4）。一张 15 GiB 的卡被干净地切成两份，互不越界，这就是硬隔离的效果。 - -### Step 8（可选）：查看 monitor 指标 - -```bash -kubectl -n kai-resource-isolator port-forward daemonset/kai-vgpu-monitor 9394:9394 & -curl -s localhost:9394/metrics | grep hami_vgpu_memory_limit_bytes -``` - -预期（示意）： - -``` -hami_vgpu_memory_limit_bytes{...} 4.147e+06 -``` - -**为什么用这个指标**：`hami_vgpu_memory_limit_bytes` 直接反映 HAMi-core 给每个容器设的显存上限，和 `nvidia-smi` 的数字对得上，是验证隔离最直接的观测点。配合 Prometheus 与 Grafana 后，每个容器的显存用量、上限与利用率都是现成的曲线。 - -## 几个值得提前知道的坑 +### 为什么完整步骤不放在博客里 -1. **`nvidia.com/gpu.memory` 节点标签必须先出现**（最重要）。KAI hamicore 在节点注册时缓存每卡显存，缺标签会导致共享 Pod 卡在 `Pending` 并报 `didn't have enough resources: GPU memory`。GKE 默认 device plugin **不打这个标签**（它只打 `nvidia.com/gpu.present`），所以本文 Step 3 手动补上。**补标签必须在装 KAI 之前完成**；万一装完 KAI 才补，重启 `kai-scheduler` 让它重新缓存即可。标签值要用 `nvidia-smi` 报的显存（本例 T4 是 15360，不是标称的 16384）。 -2. **队列必须手动创建**。Pod 靠 `kai.scheduler/queue` 标签指向队列，CRD 是 `scheduling.run.ai/v2` 的 `Queue`；队列不存在 Pod 就不会被调度。本文 Step 4 已提供清单，且要等 KAI admission webhook 就绪后再创建。 -3. **RuntimeClass `nvidia`**。KAI 的共享 reservation Pod 默认带 `runtimeClassName: nvidia`。GKE GPU 节点自带（COS 加 containerd 注册了 `nvidia` handler，真正注入 GPU），一般无需干预；自建集群若缺失，创建一个指向节点上真正执行 NVIDIA 注入的 runtime。 -4. **显存精度会四舍五入**。`gpu-memory` 是整数 MiB（无单位后缀），调度器会把它折算成两位小数的 GPU fraction 再落地为 `CUDA_DEVICE_MEMORY_LIMIT`。本文 Step 6、Step 7 在显存 15360 MiB 的 T4 上申请 `4096`，`4096 / 15360 ≈ 0.27`，实际看到的上限约为 4147 MiB 而非精确 4096（这正好是官方文档举的 T4 例子，见 [Memory value precision](https://github.com/kai-scheduler/KAI-Scheduler/blob/main/docs/gpu-sharing/hami/README.md#memory-value-precision)）。需要更精细的控制时，按 fraction 反推即可。 -5. **Chart 版本带 `-chart` 后缀**。`kai-resource-isolator` 的 OCI chart 标签是 `1.1.0-chart`，不是 `1.1.0` 也不是 `v1.1.0`。 -6. **想退出隔离也支持**：单个 Pod 加注解 `kai-resource-isolator.io/inject: "false"`；整个命名空间打标签 `kai-resource-isolator.io/webhook=ignore`，便于灰度迁移。 -7. **monitor 的 runtime 与 nodeSelector**。如果节点 NVML 必须走 NVIDIA runtime（部分自建集群），记得设置 `monitor.runtimeClassName=nvidia`；GKE GPU 节点默认满足。另外 monitor DaemonSet 默认靠 `nvidia.com/gpu.present=true` 这个 nodeSelector 选择节点；**本文 Step 3 手动补了 `gpu.present=true`，monitor Pod 能正常调度**。如果忘了打这个标签，monitor Pod 不会上来，补打或修改 nodeSelector 即可。 -8. **驱动是前置条件**。隔离栈依赖宿主机 NVIDIA 驱动。本文用 GKE 默认栈（建集群带 `gpu-driver-version=default`，GKE 自动装驱动 + device plugin + container toolkit）；自建集群请先用 NVIDIA Driver 加 container runtime 在节点上跑通 `nvidia-smi`。 -9. **GKE 上 isolator 的 DaemonSet 可能起不来（PriorityClass 配额）**。kai-resource-isolator 的 libsync 和 monitor DaemonSet 默认用 `system-node-critical` 优先级，而 GKE 不允许用户命名空间的 Pod 用这个 PriorityClass（报 `insufficient quota to match these scopes: [{PriorityClass In [system-node-critical ...]}]`）。如果遇到，把两个 DaemonSet 的 `priorityClassName` 去掉即可： +标准的 KAI + HAMi-core 链路并不长，但本次 GKE 1.35/COS/CDI 环境还涉及只读根文件系统、RuntimeClass、NVML 库路径、CDI 设备注入、PriorityClass 和 `kubectl exec` WebSocket 等兼容问题。这些内容版本相关、需要持续维护，更适合成为可执行的实验，而不是嵌在博客主线中。 - ```bash - for ds in kai-resource-isolator-libsync kai-resource-isolator-monitor; do - kubectl patch ds $ds -n kai-resource-isolator \ - --type=json -p='[{"op":"replace","path":"/spec/template/spec/priorityClassName","value":null}]' - done - ``` +完整的集群准备、安装清单、Kyverno 策略、CUDA 程序、monitor 验证、真实输出、故障排查和清理命令已整理到： - patch 后会触发滚动更新，Pod 重新调度上来。注意：`helm upgrade` 会把这个 patch 覆盖回去，所以每次升级 isolator 后要重新 patch（或给 isolator namespace 加一个允许 critical pods 的 ResourceQuota 作为更持久的解法）。 +**[实验 12：在 GKE 上验证 KAI Scheduler 与 HAMi 显存隔离](/zh/tutorials/labs/kai-scheduler-hami-gke)** -10. **共享 Pod 模板不要写 `resources.limits.nvidia.com/gpu`**。KAI 的 admission 会拒绝同时申请整卡和显存配额的 Pod（报 `cannot request both GPU and GPU memory`）。显存共享完全由 `gpu-memory` 注解控制，按 [KAI 官方 Pod 模板](https://github.com/kai-scheduler/KAI-Scheduler/blob/main/docs/gpu-sharing/hami/README.md)写即可，不加 `resources` 段。 -11. **GKE 上 KAI 的 config-deployer hook 可能失败**。`helm install kai-scheduler` 的 post-install hook（`kai-config-deployer` Job）在 GKE 1.35 上可能因 API server 认证问题（OpenAPI schema 下载报 `the server has asked for the client to provide credentials`）一直跑不完，导致 helm 报 `INSTALLATION FAILED`。但 KAI 的 CRD 和 `kai-operator` 其实已经装好了。最快的修复：手动 apply Config CR（用 `helm template` 渲染，注意 `cdiEnabled` 必须是带引号字符串 `"true"`，CRD 不接受 boolean），再 `kubectl apply --validate=false -f kai-config.yaml`；kai-operator 看到 Config CR 后会自动拉起所有组件。如果 release 状态卡在 `failed`，用 `helm upgrade ... --no-hooks` 修正状态（跳过 hook，幂等地重新确认资源）。 -12. **GKE 节点根文件系统只读，libsync 写不进 `/usr/local/vgpu`**。GKE 节点的 `/` 是 `ext2 ro` 挂载，isolator 的 libsync 默认把 `libvgpu.so` 写到 `/usr/local/vgpu`，会报 `Read-only file system` 而 CrashLoop。Step 5 的 `--set paths.hostInstallBase=/home/kubernetes/bin/nvidia` 就是解决这个（把库写到 GKE 的可写区）。注意：`paths.hostInstallBase` 只改了 libsync 的写入路径和 webhook 的 hostPath，但 chart 的 `ld.so.preload` ConfigMap 可能没跟着改，需要手动确认/修正（见 Step 5b-4）。 -13. **GKE 1.35 没有 `nvidia` RuntimeClass**。GKE 1.33+ 用 CDI 模式，节点没有注册 nvidia runtime handler。KAI 创建共享 Pod 时需要这个 RuntimeClass 存在，否则 Pod 被拒绝（`RuntimeClass "nvidia" not found`）。创建一个指向默认 `runc` handler 的即可（见 Step 5b-1）。自建集群如果已注册了真正的 nvidia runtime handler，则不需要。 -14. **reservation Pod 在 GKE CDI 模式下找不到 NVML 库**。GKE 把 NVML 库注入到容器的 `/usr/local/nvidia/lib64/`，但不设 `LD_LIBRARY_PATH`。`nvidia/cuda` 镜像自带这个变量所以能跑，但 KAI 的 reservation 镜像没有。解法是用 Kyverno MutatingWebhook 自动注入（见 Step 5b-3）。对应的 KAI issue：[#1449](https://github.com/kai-scheduler/kai-scheduler/issues/1449)。 -15. **GKE CDI 模式下共享 Pod 需要额外挂载 GPU 设备节点（已解决）**。GKE device plugin 只对 request 了 `nvidia.com/gpu` 的 Pod 注入设备；共享 Pod 用 `gpu-memory` 注解（无 resource request），所以 device plugin 不响应。解法：用 Kyverno `inject-gpu-devices` 策略自动挂载（见 Step 5b-3）+ Pod 加 `privileged: true`（HAMi-core 的 `libvgpu.so` 需要特权模式初始化 NVML，实测确认）。配齐这两项后，共享 Pod 能成功跑通 `nvidia-smi` 并看到配额内的显存（4147 MiB 而非整卡 15360 MiB）。 +如果只想理解集成原理，读到这里即可；如果要在 GKE 上复现，再进入“实验 12”按步骤操作。 ## 这意味着什么 -HAMi-core 的定位从来不是「某个调度器的附属隔离功能」，而是一个**与调度器解耦、可被各路调度器复用的隔离底座**。在 KAI Scheduler 之前，它已经支撑了 Kubernetes 原生调度器、[Kueue](/zh/docs/userguide/kueue/how-to-use-kueue)、[Volcano](/zh/docs/installation/how-to-use-volcano-vgpu)、Koordinator 等多条路径（见 [HAMi 生态集成](/zh/docs/next/core-concepts/ecosystem-integrations)）。KAI Scheduler v0.16.4 的加入，是把这个生态又扩到了 NVIDIA 官方的 AI 调度器： +HAMi-core 的定位从来不是“某个调度器的附属隔离功能”，而是一个**与调度器解耦、可被各路调度器复用的隔离底座**。在 KAI Scheduler 之前，它已经支撑了 Kubernetes 原生调度器、[Kueue](/zh/docs/userguide/kueue/how-to-use-kueue)、[Volcano](/zh/docs/installation/how-to-use-volcano-vgpu)、Koordinator 等多条路径（见 [HAMi 生态集成](/zh/docs/next/core-concepts/ecosystem-integrations)）。KAI Scheduler v0.16.4 的加入，是把这个生态又扩到了 NVIDIA 官方的 AI 调度器： - 对 **KAI 用户**：GPU 共享终于有了与之匹配的运行时硬隔离，共享不再等于缺少保障。 - 对 **HAMi 用户**：多了一条不绑死特定虚拟化设备插件、直接走 NVIDIA 官方调度器的集成路径，指标体系（`hami_*`）也保持兼容。 diff --git a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/kai-scheduler-hami-gke.md b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/kai-scheduler-hami-gke.md new file mode 100644 index 000000000..39c48a8cc --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/kai-scheduler-hami-gke.md @@ -0,0 +1,418 @@ +--- +title: "实验 12: 在 GKE 上验证 KAI Scheduler 与 HAMi 显存隔离" +description: "在 GKE 上部署 KAI Scheduler 与 kai-resource-isolator，适配 COS/CDI 运行时，并通过 CUDA 分配验证 Pod 级 GPU 显存隔离。" +sidebar_label: "实验 12: GKE 上的 KAI + HAMi" +lab: + level: Advanced + duration: 约 90 分钟 + environment: GKE 1.35、COS、containerd CDI 与 NVIDIA Tesla T4 + cost: 需要付费的 GKE GPU 节点 + authors: + - rootsongjc + verified: "2026-08-12" +tags: + - kai-scheduler + - hami-core + - gke + - gpu-sharing +toc_max_heading_level: 2 +--- + +本实验在 GKE 上部署 KAI Scheduler v0.17.0 与 `kai-resource-isolator` 1.1.0-chart，并验证共享一张 Tesla T4 的两个 Pod 都无法越过各自的显存配额。实验还记录了验证环境中实际遇到的 GKE 1.35/COS/CDI 兼容问题。 + +:::warning 环境特有的 workaround + +本文的 RuntimeClass、Kyverno、hostPath、PriorityClass 和特权容器调整只适用于验证过的 GKE 1.35/COS/CDI 路径，不是 KAI + HAMi-core 的标准安装方式。只有确认出现对应症状时才应用相应修复。 + +::: + +## 你将学到什么 + +- 启用 KAI GPU sharing 与 `hamicore` binder 插件； +- 让 `kai-resource-isolator` 适配 GKE 的只读根文件系统与 CDI 设备注入； +- 证明两个 Pod 使用同一张 GPU，并分别只看到 4147 MiB； +- 用 `cudaMalloc` 验证配额内成功、越界失败和两个 Pod 互不影响。 + +## 实验概览 + +```mermaid +%% title: GKE 上的 KAI Scheduler 与 HAMi-core 实验流程 +flowchart LR + S1["步骤 1
验证 GKE GPU"] --> S2["步骤 2
补充 GPU 标签"] + S2 --> S3["步骤 3
安装 KAI"] + S3 --> S4["步骤 4
安装 isolator"] + S4 --> S5["步骤 5
应用 GKE 适配"] + S5 --> S6["步骤 6
共享一张 T4"] + S6 --> S7["步骤 7
执行 CUDA OOM 验证"] + S7 --> S8["步骤 8
验证 monitor 指标"] +``` + +## 前提条件 + +- 已启用 GKE 与 Compute Engine API 的 GCP 项目。 +- 一套使用 COS 节点的 GKE 1.35 集群，至少有一张 NVIDIA T4。验证集群有三个 `n1-standard-2` 节点，每节点一张 T4。 +- 使用 GKE 管理的 NVIDIA 驱动、device plugin 与 container toolkit。不要在 GKE 驱动之上重复安装 GPU Operator。 +- 具有集群管理员权限的 `gcloud`、与 GKE API Server 相差不超过一个次版本的 `kubectl`，以及 Helm 3 或 4。 +- [`tutorials/labs/examples/12-kai-scheduler-hami-gke/`](https://github.com/Project-HAMi/website/tree/master/tutorials/labs/examples/12-kai-scheduler-hami-gke) 下的实验文件。 + +验证集群的控制面与节点版本均为 GKE `1.35.6-gke.1250000`。GKE 补丁版本会逐渐下线，因此先从目标可用区选择当前可用的 1.35 版本，再创建集群： + +```bash +export GKE_VERSION=$(gcloud container get-server-config \ + --zone=asia-northeast1-a \ + --format='value(validMasterVersions)' | tr ';' '\n' | grep '^1\.35\.' | head -1) +test -n "$GKE_VERSION" + +gcloud container clusters create kai-hami-test --zone=asia-northeast1-a \ + --cluster-version="$GKE_VERSION" \ + --machine-type=n1-standard-2 --num-nodes=3 \ + --image-type=COS_CONTAINERD \ + --accelerator=type=nvidia-tesla-t4,count=1,gpu-driver-version=default +gcloud container clusters get-credentials kai-hami-test \ + --zone=asia-northeast1-a +``` + +GPU 节点会产生费用，完成实验后请执行清理步骤。 + +## 步骤 1: 验证 GKE GPU 栈 + +确认 GPU 节点已经上报 `nvidia.com/gpu`： + +```bash +kubectl get nodes \ + -o custom-columns="NAME:.metadata.name,GPU:.status.capacity.nvidia\.com/gpu,ACCEL:.metadata.labels.cloud\.google\.com/gke-accelerator" +``` + +验证集群包含三个 T4 节点： + +```plaintext +NAME GPU ACCEL +gke-kai-hami-test-default-pool-... 1 nvidia-tesla-t4 +gke-kai-hami-test-default-pool-... 1 nvidia-tesla-t4 +gke-kai-hami-test-default-pool-... 1 nvidia-tesla-t4 +``` + +如果 `GPU` 为空，且节点带有 `gke-no-default-nvidia-gpu-device-plugin=true`，启用 GKE device plugin： + +```bash +kubectl label node -l cloud.google.com/gke-accelerator=nvidia-tesla-t4 \ + gke-no-default-nvidia-gpu-device-plugin- +``` + +安装 KAI 前先运行普通整卡 Pod： + +```bash +kubectl apply -f - <<'EOF' +apiVersion: v1 +kind: Pod +metadata: + name: gpu-smi-test +spec: + restartPolicy: Never + containers: + - name: cuda + image: nvidia/cuda:12.4.1-base-ubuntu22.04 + command: ["nvidia-smi"] + resources: + limits: + nvidia.com/gpu: 1 +EOF +kubectl wait --for=jsonpath='{.status.phase}'=Succeeded \ + pod/gpu-smi-test --timeout=180s +kubectl logs gpu-smi-test +kubectl delete pod gpu-smi-test +``` + +验证环境中的 T4 报告 15360 MiB 可寻址显存。 + +## 步骤 2: 添加 KAI 所需的 GPU 标签 + +GKE 默认 device plugin 不提供完整的 GPU Feature Discovery 标签。KAI 在注册节点时读取 `nvidia.com/gpu.memory`，因此要在安装 KAI 前添加： + +```bash +kubectl label node -l cloud.google.com/gke-accelerator=nvidia-tesla-t4 \ + nvidia.com/gpu.memory=15360 \ + nvidia.com/gpu.product=NVIDIA-Tesla-T4 \ + nvidia.com/gpu.count=1 \ + nvidia.com/gpu.present=true --overwrite +``` + +显存值应来自 `nvidia-smi`，不要使用 T4 标称的 16 GiB。如果 KAI 启动后才补标签，需要重启 `kai-scheduler` 刷新节点缓存。 + +## 步骤 3: 安装 KAI Scheduler 与队列 + +安装 KAI v0.17.0，启用 GPU sharing、HAMi-core 集成和 CDI： + +```bash +helm install kai-scheduler \ + oci://ghcr.io/kai-scheduler/kai-scheduler/kai-scheduler \ + --namespace kai-scheduler --create-namespace \ + --version v0.17.0 \ + --set global.gpuSharing=true \ + --set binder.plugins.hamicore.enabled=true \ + --set-string binder.plugins.gpusharing.arguments.cdiEnabled=true + +kubectl -n kai-scheduler wait --for=condition=available \ + --timeout=180s deploy --all +kubectl -n kai-scheduler wait --for=condition=Ready \ + --timeout=300s config/kai-config +kubectl apply \ + -f tutorials/labs/examples/12-kai-scheduler-hami-gke/01-queues.yaml +kubectl get queues +``` + +应出现两个队列： + +```plaintext +NAME PARENT +default +default-queue default +``` + +:::important `cdiEnabled` 必须是字符串 + +这里必须使用 `--set-string`，不能使用普通的 `--set`。KAI v0.17.0 的 Helm values 虽然能接受两种写法，但生成的 `Config` CRD 字段类型是字符串。普通的 `--set ...=true` 会渲染成布尔值，使 `kai-config-deployer` hook 报错：`cdiEnabled ... must be of type string: "boolean"`。上面的正确命令已经过 Chart 渲染，并通过当前 GKE API Server 的 server-side dry-run。 + +::: + +## 步骤 4: 安装 kai-resource-isolator + +COS 的根文件系统只读，因此将 HAMi-core 写入 GKE 可写的 NVIDIA 目录： + +```bash +helm install kai-resource-isolator \ + oci://docker.io/projecthami/kai-resource-isolator \ + --namespace kai-resource-isolator --create-namespace \ + --version 1.1.0-chart \ + --set paths.containerVgpuMount=/home/kubernetes/bin/nvidia/vgpu \ + --set-string librarySync.priorityClassName= \ + --set-string monitor.priorityClassName= \ + --set monitor.enabled=true +``` + +`containerVgpuMount` 是 1.1.0-chart 中真正控制 libsync 目标目录、preload 文件、webhook 注入路径和 monitor 缓存的值。不要使用 `paths.hostInstallBase`：它虽然出现在该 Chart 的 values 中，但模板没有引用它。两个空 PriorityClass 值用于避免 GKE 拒绝系统命名空间之外使用 `system-node-critical` 的 Pod。 + +检查渲染后的路径并等待组件就绪： + +```bash +kubectl get cm kai-resource-isolator-ldpreload \ + -n kai-resource-isolator -o jsonpath='{.data.ld\.so\.preload}' +kubectl rollout status ds/kai-resource-isolator-libsync \ + -n kai-resource-isolator --timeout=300s +kubectl rollout status ds/kai-resource-isolator-monitor \ + -n kai-resource-isolator --timeout=300s +kubectl get pods -n kai-resource-isolator +``` + +ConfigMap 输出必须为 `/home/kubernetes/bin/nvidia/vgpu/libvgpu.so`。 + +## 步骤 5: 适配 GKE CDI 设备路径 + +验证集群使用 CDI，没有注册 `nvidia` runtime handler，但 KAI reservation Pod 引用了 `runtimeClassName: nvidia`。只有 RuntimeClass 不存在时才创建兼容对象： + +```bash +kubectl get runtimeclass nvidia || kubectl apply \ + -f tutorials/labs/examples/12-kai-scheduler-hami-gke/02-runtimeclass.yaml +``` + +GKE device plugin 只向申请 `nvidia.com/gpu` 的 Pod 注入设备；KAI 共享 Pod 使用 `gpu-memory`，因此验证环境需要显式挂载设备与库。安装 Kyverno 并应用两条策略： + +```bash +helm repo add kyverno https://kyverno.github.io/kyverno/ +helm repo update +helm install kyverno kyverno/kyverno \ + --namespace kyverno --create-namespace +kubectl wait -n kyverno --for=condition=Ready pod \ + -l app.kubernetes.io/component=admission-controller --timeout=300s +kubectl apply \ + -f tutorials/labs/examples/12-kai-scheduler-hami-gke/03-gke-policies.yaml +``` + +第一条策略给 reservation Pod 添加 NVML 库路径；第二条给共享 Pod 挂载 `/dev/nvidia*`、`nvidia-smi` 与 NVIDIA 库。使用这条 workaround 时，共享 Pod 还需要 `privileged: true`。 + +:::caution 安全边界 + +本实验验证 CUDA API 层的显存限制，不是 MIG 一类硬件安全边界。GKE workaround 还使用了特权业务容器，不应将其作为不可信多租户安全方案。 + +::: + +## 步骤 6: 让两个 Pod 共享一张 T4 + +选择一台没有其他共享工作负载、且只有一张 T4 的节点。不要选择已经运行 `gpu-memory` 工作负载的节点：两个 4 GiB 实验 Pod 需要该卡至少还有 8 GiB 未被 KAI 分配。先列出候选节点与当前 Pod 分布： + +```bash +kubectl get nodes -l cloud.google.com/gke-accelerator=nvidia-tesla-t4 +kubectl get pods -A -o wide + +# 将下面的值替换为上面输出中的一台空闲单卡 T4 节点。 +export TEST_NODE= +test "$(kubectl get node "$TEST_NODE" \ + -o jsonpath='{.status.capacity.nvidia\.com/gpu}')" = "1" +kubectl label node "$TEST_NODE" hami.run/lab-12=true --overwrite + +kubectl create configmap kai-hami-lab12-source \ + --from-file=memory-limit.cu=tutorials/labs/examples/12-kai-scheduler-hami-gke/memory-limit.cu \ + --dry-run=client -o yaml | kubectl apply -f - +kubectl apply \ + -f tutorials/labs/examples/12-kai-scheduler-hami-gke/04-shared-pods.yaml +kubectl wait --for=condition=Ready \ + pod/kai-hami-lab12-a pod/kai-hami-lab12-b \ + --timeout=10m +``` + +检查节点位置，并从两个 Pod 的启动日志读取注入配额、物理 UUID 与可见显存： + +```bash +kubectl get pod kai-hami-lab12-a kai-hami-lab12-b -o wide +for pod in kai-hami-lab12-a kai-hami-lab12-b; do + echo "=== $pod ===" + kubectl logs "$pod" +done +``` + +实测时两个 Pod 都返回： + +```plaintext +limit=4147m +GPU-9acc8878-3967-5fb4-c534-43d6fd820fa6, 4147 MiB +``` + +相同 UUID 证明两个 Pod 使用同一张 T4。4147 MiB 是 KAI 把 15360 MiB 卡上的 4096 MiB 请求换算为两位小数 fraction 后的舍入结果。 + +## 步骤 7: 验证 CUDA 显存上限 + +步骤 6 创建的源码 ConfigMap 已挂载到 `/lab-source`。每个 Pod 启动时会自动编译程序，成功申请 3 GiB 后保持 30 秒，再尝试追加 2 GiB。验证通过后，Pod 会再次启动程序并持续持有 3 GiB，供步骤 8 验证实时监控指标。两个阶段均成功启动后，readiness probe 才会成功。 + +查看两个 Pod 日志中的测试区间与结果： + +```bash +for pod in kai-hami-lab12-a kai-hami-lab12-b; do + kubectl logs "$pod" | grep -E \ + 'test_(start|end)=|allocate |PASS:' +done +``` + +两个 Pod 的实测输出均为： + +```plaintext +allocate 3 GiB: no error +allocate another 2 GiB: out of memory +PASS: in-quota allocation succeeded and over-quota allocation failed +``` + +比较 `test_start` 与 `test_end`，两个 30 秒区间必须重叠。实测时两个 Pod 在同时持有 3 GiB 的情况下都返回 `PASS`。各自累计申请 5 GiB 时，HAMi-core 都记录 `Device 0 OOM 5475663872 / 4348444672`。将验证程序作为容器启动命令执行，也避免了长时间 `kubectl exec` WebSocket 中断影响判断。 + +## 步骤 8: 验证 monitor 指标 + +monitor 是 DaemonSet：每个实例只读取本节点上的 HAMi 共享内存缓存。通过 Service 访问时可能被转发到其他节点的 monitor，从而看不到这两个 Pod 的序列。因此要直接查询工作负载所在节点的 monitor 实例。这里重新从运行中的 Pod 获取节点，不再依赖步骤 6 所在 shell 导出的临时变量。 + +创建一个临时 curl Pod，读取该 monitor 的 `:9394/metrics`： + +```bash +export TEST_NODE=$(kubectl get pod kai-hami-lab12-a \ + -o jsonpath='{.spec.nodeName}') +test -n "$TEST_NODE" + +export MONITOR_IP=$(kubectl get pods -n kai-resource-isolator \ + -l app.kubernetes.io/component=kai-vgpu-monitor \ + --field-selector="spec.nodeName=$TEST_NODE" \ + -o jsonpath='{range .items[*]}{.status.podIP}{"\n"}{end}' | head -n 1) + +if test -z "$MONITOR_IP"; then + echo "$TEST_NODE 上没有运行 monitor Pod" >&2 + kubectl get pods -n kai-resource-isolator \ + -l app.kubernetes.io/component=kai-vgpu-monitor -o wide + exit 1 +fi + +kubectl delete pod lab12-monitor-check --ignore-not-found +kubectl run lab12-monitor-check \ + --image=curlimages/curl:8.15.0 --restart=Never \ + --command -- sh -lc \ + "curl -fsS http://$MONITOR_IP:9394/metrics" +kubectl wait --for=jsonpath='{.status.phase}'=Succeeded \ + pod/lab12-monitor-check --timeout=180s +kubectl logs lab12-monitor-check | grep -E \ + '^hami_vgpu_memory_(used|limit)_bytes.*pod="kai-hami-lab12-[ab]"' +``` + +实测端点为每个 Pod 分别返回一条 `used` 和 `limit` 序列： + +```plaintext +hami_vgpu_memory_limit_bytes{...,pod="kai-hami-lab12-a",...} 4.348444672e+09 +hami_vgpu_memory_limit_bytes{...,pod="kai-hami-lab12-b",...} 4.348444672e+09 +hami_vgpu_memory_used_bytes{...,pod="kai-hami-lab12-a",...} 3.328180224e+09 +hami_vgpu_memory_used_bytes{...,pod="kai-hami-lab12-b",...} 3.328180224e+09 +``` + +limit 等于 4147 MiB 对应的字节数；used 略高于 3 GiB，因为其中包含 CUDA context 与分配器开销。这证明 monitor 确实发现了两个容器的缓存并导出了实时用量，而不是只提供一个没有业务序列的 Prometheus 端点。 + +## 故障排查 + +| 现象 | 验证环境中的原因 | 处理方式 | +| :-- | :-- | :-- | +| 共享 Pod Pending，提示 `didn't have enough resources: GPU memory` | KAI 缓存节点时缺少 `nvidia.com/gpu.memory` | 补标签并重启 `kai-scheduler` | +| Queue 被拒绝 | KAI admission webhook 尚未就绪 | 等待 KAI Deployment 后再创建队列 | +| `RuntimeClass "nvidia" not found` | GKE CDI 使用 `runc`，没有 NVIDIA handler | 应用 `02-runtimeclass.yaml` | +| reservation Pod 报 `ERROR_LIBRARY_NOT_FOUND` | NVML 位于 `/usr/local/nvidia/lib64`，但不在搜索路径 | 应用 Kyverno 库路径策略 | +| 共享 Pod 看不到 `/dev/nvidia*` | 它只申请 `gpu-memory`，GKE device plugin 不执行 Allocate | 应用 Kyverno 设备挂载策略 | +| libsync 报 `Read-only file system` | COS 根文件系统只读 | 设置 `paths.containerVgpuMount=/home/kubernetes/bin/nvidia/vgpu` | +| `libvgpu.so` 无法 preload | 实际挂载路径仍指向 `/usr/local/vgpu` | 使用验证过的 `containerVgpuMount` 值重新安装 | +| DaemonSet 因 `system-node-critical` 被拒绝 | GKE PriorityClass 配额限制用户命名空间 | 将 Chart 中两个 PriorityClass 值设为空字符串 | +| `CUDA driver version is insufficient` | CUDA 镜像版本超过节点驱动支持范围 | 使用实测的 CUDA 12.4.1 或其他兼容版本 | +| `kubectl exec` 出现 WebSocket EOF | 控制面 exec 流被重置，或客户端/服务端版本偏差不受支持 | 使用兼容的 `kubectl`；本实验在容器启动时执行验证，并通过 `kubectl logs` 读取 | +| monitor 查询不到 Pod 或没有实验序列 | `$TEST_NODE` 已失效，或工作负载节点上没有 monitor | 从 `kai-hami-lab12-a` 重新获取节点，再用 `kubectl get pods -n kai-resource-isolator -l app.kubernetes.io/component=kai-vgpu-monitor -o wide` 检查 DaemonSet | + +## 清理 + +删除业务 Pod 与测试标签： + +```bash +kubectl delete \ + -f tutorials/labs/examples/12-kai-scheduler-hami-gke/04-shared-pods.yaml +kubectl delete pod lab12-monitor-check --ignore-not-found +kubectl delete configmap kai-hami-lab12-source --ignore-not-found +kubectl label node "$TEST_NODE" hami.run/lab-12- --overwrite +``` + +如果集群只用于本实验，再删除其余组件： + +```bash +kubectl delete \ + -f tutorials/labs/examples/12-kai-scheduler-hami-gke/03-gke-policies.yaml +helm uninstall kyverno -n kyverno +helm uninstall kai-resource-isolator -n kai-resource-isolator +kubectl delete \ + -f tutorials/labs/examples/12-kai-scheduler-hami-gke/01-queues.yaml +helm uninstall kai-scheduler -n kai-scheduler +``` + +只有步骤 5 创建了 RuntimeClass 时才删除 `02-runtimeclass.yaml`；如果该 RuntimeClass 原本就属于集群，应予以保留： + +```bash +kubectl delete \ + -f tutorials/labs/examples/12-kai-scheduler-hami-gke/02-runtimeclass.yaml +``` + +如果 GKE 集群专门为本实验创建，可将其删除： + +```bash +gcloud container clusters delete kai-hami-test \ + --zone=asia-northeast1-a +``` + +## 本实验验证了什么 + +| 结论 | 证据 | +| :-- | :-- | +| KAI 把两个分片工作负载调度到同一张 T4 | 两个 Pod 节点相同、GPU UUID 相同 | +| HAMi-core 改写每个 Pod 可见的显存上限 | 两个 Pod 均显示 4147 MiB，而非 15360 MiB | +| 上限被真正执行，而非只修改显示 | 3 GiB 成功，累计 5 GiB 返回 CUDA OOM | +| 一个 Pod 无法占用另一个 Pod 的配额 | Pod A 持有 3 GiB 时，Pod B 仍成功分配 3 GiB | +| 可选 monitor 能读取每容器缓存 | `:9394/metrics` 返回两个 Pod 的 4147 MiB 上限与实时 3 GiB 用量 | + +## 下一步 + +- 阅读[《KAI Scheduler 与 HAMi 的 GPU 显存硬隔离》](/zh/blog/kai-scheduler-hami-gpu-memory-hard-isolation)，了解架构与集成背景。 +- 对比[实验 3：HAMi GPU 切分](./gpu-partitioning.md)和[实验 7：k3s GPU 隔离](./hami-isolation-k3s.md)。 +- 关注 KAI 与 `kai-resource-isolator` 后续版本；当 GKE CDI 获得原生支持后，删除本实验中对应的 workaround。 diff --git a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/overview.md b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/overview.md index 32683f4b7..80667767d 100644 --- a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/overview.md @@ -19,4 +19,4 @@ import LabCardGridAuto from '@site/src/components/labs/LabCardGridAuto'; -每个实验都列出了各自的前提条件。实验 3 和 4 直接复用实验 1 搭建的集群，一次开机即可完成全部三个实验；实验 2 可在任意笔记本上运行，无需 GPU。实验 7 在租用的 GPU 虚拟机上自行搭建单节点 k3s 集群，不使用 GPU Operator。实验 8 需要已有的 Volcano GPU 集群，用于验证 Volcano vGPU、Gang 调度和队列级资源限制。实验 9 使用 Kueue 准入控制限制 HAMi vGPU 数量、显存和算力配额。实验 11 将从头搭建完整的 KServe Standard 推理环境，并通过 HAMi 原生 DRA Claim 让两个 vLLM 副本共享一张 GPU。 +每个实验都列出了各自的前提条件。实验 3 和 4 直接复用实验 1 搭建的集群，一次开机即可完成全部三个实验；实验 2 可在任意笔记本上运行，无需 GPU。实验 7 在租用的 GPU 虚拟机上自行搭建单节点 k3s 集群，不使用 GPU Operator。实验 8 需要已有的 Volcano GPU 集群，用于验证 Volcano vGPU、Gang 调度和队列级资源限制。实验 9 使用 Kueue 准入控制限制 HAMi vGPU 数量、显存和算力配额。实验 11 将从头搭建完整的 KServe Standard 推理环境，并通过 HAMi 原生 DRA Claim 让两个 vLLM 副本共享一张 GPU。实验 12 在 GKE 1.35/COS/CDI 上部署 KAI Scheduler 与 HAMi-core，并通过 CUDA 分配验证显存上限。 diff --git a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/tags.yml b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/tags.yml index 033e625ae..eafe5c2ab 100644 --- a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/tags.yml +++ b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/tags.yml @@ -10,15 +10,27 @@ "gang-scheduling": label: "gang-scheduling" permalink: "/gang-scheduling" +"gke": + label: "GKE" + permalink: "/gke" +"gpu-sharing": + label: "GPU 共享" + permalink: "/gpu-sharing" "hami": label: "hami" permalink: "/hami" +"hami-core": + label: "HAMi-core" + permalink: "/hami-core" "inference": label: "inference" permalink: "/inference" "k3s": label: "k3s" permalink: "/k-3-s" +"kai-scheduler": + label: "KAI Scheduler" + permalink: "/kai-scheduler" "kueue": label: "Kueue" permalink: "/kueue" diff --git a/sidebars-tutorials.js b/sidebars-tutorials.js index a8db41ff3..9e28572c9 100644 --- a/sidebars-tutorials.js +++ b/sidebars-tutorials.js @@ -66,6 +66,11 @@ module.exports = { id: "labs/kserve-hami-dra", customProps: { level: "Advanced", duration: "about 90 minutes" }, }, + { + type: "doc", + id: "labs/kai-scheduler-hami-gke", + customProps: { level: "Advanced", duration: "about 90 minutes" }, + }, ], }, ], diff --git a/tutorials/labs/examples/12-kai-scheduler-hami-gke/01-queues.yaml b/tutorials/labs/examples/12-kai-scheduler-hami-gke/01-queues.yaml new file mode 100644 index 000000000..a2cb72075 --- /dev/null +++ b/tutorials/labs/examples/12-kai-scheduler-hami-gke/01-queues.yaml @@ -0,0 +1,20 @@ +apiVersion: scheduling.run.ai/v2 +kind: Queue +metadata: + name: default +spec: + resources: + cpu: { quota: -1, limit: -1, overQuotaWeight: 1 } + memory: { quota: -1, limit: -1, overQuotaWeight: 1 } + gpu: { quota: -1, limit: -1, overQuotaWeight: 1 } +--- +apiVersion: scheduling.run.ai/v2 +kind: Queue +metadata: + name: default-queue +spec: + parentQueue: default + resources: + cpu: { quota: -1, limit: -1, overQuotaWeight: 1 } + memory: { quota: -1, limit: -1, overQuotaWeight: 1 } + gpu: { quota: -1, limit: -1, overQuotaWeight: 1 } diff --git a/tutorials/labs/examples/12-kai-scheduler-hami-gke/02-runtimeclass.yaml b/tutorials/labs/examples/12-kai-scheduler-hami-gke/02-runtimeclass.yaml new file mode 100644 index 000000000..5a36705dd --- /dev/null +++ b/tutorials/labs/examples/12-kai-scheduler-hami-gke/02-runtimeclass.yaml @@ -0,0 +1,5 @@ +apiVersion: node.k8s.io/v1 +kind: RuntimeClass +metadata: + name: nvidia +handler: runc diff --git a/tutorials/labs/examples/12-kai-scheduler-hami-gke/03-gke-policies.yaml b/tutorials/labs/examples/12-kai-scheduler-hami-gke/03-gke-policies.yaml new file mode 100644 index 000000000..469af90df --- /dev/null +++ b/tutorials/labs/examples/12-kai-scheduler-hami-gke/03-gke-policies.yaml @@ -0,0 +1,88 @@ +apiVersion: kyverno.io/v1 +kind: ClusterPolicy +metadata: + name: inject-nvidia-library-path +spec: + rules: + - name: add-ld-library-path + match: + any: + - resources: + kinds: [Pod] + namespaces: [kai-resource-reservation] + mutate: + foreach: + - list: "request.object.spec.containers[]" + patchStrategicMerge: + spec: + containers: + - name: "{{ element.name }}" + env: + - name: LD_LIBRARY_PATH + value: /usr/local/nvidia/lib64 +--- +apiVersion: kyverno.io/v1 +kind: ClusterPolicy +metadata: + name: inject-gpu-devices +spec: + background: false + rules: + - name: add-gpu-volumes + match: + any: + - resources: + kinds: [Pod] + preconditions: + all: + - key: '{{ request.object.metadata.annotations."gpu-memory" || '''' }}' + operator: NotEquals + value: "" + mutate: + patchesJson6902: |- + - op: add + path: /spec/volumes/- + value: {name: nvidia-dev0, hostPath: {path: /dev/nvidia0, type: CharDevice}} + - op: add + path: /spec/volumes/- + value: {name: nvidia-ctl, hostPath: {path: /dev/nvidiactl, type: CharDevice}} + - op: add + path: /spec/volumes/- + value: {name: nvidia-uvm, hostPath: {path: /dev/nvidia-uvm, type: CharDevice}} + - op: add + path: /spec/volumes/- + value: {name: nvidia-modeset, hostPath: {path: /dev/nvidia-modeset, type: CharDevice}} + - op: add + path: /spec/volumes/- + value: {name: nvidia-caps, hostPath: {path: /dev/nvidia-caps, type: Directory}} + - op: add + path: /spec/volumes/- + value: {name: nvidia-bin, hostPath: {path: /home/kubernetes/bin/nvidia/bin, type: Directory}} + - op: add + path: /spec/volumes/- + value: {name: nvidia-lib, hostPath: {path: /home/kubernetes/bin/nvidia/lib64, type: Directory}} + - name: add-gpu-volume-mounts + match: + any: + - resources: + kinds: [Pod] + preconditions: + all: + - key: '{{ request.object.metadata.annotations."gpu-memory" || '''' }}' + operator: NotEquals + value: "" + mutate: + foreach: + - list: "request.object.spec.containers[]" + patchStrategicMerge: + spec: + containers: + - name: "{{ element.name }}" + volumeMounts: + - { name: nvidia-dev0, mountPath: /dev/nvidia0 } + - { name: nvidia-ctl, mountPath: /dev/nvidiactl } + - { name: nvidia-uvm, mountPath: /dev/nvidia-uvm } + - { name: nvidia-modeset, mountPath: /dev/nvidia-modeset } + - { name: nvidia-caps, mountPath: /dev/nvidia-caps } + - { name: nvidia-bin, mountPath: /usr/local/nvidia/bin } + - { name: nvidia-lib, mountPath: /usr/local/nvidia/lib64 } diff --git a/tutorials/labs/examples/12-kai-scheduler-hami-gke/04-shared-pods.yaml b/tutorials/labs/examples/12-kai-scheduler-hami-gke/04-shared-pods.yaml new file mode 100644 index 000000000..18d67829e --- /dev/null +++ b/tutorials/labs/examples/12-kai-scheduler-hami-gke/04-shared-pods.yaml @@ -0,0 +1,111 @@ +apiVersion: v1 +kind: Pod +metadata: + name: kai-hami-lab12-a + labels: + kai.scheduler/queue: default-queue + annotations: + gpu-memory: "4096" +spec: + schedulerName: kai-scheduler + restartPolicy: Never + nodeSelector: + hami.run/lab-12: "true" + containers: + - name: gpu-workload + image: nvidia/cuda:12.4.1-devel-ubuntu22.04 + command: ["/bin/bash", "-lc"] + args: + - | + nvcc -O2 /lab-source/memory-limit.cu -o /tmp/memory-limit-test + printf 'limit=%s\n' "$CUDA_DEVICE_MEMORY_LIMIT" + /usr/local/nvidia/bin/nvidia-smi \ + --query-gpu=uuid,memory.total --format=csv,noheader + echo "test_start=$(date -u +%FT%TZ)" + HOLD_SECONDS=30 /tmp/memory-limit-test + echo "test_end=$(date -u +%FT%TZ)" + (HOLD_SECONDS=3600 /tmp/memory-limit-test \ + > /tmp/monitor-hold.log 2>&1) & + for attempt in $(seq 1 30); do + grep -q 'allocate another 2 GiB: out of memory' \ + /tmp/monitor-hold.log && break + sleep 1 + done + grep -q 'allocate another 2 GiB: out of memory' \ + /tmp/monitor-hold.log + touch /tmp/memory-test-passed + exec sleep infinity + env: + - name: LD_LIBRARY_PATH + value: /usr/local/nvidia/lib64:/usr/local/cuda/lib64 + securityContext: + privileged: true + readinessProbe: + exec: + command: ["test", "-f", "/tmp/memory-test-passed"] + periodSeconds: 2 + failureThreshold: 150 + volumeMounts: + - name: lab-source + mountPath: /lab-source + readOnly: true + volumes: + - name: lab-source + configMap: + name: kai-hami-lab12-source +--- +apiVersion: v1 +kind: Pod +metadata: + name: kai-hami-lab12-b + labels: + kai.scheduler/queue: default-queue + annotations: + gpu-memory: "4096" +spec: + schedulerName: kai-scheduler + restartPolicy: Never + nodeSelector: + hami.run/lab-12: "true" + containers: + - name: gpu-workload + image: nvidia/cuda:12.4.1-devel-ubuntu22.04 + command: ["/bin/bash", "-lc"] + args: + - | + nvcc -O2 /lab-source/memory-limit.cu -o /tmp/memory-limit-test + printf 'limit=%s\n' "$CUDA_DEVICE_MEMORY_LIMIT" + /usr/local/nvidia/bin/nvidia-smi \ + --query-gpu=uuid,memory.total --format=csv,noheader + echo "test_start=$(date -u +%FT%TZ)" + HOLD_SECONDS=30 /tmp/memory-limit-test + echo "test_end=$(date -u +%FT%TZ)" + (HOLD_SECONDS=3600 /tmp/memory-limit-test \ + > /tmp/monitor-hold.log 2>&1) & + for attempt in $(seq 1 30); do + grep -q 'allocate another 2 GiB: out of memory' \ + /tmp/monitor-hold.log && break + sleep 1 + done + grep -q 'allocate another 2 GiB: out of memory' \ + /tmp/monitor-hold.log + touch /tmp/memory-test-passed + exec sleep infinity + env: + - name: LD_LIBRARY_PATH + value: /usr/local/nvidia/lib64:/usr/local/cuda/lib64 + securityContext: + privileged: true + readinessProbe: + exec: + command: ["test", "-f", "/tmp/memory-test-passed"] + periodSeconds: 2 + failureThreshold: 150 + volumeMounts: + - name: lab-source + mountPath: /lab-source + readOnly: true + volumes: + - name: lab-source + configMap: + name: kai-hami-lab12-source diff --git a/tutorials/labs/examples/12-kai-scheduler-hami-gke/memory-limit.cu b/tutorials/labs/examples/12-kai-scheduler-hami-gke/memory-limit.cu new file mode 100644 index 000000000..db9271778 --- /dev/null +++ b/tutorials/labs/examples/12-kai-scheduler-hami-gke/memory-limit.cu @@ -0,0 +1,37 @@ +#include + +#include +#include +#include +#include + +int main() { + constexpr size_t GiB = 1024ULL * 1024ULL * 1024ULL; + void *within_limit = nullptr; + void *over_limit = nullptr; + + cudaError_t first = cudaMalloc(&within_limit, 3 * GiB); + std::cout << "allocate 3 GiB: " << cudaGetErrorString(first) << std::endl; + if (first != cudaSuccess) return 1; + + cudaError_t second = cudaMalloc(&over_limit, 2 * GiB); + std::cout << "allocate another 2 GiB: " << cudaGetErrorString(second) + << std::endl; + if (second == cudaSuccess) { + std::cerr << "ERROR: allocation exceeded the container quota" << std::endl; + cudaFree(over_limit); + cudaFree(within_limit); + return 2; + } + + const char *hold = std::getenv("HOLD_SECONDS"); + if (hold) { + std::this_thread::sleep_for(std::chrono::seconds(std::atoi(hold))); + } + + cudaFree(within_limit); + std::cout + << "PASS: in-quota allocation succeeded and over-quota allocation failed" + << std::endl; + return 0; +} diff --git a/tutorials/labs/kai-scheduler-hami-gke.md b/tutorials/labs/kai-scheduler-hami-gke.md new file mode 100644 index 000000000..2e3a245c6 --- /dev/null +++ b/tutorials/labs/kai-scheduler-hami-gke.md @@ -0,0 +1,418 @@ +--- +title: "Lab 12: Verify KAI Scheduler and HAMi Memory Isolation on GKE" +description: "Deploy KAI Scheduler and kai-resource-isolator on GKE, adapt the COS/CDI runtime path, and prove per-Pod GPU memory isolation with CUDA allocations." +sidebar_label: "Lab 12: KAI + HAMi on GKE" +lab: + level: Advanced + duration: about 90 minutes + environment: GKE 1.35 with COS, containerd CDI, and NVIDIA Tesla T4 GPUs + cost: requires billable GKE nodes with attached GPUs + authors: + - rootsongjc + verified: "2026-08-12" +tags: + - kai-scheduler + - hami-core + - gke + - gpu-sharing +toc_max_heading_level: 2 +--- + +This lab deploys KAI Scheduler v0.17.0 and `kai-resource-isolator` 1.1.0-chart on GKE, then proves that two Pods sharing one Tesla T4 cannot allocate beyond their individual memory quotas. It also documents the GKE 1.35/COS/CDI compatibility workarounds observed in the verified environment. + +:::warning Environment-specific workarounds + +The RuntimeClass, Kyverno, host path, PriorityClass, and privileged-container changes in this lab are specific to the verified GKE 1.35/COS/CDI path. They are not the standard KAI + HAMi-core installation. Apply each workaround only after confirming the matching symptom. + +::: + +## What You'll Learn + +- enable KAI GPU sharing and its `hamicore` binder plugin; +- adapt `kai-resource-isolator` to GKE's read-only root filesystem and CDI device injection; +- prove that two Pods use the same physical GPU and see separate 4147 MiB ceilings; and +- use `cudaMalloc` to prove in-quota success, over-quota failure, and cross-Pod independence. + +## Lab Overview + +```mermaid +%% title: KAI Scheduler and HAMi-core on GKE +flowchart LR + S1["Step 1
Verify GKE GPU"] --> S2["Step 2
Add GPU labels"] + S2 --> S3["Step 3
Install KAI"] + S3 --> S4["Step 4
Install isolator"] + S4 --> S5["Step 5
Apply GKE adaptations"] + S5 --> S6["Step 6
Share one T4"] + S6 --> S7["Step 7
Run CUDA OOM proof"] + S7 --> S8["Step 8
Verify monitor metrics"] +``` + +## Prerequisites + +- A GCP project with the GKE and Compute Engine APIs enabled. +- A GKE 1.35 cluster with COS nodes and at least one NVIDIA T4. The verified cluster had three `n1-standard-2` nodes, each with one T4. +- The GKE-managed NVIDIA driver, device plugin, and container toolkit. Do not install GPU Operator on top of the GKE-managed driver. +- `gcloud`, a `kubectl` version within one minor release of the GKE API server, and Helm 3 or 4 with cluster-admin permissions. +- The files under [`tutorials/labs/examples/12-kai-scheduler-hami-gke/`](https://github.com/Project-HAMi/website/tree/master/tutorials/labs/examples/12-kai-scheduler-hami-gke). + +The verified cluster ran GKE `1.35.6-gke.1250000` on both the control plane and nodes. Because GKE patch releases age out, first choose an available 1.35 version in your zone, then create the cluster with it: + +```bash +export GKE_VERSION=$(gcloud container get-server-config \ + --zone=asia-northeast1-a \ + --format='value(validMasterVersions)' | tr ';' '\n' | grep '^1\.35\.' | head -1) +test -n "$GKE_VERSION" + +gcloud container clusters create kai-hami-test --zone=asia-northeast1-a \ + --cluster-version="$GKE_VERSION" \ + --machine-type=n1-standard-2 --num-nodes=3 \ + --image-type=COS_CONTAINERD \ + --accelerator=type=nvidia-tesla-t4,count=1,gpu-driver-version=default +gcloud container clusters get-credentials kai-hami-test \ + --zone=asia-northeast1-a +``` + +GPU nodes are billable. Run the cleanup section when you finish. + +## Step 1: Verify the GKE GPU Stack + +Confirm that each GPU node reports one extended resource: + +```bash +kubectl get nodes \ + -o custom-columns="NAME:.metadata.name,GPU:.status.capacity.nvidia\.com/gpu,ACCEL:.metadata.labels.cloud\.google\.com/gke-accelerator" +``` + +The verified cluster reported three T4 nodes: + +```plaintext +NAME GPU ACCEL +gke-kai-hami-test-default-pool-... 1 nvidia-tesla-t4 +gke-kai-hami-test-default-pool-... 1 nvidia-tesla-t4 +gke-kai-hami-test-default-pool-... 1 nvidia-tesla-t4 +``` + +If `GPU` is empty and the node has `gke-no-default-nvidia-gpu-device-plugin=true`, enable the GKE device plugin: + +```bash +kubectl label node -l cloud.google.com/gke-accelerator=nvidia-tesla-t4 \ + gke-no-default-nvidia-gpu-device-plugin- +``` + +Run an ordinary whole-GPU Pod before adding KAI: + +```bash +kubectl apply -f - <<'EOF' +apiVersion: v1 +kind: Pod +metadata: + name: gpu-smi-test +spec: + restartPolicy: Never + containers: + - name: cuda + image: nvidia/cuda:12.4.1-base-ubuntu22.04 + command: ["nvidia-smi"] + resources: + limits: + nvidia.com/gpu: 1 +EOF +kubectl wait --for=jsonpath='{.status.phase}'=Succeeded \ + pod/gpu-smi-test --timeout=180s +kubectl logs gpu-smi-test +kubectl delete pod gpu-smi-test +``` + +The T4 in the verified environment reported 15360 MiB of addressable memory. + +## Step 2: Add the GPU Labels KAI Reads + +GKE's default device plugin does not provide all GPU Feature Discovery labels. KAI reads `nvidia.com/gpu.memory` when it registers the node, so add the labels before installing KAI: + +```bash +kubectl label node -l cloud.google.com/gke-accelerator=nvidia-tesla-t4 \ + nvidia.com/gpu.memory=15360 \ + nvidia.com/gpu.product=NVIDIA-Tesla-T4 \ + nvidia.com/gpu.count=1 \ + nvidia.com/gpu.present=true --overwrite +``` + +Use the value reported by `nvidia-smi`, not the T4's marketed 16 GiB. If you add the labels after KAI starts, restart `kai-scheduler` so it refreshes its node cache. + +## Step 3: Install KAI Scheduler and Its Queue + +Install KAI v0.17.0 with GPU sharing, HAMi-core integration, and CDI enabled: + +```bash +helm install kai-scheduler \ + oci://ghcr.io/kai-scheduler/kai-scheduler/kai-scheduler \ + --namespace kai-scheduler --create-namespace \ + --version v0.17.0 \ + --set global.gpuSharing=true \ + --set binder.plugins.hamicore.enabled=true \ + --set-string binder.plugins.gpusharing.arguments.cdiEnabled=true + +kubectl -n kai-scheduler wait --for=condition=available \ + --timeout=180s deploy --all +kubectl -n kai-scheduler wait --for=condition=Ready \ + --timeout=300s config/kai-config +kubectl apply \ + -f tutorials/labs/examples/12-kai-scheduler-hami-gke/01-queues.yaml +kubectl get queues +``` + +The two required queues should appear: + +```plaintext +NAME PARENT +default +default-queue default +``` + +:::important `cdiEnabled` must be a string + +Use `--set-string`, not `--set`. KAI v0.17.0's Helm values accept either representation, but the generated `Config` CRD field is a string. Plain `--set ...=true` renders a boolean and makes the `kai-config-deployer` hook fail with `cdiEnabled ... must be of type string: "boolean"`. The command above was rendered and accepted by the current GKE API server with server-side dry-run. + +::: + +## Step 4: Install kai-resource-isolator + +COS mounts the root filesystem read-only, so write HAMi-core under GKE's writable NVIDIA directory: + +```bash +helm install kai-resource-isolator \ + oci://docker.io/projecthami/kai-resource-isolator \ + --namespace kai-resource-isolator --create-namespace \ + --version 1.1.0-chart \ + --set paths.containerVgpuMount=/home/kubernetes/bin/nvidia/vgpu \ + --set-string librarySync.priorityClassName= \ + --set-string monitor.priorityClassName= \ + --set monitor.enabled=true +``` + +`containerVgpuMount` is the effective Chart value for the libsync destination, preload file, webhook injection path, and monitor cache. Do not use `paths.hostInstallBase`: it is declared in 1.1.0-chart's values but is not referenced by that Chart's templates. The two empty PriorityClass values prevent GKE from rejecting `system-node-critical` Pods outside a system namespace. + +Verify the rendered paths and wait for the components: + +```bash +kubectl get cm kai-resource-isolator-ldpreload \ + -n kai-resource-isolator -o jsonpath='{.data.ld\.so\.preload}' +kubectl rollout status ds/kai-resource-isolator-libsync \ + -n kai-resource-isolator --timeout=300s +kubectl rollout status ds/kai-resource-isolator-monitor \ + -n kai-resource-isolator --timeout=300s +kubectl get pods -n kai-resource-isolator +``` + +The ConfigMap output must be `/home/kubernetes/bin/nvidia/vgpu/libvgpu.so`. + +## Step 5: Adapt the GKE CDI Device Path + +The verified GKE 1.35 nodes used CDI and did not register an `nvidia` runtime handler, while KAI reservation Pods referenced `runtimeClassName: nvidia`. Create the compatibility RuntimeClass only if it is absent: + +```bash +kubectl get runtimeclass nvidia || kubectl apply \ + -f tutorials/labs/examples/12-kai-scheduler-hami-gke/02-runtimeclass.yaml +``` + +GKE's device plugin injects devices only into Pods requesting `nvidia.com/gpu`. KAI shared Pods request `gpu-memory` instead, so the verified environment needed explicit device and library mounts. Install Kyverno and apply the two policies: + +```bash +helm repo add kyverno https://kyverno.github.io/kyverno/ +helm repo update +helm install kyverno kyverno/kyverno \ + --namespace kyverno --create-namespace +kubectl wait -n kyverno --for=condition=Ready pod \ + -l app.kubernetes.io/component=admission-controller --timeout=300s +kubectl apply \ + -f tutorials/labs/examples/12-kai-scheduler-hami-gke/03-gke-policies.yaml +``` + +The first policy adds the NVML library path to reservation Pods. The second mounts `/dev/nvidia*`, `nvidia-smi`, and NVIDIA libraries into shared Pods. In this workaround path, shared Pods also require `privileged: true`. + +:::caution Security boundary + +This lab proves CUDA API-level memory enforcement. It is not a MIG-like hardware security boundary, and the GKE workaround uses privileged workload containers. Do not treat it as an untrusted multi-tenant security design. + +::: + +## Step 6: Place Two Pods on One T4 + +Choose an otherwise idle T4 node with exactly one GPU. Do not select a node that already runs a `gpu-memory` workload: two 4 GiB Lab Pods need at least 8 GiB of unallocated KAI memory on that card. List the candidates and current Pod placement first: + +```bash +kubectl get nodes -l cloud.google.com/gke-accelerator=nvidia-tesla-t4 +kubectl get pods -A -o wide + +# Replace this value with one idle, single-T4 node from the output above. +export TEST_NODE= +test "$(kubectl get node "$TEST_NODE" \ + -o jsonpath='{.status.capacity.nvidia\.com/gpu}')" = "1" +kubectl label node "$TEST_NODE" hami.run/lab-12=true --overwrite + +kubectl create configmap kai-hami-lab12-source \ + --from-file=memory-limit.cu=tutorials/labs/examples/12-kai-scheduler-hami-gke/memory-limit.cu \ + --dry-run=client -o yaml | kubectl apply -f - +kubectl apply \ + -f tutorials/labs/examples/12-kai-scheduler-hami-gke/04-shared-pods.yaml +kubectl wait --for=condition=Ready \ + pod/kai-hami-lab12-a pod/kai-hami-lab12-b \ + --timeout=10m +``` + +Verify the node placement, then read the injected quota, physical UUID, and visible memory from each Pod's startup log: + +```bash +kubectl get pod kai-hami-lab12-a kai-hami-lab12-b -o wide +for pod in kai-hami-lab12-a kai-hami-lab12-b; do + echo "=== $pod ===" + kubectl logs "$pod" +done +``` + +Both Pods in the verified run reported: + +```plaintext +limit=4147m +GPU-9acc8878-3967-5fb4-c534-43d6fd820fa6, 4147 MiB +``` + +The matching UUID proves both Pods use the same T4; the 4147 MiB value is KAI's two-decimal fraction rounding of a 4096 MiB request on a 15360 MiB card. + +## Step 7: Prove the CUDA Memory Ceiling + +The source ConfigMap created in Step 6 is mounted at `/lab-source`. On startup, each Pod compiles the program, holds a successful 3 GiB allocation for 30 seconds, and attempts to allocate another 2 GiB. After that proof passes, it starts the same program again and keeps 3 GiB allocated so Step 8 can verify the live monitor gauges. The readiness probe succeeds only after both phases start successfully. + +Inspect the test interval and result recorded in each Pod log: + +```bash +for pod in kai-hami-lab12-a kai-hami-lab12-b; do + kubectl logs "$pod" | grep -E \ + 'test_(start|end)=|allocate |PASS:' +done +``` + +The verified output from each Pod was: + +```plaintext +allocate 3 GiB: no error +allocate another 2 GiB: out of memory +PASS: in-quota allocation succeeded and over-quota allocation failed +``` + +Compare `test_start` and `test_end`: the two 30-second intervals must overlap. In the verified run, both Pods returned `PASS` while holding 3 GiB concurrently. HAMi-core logged `Device 0 OOM 5475663872 / 4348444672` for each Pod's 5 GiB cumulative request. Running the proof as the container startup command also avoids making the result depend on a long-lived `kubectl exec` WebSocket. + +## Step 8: Verify the Monitor Metrics + +The monitor is a DaemonSet: each instance reads the HAMi shared-memory cache on its own node. A Service can forward to a monitor on a different node and return no series for these Pods, so query the monitor instance on the workload node directly. Derive that node again from the running Pod instead of relying on the shell variable exported in Step 6. + +Create a short-lived curl Pod that reads that monitor's `:9394/metrics` endpoint: + +```bash +export TEST_NODE=$(kubectl get pod kai-hami-lab12-a \ + -o jsonpath='{.spec.nodeName}') +test -n "$TEST_NODE" + +export MONITOR_IP=$(kubectl get pods -n kai-resource-isolator \ + -l app.kubernetes.io/component=kai-vgpu-monitor \ + --field-selector="spec.nodeName=$TEST_NODE" \ + -o jsonpath='{range .items[*]}{.status.podIP}{"\n"}{end}' | head -n 1) + +if test -z "$MONITOR_IP"; then + echo "No monitor Pod is running on $TEST_NODE" >&2 + kubectl get pods -n kai-resource-isolator \ + -l app.kubernetes.io/component=kai-vgpu-monitor -o wide + exit 1 +fi + +kubectl delete pod lab12-monitor-check --ignore-not-found +kubectl run lab12-monitor-check \ + --image=curlimages/curl:8.15.0 --restart=Never \ + --command -- sh -lc \ + "curl -fsS http://$MONITOR_IP:9394/metrics" +kubectl wait --for=jsonpath='{.status.phase}'=Succeeded \ + pod/lab12-monitor-check --timeout=180s +kubectl logs lab12-monitor-check | grep -E \ + '^hami_vgpu_memory_(used|limit)_bytes.*pod="kai-hami-lab12-[ab]"' +``` + +The verified endpoint returned one `used` and one `limit` series for each Pod: + +```plaintext +hami_vgpu_memory_limit_bytes{...,pod="kai-hami-lab12-a",...} 4.348444672e+09 +hami_vgpu_memory_limit_bytes{...,pod="kai-hami-lab12-b",...} 4.348444672e+09 +hami_vgpu_memory_used_bytes{...,pod="kai-hami-lab12-a",...} 3.328180224e+09 +hami_vgpu_memory_used_bytes{...,pod="kai-hami-lab12-b",...} 3.328180224e+09 +``` + +The limit equals 4147 MiB in bytes. The used value is slightly above 3 GiB because it includes CUDA context and allocator overhead. This proves the monitor found both per-container caches and exported live usage rather than only serving an empty Prometheus endpoint. + +## Troubleshooting + +| Symptom | Cause in the verified environment | Action | +| :-- | :-- | :-- | +| Shared Pod stays Pending with `didn't have enough resources: GPU memory` | `nvidia.com/gpu.memory` was missing when KAI cached the node | Add the label and restart `kai-scheduler` | +| Queue object is rejected | KAI admission webhook is not ready | Wait for KAI Deployments, then apply the queues | +| `RuntimeClass "nvidia" not found` | GKE CDI uses `runc` and has no NVIDIA handler | Apply `02-runtimeclass.yaml` | +| Reservation Pod reports `ERROR_LIBRARY_NOT_FOUND` | NVML exists under `/usr/local/nvidia/lib64` but is not in the search path | Apply the Kyverno library-path policy | +| Shared Pod cannot see `/dev/nvidia*` | It requests `gpu-memory`, so the GKE device plugin does not run Allocate | Apply the Kyverno device-mount policy | +| libsync reports `Read-only file system` | COS root filesystem is read-only | Set `paths.containerVgpuMount=/home/kubernetes/bin/nvidia/vgpu` | +| `libvgpu.so` cannot be preloaded | The effective mount path still points to `/usr/local/vgpu` | Reinstall with the verified `containerVgpuMount` value | +| DaemonSet rejected for `system-node-critical` | GKE PriorityClass quota blocks user namespaces | Set both Chart PriorityClass values to an empty string | +| `CUDA driver version is insufficient` | CUDA image is newer than the node driver supports | Use the verified CUDA 12.4.1 image or another compatible version | +| `kubectl exec` ends with WebSocket EOF | Control-plane exec stream reset or unsupported client/server skew | Use a compatible `kubectl`; the Lab proof runs at container startup and is read with `kubectl logs` | +| Monitor lookup returns no Pod or no series | `$TEST_NODE` was stale, or the monitor is absent from the workload node | Derive the node from `kai-hami-lab12-a`, then inspect the monitor DaemonSet with `kubectl get pods -n kai-resource-isolator -l app.kubernetes.io/component=kai-vgpu-monitor -o wide` | + +## Cleanup + +Remove the workload and test label: + +```bash +kubectl delete \ + -f tutorials/labs/examples/12-kai-scheduler-hami-gke/04-shared-pods.yaml +kubectl delete pod lab12-monitor-check --ignore-not-found +kubectl delete configmap kai-hami-lab12-source --ignore-not-found +kubectl label node "$TEST_NODE" hami.run/lab-12- --overwrite +``` + +If this cluster is dedicated to the lab, remove the remaining components: + +```bash +kubectl delete \ + -f tutorials/labs/examples/12-kai-scheduler-hami-gke/03-gke-policies.yaml +helm uninstall kyverno -n kyverno +helm uninstall kai-resource-isolator -n kai-resource-isolator +kubectl delete \ + -f tutorials/labs/examples/12-kai-scheduler-hami-gke/01-queues.yaml +helm uninstall kai-scheduler -n kai-scheduler +``` + +Delete `02-runtimeclass.yaml` only if Step 5 created it; preserve a RuntimeClass that already belonged to the cluster: + +```bash +kubectl delete \ + -f tutorials/labs/examples/12-kai-scheduler-hami-gke/02-runtimeclass.yaml +``` + +Delete the GKE cluster if you created it only for this exercise: + +```bash +gcloud container clusters delete kai-hami-test \ + --zone=asia-northeast1-a +``` + +## What This Lab Proved + +| Claim | Evidence | +| :-- | :-- | +| KAI schedules two fractional workloads onto one T4 | Same node and same GPU UUID in both Pods | +| HAMi-core changes the visible per-Pod memory ceiling | Both Pods report 4147 MiB instead of 15360 MiB | +| The ceiling is enforced, not only displayed | 3 GiB succeeds; the cumulative 5 GiB request returns CUDA OOM | +| One Pod cannot consume the other's quota | Pod B succeeds while Pod A holds 3 GiB | +| The optional monitor reads per-container caches | `:9394/metrics` reports both Pods' 4147 MiB limits and live 3 GiB usage | + +## Next Steps + +- Read [GPU Memory Hard Isolation with KAI Scheduler and HAMi](/blog/kai-scheduler-hami-gpu-memory-hard-isolation) for the architecture and integration background. +- Compare this path with [Lab 3: GPU Partitioning](./gpu-partitioning.md) and [Lab 7: k3s Isolation](./hami-isolation-k3s.md). +- Track upstream KAI and `kai-resource-isolator` releases; once GKE CDI support is native, remove the matching workaround from this lab. diff --git a/tutorials/overview.md b/tutorials/overview.md index 010b9373a..5cfd995f1 100644 --- a/tutorials/overview.md +++ b/tutorials/overview.md @@ -17,4 +17,4 @@ Background knowledge that the labs build on. ## Labs - Each lab lists its own prerequisites. Labs 3 and 4 continue from the cluster Lab 1 builds, so a single session covers all three; Lab 2 runs on any laptop with no GPU required. Lab 7 brings up its own single-node k3s cluster on a rented GPU VM, without the GPU Operator. Lab 8 requires an existing Volcano GPU cluster and validates Volcano vGPU, Gang scheduling, and queue-level limits. Lab 9 uses Kueue admission control to enforce HAMi vGPU count, memory, and compute quotas. Lab 11 builds a complete KServe Standard inference stack and runs two vLLM replicas on one GPU through native HAMi DRA claims. + Each lab lists its own prerequisites. Labs 3 and 4 continue from the cluster Lab 1 builds, so a single session covers all three; Lab 2 runs on any laptop with no GPU required. Lab 7 brings up its own single-node k3s cluster on a rented GPU VM, without the GPU Operator. Lab 8 requires an existing Volcano GPU cluster and validates Volcano vGPU, Gang scheduling, and queue-level limits. Lab 9 uses Kueue admission control to enforce HAMi vGPU count, memory, and compute quotas. Lab 11 builds a complete KServe Standard inference stack and runs two vLLM replicas on one GPU through native HAMi DRA claims. Lab 12 deploys KAI Scheduler and HAMi-core on GKE 1.35/COS/CDI and proves the memory ceiling with CUDA allocations. From 9fe29c22ac7aa380bb63a5aa36c76672a6b4ed0c Mon Sep 17 00:00:00 2001 From: Jimmy Song Date: Wed, 12 Aug 2026 17:51:37 +0800 Subject: [PATCH 3/6] fix: streamline tags formatting in GPU Memory Hard Isolation documentation Signed-off-by: Jimmy Song --- .../index.md | 10 +--------- 1 file changed, 1 insertion(+), 9 deletions(-) diff --git a/blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md b/blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md index fe37541a6..8505059ce 100644 --- a/blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md +++ b/blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md @@ -3,15 +3,7 @@ title: "GPU Memory Hard Isolation with KAI Scheduler and HAMi: How It Works and date: "2026-08-11" description: "From the CUDA interception principle to a reproducible verification on GKE: use KAI Scheduler v0.17.0 for shared GPU scheduling and kai-resource-isolator 1.1.0-chart (built on HAMi-core) for CUDA level memory isolation, so two pods on a single NVIDIA T4 each see only their own memory quota. Explains the CUDA_DEVICE_MEMORY_LIMIT contract between the scheduling layer and the isolation layer, plus the node label, queue, and RuntimeClass pitfalls." authors: [rootsongjc] -tags: - [ - "HAMi", - "KAI Scheduler", - "GPU Memory Hard Isolation", - "GPU Sharing", - "Kubernetes", - "Cloud Native", - ] +tags: ["HAMi", "KAI Scheduler", "Hard Isolation", "GPU Sharing", "Kubernetes", "Cloud Native"] --- GPU sharing has been discussed in the Kubernetes ecosystem for years, but the scheduling layer and the isolation layer have long operated in isolation from each other. The scheduler places several pods onto the same card, yet once a container touches the GPU it still sees the full device memory. Whichever container calls `cudaMalloc` first can occupy everything, so the isolation is effectively absent. So called "sharing" is really just "grabbing", with no resource guarantee at all. From 7a5e983b4bb7e708db3bbe6f4fedfa55f7f4e345 Mon Sep 17 00:00:00 2001 From: Jimmy Song Date: Wed, 12 Aug 2026 19:49:13 +0800 Subject: [PATCH 4/6] fix: correct typos and improve clarity in GPU Memory Hard Isolation documentation Signed-off-by: Jimmy Song --- .../index.md | 15 +++++++------- .../index.md | 9 +++++---- .../current/labs/kai-scheduler-hami-gke.md | 17 ++++++++-------- .../12-kai-scheduler-hami-gke/01-queues.yaml | 20 ------------------- tutorials/labs/kai-scheduler-hami-gke.md | 17 ++++++++-------- 5 files changed, 29 insertions(+), 49 deletions(-) delete mode 100644 tutorials/labs/examples/12-kai-scheduler-hami-gke/01-queues.yaml diff --git a/blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md b/blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md index 8505059ce..3ca7372eb 100644 --- a/blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md +++ b/blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md @@ -6,7 +6,7 @@ authors: [rootsongjc] tags: ["HAMi", "KAI Scheduler", "Hard Isolation", "GPU Sharing", "Kubernetes", "Cloud Native"] --- -GPU sharing has been discussed in the Kubernetes ecosystem for years, but the scheduling layer and the isolation layer have long operated in isolation from each other. The scheduler places several pods onto the same card, yet once a container touches the GPU it still sees the full device memory. Whichever container calls `cudaMalloc` first can occupy everything, so the isolation is effectively absent. So called "sharing" is really just "grabbing", with no resource guarantee at all. +GPU sharing has been discussed in the Kubernetes ecosystem for years, but the scheduling layer and the isolation layer have long operated in isolation from each other. The scheduler places several pods onto the same card, yet once a container touches the GPU it still sees the full device memory. Whichever container calls `cudaMalloc` first can occupy everything, so the isolation is effectively absent. So-called "sharing" is really just "grabbing", with no resource guarantee at all. Solving this requires the scheduling layer (deciding "who uses which GPU, and how much") and the isolation layer (guaranteeing "once a quota is set, it cannot be exceeded") to cooperate. **HAMi-core** is exactly such an isolation engine, reusable by multiple schedulers. Before KAI Scheduler, it already supported the Kubernetes native scheduler (via HAMi's own `hami-scheduler` extender), [Kueue](/docs/userguide/kueue/how-to-use-kueue), [Volcano](/docs/installation/how-to-use-volcano-vgpu), and more (see the full landscape in [Ecosystem Integrations](/docs/next/core-concepts/ecosystem-integrations)). @@ -46,7 +46,7 @@ The three components have similar-sounding names, so getting their roles straigh - **HAMi-core (`libvgpu.so`)**: HAMi's CUDA interception library (CNCF incubating), and **the isolation engine itself**. It intercepts CUDA calls (like `cudaMalloc`) inside the container via `LD_PRELOAD` and enforces a memory quota. It does not care who provided the quota: any scheduler that hands in the quota by convention gets isolation for free. Before KAI, it was already reused by HAMi's own device-plugin/webhook, Volcano's `volcano-vgpu-device-plugin`, and others. -- **KAI Scheduler**: NVIDIA's open source Kubernetes scheduler for AI workloads. It only owns the **scheduling layer**, deciding which node a pod lands on, which GPU it uses, and how much memory it gets. Since v0.16.4, its `hamicore` plugin writes the computed memory quota into the container's environment variable at bind time. +- **KAI Scheduler**: NVIDIA's open-source Kubernetes scheduler for AI workloads. It only owns the **scheduling layer**, deciding which node a pod lands on, which GPU it uses, and how much memory it gets. Since v0.16.4, its `hamicore` plugin writes the computed memory quota into the container's environment variable at bind time. - **`kai-resource-isolator`**: a companion component **provided by the HAMi project specifically for the KAI Scheduler integration path**. It ships HAMi-core's `libvgpu.so` to every GPU node and uses a MutatingWebhook to rewrite pods, injecting the library and `ld.so.preload`. In other words, it is the bridge that turns KAI's scheduling decision into isolation HAMi-core can actually enforce. @@ -105,8 +105,9 @@ graph TD RUN["3. Container starts
libvgpu.so intercepts cudaMalloc via LD_PRELOAD"] ENF["Rejects over limit allocation
nvidia-smi shows only the quota"] - KAI --> ENV --> ISO --> RUN --> ENF - LIB -. "inject hostPath and ld.so.preload" .-> ISO + KAI --> ENV --> ISO + ISO -->|"injects hostPath and ld.so.preload"| RUN --> ENF + LIB -. "provides libvgpu.so on the node" .-> RUN MON -. "collects per container memory" .-> RUN style KAI fill:#d9f99d,stroke:#4f7d00,stroke-width:2px,color:#1f2937 @@ -149,13 +150,13 @@ This integration path is the result of more than a year of work between the HAMi KAI Scheduler's support for HAMi-core first appeared in **v0.16.4**. The current integration documentation requires KAI Scheduler v0.17.0 or later, and this post tests v0.17.0. The key piece is the `hamicore` plugin: once enabled, when KAI binds a shared GPU pod to a node, it injects the `CUDA_DEVICE_MEMORY_LIMIT` environment variable into the container based on the `gpu-memory` (or `gpu-fraction`) annotation, exactly the quota that HAMi-core needs to enforce isolation, per the contract above. -Other GPU related changes in v0.17.0 include: fixing invalid volume names caused by `/` in shared pod names; correcting the allocation math for `MinNodeGPUMemoryMiB` and fractional `gpu-memory`; and using the largest GPU profile in the cluster for overLimit decisions. It also adds preemption-delay (a time window for Cluster Autoscaler to bring up nodes), NUMA aware scoring, and GitOps and ArgoCD installation support. +Other GPU-related changes in v0.17.0 include: fixing invalid volume names caused by `/` in shared pod names; correcting the allocation math for `MinNodeGPUMemoryMiB` and fractional `gpu-memory`; and using the largest GPU profile in the cluster for overLimit decisions. It also adds preemption-delay (a time window for Cluster Autoscaler to bring up nodes), NUMA-aware scoring, and GitOps and ArgoCD installation support. ### kai-resource-isolator 1.1.0-chart This is the isolator shipped alongside HAMi. It receives the quota injected by KAI and, before the container actually starts, puts the HAMi-core `libvgpu.so` in place. Compared with the first release, 1.1.0 adds several operational improvements: -- **New `kai-vgpu-monitor`**: runs as a DaemonSet, exposes HAMi compatible metrics on `:9394` (`hami_vgpu_memory_used_bytes`, `hami_vgpu_memory_limit_bytes`, `hami_container_device_utilization_ratio`), supports ServiceMonitor, and can be scraped directly by Prometheus. +- **New `kai-vgpu-monitor`**: runs as a DaemonSet, exposes HAMi-compatible metrics on `:9394` (`hami_vgpu_memory_used_bytes`, `hami_vgpu_memory_limit_bytes`, `hami_container_device_utilization_ratio`), supports ServiceMonitor, and can be scraped directly by Prometheus. - **Multi container injection fix**: when a pod has multiple containers, the webhook now handles them correctly and no longer skips any. - **Security tightening**: the webhook now uses a namespaced Issuer (instead of a ClusterIssuer), and the ClusterRole no longer reads Secrets. - **Global image repository** precedence cleaned up, and `hamicore` installation parameters corrected. @@ -215,4 +216,4 @@ Behind this is more than a year of careful alignment between the KAI Scheduler t - The background story: [HAMi-core adopted by NVIDIA KAI Scheduler](/blog/hami-core-adopted-by-nvidia-kai-scheduler) - User docs: [How to use HAMi with KAI Scheduler](/docs/next/userguide/kai-scheduler/how-to-use-kai-scheduler) - Related repos: [Project-HAMi/KAI-resource-isolator](https://github.com/Project-HAMi/KAI-resource-isolator), [Project-HAMi/HAMi-core](https://github.com/Project-HAMi/HAMi-core) (the CNCF incubating CUDA interception library), [kai-scheduler/KAI-Scheduler](https://github.com/kai-scheduler/KAI-Scheduler), [KAI Scheduler HAMi resource isolation docs](https://github.com/kai-scheduler/KAI-Scheduler/blob/main/docs/gpu-sharing/hami/README.md) -- Run it on your own GKE, AWS, or self built cluster, and share real results in an issue or the community group. If HAMi-core lacks support for a certain card or CUDA version, open an issue at [Project-HAMi/HAMi](https://github.com/Project-HAMi/HAMi). This is the feedback the community values most. +- Run it on your own GKE, AWS, or self-built cluster, and share real results in an issue or the community group. If HAMi-core lacks support for a certain card or CUDA version, open an issue at [Project-HAMi/HAMi](https://github.com/Project-HAMi/HAMi). This is the feedback the community values most. diff --git a/i18n/zh/docusaurus-plugin-content-blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md b/i18n/zh/docusaurus-plugin-content-blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md index b7ed8a197..0517ac3a0 100644 --- a/i18n/zh/docusaurus-plugin-content-blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md +++ b/i18n/zh/docusaurus-plugin-content-blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md @@ -46,7 +46,7 @@ HAMi 社区多年打磨的 HAMi-core（CNCF 孵化项目）正是这样的隔离 - **HAMi-core（`libvgpu.so`）**：HAMi 项目的 CUDA 拦截库，是**隔离引擎本身**。它通过 `LD_PRELOAD` 拦截容器里的 CUDA 调用（如 `cudaMalloc`），按一个显存配额强制限制。它不关心配额是谁给的：任何调度器只要按约定把配额传进来，它都能执行隔离。在 KAI 之前，它已经被 HAMi 自带的 device-plugin/webhook、Volcano 的 `volcano-vgpu-device-plugin` 等复用。 -- **KAI Scheduler**：NVIDIA 开源的 Kubernetes AI 工作负载调度器。它只负责**调度层**，即决定 Pod 落到哪个节点、用哪张 GPU、分多少显存。自 v0.16.4 起，它的 `hamicore` 插件在绑定 Pod 时，会把算好的显存配额写进容器的环境变量。 +- **KAI Scheduler**：NVIDIA 开源的 Kubernetes AI 工作负载调度器。它只负责**调度层**，即决定 Pod 落到哪个节点、用哪张 GPU、分多少显存。自 v0.16.4 起，它的 `hamicore` 插件在绑定 Pod 时，会把计算出的显存配额写进容器的环境变量。 - **`kai-resource-isolator`**：HAMi 项目侧为 KAI Scheduler 这条集成路径**专门提供的配套组件**。它把 HAMi-core 的 `libvgpu.so` 分发到每个 GPU 节点，并用 MutatingWebhook 改写 Pod，把库和 `ld.so.preload` 注入进去。换言之，它是“把 KAI 的调度决策落地成 HAMi-core 能执行的隔离”的桥梁。 @@ -105,8 +105,9 @@ graph TD RUN["3. 容器启动
libvgpu.so 经 LD_PRELOAD 拦截 cudaMalloc"] ENF["按 LIMIT 拒绝超额分配
nvidia-smi 仅显示配额内显存"] - KAI --> ENV --> ISO --> RUN --> ENF - LIB -. "注入 hostPath 与 ld.so.preload" .-> ISO + KAI --> ENV --> ISO + ISO -->|"注入 hostPath 与 ld.so.preload"| RUN --> ENF + LIB -. "在节点上提供 libvgpu.so" .-> RUN MON -. "采集各容器显存" .-> RUN style KAI fill:#d9f99d,stroke:#4f7d00,stroke-width:2px,color:#1f2937 @@ -131,7 +132,7 @@ graph TD 隔离的“最后一公里”发生在容器进程里，链路是这样的： 1. **KAI 在调度时注入 `CUDA_DEVICE_MEMORY_LIMIT`**（带上 Pod 申请的显存配额，单位 MiB）。 -2. **kai-resource-isolator 的 webhook 改写 Pod**：挂载宿主机上的 `libvgpu.so`，并把 `/etc/ld.so.preload` 指向它。`ld.so.preload` 是动态链接器的机制，被列在里面的共享库会在所有其他库之前加载。 +2. **kai-resource-isolator 的 webhook 改写 Pod**：挂载宿主机上的 `libvgpu.so`，并把 `/etc/ld.so.preload` 指向它。`ld.so.preload` 是动态链接器的机制，列在其中的共享库会在所有其他库之前加载。 3. **容器进程启动后**，任何对 CUDA 运行时（`libcudart`）或驱动 API 的调用，都会先经过 `libvgpu.so`。后者拦截 `cudaMalloc` 之类的显存分配调用，从 `CUDA_DEVICE_MEMORY_LIMIT` 读出配额，累计该容器的显存用量；一旦超额就拒绝分配。 4. **对外可见的效果**：`nvidia-smi` 只显示配额内的显存（HAMi-core 会改写设备查询的返回值），容器再怎么 `cudaMalloc` 也越不过这条线。 diff --git a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/kai-scheduler-hami-gke.md b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/kai-scheduler-hami-gke.md index 39c48a8cc..be244f49a 100644 --- a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/kai-scheduler-hami-gke.md +++ b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/kai-scheduler-hami-gke.md @@ -139,7 +139,7 @@ kubectl label node -l cloud.google.com/gke-accelerator=nvidia-tesla-t4 \ 显存值应来自 `nvidia-smi`，不要使用 T4 标称的 16 GiB。如果 KAI 启动后才补标签，需要重启 `kai-scheduler` 刷新节点缓存。 -## 步骤 3: 安装 KAI Scheduler 与队列 +## 步骤 3: 安装 KAI Scheduler 与默认队列 安装 KAI v0.17.0，启用 GPU sharing、HAMi-core 集成和 CDI： @@ -156,17 +156,15 @@ kubectl -n kai-scheduler wait --for=condition=available \ --timeout=180s deploy --all kubectl -n kai-scheduler wait --for=condition=Ready \ --timeout=300s config/kai-config -kubectl apply \ - -f tutorials/labs/examples/12-kai-scheduler-hami-gke/01-queues.yaml kubectl get queues ``` -应出现两个队列： +KAI v0.17.0 会自动创建默认的父子队列： ```plaintext -NAME PARENT -default -default-queue default +NAME PARENT +default-parent-queue +default-queue default-parent-queue ``` :::important `cdiEnabled` 必须是字符串 @@ -382,11 +380,12 @@ kubectl delete \ -f tutorials/labs/examples/12-kai-scheduler-hami-gke/03-gke-policies.yaml helm uninstall kyverno -n kyverno helm uninstall kai-resource-isolator -n kai-resource-isolator -kubectl delete \ - -f tutorials/labs/examples/12-kai-scheduler-hami-gke/01-queues.yaml +kubectl delete queues default-queue default-parent-queue --ignore-not-found helm uninstall kai-scheduler -n kai-scheduler ``` +这里显式删除 Queue 是有意为之：KAI 给默认队列添加了 `helm.sh/resource-policy: keep` 注解，因此 Helm 卸载时会保留它们。 + 只有步骤 5 创建了 RuntimeClass 时才删除 `02-runtimeclass.yaml`；如果该 RuntimeClass 原本就属于集群，应予以保留： ```bash diff --git a/tutorials/labs/examples/12-kai-scheduler-hami-gke/01-queues.yaml b/tutorials/labs/examples/12-kai-scheduler-hami-gke/01-queues.yaml deleted file mode 100644 index a2cb72075..000000000 --- a/tutorials/labs/examples/12-kai-scheduler-hami-gke/01-queues.yaml +++ /dev/null @@ -1,20 +0,0 @@ -apiVersion: scheduling.run.ai/v2 -kind: Queue -metadata: - name: default -spec: - resources: - cpu: { quota: -1, limit: -1, overQuotaWeight: 1 } - memory: { quota: -1, limit: -1, overQuotaWeight: 1 } - gpu: { quota: -1, limit: -1, overQuotaWeight: 1 } ---- -apiVersion: scheduling.run.ai/v2 -kind: Queue -metadata: - name: default-queue -spec: - parentQueue: default - resources: - cpu: { quota: -1, limit: -1, overQuotaWeight: 1 } - memory: { quota: -1, limit: -1, overQuotaWeight: 1 } - gpu: { quota: -1, limit: -1, overQuotaWeight: 1 } diff --git a/tutorials/labs/kai-scheduler-hami-gke.md b/tutorials/labs/kai-scheduler-hami-gke.md index 2e3a245c6..334254d06 100644 --- a/tutorials/labs/kai-scheduler-hami-gke.md +++ b/tutorials/labs/kai-scheduler-hami-gke.md @@ -139,7 +139,7 @@ kubectl label node -l cloud.google.com/gke-accelerator=nvidia-tesla-t4 \ Use the value reported by `nvidia-smi`, not the T4's marketed 16 GiB. If you add the labels after KAI starts, restart `kai-scheduler` so it refreshes its node cache. -## Step 3: Install KAI Scheduler and Its Queue +## Step 3: Install KAI Scheduler and Its Default Queues Install KAI v0.17.0 with GPU sharing, HAMi-core integration, and CDI enabled: @@ -156,17 +156,15 @@ kubectl -n kai-scheduler wait --for=condition=available \ --timeout=180s deploy --all kubectl -n kai-scheduler wait --for=condition=Ready \ --timeout=300s config/kai-config -kubectl apply \ - -f tutorials/labs/examples/12-kai-scheduler-hami-gke/01-queues.yaml kubectl get queues ``` -The two required queues should appear: +KAI v0.17.0 creates its default parent and child queues automatically: ```plaintext -NAME PARENT -default -default-queue default +NAME PARENT +default-parent-queue +default-queue default-parent-queue ``` :::important `cdiEnabled` must be a string @@ -382,11 +380,12 @@ kubectl delete \ -f tutorials/labs/examples/12-kai-scheduler-hami-gke/03-gke-policies.yaml helm uninstall kyverno -n kyverno helm uninstall kai-resource-isolator -n kai-resource-isolator -kubectl delete \ - -f tutorials/labs/examples/12-kai-scheduler-hami-gke/01-queues.yaml +kubectl delete queues default-queue default-parent-queue --ignore-not-found helm uninstall kai-scheduler -n kai-scheduler ``` +The explicit Queue deletion is intentional: KAI annotates its default queues with `helm.sh/resource-policy: keep`, so Helm preserves them during uninstall. + Delete `02-runtimeclass.yaml` only if Step 5 created it; preserve a RuntimeClass that already belonged to the cluster: ```bash From 58a719a3c7ec0de709714abfc5b536adc0ed91cc Mon Sep 17 00:00:00 2001 From: Jimmy Song Date: Thu, 13 Aug 2026 11:42:43 +0800 Subject: [PATCH 5/6] docs(tutorials): update kai-scheduler-hami-gke documentation with verified outputs and notes - Added a note about the output blocks indicating the verification date and environment-specific details. - Updated GPU node reporting examples with actual node names from the verified run. - Included relevant `nvidia-smi` output for clarity on GPU memory reporting. - Enhanced steps with actual command outputs and statuses from the verified installation. - Clarified the requirements for KAI labels and the expected outputs for various components. - Documented the successful verification of Pods and their resource allocation results. Signed-off-by: Jimmy Song --- .../index.md | 308 +++++++++-------- .../index.md | 320 ++++++++++-------- .../current/labs/kai-scheduler-hami-gke.md | 110 +++++- tutorials/labs/kai-scheduler-hami-gke.md | 110 +++++- 4 files changed, 556 insertions(+), 292 deletions(-) diff --git a/blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md b/blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md index 3ca7372eb..57dc3a5b5 100644 --- a/blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md +++ b/blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md @@ -1,171 +1,224 @@ --- title: "GPU Memory Hard Isolation with KAI Scheduler and HAMi: How It Works and How to Verify It" date: "2026-08-11" -description: "From the CUDA interception principle to a reproducible verification on GKE: use KAI Scheduler v0.17.0 for shared GPU scheduling and kai-resource-isolator 1.1.0-chart (built on HAMi-core) for CUDA level memory isolation, so two pods on a single NVIDIA T4 each see only their own memory quota. Explains the CUDA_DEVICE_MEMORY_LIMIT contract between the scheduling layer and the isolation layer, plus the node label, queue, and RuntimeClass pitfalls." +description: "A concise explanation of the KAI Scheduler and HAMi-core isolation path, plus reproducible GKE evidence that two Pods sharing one NVIDIA T4 cannot exceed their individual GPU memory quotas." authors: [rootsongjc] tags: ["HAMi", "KAI Scheduler", "Hard Isolation", "GPU Sharing", "Kubernetes", "Cloud Native"] --- -GPU sharing has been discussed in the Kubernetes ecosystem for years, but the scheduling layer and the isolation layer have long operated in isolation from each other. The scheduler places several pods onto the same card, yet once a container touches the GPU it still sees the full device memory. Whichever container calls `cudaMalloc` first can occupy everything, so the isolation is effectively absent. So-called "sharing" is really just "grabbing", with no resource guarantee at all. +The companion post [HAMi-core adopted by NVIDIA KAI Scheduler](/blog/hami-core-adopted-by-nvidia-kai-scheduler) already introduces KAI Scheduler and the collaboration behind this integration. This post skips that background and focuses on one question: **when KAI Scheduler places two Pods on one GPU, does HAMi-core actually enforce each Pod's memory quota?** -Solving this requires the scheduling layer (deciding "who uses which GPU, and how much") and the isolation layer (guaranteeing "once a quota is set, it cannot be exceeded") to cooperate. **HAMi-core** is exactly such an isolation engine, reusable by multiple schedulers. Before KAI Scheduler, it already supported the Kubernetes native scheduler (via HAMi's own `hami-scheduler` extender), [Kueue](/docs/userguide/kueue/how-to-use-kueue), [Volcano](/docs/installation/how-to-use-volcano-vgpu), and more (see the full landscape in [Ecosystem Integrations](/docs/next/core-concepts/ecosystem-integrations)). +We verified the currently documented combination—KAI Scheduler v0.17.0 and `kai-resource-isolator` 1.1.0-chart—on GKE 1.35/COS/CDI. Both Pods shared the same NVIDIA T4, each saw a 4147 MiB ceiling, a 3 GiB CUDA allocation succeeded, and a cumulative 5 GiB allocation failed. The optional monitor also exported live limit and usage metrics for both Pods. -**Starting with KAI Scheduler v0.16.4, NVIDIA's KAI Scheduler officially joined this list**, building in HAMi-core as the isolation engine for its GPU sharing. This post verifies the currently documented supported combination, KAI Scheduler v0.17.0 plus `kai-resource-isolator` 1.1.0-chart. That means when you schedule GPU workloads with this integration enabled, you no longer get only "cooperative sharing"; you get CUDA API-level memory enforcement. This post covers two things: +:::note About the captured output -- **Principles**: what each of KAI Scheduler, `kai-resource-isolator`, and HAMi-core is responsible for, how they connect, and how the `CUDA_DEVICE_MEMORY_LIMIT` contract ties the scheduling layer to the isolation layer. -- **Practice**: an end-to-end GKE verification on one NVIDIA T4, including a direct `cudaMalloc` proof that the per-Pod memory quota cannot be exceeded. The complete procedure is in Lab 12. - -The background story and collaboration timeline are in the companion post [HAMi-core adopted by NVIDIA KAI Scheduler: GPU sharing enters the hard isolation era](/blog/hami-core-adopted-by-nvidia-kai-scheduler). - -:::note About the output in this post - -The GKE UUID, memory ceiling, error message, and CUDA allocation results marked as captured came from the verified GKE 1.35/COS/CDI run. Resource names are shortened where necessary; values in another cluster will differ. +The UUID, memory ceiling, CUDA allocation results, and monitor metrics below came from the verified GKE run. Resource suffixes and addresses will differ in another cluster. ::: -## Background: why sharing is not isolation +## How the integration works -| Layer | Owner | What it solves | What it does not solve | -| :-- | :-- | :-- | :-- | -| Scheduling | KAI Scheduler, Volcano, Kueue, ... | Multiple pods can land on the same GPU | The container still sees all the memory | -| Runtime | HAMi-core (`libvgpu.so`) | Intercepts CUDA calls, enforces a memory quota | On its own, does not know how much each pod should get | +The path has three responsibilities: -Real GPU sharing needs both layers, and they must cooperate: the scheduling layer decides "who uses which GPU, and how much", and the isolation layer guarantees "the agreed amount is all you get". The catch is that **the isolation layer needs to know "how much", a number it cannot compute on its own, because that number comes from the scheduling layer**. +- **KAI Scheduler** decides which GPU a Pod uses and injects the computed quota through `CUDA_DEVICE_MEMORY_LIMIT`. +- **`kai-resource-isolator`** distributes `libvgpu.so`; its webhook mounts the library and configures `ld.so.preload` in the workload Pod. Its optional monitor reads node-local HAMi caches and exposes `hami_*` metrics on `:9394`. +- **HAMi-core (`libvgpu.so`)** intercepts CUDA calls such as `cudaMalloc` and rejects allocations beyond the quota. -HAMi-core, refined by the HAMi community over years, is exactly that isolation layer, and it is **decoupled from any specific scheduler**: well before KAI Scheduler, HAMi-core already worked with the Kubernetes native scheduler (via HAMi's own `hami-scheduler` extender), [Kueue](/docs/userguide/kueue/how-to-use-kueue), [Volcano](/docs/installation/how-to-use-volcano-vgpu), Koordinator, and others (see the full landscape in [Ecosystem Integrations](/docs/next/core-concepts/ecosystem-integrations)). +```mermaid +%% title: KAI Scheduler and HAMi-core isolation path +graph TD + KAI["KAI Scheduler
computes the GPU memory quota"] + ENV["Inject CUDA_DEVICE_MEMORY_LIMIT"] + WEBHOOK["kai-resource-isolator webhook
mounts hostPath and ld.so.preload"] + LIB["libsync DaemonSet
provides libvgpu.so on each GPU node"] + RUN["HAMi-core intercepts CUDA calls"] + ENF["Reject over-quota allocations"] + MON["monitor DaemonSet
exports hami_* metrics on :9394"] + + KAI --> ENV --> WEBHOOK --> RUN --> ENF + LIB -. "provides libvgpu.so" .-> RUN + RUN -. "writes node-local cache" .-> MON -**KAI Scheduler joined this list in v0.16.4**, building HAMi-core in as the isolation engine for its GPU sharing. What this post explains is the role of each of the three components on the KAI Scheduler integration path, and how they connect. + style KAI fill:#d9f99d,stroke:#4f7d00,stroke-width:2px,color:#1f2937 + style WEBHOOK fill:#dbeafe,stroke:#1a5fb4,stroke-width:2px,color:#1f2937 + style LIB fill:#dbeafe,stroke:#1a5fb4,stroke-width:2px,color:#1f2937 + style RUN fill:#fef3c7,stroke:#b45309,stroke-width:2px,color:#1f2937 + style ENF fill:#dcfce7,stroke:#0b6b3c,stroke-width:2px,color:#1f2937 + style MON fill:#dbeafe,stroke:#1a5fb4,stroke-width:2px,color:#1f2937 +``` -## How it works: one contract, three steps +`CUDA_DEVICE_MEMORY_LIMIT` is the contract between scheduling and enforcement. KAI does not need to know how CUDA calls are intercepted, and HAMi-core does not need to know how the quota was calculated. KAI retains its own scheduling logic; it integrates with HAMi-core rather than replacing its scheduler with the full HAMi platform. -### What each of the three components is and does +At runtime, the dynamic linker loads `libvgpu.so` before the CUDA libraries. HAMi-core reads the injected quota, tracks the container's memory use, rewrites device queries so tools such as `nvidia-smi` show the quota, and returns an error when a new allocation would cross the limit. This is CUDA API-level enforcement, rather than an application voluntarily respecting a value. -The three components have similar-sounding names, so getting their roles straight is the prerequisite for understanding the whole chain: +## Standard integration on an existing GPU cluster -- **HAMi-core (`libvgpu.so`)**: HAMi's CUDA interception library (CNCF incubating), and **the isolation engine itself**. It intercepts CUDA calls (like `cudaMalloc`) inside the container via `LD_PRELOAD` and enforces a memory quota. It does not care who provided the quota: any scheduler that hands in the quota by convention gets isolation for free. Before KAI, it was already reused by HAMi's own device-plugin/webhook, Volcano's `volcano-vgpu-device-plugin`, and others. +This section assumes that the Kubernetes cluster already has working NVIDIA GPUs: nodes advertise `nvidia.com/gpu`, and an ordinary whole-GPU Pod can run `nvidia-smi`. It shows the standard integration path without the GKE-specific workarounds from Lab 12. -- **KAI Scheduler**: NVIDIA's open-source Kubernetes scheduler for AI workloads. It only owns the **scheduling layer**, deciding which node a pod lands on, which GPU it uses, and how much memory it gets. Since v0.16.4, its `hamicore` plugin writes the computed memory quota into the container's environment variable at bind time. +### 1. Install KAI Scheduler -- **`kai-resource-isolator`**: a companion component **provided by the HAMi project specifically for the KAI Scheduler integration path**. It ships HAMi-core's `libvgpu.so` to every GPU node and uses a MutatingWebhook to rewrite pods, injecting the library and `ld.so.preload`. In other words, it is the bridge that turns KAI's scheduling decision into isolation HAMi-core can actually enforce. +Enable GPU sharing and the `hamicore` binder plugin: -```mermaid -%% title: Responsibilities of the three components and how they connect -graph LR - subgraph SCH["Scheduling layer"] - KAI["KAI Scheduler
(hamicore plugin)

Role: compute the quota
bind Pod to a node
inject CUDA_DEVICE_MEMORY_LIMIT"] - end - subgraph BRIDGE["Bridge layer (provided by the HAMi project)"] - ISO["kai-resource-isolator

Role: ship libvgpu.so
webhook injects library + ld.so.preload"] - end - subgraph ISO2["Isolation layer"] - HAMI["HAMi-core / libvgpu.so
(CNCF incubating)

Role: intercept cudaMalloc
enforce the memory quota"] - end - - KAI -->|"CUDA_DEVICE_MEMORY_LIMIT
+ Pod bind result"| ISO - ISO -->|"libvgpu.so in place
ld.so.preload points at it"| HAMI - HAMI -.->|"same engine reused
(also by K8s / Volcano / Kueue / ...)"| SCH +```bash +helm install kai-scheduler \ + oci://ghcr.io/kai-scheduler/kai-scheduler/kai-scheduler \ + --namespace kai-scheduler --create-namespace \ + --version v0.17.0 \ + --set global.gpuSharing=true \ + --set binder.plugins.hamicore.enabled=true - style KAI fill:#d9f99d,stroke:#4f7d00,stroke-width:2px,color:#1f2937 - style ISO fill:#dbeafe,stroke:#1a5fb4,stroke-width:2px,color:#1f2937 - style HAMI fill:#fef3c7,stroke:#b45309,stroke-width:2px,color:#1f2937 - style SCH fill:#f7fee7,stroke:#4f7d00,stroke-width:1px,color:#1f2937 - style BRIDGE fill:#eff6ff,stroke:#1a5fb4,stroke-width:1px,color:#1f2937 - style ISO2 fill:#fffbeb,stroke:#b45309,stroke-width:1px,color:#1f2937 +kubectl -n kai-scheduler wait --for=condition=available \ + --timeout=180s deploy --all +kubectl -n kai-scheduler wait --for=condition=Ready \ + --timeout=300s config/kai-config +kubectl get pods -n kai-scheduler +kubectl get queues ``` -In one sentence: **KAI Scheduler computes the quota, `kai-resource-isolator` puts the isolation library in place, and HAMi-core actually enforces the isolation at runtime.** - -:::note Why HAMi-core, not the full HAMi platform - -KAI Scheduler integrates with **HAMi-core itself**, not the full HAMi platform. KAI keeps its own scheduling capability (it is not replaced by `hami-scheduler`) and only brings in HAMi-core for GPU memory isolation. This mirrors how Volcano works (using `volcano-vgpu-device-plugin` + HAMi-core): each scheduler keeps its own, and they share HAMi-core as the common isolation engine. - -::: - -### The contract: `CUDA_DEVICE_MEMORY_LIMIT` +A healthy installation has all KAI components running and the default queue hierarchy available. Pod suffixes vary: -The whole chain works because the scheduling layer and the isolation layer agreed on a minimal hand off point: the environment variable **`CUDA_DEVICE_MEMORY_LIMIT`**. - -- **KAI Scheduler (scheduling layer)** computes "how much memory this pod may use" and writes it into the container's environment variables when it binds the pod to a node. -- **HAMi-core (isolation layer)** reads that variable and, at runtime, actually keeps memory usage under that ceiling. +```text +NAME READY STATUS +admission-... 1/1 Running +binder-... 1/1 Running +kai-operator-... 1/1 Running +kai-scheduler-default-... 1/1 Running +pod-grouper-... 1/1 Running +podgroup-controller-... 1/1 Running +queue-controller-... 1/1 Running + +NAME PARENT +default-parent-queue +default-queue default-parent-queue +``` -This contract matters because it **fully decouples the two sides**: KAI does not need to know how CUDA is intercepted, and HAMi-core does not need to know how the share was computed. As long as both honor that one variable, any scheduler can reuse the same isolation engine. That is exactly why HAMi-core can support multiple schedulers at once (see "What this means" at the end). +### 2. Install kai-resource-isolator -### The three steps +Install the HAMi-core library distributor, injection webhook, and optional monitor: -```mermaid -%% title: Three step cooperation between KAI Scheduler and kai-resource-isolator -graph TD - KAI["KAI Scheduler
hamicore plugin
reads gpu-memory annotation"] - ENV["1. Inject at scheduling time
CUDA_DEVICE_MEMORY_LIMIT env var"] - ISO["2. kai-resource-isolator
Mutating Webhook rewrites Pod"] - LIB["libsync DaemonSet
distributes libvgpu.so to /usr/local/vgpu"] - MON["monitor DaemonSet
exposes hami_* metrics on :9394"] - RUN["3. Container starts
libvgpu.so intercepts cudaMalloc via LD_PRELOAD"] - ENF["Rejects over limit allocation
nvidia-smi shows only the quota"] - - KAI --> ENV --> ISO - ISO -->|"injects hostPath and ld.so.preload"| RUN --> ENF - LIB -. "provides libvgpu.so on the node" .-> RUN - MON -. "collects per container memory" .-> RUN +```bash +helm install kai-resource-isolator \ + oci://docker.io/projecthami/kai-resource-isolator \ + --namespace kai-resource-isolator --create-namespace \ + --version 1.1.0-chart \ + --set monitor.enabled=true - style KAI fill:#d9f99d,stroke:#4f7d00,stroke-width:2px,color:#1f2937 - style ENV fill:#d9f99d,stroke:#4f7d00,stroke-width:2px,color:#1f2937 - style ISO fill:#dbeafe,stroke:#1a5fb4,stroke-width:2px,color:#1f2937 - style LIB fill:#dbeafe,stroke:#1a5fb4,stroke-width:2px,color:#1f2937 - style MON fill:#dbeafe,stroke:#1a5fb4,stroke-width:2px,color:#1f2937 - style RUN fill:#fef3c7,stroke:#b45309,stroke-width:2px,color:#1f2937 - style ENF fill:#dcfce7,stroke:#0b6b3c,stroke-width:2px,color:#1f2937 +kubectl rollout status ds/kai-resource-isolator-libsync \ + -n kai-resource-isolator --timeout=300s +kubectl rollout status ds/kai-resource-isolator-monitor \ + -n kai-resource-isolator --timeout=300s +kubectl get pods -n kai-resource-isolator ``` -In one sentence: **KAI says you may only use this much, and the isolator makes sure you really can only use this much.** +There should be one ready libsync Pod and one ready monitor Pod per GPU node, plus a ready webhook: -The three components divide the work as follows (see the [KAI Scheduler docs on HAMi resource isolation](https://github.com/kai-scheduler/KAI-Scheduler/blob/main/docs/gpu-sharing/hami/README.md)): +```text +NAME READY STATUS +kai-resource-isolator-libsync-... 1/1 Running +kai-resource-isolator-monitor-... 1/1 Running +kai-resource-isolator-webhook-... 1/1 Running +``` -- **libsync DaemonSet** copies `libvgpu.so` to `/usr/local/vgpu` on every GPU node. -- **mutating webhook** injects the hostPath volume mount into the pod, points `/etc/ld.so.preload` at `libvgpu.so`, and writes the `POD_UID`, `CONTAINER_NAME`, and `CONTAINER_VGPU_MOUNT` environment variables. -- **monitor DaemonSet** (optional) reads each container shared memory cache and exposes metrics on `:9394`. +### 3. Run a shared GPU Pod + +The `gpu-memory` annotation is an integer number of MiB without a suffix. The queue label and `schedulerName` send the Pod through KAI Scheduler: + +```bash +kubectl apply -f - <<'EOF' +apiVersion: v1 +kind: Pod +metadata: + name: kai-hami-check + labels: + kai.scheduler/queue: default-queue + annotations: + gpu-memory: "4096" +spec: + schedulerName: kai-scheduler + containers: + - name: cuda + image: nvidia/cuda:12.4.1-base-ubuntu22.04 + command: ["sleep", "infinity"] +EOF + +kubectl wait --for=condition=Ready pod/kai-hami-check --timeout=5m +kubectl get pod kai-hami-check -o wide +``` -### How the CUDA interception works +The Pod should be `Running` on a GPU node: -The "last mile" of isolation happens inside the container process. The chain is: +```text +NAME READY STATUS NODE +kai-hami-check 1/1 Running gpu-node-1 +``` -1. **KAI injects `CUDA_DEVICE_MEMORY_LIMIT` at scheduling time** (carrying the pod's requested memory quota, in MiB). -2. **The kai-resource-isolator webhook rewrites the pod**: it mounts the host's `libvgpu.so` and points `/etc/ld.so.preload` at it. `ld.so.preload` is a dynamic linker mechanism: any shared library listed there is loaded before all other libraries. -3. **Once the container process starts**, every call into the CUDA runtime (`libcudart`) or driver API passes through `libvgpu.so` first. It intercepts memory allocation calls like `cudaMalloc`, reads the quota from `CUDA_DEVICE_MEMORY_LIMIT`, accumulates the container's memory usage, and rejects any allocation that would exceed the quota. -4. **The visible effect**: `nvidia-smi` shows only the quota memory (HAMi-core rewrites the device query responses), and no matter how hard the container calls `cudaMalloc`, it cannot cross that line. +### 4. Check the scheduling-to-runtime handoff -That is what "hard isolation" means: it is not left to application discipline, but enforced at the CUDA call layer. +Inspect the quota from KAI, the preload file injected by the isolator, and the memory visible through HAMi-core: -:::tip How this integration came together +```bash +kubectl exec kai-hami-check -- sh -lc ' + printf "limit=%s\n" "$CUDA_DEVICE_MEMORY_LIMIT" + cat /etc/ld.so.preload + nvidia-smi --query-gpu=uuid,memory.total --format=csv,noheader +' +``` -This integration path is the result of more than a year of work between the HAMi community and the NVIDIA KAI Scheduler team. The split is clean: KAI injects the environment variable, HAMi provides the resource isolation components. The full timeline and contributors are in the companion post [HAMi-core adopted by NVIDIA KAI Scheduler](/blog/hami-core-adopted-by-nvidia-kai-scheduler). +For a 4096 MiB request on a 15360 MiB T4, the verified output was: -::: +```text +limit=4147m +/usr/local/vgpu/libvgpu.so +GPU-9acc8878-3967-5fb4-c534-43d6fd820fa6, 4147 MiB +``` -## What these two versions bring +These three lines prove that the integration chain is active: KAI supplied a quota, the isolator injected HAMi-core, and the container sees the resulting limit instead of the full card. The value is 4147 rather than exactly 4096 MiB because KAI converts the request to a two-decimal GPU fraction before calculating the enforced limit. + +### 5. Check the optional monitor + +Each monitor reads caches from its own node, so select the monitor Pod running on the workload node. Keep the port-forward command running in one terminal: + +```bash +export WORKLOAD_NODE=$(kubectl get pod kai-hami-check \ + -o jsonpath='{.spec.nodeName}') +export MONITOR_POD=$(kubectl get pods -n kai-resource-isolator \ + -l app.kubernetes.io/component=kai-vgpu-monitor \ + --field-selector="spec.nodeName=$WORKLOAD_NODE" \ + -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' | head -n 1) +test -n "$MONITOR_POD" || { + echo "No monitor Pod is running on $WORKLOAD_NODE" >&2 + exit 1 +} +kubectl port-forward -n kai-resource-isolator \ + "pod/$MONITOR_POD" 9394:9394 +``` -### KAI Scheduler (HAMi-core hard isolation built in since v0.16.4) +From another terminal, confirm that the endpoint contains a series for the Pod: -KAI Scheduler's support for HAMi-core first appeared in **v0.16.4**. The current integration documentation requires KAI Scheduler v0.17.0 or later, and this post tests v0.17.0. The key piece is the `hamicore` plugin: once enabled, when KAI binds a shared GPU pod to a node, it injects the `CUDA_DEVICE_MEMORY_LIMIT` environment variable into the container based on the `gpu-memory` (or `gpu-fraction`) annotation, exactly the quota that HAMi-core needs to enforce isolation, per the contract above. +```bash +curl -s http://127.0.0.1:9394/metrics | grep \ + 'hami_vgpu_memory_limit_bytes.*pod="kai-hami-check"' +``` -Other GPU-related changes in v0.17.0 include: fixing invalid volume names caused by `/` in shared pod names; correcting the allocation math for `MinNodeGPUMemoryMiB` and fractional `gpu-memory`; and using the largest GPU profile in the cluster for overLimit decisions. It also adds preemption-delay (a time window for Cluster Autoscaler to bring up nodes), NUMA-aware scoring, and GitOps and ArgoCD installation support. +Expected shape: -### kai-resource-isolator 1.1.0-chart +```text +hami_vgpu_memory_limit_bytes{...,pod="kai-hami-check",...} 4.348444672e+09 +``` -This is the isolator shipped alongside HAMi. It receives the quota injected by KAI and, before the container actually starts, puts the HAMi-core `libvgpu.so` in place. Compared with the first release, 1.1.0 adds several operational improvements: +This is enough for a basic integration check. To prove that the displayed ceiling is actually enforced, run a CUDA allocation test that succeeds below the quota and fails above it; the next section summarizes that result, and Lab 12 provides the complete program and procedure. -- **New `kai-vgpu-monitor`**: runs as a DaemonSet, exposes HAMi-compatible metrics on `:9394` (`hami_vgpu_memory_used_bytes`, `hami_vgpu_memory_limit_bytes`, `hami_container_device_utilization_ratio`), supports ServiceMonitor, and can be scraped directly by Prometheus. -- **Multi container injection fix**: when a pod has multiple containers, the webhook now handles them correctly and no longer skips any. -- **Security tightening**: the webhook now uses a namespaced Issuer (instead of a ClusterIssuer), and the ClusterRole no longer reads Secrets. -- **Global image repository** precedence cleaned up, and `hamicore` installation parameters corrected. +Remove the basic test Pod when finished: -## GKE Verification: Does the Isolation Actually Hold? +```bash +kubectl delete pod kai-hami-check +``` -We completed an end-to-end test on a GKE 1.35/COS/CDI cluster with three `n1-standard-2` nodes, each carrying one NVIDIA T4. KAI Scheduler v0.17.0 handled shared scheduling, while `kai-resource-isolator` 1.1.0-chart injected HAMi-core. +## GKE verification: does the isolation hold? -The verification went beyond `nvidia-smi`: +We ran the integration on a GKE 1.35/COS/CDI cluster with three `n1-standard-2` nodes, each carrying one NVIDIA T4. KAI Scheduler v0.17.0 handled shared scheduling, while `kai-resource-isolator` 1.1.0-chart injected HAMi-core. | Check | Captured result | What it proves | | :-- | :-- | :-- | @@ -173,7 +226,7 @@ The verification went beyond `nvidia-smi`: | Visible memory | Both Pods reported `4147 MiB`; the full card reported `15360 MiB` | HAMi-core exposed KAI's per-Pod quota | | CUDA allocation | 3 GiB succeeded; a cumulative 5 GiB returned `out of memory` | The ceiling was enforced, not merely displayed | | Concurrent isolation | Pod B still allocated its own 3 GiB while Pod A held 3 GiB | One Pod could not consume the other's quota | -| Monitor metrics | The same-node `:9394/metrics` endpoint reported both Pods' 4.348 GB limits and 3.328 GB live usage | The monitor read each container's shared-memory cache and exported non-empty per-Pod gauges | +| Monitor metrics | The same-node `:9394/metrics` endpoint reported both Pods' 4.348 GB limits and 3.328 GB live usage | The monitor exported non-empty per-Pod gauges from each container's cache | HAMi-core logged the over-quota allocation as: @@ -183,7 +236,7 @@ allocate another 2 GiB: out of memory PASS: in-quota allocation succeeded and over-quota allocation failed ``` -Together, these checks connect scheduling onto one card, per-container visibility, actual CUDA allocation enforcement, and observability into one evidence chain. Because the monitor runs as a DaemonSet and reads node-local caches, Lab 12 queries the instance on the workload node directly instead of relying on a Service that may select another node. +Together, these checks connect scheduling onto one card, per-container visibility, actual CUDA allocation enforcement, and observability into one evidence chain. Because the monitor reads node-local caches, the lab queries the monitor instance on the workload node directly instead of using a Service that may select another node. :::note Isolation boundary @@ -191,29 +244,14 @@ This proves CUDA API-level memory enforcement, not a MIG-like hardware security ::: -### Why the full procedure is not in this post +## Reproduce the result -The standard KAI + HAMi-core path is short. GKE 1.35/COS/CDI adds version-specific concerns around the read-only root filesystem, RuntimeClass, NVML library paths, CDI device injection, PriorityClass, and `kubectl exec` WebSocket resets. Those steps need independent maintenance and fit a reproducible lab better than the narrative of this post. - -The complete cluster setup, manifests, Kyverno policies, CUDA program, monitor verification, captured outputs, troubleshooting table, and cleanup commands are in: +The standard KAI + HAMi-core installation is short. GKE 1.35/COS/CDI additionally requires environment-specific handling for the read-only root filesystem, RuntimeClass, NVML library paths, CDI device injection, and PriorityClass. Those operational details and their captured command output are maintained in: **[Lab 12: Verify KAI Scheduler and HAMi Memory Isolation on GKE](/tutorials/labs/kai-scheduler-hami-gke)** -If you only need the architecture, stop here. If you want to reproduce it on GKE, continue with Lab 12. - -## What this means - -HAMi-core was never positioned as "the isolation feature of some particular scheduler"; it is a **scheduler-decoupled, reusable isolation base**. Well before KAI Scheduler, it already powered the Kubernetes native scheduler, [Kueue](/docs/userguide/kueue/how-to-use-kueue), [Volcano](/docs/installation/how-to-use-volcano-vgpu), Koordinator, and other paths (see [Ecosystem Integrations](/docs/next/core-concepts/ecosystem-integrations)). KAI Scheduler v0.16.4 joining the list extends this ecosystem to NVIDIA's official AI scheduler: - -- For **KAI users**: GPU sharing finally has a matching runtime hard isolation, so sharing no longer means running without guarantees. -- For **HAMi users**: there is now a path that does not lock you to a specific virtual device plugin and goes directly through NVIDIA's official scheduler, while keeping the metric surface (`hami_*`) compatible. -- For the **community**: the contract between the scheduling layer and the isolation layer (`CUDA_DEVICE_MEMORY_LIMIT`) is validated once more; any future scheduler that honors it can reuse the same isolation engine. - -Behind this is more than a year of careful alignment between the KAI Scheduler team (Run:ai) and the HAMi maintainers. The `LD_PRELOAD`, the webhook, the metrics port, and the opt-out switch were each settled by both sides; the security tightening in `kai-resource-isolator` 1.1.0 (namespaced Issuer, tightened ClusterRole) was polished through community review line by line. - ## Next steps -- The background story: [HAMi-core adopted by NVIDIA KAI Scheduler](/blog/hami-core-adopted-by-nvidia-kai-scheduler) -- User docs: [How to use HAMi with KAI Scheduler](/docs/next/userguide/kai-scheduler/how-to-use-kai-scheduler) -- Related repos: [Project-HAMi/KAI-resource-isolator](https://github.com/Project-HAMi/KAI-resource-isolator), [Project-HAMi/HAMi-core](https://github.com/Project-HAMi/HAMi-core) (the CNCF incubating CUDA interception library), [kai-scheduler/KAI-Scheduler](https://github.com/kai-scheduler/KAI-Scheduler), [KAI Scheduler HAMi resource isolation docs](https://github.com/kai-scheduler/KAI-Scheduler/blob/main/docs/gpu-sharing/hami/README.md) -- Run it on your own GKE, AWS, or self-built cluster, and share real results in an issue or the community group. If HAMi-core lacks support for a certain card or CUDA version, open an issue at [Project-HAMi/HAMi](https://github.com/Project-HAMi/HAMi). This is the feedback the community values most. +- Background and collaboration history: [HAMi-core adopted by NVIDIA KAI Scheduler](/blog/hami-core-adopted-by-nvidia-kai-scheduler) +- User documentation: [How to use HAMi with KAI Scheduler](/docs/next/userguide/kai-scheduler/how-to-use-kai-scheduler) +- Repositories: [KAI-resource-isolator](https://github.com/Project-HAMi/KAI-resource-isolator), [HAMi-core](https://github.com/Project-HAMi/HAMi-core), and [KAI-Scheduler](https://github.com/kai-scheduler/KAI-Scheduler) diff --git a/i18n/zh/docusaurus-plugin-content-blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md b/i18n/zh/docusaurus-plugin-content-blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md index 0517ac3a0..74fa1d917 100644 --- a/i18n/zh/docusaurus-plugin-content-blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md +++ b/i18n/zh/docusaurus-plugin-content-blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md @@ -1,181 +1,234 @@ --- title: "KAI Scheduler 与 HAMi 的 GPU 显存硬隔离：运行原理与实践" date: "2026-08-11" -description: "从 CUDA 拦截原理到 GKE 上的可复现验证：用 KAI Scheduler v0.17.0 做 GPU 共享调度、kai-resource-isolator 1.1.0-chart（基于 HAMi-core）做 CUDA 级显存隔离，在单张 NVIDIA T4 上让两个 Pod 各自只见自己的显存配额。讲清 CUDA_DEVICE_MEMORY_LIMIT 这一调度层与隔离层的契约，以及节点标签、队列、RuntimeClass 等关键注意事项。" +description: "简要说明 KAI Scheduler 与 HAMi-core 的隔离链路，并通过可复现的 GKE 实测证明：共享同一张 NVIDIA T4 的两个 Pod 都无法越过各自的显存配额。" authors: [rootsongjc] tags: ["HAMi", "KAI Scheduler", "硬隔离", "GPU 共享", "Kubernetes", "云原生"] --- -GPU 共享在 Kubernetes 生态里讨论了很多年，但调度层与隔离层长期各自为政：调度器把几个 Pod 分到同一张卡，容器进入 GPU 后却依然能看到整张卡的显存，谁先发起 `cudaMalloc` 谁就占满，隔离形同虚设。所谓“共享”其实只是“抢”，没有任何资源保障可言。 +上一篇 [《HAMi-core 被 NVIDIA KAI Scheduler 采用》](/zh/blog/hami-core-adopted-by-nvidia-kai-scheduler)已经介绍了 KAI Scheduler 和这项集成的协作背景。本文不再重复铺垫，只回答一个问题：**KAI Scheduler 把两个 Pod 调度到同一张 GPU 后，HAMi-core 是否真的能限制每个 Pod 的显存用量？** -要解决这个问题，需要调度层（决定“谁能用哪张卡、用多少”）和隔离层（保证“说好用多少就只能用多少”）协同。**HAMi-core** 正是这样一个可被多种调度器复用的隔离底座。在 KAI Scheduler 之前，它已经支持 Kubernetes 原生调度器（经 HAMi 自带的 `hami-scheduler` 扩展）、[Kueue](/zh/docs/userguide/kueue/how-to-use-kueue)、[Volcano](/zh/docs/installation/how-to-use-volcano-vgpu) 等（完整生态见 [HAMi 生态集成](/zh/docs/next/core-concepts/ecosystem-integrations)）。 +我们在 GKE 1.35/COS/CDI 上验证了当前文档支持的组合：KAI Scheduler v0.17.0 与 `kai-resource-isolator` 1.1.0-chart。两个 Pod 共享同一张 NVIDIA T4，各自看到 4147 MiB 上限；申请 3 GiB 成功，累计申请 5 GiB 失败。可选的 monitor 也导出了两个 Pod 的实时显存上限与用量。 -**自 KAI Scheduler v0.16.4 起，NVIDIA 的 KAI Scheduler 也正式加入这一行列**，把 HAMi-core 作为其 GPU 共享的隔离引擎内置支持。本文验证当前文档支持的组合：KAI Scheduler v0.17.0 与 `kai-resource-isolator` 1.1.0-chart。启用这条集成后，GPU 共享不再只是“协作式共享”，而是在 CUDA API 层执行显存上限。本文讲清两件事： +:::note 关于实测输出 -- **原理**：KAI Scheduler、`kai-resource-isolator`、HAMi-core 三者各自负责什么、如何衔接，以及 `CUDA_DEVICE_MEMORY_LIMIT` 这一契约如何把调度层与隔离层连起来。 -- **实践**：一套在 GKE 上完成的端到端验证（单张 NVIDIA T4 卡，两个 Pod 共享），并通过 `cudaMalloc` 证明显存配额无法被越过。完整复现步骤见实验 12。 - -背景故事与协作时间线见 [《HAMi-core 被 NVIDIA KAI Scheduler 采用：GPU 共享正式迈入硬隔离时代》](/zh/blog/hami-core-adopted-by-nvidia-kai-scheduler)。 - -:::note 关于本文的输出 - -下半部分的命令已在本文的 GKE 1.35/COS/CDI 集群上实际执行。标为“实测”的 UUID、显存上限、错误信息和 CUDA 分配结果来自该次运行；标为“示意”的资源名称与地址经过脱敏，实际值以你的集群环境为准。 +下文的 UUID、显存上限、CUDA 分配结果和 monitor 指标均来自该次 GKE 实测。资源后缀和地址在不同集群中会变化。 ::: -## 背景：为什么“共享”不等于“隔离” +## 集成如何工作 -| 层次 | 负责方 | 解决了什么 | 没有解决什么 | -| :-- | :-- | :-- | :-- | -| 调度层 | KAI Scheduler、Volcano、Kueue 等 | 多个 Pod 能被分到同一张 GPU | 容器内仍可见全部显存 | -| 运行时层 | HAMi-core（`libvgpu.so`） | 拦截 CUDA 调用，按配额限制显存 | 单独使用时，不知道每个 Pod 该分多少 | +整条链路只有三项职责： -要实现真正的 GPU 共享，这两层缺一不可，而且必须协同：调度层决定“谁能用哪张卡、用多少”，隔离层保证“说好用多少就只能用多少”。问题是，**隔离层需要知道“到底用多少”这个数字，而它本身是算不出来的，这个数字来自调度层**。 +- **KAI Scheduler** 决定 Pod 使用哪张 GPU，并通过 `CUDA_DEVICE_MEMORY_LIMIT` 注入计算出的显存配额。 +- **`kai-resource-isolator`** 分发 `libvgpu.so`；其 webhook 为业务 Pod 挂载该库并配置 `ld.so.preload`。可选的 monitor 读取节点本地 HAMi 缓存，在 `:9394` 暴露 `hami_*` 指标。 +- **HAMi-core（`libvgpu.so`）** 拦截 `cudaMalloc` 等 CUDA 调用，拒绝超出配额的显存分配。 -HAMi 社区多年打磨的 HAMi-core（CNCF 孵化项目）正是这样的隔离层，而且它是**与调度器解耦**的：在 KAI Scheduler 之前，HAMi-core 已经通过不同的方式与 Kubernetes 原生调度器（经 HAMi 自带的 `hami-scheduler` 扩展）、[Kueue](/zh/docs/userguide/kueue/how-to-use-kueue)、[Volcano](/zh/docs/installation/how-to-use-volcano-vgpu)、Koordinator 等协同工作（完整生态见 [HAMi 生态集成](/zh/docs/next/core-concepts/ecosystem-integrations)）。 +```mermaid +%% title: KAI Scheduler 与 HAMi-core 的隔离链路 +graph TD + KAI["KAI Scheduler
计算 GPU 显存配额"] + ENV["注入 CUDA_DEVICE_MEMORY_LIMIT"] + WEBHOOK["kai-resource-isolator webhook
挂载 hostPath 与 ld.so.preload"] + LIB["libsync DaemonSet
在各 GPU 节点提供 libvgpu.so"] + RUN["HAMi-core 拦截 CUDA 调用"] + ENF["拒绝超出配额的分配"] + MON["monitor DaemonSet
在 :9394 暴露 hami_* 指标"] + + KAI --> ENV --> WEBHOOK --> RUN --> ENF + LIB -. "提供 libvgpu.so" .-> RUN + RUN -. "写入节点本地缓存" .-> MON -**KAI Scheduler 在 v0.16.4 正式加入这一行列**，把 HAMi-core 作为其 GPU 共享的隔离引擎内置支持。本文要讲清的，就是 KAI Scheduler 这条集成路径上三个组件各自的角色，以及它们之间如何衔接。 + style KAI fill:#d9f99d,stroke:#4f7d00,stroke-width:2px,color:#1f2937 + style WEBHOOK fill:#dbeafe,stroke:#1a5fb4,stroke-width:2px,color:#1f2937 + style LIB fill:#dbeafe,stroke:#1a5fb4,stroke-width:2px,color:#1f2937 + style RUN fill:#fef3c7,stroke:#b45309,stroke-width:2px,color:#1f2937 + style ENF fill:#dcfce7,stroke:#0b6b3c,stroke-width:2px,color:#1f2937 + style MON fill:#dbeafe,stroke:#1a5fb4,stroke-width:2px,color:#1f2937 +``` -## 运行原理：一个契约、三步协作 +`CUDA_DEVICE_MEMORY_LIMIT` 是调度层与隔离层之间的契约。KAI 不需要知道 CUDA 调用如何被拦截，HAMi-core 也不需要知道配额如何计算。KAI 保留自己的调度逻辑；它集成的是 HAMi-core，而不是用完整的 HAMi 平台替换自身调度器。 -### 三个组件各是什么、各做什么 +容器启动时，动态链接器会先于 CUDA 库加载 `libvgpu.so`。HAMi-core 读取注入的配额，跟踪容器显存用量，改写设备查询结果，使 `nvidia-smi` 等工具只显示该配额；新的分配一旦越界就返回错误。这是在 CUDA API 层强制执行，并非依赖应用自觉遵守一个数值。 -先把三个名字容易混淆的组件讲清楚，这是理解整条链路的前提： +## 在已有 GPU 集群上的标准集成步骤 -- **HAMi-core（`libvgpu.so`）**：HAMi 项目的 CUDA 拦截库，是**隔离引擎本身**。它通过 `LD_PRELOAD` 拦截容器里的 CUDA 调用（如 `cudaMalloc`），按一个显存配额强制限制。它不关心配额是谁给的：任何调度器只要按约定把配额传进来，它都能执行隔离。在 KAI 之前，它已经被 HAMi 自带的 device-plugin/webhook、Volcano 的 `volcano-vgpu-device-plugin` 等复用。 +下面默认 Kubernetes 集群中的 NVIDIA GPU 已经可用：节点能够上报 `nvidia.com/gpu`，普通整卡 Pod 也能正常执行 `nvidia-smi`。这里只写标准集成路径，不包含实验 12 中针对 GKE 的特殊适配。 -- **KAI Scheduler**：NVIDIA 开源的 Kubernetes AI 工作负载调度器。它只负责**调度层**，即决定 Pod 落到哪个节点、用哪张 GPU、分多少显存。自 v0.16.4 起，它的 `hamicore` 插件在绑定 Pod 时，会把计算出的显存配额写进容器的环境变量。 +### 1. 安装 KAI Scheduler -- **`kai-resource-isolator`**：HAMi 项目侧为 KAI Scheduler 这条集成路径**专门提供的配套组件**。它把 HAMi-core 的 `libvgpu.so` 分发到每个 GPU 节点，并用 MutatingWebhook 改写 Pod，把库和 `ld.so.preload` 注入进去。换言之，它是“把 KAI 的调度决策落地成 HAMi-core 能执行的隔离”的桥梁。 +启用 GPU 共享与 `hamicore` binder 插件： -```mermaid -%% title: 三个组件的职责与衔接 -graph LR - subgraph SCH["调度层"] - KAI["KAI Scheduler
（hamicore 插件）

职责：算配额
把 Pod 绑定到节点
注入 CUDA_DEVICE_MEMORY_LIMIT"] - end - subgraph BRIDGE["衔接层（HAMi 项目提供）"] - ISO["kai-resource-isolator

职责：分发 libvgpu.so
webhook 注入库与 ld.so.preload"] - end - subgraph ISO2["隔离层"] - HAMI["HAMi-core / libvgpu.so
（CNCF 孵化）

职责：拦截 cudaMalloc
按配额强制限制显存"] - end - - KAI -->|"CUDA_DEVICE_MEMORY_LIMIT
+ Pod 绑定结果"| ISO - ISO -->|"libvgpu.so 就位
ld.so.preload 指向它"| HAMI - HAMI -.->|"复用同一引擎
（也被 K8s/Volcano/Kueue 等复用）"| SCH +```bash +helm install kai-scheduler \ + oci://ghcr.io/kai-scheduler/kai-scheduler/kai-scheduler \ + --namespace kai-scheduler --create-namespace \ + --version v0.17.0 \ + --set global.gpuSharing=true \ + --set binder.plugins.hamicore.enabled=true - style KAI fill:#d9f99d,stroke:#4f7d00,stroke-width:2px,color:#1f2937 - style ISO fill:#dbeafe,stroke:#1a5fb4,stroke-width:2px,color:#1f2937 - style HAMI fill:#fef3c7,stroke:#b45309,stroke-width:2px,color:#1f2937 - style SCH fill:#f7fee7,stroke:#4f7d00,stroke-width:1px,color:#1f2937 - style BRIDGE fill:#eff6ff,stroke:#1a5fb4,stroke-width:1px,color:#1f2937 - style ISO2 fill:#fffbeb,stroke:#b45309,stroke-width:1px,color:#1f2937 +kubectl -n kai-scheduler wait --for=condition=available \ + --timeout=180s deploy --all +kubectl -n kai-scheduler wait --for=condition=Ready \ + --timeout=300s config/kai-config +kubectl get pods -n kai-scheduler +kubectl get queues ``` -一句话：**KAI Scheduler 算配额，`kai-resource-isolator` 把隔离库就位，HAMi-core 在运行时真正执行隔离。** - -:::note 为什么是 HAMi-core，不是完整 HAMi 平台 - -KAI Scheduler 的集成目标是 **HAMi-core 本身**，而不是完整的 HAMi 平台。KAI 保留自己的调度能力（不替换成 `hami-scheduler`），只引入 HAMi-core 来做 GPU 显存隔离。这与 Volcano（用 `volcano-vgpu-device-plugin` + HAMi-core）的分工模式是同一思路：调度器各用各的，隔离引擎共用 HAMi-core。 - -::: - -### 契约：`CUDA_DEVICE_MEMORY_LIMIT` +正常情况下，KAI 的所有组件都处于运行状态，并且默认父子队列已经创建。Pod 后缀会因环境而异： -整条链路之所以能成立，是因为调度层和隔离层约定了一个极简的交接点：环境变量 **`CUDA_DEVICE_MEMORY_LIMIT`**。 - -- **KAI Scheduler（调度层）** 负责算出“这个 Pod 能用多少显存”，并在绑定节点时把它写进容器的环境变量。 -- **HAMi-core（隔离层）** 负责读这个环境变量，并在运行时真正把显存用量卡在这个上限以内。 +```text +NAME READY STATUS +admission-... 1/1 Running +binder-... 1/1 Running +kai-operator-... 1/1 Running +kai-scheduler-default-... 1/1 Running +pod-grouper-... 1/1 Running +podgroup-controller-... 1/1 Running +queue-controller-... 1/1 Running + +NAME PARENT +default-parent-queue +default-queue default-parent-queue +``` -这个契约之所以重要，是因为它**把两件事彻底解耦**：KAI 不需要知道 CUDA 怎么被拦截，HAMi-core 不需要知道份额是怎么算出来的。两边只要都遵守 `CUDA_DEVICE_MEMORY_LIMIT` 这一个变量，任何调度器都能复用同一套隔离引擎。这正是 HAMi-core 能同时支持多个调度器的根本原因（见文末“这意味着什么”）。 +### 2. 安装 kai-resource-isolator -### 三步协作 +安装 HAMi-core 库分发组件、注入 webhook 和可选的 monitor： -```mermaid -%% title: KAI Scheduler 与 kai-resource-isolator 的三步协作 -graph TD - KAI["KAI Scheduler
hamicore 插件
读取 gpu-memory 注解"] - ENV["1. 调度时注入
CUDA_DEVICE_MEMORY_LIMIT 环境变量"] - ISO["2. kai-resource-isolator
Mutating Webhook 改写 Pod"] - LIB["libsync DaemonSet
分发 libvgpu.so 到 /usr/local/vgpu"] - MON["monitor DaemonSet
:9394 暴露 hami_* 指标"] - RUN["3. 容器启动
libvgpu.so 经 LD_PRELOAD 拦截 cudaMalloc"] - ENF["按 LIMIT 拒绝超额分配
nvidia-smi 仅显示配额内显存"] - - KAI --> ENV --> ISO - ISO -->|"注入 hostPath 与 ld.so.preload"| RUN --> ENF - LIB -. "在节点上提供 libvgpu.so" .-> RUN - MON -. "采集各容器显存" .-> RUN +```bash +helm install kai-resource-isolator \ + oci://docker.io/projecthami/kai-resource-isolator \ + --namespace kai-resource-isolator --create-namespace \ + --version 1.1.0-chart \ + --set monitor.enabled=true - style KAI fill:#d9f99d,stroke:#4f7d00,stroke-width:2px,color:#1f2937 - style ENV fill:#d9f99d,stroke:#4f7d00,stroke-width:2px,color:#1f2937 - style ISO fill:#dbeafe,stroke:#1a5fb4,stroke-width:2px,color:#1f2937 - style LIB fill:#dbeafe,stroke:#1a5fb4,stroke-width:2px,color:#1f2937 - style MON fill:#dbeafe,stroke:#1a5fb4,stroke-width:2px,color:#1f2937 - style RUN fill:#fef3c7,stroke:#b45309,stroke-width:2px,color:#1f2937 - style ENF fill:#dcfce7,stroke:#0b6b3c,stroke-width:2px,color:#1f2937 +kubectl rollout status ds/kai-resource-isolator-libsync \ + -n kai-resource-isolator --timeout=300s +kubectl rollout status ds/kai-resource-isolator-monitor \ + -n kai-resource-isolator --timeout=300s +kubectl get pods -n kai-resource-isolator ``` -一句话概括：**KAI 说“你只能用这么多”，isolator 负责“让你真的只能用这么多”。** +每个 GPU 节点上都应该有一个就绪的 libsync Pod 和一个就绪的 monitor Pod，同时 webhook 也应处于就绪状态： -三个组件的分工如下（详见 [KAI Scheduler 官方文档“HAMi 资源隔离”](https://github.com/kai-scheduler/KAI-Scheduler/blob/main/docs/gpu-sharing/hami/README.md)）： +```text +NAME READY STATUS +kai-resource-isolator-libsync-... 1/1 Running +kai-resource-isolator-monitor-... 1/1 Running +kai-resource-isolator-webhook-... 1/1 Running +``` -- **libsync DaemonSet**：把 `libvgpu.so` 复制到每个 GPU 节点的 `/usr/local/vgpu`。 -- **mutating webhook**：给 Pod 注入 hostPath 卷挂载，把 `/etc/ld.so.preload` 指向 `libvgpu.so`，并写入 `POD_UID`、`CONTAINER_NAME`、`CONTAINER_VGPU_MOUNT` 环境变量。 -- **monitor DaemonSet**（可选）：读取各容器的共享内存缓存，在 `:9394` 暴露指标。 +### 3. 运行一个共享 GPU Pod + +`gpu-memory` 注解使用不带单位后缀的整数 MiB。队列标签与 `schedulerName` 会让 Pod 经 KAI Scheduler 调度： + +```bash +kubectl apply -f - <<'EOF' +apiVersion: v1 +kind: Pod +metadata: + name: kai-hami-check + labels: + kai.scheduler/queue: default-queue + annotations: + gpu-memory: "4096" +spec: + schedulerName: kai-scheduler + containers: + - name: cuda + image: nvidia/cuda:12.4.1-base-ubuntu22.04 + command: ["sleep", "infinity"] +EOF + +kubectl wait --for=condition=Ready pod/kai-hami-check --timeout=5m +kubectl get pod kai-hami-check -o wide +``` -### CUDA 拦截是如何生效的 +Pod 应该处于 `Running` 状态，并被调度到一个 GPU 节点： -隔离的“最后一公里”发生在容器进程里，链路是这样的： +```text +NAME READY STATUS NODE +kai-hami-check 1/1 Running gpu-node-1 +``` -1. **KAI 在调度时注入 `CUDA_DEVICE_MEMORY_LIMIT`**（带上 Pod 申请的显存配额，单位 MiB）。 -2. **kai-resource-isolator 的 webhook 改写 Pod**：挂载宿主机上的 `libvgpu.so`，并把 `/etc/ld.so.preload` 指向它。`ld.so.preload` 是动态链接器的机制，列在其中的共享库会在所有其他库之前加载。 -3. **容器进程启动后**，任何对 CUDA 运行时（`libcudart`）或驱动 API 的调用，都会先经过 `libvgpu.so`。后者拦截 `cudaMalloc` 之类的显存分配调用，从 `CUDA_DEVICE_MEMORY_LIMIT` 读出配额，累计该容器的显存用量；一旦超额就拒绝分配。 -4. **对外可见的效果**：`nvidia-smi` 只显示配额内的显存（HAMi-core 会改写设备查询的返回值），容器再怎么 `cudaMalloc` 也越不过这条线。 +### 4. 检查调度层到运行时的交接 -这就是“硬隔离”的含义：不是靠应用自觉，而是在 CUDA 调用这一层强制执行。 +检查 KAI 提供的配额、isolator 注入的 preload 文件，以及 HAMi-core 向容器暴露的显存： -:::tip 集成的来龙去脉 +```bash +kubectl exec kai-hami-check -- sh -lc ' + printf "limit=%s\n" "$CUDA_DEVICE_MEMORY_LIMIT" + cat /etc/ld.so.preload + nvidia-smi --query-gpu=uuid,memory.total --format=csv,noheader +' +``` -这条集成路径是 HAMi 社区与 NVIDIA KAI Scheduler 团队一年多协作的结果，分工很清晰：KAI 负责注入环境变量，HAMi 负责资源隔离组件。完整的时间线与参与人员见姊妹篇 [《HAMi-core 被 NVIDIA KAI Scheduler 采用》](/zh/blog/hami-core-adopted-by-nvidia-kai-scheduler)。 +在一张 15360 MiB T4 上申请 4096 MiB 时，实测输出为： -::: +```text +limit=4147m +/usr/local/vgpu/libvgpu.so +GPU-9acc8878-3967-5fb4-c534-43d6fd820fa6, 4147 MiB +``` -## 这两个版本带来了什么 +这三行可以证明集成链路已经生效：KAI 提供了显存配额，isolator 注入了 HAMi-core，容器看到的是隔离后的上限而非整卡显存。结果是 4147 而不是刚好 4096 MiB，是因为 KAI 会先把请求换算成两位小数的 GPU fraction，再计算最终强制上限。 + +### 5. 检查可选的 monitor + +每个 monitor 只读取本节点上的缓存，因此要选择与业务 Pod 位于同一节点的 monitor。在一个终端中保持下面的端口转发命令运行： + +```bash +export WORKLOAD_NODE=$(kubectl get pod kai-hami-check \ + -o jsonpath='{.spec.nodeName}') +export MONITOR_POD=$(kubectl get pods -n kai-resource-isolator \ + -l app.kubernetes.io/component=kai-vgpu-monitor \ + --field-selector="spec.nodeName=$WORKLOAD_NODE" \ + -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' | head -n 1) +test -n "$MONITOR_POD" || { + echo "$WORKLOAD_NODE 上没有运行 monitor Pod" >&2 + exit 1 +} +kubectl port-forward -n kai-resource-isolator \ + "pod/$MONITOR_POD" 9394:9394 +``` -### KAI Scheduler（自 v0.16.4 起内置 HAMi-core 硬隔离） +在另一个终端确认端点中存在该 Pod 的指标： -KAI Scheduler 对 HAMi-core 的支持最早出现在 **v0.16.4**。当前集成文档要求 KAI Scheduler v0.17.0 或更高版本，本文实测 v0.17.0。关键在于 `hamicore` 插件：启用后，KAI 在把共享 GPU 的 Pod 绑定到节点时，会按 `gpu-memory`（或 `gpu-fraction`）注解向容器注入 `CUDA_DEVICE_MEMORY_LIMIT` 环境变量，这正是上一节契约里 HAMi-core 执行隔离所需的配额。 +```bash +curl -s http://127.0.0.1:9394/metrics | grep \ + 'hami_vgpu_memory_limit_bytes.*pod="kai-hami-check"' +``` -v0.17.0 中其他与 GPU 相关的改动还包括：修正共享 Pod 名称带“`/`”造成的非法卷名问题；修正 `MinNodeGPUMemoryMiB` 与 fractional `gpu-memory` 的分配计算；overLimit 判定改用集群最大 GPU 规格。此外还有 preemption-delay（为 Cluster Autoscaler 留出拉起节点的时间窗）、NUMA 感知打分、GitOps 与 ArgoCD 安装支持等。 +输出格式应类似： -### kai-resource-isolator 1.1.0-chart +```text +hami_vgpu_memory_limit_bytes{...,pod="kai-hami-check",...} 4.348444672e+09 +``` -这是与 HAMi 配套发布的隔离器。它接收 KAI 注入的配额，在容器真正运行起来之前，把 HAMi-core 的 `libvgpu.so` 注入到位。相对首个版本，1.1.0 增加了一组运维改进： +以上检查足以确认基本集成链路正确。要证明显示出来的上限确实无法越过，还需要执行一次 CUDA 分配测试：配额内成功、超额失败。下一节汇总了这项实测结果，实验 12 则提供完整程序和操作过程。 -- **新增 `kai-vgpu-monitor`**：以 DaemonSet 形式运行，在 `:9394` 暴露 HAMi 兼容指标（`hami_vgpu_memory_used_bytes`、`hami_vgpu_memory_limit_bytes`、`hami_container_device_utilization_ratio`），支持 ServiceMonitor，可被 Prometheus 直接抓取。 -- **多容器注入修复**：一个 Pod 内有多个容器时，webhook 现在能正确处理，不再漏注。 -- **安全收紧**：webhook 改用命名空间内 Issuer（不再使用 ClusterIssuer），ClusterRole 收回读 Secret 的权限。 -- **全局镜像仓库**优先级理顺，`hamicore` 安装参数修正。 +完成基本检查后删除测试 Pod： -## GKE 实测：隔离是否真的生效 +```bash +kubectl delete pod kai-hami-check +``` -我们在一套 GKE 1.35/COS/CDI 集群上完成了端到端验证：3 个 `n1-standard-2` 节点，每节点一张 NVIDIA T4；KAI Scheduler v0.17.0 负责共享调度，`kai-resource-isolator` 1.1.0-chart 负责注入 HAMi-core。 +## GKE 实测：隔离是否真的生效？ -验证没有止步于 `nvidia-smi`。两个 Pod 被固定到一个只有一张 T4 的节点上，并完成了三层检查： +验证环境是 GKE 1.35/COS/CDI 集群，包含三个 `n1-standard-2` 节点，每个节点有一张 NVIDIA T4。KAI Scheduler v0.17.0 负责共享调度，`kai-resource-isolator` 1.1.0-chart 负责注入 HAMi-core。 -| 检查 | 实测结果 | 证明了什么 | +| 检查项 | 实测结果 | 证明了什么 | | :-- | :-- | :-- | -| 节点与 GPU UUID | 两个 Pod 位于同一节点，UUID 均为 `GPU-9acc8878-...` | 两个 Pod 确实共享同一张物理卡 | -| 可见显存 | 两个 Pod 均为 `4147 MiB`，整卡为 `15360 MiB` | KAI 注入的配额已由 HAMi-core 呈现到容器 | -| CUDA 分配 | 3 GiB 成功，累计申请 5 GiB 返回 `out of memory` | 显存上限被实际执行，不只是修改显示 | -| 并发隔离 | Pod A 持有 3 GiB 时，Pod B 仍成功获得自己的 3 GiB | 一个 Pod 无法占用另一个 Pod 的配额 | -| monitor 指标 | 同节点 `:9394/metrics` 返回两个 Pod 各自 4.348 GB 上限与 3.328 GB 实时用量 | monitor 读取了每个容器的共享内存缓存，并导出非空的 Pod 级指标 | +| 节点与 GPU UUID | 两个 Pod 位于同一单卡节点，并返回 `GPU-9acc8878-...` | 它们共享同一张物理 T4 | +| 可见显存 | 两个 Pod 均报告 `4147 MiB`，整卡报告 `15360 MiB` | HAMi-core 暴露了 KAI 分配给各 Pod 的配额 | +| CUDA 分配 | 3 GiB 成功，累计 5 GiB 返回 `out of memory` | 上限确实执行，而不只是显示值变化 | +| 并发隔离 | Pod A 持有 3 GiB 时，Pod B 仍能申请自己的 3 GiB | 一个 Pod 无法占用另一个 Pod 的配额 | +| Monitor 指标 | 同节点 `:9394/metrics` 返回两个 Pod 的 4.348 GB 上限和 3.328 GB 实时用量 | monitor 从容器缓存导出了非空的 Pod 级指标 | -HAMi-core 在越界时记录了： +超额分配时，HAMi-core 记录： ```text Device 0 OOM 5475663872 / 4348444672 @@ -183,37 +236,22 @@ allocate another 2 GiB: out of memory PASS: in-quota allocation succeeded and over-quota allocation failed ``` -这组结果把“调度到同一张卡”“容器只看到自己的配额”“CUDA 无法越过配额”和“指标可观测”连成了完整证据链。monitor 以 DaemonSet 运行并读取节点本地缓存，因此实验 12 会直接查询工作负载所在节点的实例，而不是依赖可能选中其他节点的 Service。 +这些结果把“调度到同一张卡”“容器内只见自身配额”“CUDA 分配确实越不过上限”和“monitor 能观测实时用量”连成了一条完整证据链。由于 monitor 读取节点本地缓存，实验会直接查询业务 Pod 所在节点的 monitor 实例，避免 Service 把请求转发到其他节点。 :::note 隔离边界 -这里验证的是 HAMi-core 在 CUDA API 层执行的显存配额，不是 MIG 一类硬件安全边界。本次 GKE 兼容路径还使用了特权业务容器，因此不应直接解读为面向不可信多租户的安全隔离方案。 +这里验证的是 CUDA API 层的显存限制，不是 MIG 一类硬件安全边界。本次 GKE 兼容路径还使用了特权业务容器，因此不应作为不可信多租户安全方案。 ::: -### 为什么完整步骤不放在博客里 +## 复现实测结果 -标准的 KAI + HAMi-core 链路并不长，但本次 GKE 1.35/COS/CDI 环境还涉及只读根文件系统、RuntimeClass、NVML 库路径、CDI 设备注入、PriorityClass 和 `kubectl exec` WebSocket 等兼容问题。这些内容版本相关、需要持续维护，更适合成为可执行的实验，而不是嵌在博客主线中。 - -完整的集群准备、安装清单、Kyverno 策略、CUDA 程序、monitor 验证、真实输出、故障排查和清理命令已整理到： +标准 KAI + HAMi-core 安装链路并不长。GKE 1.35/COS/CDI 还需要针对只读根文件系统、RuntimeClass、NVML 库路径、CDI 设备注入和 PriorityClass 做环境适配。完整操作、实测命令输出与故障排查统一维护在： **[实验 12：在 GKE 上验证 KAI Scheduler 与 HAMi 显存隔离](/zh/tutorials/labs/kai-scheduler-hami-gke)** -如果只想理解集成原理，读到这里即可；如果要在 GKE 上复现，再进入“实验 12”按步骤操作。 - -## 这意味着什么 - -HAMi-core 的定位从来不是“某个调度器的附属隔离功能”，而是一个**与调度器解耦、可被各路调度器复用的隔离底座**。在 KAI Scheduler 之前，它已经支撑了 Kubernetes 原生调度器、[Kueue](/zh/docs/userguide/kueue/how-to-use-kueue)、[Volcano](/zh/docs/installation/how-to-use-volcano-vgpu)、Koordinator 等多条路径（见 [HAMi 生态集成](/zh/docs/next/core-concepts/ecosystem-integrations)）。KAI Scheduler v0.16.4 的加入，是把这个生态又扩到了 NVIDIA 官方的 AI 调度器： - -- 对 **KAI 用户**：GPU 共享终于有了与之匹配的运行时硬隔离，共享不再等于缺少保障。 -- 对 **HAMi 用户**：多了一条不绑死特定虚拟化设备插件、直接走 NVIDIA 官方调度器的集成路径，指标体系（`hami_*`）也保持兼容。 -- 对 **社区**：调度层与隔离层之间的契约（`CUDA_DEVICE_MEMORY_LIMIT`）被又一次验证可行，后续任何调度器只要遵守它，就能复用同一套隔离引擎。 - -这背后是 KAI Scheduler 团队（Run:ai）与 HAMi 维护者一年多反复对齐的结果。`LD_PRELOAD`、webhook、指标端口、退出开关，每一个都是两边共同敲定的；`kai-resource-isolator` 1.1.0 里那批安全收紧（命名空间内 Issuer、收紧 ClusterRole），也都是社区评审一条条打磨出来的。 - ## 下一步 -- 背景故事：[《HAMi-core 被 NVIDIA KAI Scheduler 采用》](/zh/blog/hami-core-adopted-by-nvidia-kai-scheduler) +- 背景与协作过程：[《HAMi-core 被 NVIDIA KAI Scheduler 采用》](/zh/blog/hami-core-adopted-by-nvidia-kai-scheduler) - 用户文档：[如何在 KAI Scheduler 中使用 HAMi](/zh/docs/next/userguide/kai-scheduler/how-to-use-kai-scheduler) -- 相关仓库：[Project-HAMi/KAI-resource-isolator](https://github.com/Project-HAMi/KAI-resource-isolator) · [Project-HAMi/HAMi-core](https://github.com/Project-HAMi/HAMi-core)（CNCF 孵化的 CUDA 拦截库） · [kai-scheduler/KAI-Scheduler](https://github.com/kai-scheduler/KAI-Scheduler) · [KAI Scheduler HAMi 资源隔离文档](https://github.com/kai-scheduler/KAI-Scheduler/blob/main/docs/gpu-sharing/hami/README.md) -- 欢迎在你自己的 GKE、AWS 或自建集群上跑一遍，并把真实结果反馈到 issue 或社区群。若缺 HAMi-core 在某种卡或某种 CUDA 版本上的支持，直接到 [Project-HAMi/HAMi](https://github.com/Project-HAMi/HAMi) 提 issue，这是社区最看重的反馈。 +- 相关仓库：[KAI-resource-isolator](https://github.com/Project-HAMi/KAI-resource-isolator)、[HAMi-core](https://github.com/Project-HAMi/HAMi-core) 和 [KAI-Scheduler](https://github.com/kai-scheduler/KAI-Scheduler) diff --git a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/kai-scheduler-hami-gke.md b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/kai-scheduler-hami-gke.md index be244f49a..3ea42af28 100644 --- a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/kai-scheduler-hami-gke.md +++ b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/kai-scheduler-hami-gke.md @@ -74,6 +74,12 @@ gcloud container clusters get-credentials kai-hami-test \ GPU 节点会产生费用，完成实验后请执行清理步骤。 +:::note 关于输出块 + +下文输出均采集自 2026-08-12 的验证运行。Pod 后缀、运行时长、IP 地址和节点名称因环境而异；复现时应重点比较组件名称、就绪状态、调度位置和测量值。 + +::: + ## 步骤 1: 验证 GKE GPU 栈 确认 GPU 节点已经上报 `nvidia.com/gpu`： @@ -87,9 +93,9 @@ kubectl get nodes \ ```plaintext NAME GPU ACCEL -gke-kai-hami-test-default-pool-... 1 nvidia-tesla-t4 -gke-kai-hami-test-default-pool-... 1 nvidia-tesla-t4 -gke-kai-hami-test-default-pool-... 1 nvidia-tesla-t4 +gke-kai-hami-test-default-pool-370c394b-fxh2 1 nvidia-tesla-t4 +gke-kai-hami-test-default-pool-370c394b-pm4j 1 nvidia-tesla-t4 +gke-kai-hami-test-default-pool-370c394b-r8n5 1 nvidia-tesla-t4 ``` 如果 `GPU` 为空，且节点带有 `gke-no-default-nvidia-gpu-device-plugin=true`，启用 GKE device plugin： @@ -123,7 +129,15 @@ kubectl logs gpu-smi-test kubectl delete pod gpu-smi-test ``` -验证环境中的 T4 报告 15360 MiB 可寻址显存。 +实测 `nvidia-smi` 中与本实验相关的输出为： + +```plaintext +GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC + 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 +... 0MiB / 15360MiB +``` + +驱动补丁版本可能不同，但业务镜像必须与之兼容。记录这里报告的显存值，供步骤 2 使用。 ## 步骤 2: 添加 KAI 所需的 GPU 标签 @@ -135,6 +149,18 @@ kubectl label node -l cloud.google.com/gke-accelerator=nvidia-tesla-t4 \ nvidia.com/gpu.product=NVIDIA-Tesla-T4 \ nvidia.com/gpu.count=1 \ nvidia.com/gpu.present=true --overwrite + +kubectl get nodes -o custom-columns=\ +'NAME:.metadata.name,GPU.MEMORY:.metadata.labels.nvidia\.com/gpu\.memory,GPU.PRODUCT:.metadata.labels.nvidia\.com/gpu\.product' +``` + +实测节点随后暴露了 KAI 读取的标签值： + +```plaintext +NAME GPU.MEMORY GPU.PRODUCT +gke-kai-hami-test-default-pool-370c394b-fxh2 15360 NVIDIA-Tesla-T4 +gke-kai-hami-test-default-pool-370c394b-pm4j 15360 NVIDIA-Tesla-T4 +gke-kai-hami-test-default-pool-370c394b-r8n5 15360 NVIDIA-Tesla-T4 ``` 显存值应来自 `nvidia-smi`，不要使用 T4 标称的 16 GiB。如果 KAI 启动后才补标签，需要重启 `kai-scheduler` 刷新节点缓存。 @@ -156,10 +182,24 @@ kubectl -n kai-scheduler wait --for=condition=available \ --timeout=180s deploy --all kubectl -n kai-scheduler wait --for=condition=Ready \ --timeout=300s config/kai-config +kubectl get pods -n kai-scheduler kubectl get queues ``` -KAI v0.17.0 会自动创建默认的父子队列： +实测安装中的七个 KAI 控制面组件均处于运行状态。自动生成的 Pod 后缀会不同： + +```plaintext +NAME READY STATUS RESTARTS AGE +admission-759b9bb99c-... 1/1 Running 0 4m +binder-54665cc5d9-... 1/1 Running 0 4m +kai-operator-997c6886c-... 1/1 Running 0 4m +kai-scheduler-default-d85d7dbdf-... 1/1 Running 0 4m +pod-grouper-68f4fb47-... 1/1 Running 0 4m +podgroup-controller-5947b5b4dd-... 1/1 Running 0 4m +queue-controller-6cc8c844c8-... 1/1 Running 0 4m +``` + +KAI v0.17.0 还会自动创建默认的父子队列： ```plaintext NAME PARENT @@ -202,7 +242,22 @@ kubectl rollout status ds/kai-resource-isolator-monitor \ kubectl get pods -n kai-resource-isolator ``` -ConfigMap 输出必须为 `/home/kubernetes/bin/nvidia/vgpu/libvgpu.so`。 +实测路径和组件状态如下： + +```plaintext +/home/kubernetes/bin/nvidia/vgpu/libvgpu.so + +NAME READY STATUS RESTARTS AGE +kai-resource-isolator-libsync-... 1/1 Running 0 2m +kai-resource-isolator-libsync-... 1/1 Running 0 2m +kai-resource-isolator-libsync-... 1/1 Running 0 2m +kai-resource-isolator-monitor-26bj8 1/1 Running 0 2m +kai-resource-isolator-monitor-hf67f 1/1 Running 0 2m +kai-resource-isolator-monitor-tnj9l 1/1 Running 0 2m +kai-resource-isolator-webhook-... 1/1 Running 0 2m +``` + +每个 GPU 节点都应有一个 libsync 和一个 monitor Pod，并且 webhook Pod 已就绪。 ## 步骤 5: 适配 GKE CDI 设备路径 @@ -224,6 +279,27 @@ kubectl wait -n kyverno --for=condition=Ready pod \ -l app.kubernetes.io/component=admission-controller --timeout=300s kubectl apply \ -f tutorials/labs/examples/12-kai-scheduler-hami-gke/03-gke-policies.yaml +kubectl get runtimeclass nvidia +kubectl get pods -n kyverno +kubectl get clusterpolicy \ + inject-nvidia-library-path inject-gpu-devices +``` + +实测时兼容对象与 Kyverno 控制器均已就绪： + +```plaintext +NAME HANDLER AGE +nvidia runc 3m + +NAME READY STATUS RESTARTS AGE +kyverno-admission-controller-7cdf5b9c-... 1/1 Running 0 2m +kyverno-background-controller-7b54965bf9-... 1/1 Running 0 2m +kyverno-cleanup-controller-59c8fdfb66-... 1/1 Running 0 2m +kyverno-reports-controller-5c96886c9-... 1/1 Running 0 2m + +NAME ADMISSION BACKGROUND READY +inject-nvidia-library-path true true true +inject-gpu-devices true true true ``` 第一条策略给 reservation Pod 添加 NVML 库路径；第二条给共享 Pod 挂载 `/dev/nvidia*`、`nvidia-smi` 与 NVIDIA 库。使用这条 workaround 时，共享 Pod 还需要 `privileged: true`。 @@ -268,7 +344,15 @@ for pod in kai-hami-lab12-a kai-hami-lab12-b; do done ``` -实测时两个 Pod 都返回： +实测时两个 Pod 在同一节点就绪： + +```plaintext +NAME READY STATUS RESTARTS IP NODE +kai-hami-lab12-a 1/1 Running 0 10.84.2.66 gke-kai-hami-test-default-pool-370c394b-pm4j +kai-hami-lab12-b 1/1 Running 0 10.84.2.67 gke-kai-hami-test-default-pool-370c394b-pm4j +``` + +两者的启动日志都返回： ```plaintext limit=4147m @@ -290,12 +374,22 @@ for pod in kai-hami-lab12-a kai-hami-lab12-b; do done ``` -两个 Pod 的实测输出均为： +实测时间区间互相重叠，并且每个 Pod 都返回 `PASS`： ```plaintext +=== kai-hami-lab12-a === +test_start=2026-08-12T05:11:13Z +allocate 3 GiB: no error +allocate another 2 GiB: out of memory +PASS: in-quota allocation succeeded and over-quota allocation failed +test_end=2026-08-12T05:11:44Z + +=== kai-hami-lab12-b === +test_start=2026-08-12T05:11:11Z allocate 3 GiB: no error allocate another 2 GiB: out of memory PASS: in-quota allocation succeeded and over-quota allocation failed +test_end=2026-08-12T05:11:43Z ``` 比较 `test_start` 与 `test_end`，两个 30 秒区间必须重叠。实测时两个 Pod 在同时持有 3 GiB 的情况下都返回 `PASS`。各自累计申请 5 GiB 时，HAMi-core 都记录 `Device 0 OOM 5475663872 / 4348444672`。将验证程序作为容器启动命令执行，也避免了长时间 `kubectl exec` WebSocket 中断影响判断。 diff --git a/tutorials/labs/kai-scheduler-hami-gke.md b/tutorials/labs/kai-scheduler-hami-gke.md index 334254d06..1b6fd5e2c 100644 --- a/tutorials/labs/kai-scheduler-hami-gke.md +++ b/tutorials/labs/kai-scheduler-hami-gke.md @@ -74,6 +74,12 @@ gcloud container clusters get-credentials kai-hami-test \ GPU nodes are billable. Run the cleanup section when you finish. +:::note About the output blocks + +The output blocks below were captured from the verified run on 2026-08-12. Pod suffixes, ages, IP addresses, and node names are environment-specific; compare the component names, readiness, placement, and measured values. + +::: + ## Step 1: Verify the GKE GPU Stack Confirm that each GPU node reports one extended resource: @@ -87,9 +93,9 @@ The verified cluster reported three T4 nodes: ```plaintext NAME GPU ACCEL -gke-kai-hami-test-default-pool-... 1 nvidia-tesla-t4 -gke-kai-hami-test-default-pool-... 1 nvidia-tesla-t4 -gke-kai-hami-test-default-pool-... 1 nvidia-tesla-t4 +gke-kai-hami-test-default-pool-370c394b-fxh2 1 nvidia-tesla-t4 +gke-kai-hami-test-default-pool-370c394b-pm4j 1 nvidia-tesla-t4 +gke-kai-hami-test-default-pool-370c394b-r8n5 1 nvidia-tesla-t4 ``` If `GPU` is empty and the node has `gke-no-default-nvidia-gpu-device-plugin=true`, enable the GKE device plugin: @@ -123,7 +129,15 @@ kubectl logs gpu-smi-test kubectl delete pod gpu-smi-test ``` -The T4 in the verified environment reported 15360 MiB of addressable memory. +The relevant lines in the verified `nvidia-smi` output were: + +```plaintext +GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC + 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 +... 0MiB / 15360MiB +``` + +The driver patch may differ, but the workload image must be compatible with it. Record the reported memory value for Step 2. ## Step 2: Add the GPU Labels KAI Reads @@ -135,6 +149,18 @@ kubectl label node -l cloud.google.com/gke-accelerator=nvidia-tesla-t4 \ nvidia.com/gpu.product=NVIDIA-Tesla-T4 \ nvidia.com/gpu.count=1 \ nvidia.com/gpu.present=true --overwrite + +kubectl get nodes -o custom-columns=\ +'NAME:.metadata.name,GPU.MEMORY:.metadata.labels.nvidia\.com/gpu\.memory,GPU.PRODUCT:.metadata.labels.nvidia\.com/gpu\.product' +``` + +The verified nodes then exposed the values KAI reads: + +```plaintext +NAME GPU.MEMORY GPU.PRODUCT +gke-kai-hami-test-default-pool-370c394b-fxh2 15360 NVIDIA-Tesla-T4 +gke-kai-hami-test-default-pool-370c394b-pm4j 15360 NVIDIA-Tesla-T4 +gke-kai-hami-test-default-pool-370c394b-r8n5 15360 NVIDIA-Tesla-T4 ``` Use the value reported by `nvidia-smi`, not the T4's marketed 16 GiB. If you add the labels after KAI starts, restart `kai-scheduler` so it refreshes its node cache. @@ -156,10 +182,24 @@ kubectl -n kai-scheduler wait --for=condition=available \ --timeout=180s deploy --all kubectl -n kai-scheduler wait --for=condition=Ready \ --timeout=300s config/kai-config +kubectl get pods -n kai-scheduler kubectl get queues ``` -KAI v0.17.0 creates its default parent and child queues automatically: +All seven KAI control-plane components were running in the verified installation. Generated Pod suffixes will differ: + +```plaintext +NAME READY STATUS RESTARTS AGE +admission-759b9bb99c-... 1/1 Running 0 4m +binder-54665cc5d9-... 1/1 Running 0 4m +kai-operator-997c6886c-... 1/1 Running 0 4m +kai-scheduler-default-d85d7dbdf-... 1/1 Running 0 4m +pod-grouper-68f4fb47-... 1/1 Running 0 4m +podgroup-controller-5947b5b4dd-... 1/1 Running 0 4m +queue-controller-6cc8c844c8-... 1/1 Running 0 4m +``` + +KAI v0.17.0 also created its default parent and child queues automatically: ```plaintext NAME PARENT @@ -202,7 +242,22 @@ kubectl rollout status ds/kai-resource-isolator-monitor \ kubectl get pods -n kai-resource-isolator ``` -The ConfigMap output must be `/home/kubernetes/bin/nvidia/vgpu/libvgpu.so`. +The path and component status in the verified run were: + +```plaintext +/home/kubernetes/bin/nvidia/vgpu/libvgpu.so + +NAME READY STATUS RESTARTS AGE +kai-resource-isolator-libsync-... 1/1 Running 0 2m +kai-resource-isolator-libsync-... 1/1 Running 0 2m +kai-resource-isolator-libsync-... 1/1 Running 0 2m +kai-resource-isolator-monitor-26bj8 1/1 Running 0 2m +kai-resource-isolator-monitor-hf67f 1/1 Running 0 2m +kai-resource-isolator-monitor-tnj9l 1/1 Running 0 2m +kai-resource-isolator-webhook-... 1/1 Running 0 2m +``` + +There must be one libsync and one monitor Pod per GPU node, plus a ready webhook Pod. ## Step 5: Adapt the GKE CDI Device Path @@ -224,6 +279,27 @@ kubectl wait -n kyverno --for=condition=Ready pod \ -l app.kubernetes.io/component=admission-controller --timeout=300s kubectl apply \ -f tutorials/labs/examples/12-kai-scheduler-hami-gke/03-gke-policies.yaml +kubectl get runtimeclass nvidia +kubectl get pods -n kyverno +kubectl get clusterpolicy \ + inject-nvidia-library-path inject-gpu-devices +``` + +The compatibility objects and Kyverno controllers were ready in the verified run: + +```plaintext +NAME HANDLER AGE +nvidia runc 3m + +NAME READY STATUS RESTARTS AGE +kyverno-admission-controller-7cdf5b9c-... 1/1 Running 0 2m +kyverno-background-controller-7b54965bf9-... 1/1 Running 0 2m +kyverno-cleanup-controller-59c8fdfb66-... 1/1 Running 0 2m +kyverno-reports-controller-5c96886c9-... 1/1 Running 0 2m + +NAME ADMISSION BACKGROUND READY +inject-nvidia-library-path true true true +inject-gpu-devices true true true ``` The first policy adds the NVML library path to reservation Pods. The second mounts `/dev/nvidia*`, `nvidia-smi`, and NVIDIA libraries into shared Pods. In this workaround path, shared Pods also require `privileged: true`. @@ -268,7 +344,15 @@ for pod in kai-hami-lab12-a kai-hami-lab12-b; do done ``` -Both Pods in the verified run reported: +Both Pods were ready on the same node in the verified run: + +```plaintext +NAME READY STATUS RESTARTS IP NODE +kai-hami-lab12-a 1/1 Running 0 10.84.2.66 gke-kai-hami-test-default-pool-370c394b-pm4j +kai-hami-lab12-b 1/1 Running 0 10.84.2.67 gke-kai-hami-test-default-pool-370c394b-pm4j +``` + +Their startup logs both reported: ```plaintext limit=4147m @@ -290,12 +374,22 @@ for pod in kai-hami-lab12-a kai-hami-lab12-b; do done ``` -The verified output from each Pod was: +The captured intervals overlapped, and each Pod returned `PASS`: ```plaintext +=== kai-hami-lab12-a === +test_start=2026-08-12T05:11:13Z +allocate 3 GiB: no error +allocate another 2 GiB: out of memory +PASS: in-quota allocation succeeded and over-quota allocation failed +test_end=2026-08-12T05:11:44Z + +=== kai-hami-lab12-b === +test_start=2026-08-12T05:11:11Z allocate 3 GiB: no error allocate another 2 GiB: out of memory PASS: in-quota allocation succeeded and over-quota allocation failed +test_end=2026-08-12T05:11:43Z ``` Compare `test_start` and `test_end`: the two 30-second intervals must overlap. In the verified run, both Pods returned `PASS` while holding 3 GiB concurrently. HAMi-core logged `Device 0 OOM 5475663872 / 4348444672` for each Pod's 5 GiB cumulative request. Running the proof as the container startup command also avoids making the result depend on a long-lived `kubectl exec` WebSocket. From 94a667f725af6543388ab8b35783bfc618a85467 Mon Sep 17 00:00:00 2001 From: Jimmy Song Date: Thu, 13 Aug 2026 12:26:52 +0800 Subject: [PATCH 6/6] feat(tutorials): enhance GKE lab with additional resource checks and deployment status commands Signed-off-by: Jimmy Song --- blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md | 4 ++++ .../kai-scheduler-hami-gpu-memory-hard-isolation/index.md | 4 ++++ .../current/labs/kai-scheduler-hami-gke.md | 5 +++++ .../labs/examples/12-kai-scheduler-hami-gke/memory-limit.cu | 6 ++++++ tutorials/labs/kai-scheduler-hami-gke.md | 5 +++++ 5 files changed, 24 insertions(+) diff --git a/blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md b/blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md index 57dc3a5b5..62b1b04f3 100644 --- a/blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md +++ b/blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md @@ -109,6 +109,8 @@ kubectl rollout status ds/kai-resource-isolator-libsync \ -n kai-resource-isolator --timeout=300s kubectl rollout status ds/kai-resource-isolator-monitor \ -n kai-resource-isolator --timeout=300s +kubectl rollout status deploy/kai-resource-isolator-webhook \ + -n kai-resource-isolator --timeout=300s kubectl get pods -n kai-resource-isolator ``` @@ -160,6 +162,8 @@ Inspect the quota from KAI, the preload file injected by the isolator, and the m ```bash kubectl exec kai-hami-check -- sh -lc ' + test -n "$CUDA_DEVICE_MEMORY_LIMIT" + test -f /usr/local/vgpu/libvgpu.so printf "limit=%s\n" "$CUDA_DEVICE_MEMORY_LIMIT" cat /etc/ld.so.preload nvidia-smi --query-gpu=uuid,memory.total --format=csv,noheader diff --git a/i18n/zh/docusaurus-plugin-content-blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md b/i18n/zh/docusaurus-plugin-content-blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md index 74fa1d917..35d4ec23e 100644 --- a/i18n/zh/docusaurus-plugin-content-blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md +++ b/i18n/zh/docusaurus-plugin-content-blog/kai-scheduler-hami-gpu-memory-hard-isolation/index.md @@ -109,6 +109,8 @@ kubectl rollout status ds/kai-resource-isolator-libsync \ -n kai-resource-isolator --timeout=300s kubectl rollout status ds/kai-resource-isolator-monitor \ -n kai-resource-isolator --timeout=300s +kubectl rollout status deploy/kai-resource-isolator-webhook \ + -n kai-resource-isolator --timeout=300s kubectl get pods -n kai-resource-isolator ``` @@ -160,6 +162,8 @@ kai-hami-check 1/1 Running gpu-node-1 ```bash kubectl exec kai-hami-check -- sh -lc ' + test -n "$CUDA_DEVICE_MEMORY_LIMIT" + test -f /usr/local/vgpu/libvgpu.so printf "limit=%s\n" "$CUDA_DEVICE_MEMORY_LIMIT" cat /etc/ld.so.preload nvidia-smi --query-gpu=uuid,memory.total --format=csv,noheader diff --git a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/kai-scheduler-hami-gke.md b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/kai-scheduler-hami-gke.md index 3ea42af28..d0fa92d29 100644 --- a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/kai-scheduler-hami-gke.md +++ b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/kai-scheduler-hami-gke.md @@ -239,6 +239,8 @@ kubectl rollout status ds/kai-resource-isolator-libsync \ -n kai-resource-isolator --timeout=300s kubectl rollout status ds/kai-resource-isolator-monitor \ -n kai-resource-isolator --timeout=300s +kubectl rollout status deploy/kai-resource-isolator-webhook \ + -n kai-resource-isolator --timeout=300s kubectl get pods -n kai-resource-isolator ``` @@ -279,6 +281,9 @@ kubectl wait -n kyverno --for=condition=Ready pod \ -l app.kubernetes.io/component=admission-controller --timeout=300s kubectl apply \ -f tutorials/labs/examples/12-kai-scheduler-hami-gke/03-gke-policies.yaml +kubectl wait --for=condition=Ready --timeout=180s \ + clusterpolicy/inject-nvidia-library-path \ + clusterpolicy/inject-gpu-devices kubectl get runtimeclass nvidia kubectl get pods -n kyverno kubectl get clusterpolicy \ diff --git a/tutorials/labs/examples/12-kai-scheduler-hami-gke/memory-limit.cu b/tutorials/labs/examples/12-kai-scheduler-hami-gke/memory-limit.cu index db9271778..aea1e8c40 100644 --- a/tutorials/labs/examples/12-kai-scheduler-hami-gke/memory-limit.cu +++ b/tutorials/labs/examples/12-kai-scheduler-hami-gke/memory-limit.cu @@ -23,6 +23,12 @@ int main() { cudaFree(within_limit); return 2; } + if (second != cudaErrorMemoryAllocation) { + std::cerr << "ERROR: expected out of memory, got " + << cudaGetErrorString(second) << std::endl; + cudaFree(within_limit); + return 3; + } const char *hold = std::getenv("HOLD_SECONDS"); if (hold) { diff --git a/tutorials/labs/kai-scheduler-hami-gke.md b/tutorials/labs/kai-scheduler-hami-gke.md index 1b6fd5e2c..acfd9ff26 100644 --- a/tutorials/labs/kai-scheduler-hami-gke.md +++ b/tutorials/labs/kai-scheduler-hami-gke.md @@ -239,6 +239,8 @@ kubectl rollout status ds/kai-resource-isolator-libsync \ -n kai-resource-isolator --timeout=300s kubectl rollout status ds/kai-resource-isolator-monitor \ -n kai-resource-isolator --timeout=300s +kubectl rollout status deploy/kai-resource-isolator-webhook \ + -n kai-resource-isolator --timeout=300s kubectl get pods -n kai-resource-isolator ``` @@ -279,6 +281,9 @@ kubectl wait -n kyverno --for=condition=Ready pod \ -l app.kubernetes.io/component=admission-controller --timeout=300s kubectl apply \ -f tutorials/labs/examples/12-kai-scheduler-hami-gke/03-gke-policies.yaml +kubectl wait --for=condition=Ready --timeout=180s \ + clusterpolicy/inject-nvidia-library-path \ + clusterpolicy/inject-gpu-devices kubectl get runtimeclass nvidia kubectl get pods -n kyverno kubectl get clusterpolicy \