From 11c83317806c9df6c75110f87125ef8b2b8b461c Mon Sep 17 00:00:00 2001 From: Jifei Wang Date: Mon, 3 Aug 2026 11:35:18 +0800 Subject: [PATCH 1/3] add amd support Signed-off-by: Jifei Wang --- docs/contributor/roadmap.md | 2 +- .../amd-device/enable-amd-gpu-sharing.md | 176 +++++++++++++++++ .../examples/allocate-core-and-memory.md | 31 +++ docs/userguide/device-supported.md | 1 + .../current.json | 8 + .../current/contributor/roadmap.md | 2 +- .../amd-device/enable-amd-gpu-sharing.md | 177 ++++++++++++++++++ .../examples/allocate-core-and-memory.md | 32 ++++ .../current/userguide/device-supported.md | 1 + sidebars.js | 15 ++ 10 files changed, 443 insertions(+), 2 deletions(-) create mode 100644 docs/userguide/amd-device/enable-amd-gpu-sharing.md create mode 100644 docs/userguide/amd-device/examples/allocate-core-and-memory.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/userguide/amd-device/enable-amd-gpu-sharing.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/userguide/amd-device/examples/allocate-core-and-memory.md diff --git a/docs/contributor/roadmap.md b/docs/contributor/roadmap.md index aef0c7461..1eeafd24e 100644 --- a/docs/contributor/roadmap.md +++ b/docs/contributor/roadmap.md @@ -33,5 +33,5 @@ sidebar_label: Roadmap - [ ] Rich observability support - [x] DRA support - [ ] Support Intel GPU device -- [ ] Support AMD GPU device +- [x] Support AMD GPU device - [x] Support Enflame GCU device diff --git a/docs/userguide/amd-device/enable-amd-gpu-sharing.md b/docs/userguide/amd-device/enable-amd-gpu-sharing.md new file mode 100644 index 000000000..193727563 --- /dev/null +++ b/docs/userguide/amd-device/enable-amd-gpu-sharing.md @@ -0,0 +1,176 @@ +--- +title: Enable AMD GPU Sharing +sidebar_label: GPU Sharing +--- + +## Introduction + +HAMi supports sharing AMD Instinct/ROCm GPUs. Workloads request device memory and compute-unit (CU) share through standard Kubernetes resources, without application code changes. + +**GPU sharing**: Multiple tasks can share one AMD GPU instead of occupying a whole card. + +**Device memory control**: Allocate a specific amount of device memory (MiB). HAMi enforces a hard limit so usage cannot exceed the allocation. + +**Device compute core limitation**: Allocate a percentage of compute units (`amd.com/gpucores: 25` means about 25% of the device CUs). + +:::caution + +Use the [amd-device-plugin](https://github.com/Project-HAMi/amd-device-plugin) image and manifests that match your HAMi version. Do not deploy the upstream ROCm `k8s-device-plugin` image for HAMi soft vGPU. + +::: + +## Prerequisites + +Deploy these components: + +| Component | Role | Key requirement | +| --- | --- | --- | +| HAMi | Scheduling, allocation, and admission | Scheduler is running and manages the three AMD resources | +| AMD GPU Operator (recommended) | Driver and ROCm environment | Disable the Operator native device-plugin | +| [amd-device-plugin](https://github.com/Project-HAMi/amd-device-plugin) | Register AMD resources, allocate CUs, inject runtime limits | Deploy the HAMi fork; it discovers VRAM/CU via amd-smi/libdrm | + +Nodes also need a working AMD driver and ROCm. Verify with: + +```bash +amd-smi static --gpu 0 +``` + +The output should include the device model, VRAM, and `NUM_COMPUTE_UNITS`. + +## Enabling AMD GPU Sharing + +### Configure HAMi + +After installing HAMi, confirm the scheduler manages all AMD vGPU resources. The values file should include: + +```yaml +devices: + amd: + customresources: + - amd.com/gpu + - amd.com/gpumem + - amd.com/gpucores +``` + +Confirm the scheduler is running: + +```bash +kubectl -n kube-system get pods | grep hami-scheduler +``` + +### Disable the AMD GPU Operator device-plugin + +If you use the [AMD GPU Operator](https://github.com/ROCm/gpu-operator) for drivers and ROCm, disable its native device-plugin so it does not compete with HAMi `amd-device-plugin` for `amd.com/gpu`: + +```bash +kubectl -n kube-amd-gpu patch deviceconfig default --type=merge -p \ + '{"spec":{"devicePlugin":{"enableDevicePlugin":false}}}' +``` + +`amd-device-plugin` reads per-device VRAM, CU count, UUID, and product name through amd-smi/libdrm, so the Operator node-labeller is optional for HAMi soft vGPU. + +### Deploy amd-device-plugin + +Deploy [amd-device-plugin](https://github.com/Project-HAMi/amd-device-plugin) to all AMD GPU nodes. Prefer the Helm chart in that repository: + +```bash +git clone https://github.com/Project-HAMi/amd-device-plugin.git +helm install amd-gpu ./amd-device-plugin/helm/amd-gpu -n kube-system \ + --set dp.image.repository= \ + --set dp.image.tag= +``` + +Replace `` and `` with the image that matches your HAMi version. The chart mounts `/var/lib/kubelet/device-plugins`, `/sys`, and the vGPU hook path, and sets `NODE_NAME` from `spec.nodeName`. + +Wait for the DaemonSet: + +```bash +kubectl -n kube-system rollout status ds/amd-gpu-device-plugin-daemonset +``` + +Confirm the device-plugin registered full device info with HAMi: + +```bash +kubectl get node -o json | \ + jq -r '.metadata.annotations["hami.io/node-amd-register"]' +``` + +The result must include `devmem` and `devcore`. Example for MI300X VF: + +```json +[{"count":10,"devmem":196608,"devcore":304,"type":"AMD_Instinct_MI300X_VF"}] +``` + +## Running AMD vGPU Jobs + +Request AMD GPUs with `amd.com/gpu`, `amd.com/gpumem`, and `amd.com/gpucores`: + +- `amd.com/gpu`: number of AMD GPUs +- `amd.com/gpumem`: device memory quota per GPU, in MiB +- `amd.com/gpucores`: CU quota percentage per GPU, range 0-100; for example `25` allocates about 76 CUs on a 304-CU device + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: amd-vgpu-example +spec: + schedulerName: hami-scheduler + restartPolicy: Never + containers: + - name: pytorch + image: rocm/pytorch:latest + command: ["bash", "-c"] + args: + - | + env | grep -E 'LD_AUDIT|HIP_DEVICE_MEMORY_LIMIT' + python3 -c 'import torch; print(torch.cuda.mem_get_info(0)); print(torch.cuda.get_device_name(0))' + sleep 300 + resources: + requests: + amd.com/gpu: 1 + amd.com/gpumem: 49152 + amd.com/gpucores: 25 + limits: + amd.com/gpu: 1 + amd.com/gpumem: 49152 + amd.com/gpucores: 25 +``` + +```bash +kubectl apply -f amd-vgpu-example.yaml +kubectl get pod amd-vgpu-example -o wide +kubectl logs amd-vgpu-example +``` + +On success, logs look like: + +```text +LD_AUDIT=/usr/local/vgpu/libamvgpu.so +HIP_DEVICE_MEMORY_LIMIT=49152m +(51539607552, 51539607552) +AMD Instinct MI300X VF +``` + +`51539607552` is bytes, about 48 GiB. You can submit two identical 48 GiB / 25% CU workloads to verify sharing. + +## Troubleshooting + +| Symptom | Action | +| --- | --- | +| `node unregistered` | Check that `amd-device-plugin` is running and `hami.io/node-amd-register` contains `devmem` and `devcore`. Restart the DaemonSet if needed. | +| `CardInsufficientMemory` | The Pod requests more memory than the device has free. Lower `amd.com/gpumem` or wait for other workloads to finish. | +| `insufficient free CUs` | Delete finished AMD vGPU test Pods and restart `amd-device-plugin` to clear stale allocations. | +| Memory inside the container still shows the full physical size | Check that the Pod env includes `LD_AUDIT` and `HIP_DEVICE_MEMORY_LIMIT`. | + +Clean up after testing: + +```bash +kubectl delete pod amd-vgpu-example +``` + +## Notes + +1. Deploy [amd-device-plugin](https://github.com/Project-HAMi/amd-device-plugin); use an image that matches your HAMi version. +2. Keep the AMD GPU Operator native device-plugin disabled while using HAMi AMD soft vGPU sharing. +3. Omitting `amd.com/gpucores` allocates all CUs on each requested GPU. diff --git a/docs/userguide/amd-device/examples/allocate-core-and-memory.md b/docs/userguide/amd-device/examples/allocate-core-and-memory.md new file mode 100644 index 000000000..47ccf392d --- /dev/null +++ b/docs/userguide/amd-device/examples/allocate-core-and-memory.md @@ -0,0 +1,31 @@ +--- +title: Allocate device core and memory resource +--- + +To allocate part of an AMD GPU, set `amd.com/gpucores` and `amd.com/gpumem` together with the number of GPUs in `amd.com/gpu`. + +The example below requests one GPU with 48 GiB device memory and 25% compute units: + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: amd-vgpu-example +spec: + schedulerName: hami-scheduler + restartPolicy: Never + containers: + - name: pytorch + image: rocm/pytorch:latest + command: ["bash", "-c"] + args: + - | + env | grep -E 'LD_AUDIT|HIP_DEVICE_MEMORY_LIMIT' + python3 -c 'import torch; print(torch.cuda.mem_get_info(0)); print(torch.cuda.get_device_name(0))' + sleep 300 + resources: + limits: + amd.com/gpu: 1 # requesting one AMD GPU + amd.com/gpumem: 49152 # each GPU requires 49152 MiB device memory + amd.com/gpucores: 25 # each GPU uses 25% of total compute units +``` diff --git a/docs/userguide/device-supported.md b/docs/userguide/device-supported.md index a6beae4d0..3b39d989b 100644 --- a/docs/userguide/device-supported.md +++ b/docs/userguide/device-supported.md @@ -18,4 +18,5 @@ The table below lists the devices supported by HAMi: | XPU | Kunlunxin | P800 | Yes | Yes | No | | GPU | Vastai | VA16 | Yes | Yes | No | | GPU | Biren | Biren166M | Yes | Yes | No | +| GPU | AMD | Instinct / ROCm | Yes | Yes | No | | DPU | Teco | Checking | In progress | In progress | No | diff --git a/i18n/zh/docusaurus-plugin-content-docs/current.json b/i18n/zh/docusaurus-plugin-content-docs/current.json index 4f107183e..85a18a02b 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current.json +++ b/i18n/zh/docusaurus-plugin-content-docs/current.json @@ -71,6 +71,14 @@ "message": "共享燧原 GCU 设备", "description": "The label for category 'Share Enflame GCU devices' in sidebar 'docs'" }, + "sidebar.docs.category.Share AMD GPU devices": { + "message": "共享 AMD GPU 设备", + "description": "The label for category 'Share AMD GPU devices' in sidebar 'docs'" + }, + "sidebar.docs.category.amd-examples": { + "message": "示例", + "description": "The label for category 'Examples' in sidebar 'docs'" + }, "sidebar.docs.category.Share MetaX GPU devices": { "message": "共享沐曦 GPU 设备", "description": "The label for category 'Share MetaX GPU devices' in sidebar 'docs'" diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor/roadmap.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor/roadmap.md index 0622837f0..8482b0a89 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor/roadmap.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor/roadmap.md @@ -34,5 +34,5 @@ sidebar_label: 路线图 - [ ] 丰富的可观测性支持 - [x] 支持 DRA - [ ] 支持 Intel GPU 设备 -- [ ] 支持 AMD GPU 设备 +- [x] 支持 AMD GPU 设备 - [x] 支持 Enflame GCU 设备 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/userguide/amd-device/enable-amd-gpu-sharing.md b/i18n/zh/docusaurus-plugin-content-docs/current/userguide/amd-device/enable-amd-gpu-sharing.md new file mode 100644 index 000000000..e50f3bd3e --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/userguide/amd-device/enable-amd-gpu-sharing.md @@ -0,0 +1,177 @@ +--- +title: 启用 AMD GPU 共享 +sidebar_label: GPU 共享 +translated: true +--- + +## 简介 + +HAMi 支持共享 AMD Instinct/ROCm GPU。工作负载通过标准 Kubernetes 资源请求显存和算力份额，无需修改应用代码。 + +**GPU 共享**: 每个任务可以只占用一部分显卡，多个任务可以共享一张显卡。 + +**可限制分配的显存大小**: 可以用显存值（MiB）分配 GPU，HAMi 会确保任务使用的显存不会超过分配数值。 + +**可限制计算单元数量**: 可以指定任务使用的算力比例（例如 `amd.com/gpucores: 25` 代表使用约 25% CU）。 + +:::caution + +请使用与 HAMi 版本匹配的 [amd-device-plugin](https://github.com/Project-HAMi/amd-device-plugin) 镜像和部署清单。不要用上游 ROCm `k8s-device-plugin` 镜像来跑 HAMi soft vGPU。 + +::: + +## 节点需求 + +需要部署以下组件： + +| 组件 | 作用 | 关键要求 | +| --- | --- | --- | +| HAMi | 调度、设备分配和准入 | scheduler 正常运行，并管理三个 AMD 资源 | +| AMD GPU Operator（推荐） | 驱动和 ROCm 环境 | 禁用 Operator 原生 device-plugin | +| [amd-device-plugin](https://github.com/Project-HAMi/amd-device-plugin) | 注册 AMD 资源、分配 CU、注入容器运行时限制 | 部署 HAMi fork；通过 amd-smi/libdrm 发现显存与 CU | + +节点还必须具备可用的 AMD 驱动和 ROCm。可用以下命令确认： + +```bash +amd-smi static --gpu 0 +``` + +输出应包含设备型号、VRAM 和 `NUM_COMPUTE_UNITS`。 + +## 开启 AMD GPU 共享 + +### 配置 HAMi + +安装 HAMi 后，确认 scheduler 管理所有 AMD vGPU 资源。values 文件中应包含： + +```yaml +devices: + amd: + customresources: + - amd.com/gpu + - amd.com/gpumem + - amd.com/gpucores +``` + +安装或升级后确认 scheduler 正常运行： + +```bash +kubectl -n kube-system get pods | grep hami-scheduler +``` + +### 禁用 AMD GPU Operator device-plugin + +若使用 [AMD GPU Operator](https://github.com/ROCm/gpu-operator) 管理驱动和 ROCm，请关闭其原生 device-plugin，避免与 HAMi `amd-device-plugin` 竞争 `amd.com/gpu`： + +```bash +kubectl -n kube-amd-gpu patch deviceconfig default --type=merge -p \ + '{"spec":{"devicePlugin":{"enableDevicePlugin":false}}}' +``` + +`amd-device-plugin` 通过 amd-smi/libdrm 读取每张卡的显存、CU、UUID 和产品名称，因此对 HAMi soft vGPU 而言，Operator node-labeller 是可选的。 + +### 部署 amd-device-plugin + +在所有 AMD GPU 节点上部署 [amd-device-plugin](https://github.com/Project-HAMi/amd-device-plugin)。推荐使用该仓库中的 Helm chart： + +```bash +git clone https://github.com/Project-HAMi/amd-device-plugin.git +helm install amd-gpu ./amd-device-plugin/helm/amd-gpu -n kube-system \ + --set dp.image.repository= \ + --set dp.image.tag= +``` + +将 `` 和 `` 替换为与当前 HAMi 版本匹配的镜像。Chart 会挂载 `/var/lib/kubelet/device-plugins`、`/sys` 和 vGPU hook 路径，并将 `NODE_NAME` 设为 `spec.nodeName`。 + +等待 DaemonSet 就绪： + +```bash +kubectl -n kube-system rollout status ds/amd-gpu-device-plugin-daemonset +``` + +确认 device-plugin 已将完整设备信息注册给 HAMi： + +```bash +kubectl get node -o json | \ + jq -r '.metadata.annotations["hami.io/node-amd-register"]' +``` + +结果必须包含 `devmem` 和 `devcore`。例如 MI300X VF： + +```json +[{"count":10,"devmem":196608,"devcore":304,"type":"AMD_Instinct_MI300X_VF"}] +``` + +## 运行 AMD vGPU 任务 + +通过 `amd.com/gpu`、`amd.com/gpumem` 和 `amd.com/gpucores` 请求 AMD GPU： + +- `amd.com/gpu`：Pod 需要的 AMD GPU 数量 +- `amd.com/gpumem`：每张 GPU 的显存配额，单位为 MiB +- `amd.com/gpucores`：每张 GPU 的 CU 配额百分比，范围为 0-100；例如 `25` 在 304 CU 的设备上约分配 76 CU + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: amd-vgpu-example +spec: + schedulerName: hami-scheduler + restartPolicy: Never + containers: + - name: pytorch + image: rocm/pytorch:latest + command: ["bash", "-c"] + args: + - | + env | grep -E 'LD_AUDIT|HIP_DEVICE_MEMORY_LIMIT' + python3 -c 'import torch; print(torch.cuda.mem_get_info(0)); print(torch.cuda.get_device_name(0))' + sleep 300 + resources: + requests: + amd.com/gpu: 1 + amd.com/gpumem: 49152 + amd.com/gpucores: 25 + limits: + amd.com/gpu: 1 + amd.com/gpumem: 49152 + amd.com/gpucores: 25 +``` + +```bash +kubectl apply -f amd-vgpu-example.yaml +kubectl get pod amd-vgpu-example -o wide +kubectl logs amd-vgpu-example +``` + +成功时，日志应包含类似内容： + +```text +LD_AUDIT=/usr/local/vgpu/libamvgpu.so +HIP_DEVICE_MEMORY_LIMIT=49152m +(51539607552, 51539607552) +AMD Instinct MI300X VF +``` + +`51539607552` 为字节，约等于 48 GiB。可同时提交两个相同的 48 GiB / 25% CU 工作负载，验证共享和并发。 + +## 常见问题 + +| 症状 | 处理 | +| --- | --- | +| `node unregistered` | 检查 `amd-device-plugin` 是否运行，以及 `hami.io/node-amd-register` 是否包含 `devmem`、`devcore`。必要时重启 DaemonSet。 | +| `CardInsufficientMemory` | Pod 请求的显存超过设备剩余显存；降低 `amd.com/gpumem` 或等待其他工作负载结束。 | +| `insufficient free CUs` | 删除已完成的 AMD vGPU 测试 Pod，并重启 `amd-device-plugin` 清理过期分配。 | +| 容器内显存仍为物理卡大小 | 检查 Pod 环境是否有 `LD_AUDIT` 和 `HIP_DEVICE_MEMORY_LIMIT`。 | + +测试完成后删除工作负载： + +```bash +kubectl delete pod amd-vgpu-example +``` + +## 注意事项 + +1. 请部署 [amd-device-plugin](https://github.com/Project-HAMi/amd-device-plugin)，并使用与 HAMi 版本匹配的镜像。 +2. 使用 HAMi AMD soft vGPU 共享时，请保持 AMD GPU Operator 原生 device-plugin 关闭。 +3. 未设置 `amd.com/gpucores` 时，容器会获得每张已分配 GPU 的全部 CU。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/userguide/amd-device/examples/allocate-core-and-memory.md b/i18n/zh/docusaurus-plugin-content-docs/current/userguide/amd-device/examples/allocate-core-and-memory.md new file mode 100644 index 000000000..258dc2b3f --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/userguide/amd-device/examples/allocate-core-and-memory.md @@ -0,0 +1,32 @@ +--- +title: 分配设备核心和显存资源 +translated: true +--- + +若要分配部分 AMD GPU 资源，只需在请求 GPU 数量 `amd.com/gpu` 的同时设置 `amd.com/gpucores` 和 `amd.com/gpumem`。 + +以下示例请求一张 GPU，并分配 48 GiB 显存和 25% 计算单元： + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: amd-vgpu-example +spec: + schedulerName: hami-scheduler + restartPolicy: Never + containers: + - name: pytorch + image: rocm/pytorch:latest + command: ["bash", "-c"] + args: + - | + env | grep -E 'LD_AUDIT|HIP_DEVICE_MEMORY_LIMIT' + python3 -c 'import torch; print(torch.cuda.mem_get_info(0)); print(torch.cuda.get_device_name(0))' + sleep 300 + resources: + limits: + amd.com/gpu: 1 # 请求一个 AMD GPU + amd.com/gpumem: 49152 # 每个 GPU 包含 49152 MiB 设备显存 + amd.com/gpucores: 25 # 每个 GPU 分配 25% 的计算单元 +``` diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/userguide/device-supported.md b/i18n/zh/docusaurus-plugin-content-docs/current/userguide/device-supported.md index 16ad1262b..ae0c03759 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/userguide/device-supported.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/userguide/device-supported.md @@ -18,4 +18,5 @@ HAMi 支持的设备如下表所示： | GCU | 燧原科技（Enflame） | S60 | 是 | 是 | 否 | | XPU | 昆仑芯（Kunlunxin） | P800 | 是 | 是 | 否 | | GPU | 瀚博（Vastai） | VA16 | 是 | 是 | 否 | +| GPU | AMD | Instinct / ROCm | 是 | 是 | 否 | | DPU | 太初元碁（Teco） | 检查中 | 进行中 | 进行中 | 否 | diff --git a/sidebars.js b/sidebars.js index 828ff4e7c..445b9b301 100644 --- a/sidebars.js +++ b/sidebars.js @@ -193,6 +193,21 @@ module.exports = { label: "Share Enflame GCU devices", items: ["userguide/enflame-device/enable-enflame-gcu-sharing"], }, + { + type: "category", + label: "Share AMD GPU devices", + items: [ + "userguide/amd-device/enable-amd-gpu-sharing", + { + type: "category", + label: "Examples", + key: "amd-examples", + items: [ + "userguide/amd-device/examples/allocate-core-and-memory", + ], + }, + ], + }, { type: "category", label: "Managing AWS Neuron devices", From d1b432cdad19b5cc0708a2585b104d6ad102d4ec Mon Sep 17 00:00:00 2001 From: Jifei Wang Date: Mon, 3 Aug 2026 11:58:43 +0800 Subject: [PATCH 2/3] fix(amd): prettier format and helm --dependency-update Signed-off-by: Jifei Wang Co-authored-by: Cursor --- docs/userguide/amd-device/enable-amd-gpu-sharing.md | 3 ++- .../current/userguide/amd-device/enable-amd-gpu-sharing.md | 3 ++- sidebars.js | 4 +--- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/userguide/amd-device/enable-amd-gpu-sharing.md b/docs/userguide/amd-device/enable-amd-gpu-sharing.md index 193727563..ef4b3b077 100644 --- a/docs/userguide/amd-device/enable-amd-gpu-sharing.md +++ b/docs/userguide/amd-device/enable-amd-gpu-sharing.md @@ -76,6 +76,7 @@ Deploy [amd-device-plugin](https://github.com/Project-HAMi/amd-device-plugin) to ```bash git clone https://github.com/Project-HAMi/amd-device-plugin.git helm install amd-gpu ./amd-device-plugin/helm/amd-gpu -n kube-system \ + --dependency-update \ --set dp.image.repository= \ --set dp.image.tag= ``` @@ -98,7 +99,7 @@ kubectl get node -o json | \ The result must include `devmem` and `devcore`. Example for MI300X VF: ```json -[{"count":10,"devmem":196608,"devcore":304,"type":"AMD_Instinct_MI300X_VF"}] +[{ "count": 10, "devmem": 196608, "devcore": 304, "type": "AMD_Instinct_MI300X_VF" }] ``` ## Running AMD vGPU Jobs diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/userguide/amd-device/enable-amd-gpu-sharing.md b/i18n/zh/docusaurus-plugin-content-docs/current/userguide/amd-device/enable-amd-gpu-sharing.md index e50f3bd3e..c9bcec367 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/userguide/amd-device/enable-amd-gpu-sharing.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/userguide/amd-device/enable-amd-gpu-sharing.md @@ -77,6 +77,7 @@ kubectl -n kube-amd-gpu patch deviceconfig default --type=merge -p \ ```bash git clone https://github.com/Project-HAMi/amd-device-plugin.git helm install amd-gpu ./amd-device-plugin/helm/amd-gpu -n kube-system \ + --dependency-update \ --set dp.image.repository= \ --set dp.image.tag= ``` @@ -99,7 +100,7 @@ kubectl get node -o json | \ 结果必须包含 `devmem` 和 `devcore`。例如 MI300X VF： ```json -[{"count":10,"devmem":196608,"devcore":304,"type":"AMD_Instinct_MI300X_VF"}] +[{ "count": 10, "devmem": 196608, "devcore": 304, "type": "AMD_Instinct_MI300X_VF" }] ``` ## 运行 AMD vGPU 任务 diff --git a/sidebars.js b/sidebars.js index 445b9b301..3487c0e9f 100644 --- a/sidebars.js +++ b/sidebars.js @@ -202,9 +202,7 @@ module.exports = { type: "category", label: "Examples", key: "amd-examples", - items: [ - "userguide/amd-device/examples/allocate-core-and-memory", - ], + items: ["userguide/amd-device/examples/allocate-core-and-memory"], }, ], }, From 991595d5389f4f3fe21c89c73b672965a999b65e Mon Sep 17 00:00:00 2001 From: Jifei Wang Date: Mon, 3 Aug 2026 14:02:48 +0800 Subject: [PATCH 3/3] docs(amd): sync with amd-device-plugin v0.0.1 release Signed-off-by: Jifei Wang Co-authored-by: Cursor --- .../amd-device/enable-amd-gpu-sharing.md | 49 +++++++++++++------ .../amd-device/enable-amd-gpu-sharing.md | 49 +++++++++++++------ 2 files changed, 68 insertions(+), 30 deletions(-) diff --git a/docs/userguide/amd-device/enable-amd-gpu-sharing.md b/docs/userguide/amd-device/enable-amd-gpu-sharing.md index ef4b3b077..16b581cad 100644 --- a/docs/userguide/amd-device/enable-amd-gpu-sharing.md +++ b/docs/userguide/amd-device/enable-amd-gpu-sharing.md @@ -15,7 +15,9 @@ HAMi supports sharing AMD Instinct/ROCm GPUs. Workloads request device memory an :::caution -Use the [amd-device-plugin](https://github.com/Project-HAMi/amd-device-plugin) image and manifests that match your HAMi version. Do not deploy the upstream ROCm `k8s-device-plugin` image for HAMi soft vGPU. +Use the [amd-device-plugin](https://github.com/Project-HAMi/amd-device-plugin) image and Helm chart. Do not deploy the upstream ROCm `k8s-device-plugin` image for HAMi soft vGPU. + +Fractional memory isolation loads `libamvgpu.so` through glibc `LD_AUDIT` and currently requires glibc symbols through `GLIBC_2.34`. Workload images based on older glibc (for example Ubuntu 20.04 or RHEL 8) or musl/Alpine are not supported yet. See [HAMi#2265](https://github.com/Project-HAMi/HAMi/issues/2265). ::: @@ -27,9 +29,9 @@ Deploy these components: | --- | --- | --- | | HAMi | Scheduling, allocation, and admission | Scheduler is running and manages the three AMD resources | | AMD GPU Operator (recommended) | Driver and ROCm environment | Disable the Operator native device-plugin | -| [amd-device-plugin](https://github.com/Project-HAMi/amd-device-plugin) | Register AMD resources, allocate CUs, inject runtime limits | Deploy the HAMi fork; it discovers VRAM/CU via amd-smi/libdrm | +| [amd-device-plugin](https://github.com/Project-HAMi/amd-device-plugin) | Register AMD resources, allocate CUs, inject runtime limits | Deploy chart/image `0.0.1` or newer; discovers VRAM/CU via amd-smi/libdrm | -Nodes also need a working AMD driver and ROCm. Verify with: +Nodes also need a working AMD driver and ROCm (validated with ROCm 7.0.2). Verify with: ```bash amd-smi static --gpu 0 @@ -71,17 +73,16 @@ kubectl -n kube-amd-gpu patch deviceconfig default --type=merge -p \ ### Deploy amd-device-plugin -Deploy [amd-device-plugin](https://github.com/Project-HAMi/amd-device-plugin) to all AMD GPU nodes. Prefer the Helm chart in that repository: +Deploy [amd-device-plugin](https://github.com/Project-HAMi/amd-device-plugin) to all AMD GPU nodes. Chart `0.0.1` defaults to image `ghcr.io/project-hami/amd-device-plugin:0.0.1` and installs the bundled `libamvgpu.so` hook onto the node through a `postStart` lifecycle hook: ```bash -git clone https://github.com/Project-HAMi/amd-device-plugin.git -helm install amd-gpu ./amd-device-plugin/helm/amd-gpu -n kube-system \ - --dependency-update \ - --set dp.image.repository= \ - --set dp.image.tag= +helm upgrade --install amd-gpu \ + https://github.com/Project-HAMi/amd-device-plugin/releases/download/amd-gpu-helm-0.0.1/amd-gpu-0.0.1.tgz \ + --namespace kube-system \ + --create-namespace ``` -Replace `` and `` with the image that matches your HAMi version. The chart mounts `/var/lib/kubelet/device-plugins`, `/sys`, and the vGPU hook path, and sets `NODE_NAME` from `spec.nodeName`. +If the GHCR package is private in your environment, configure `imagePullSecrets`. You can also clone the repository and install from `./helm/amd-gpu`. Wait for the DaemonSet: @@ -92,14 +93,28 @@ kubectl -n kube-system rollout status ds/amd-gpu-device-plugin-daemonset Confirm the device-plugin registered full device info with HAMi: ```bash -kubectl get node -o json | \ - jq -r '.metadata.annotations["hami.io/node-amd-register"]' +kubectl get node -o jsonpath='{.metadata.annotations.hami\.io/node-amd-register}' ``` The result must include `devmem` and `devcore`. Example for MI300X VF: ```json -[{ "count": 10, "devmem": 196608, "devcore": 304, "type": "AMD_Instinct_MI300X_VF" }] +[ + { + "id": "8eff74b5-0000-1000-801b-b56457addd1b", + "index": 0, + "count": 10, + "devmem": 196288, + "devcore": 304, + "type": "AMD Instinct MI300X VF", + "numa": 0, + "health": true, + "devicevendor": "amd", + "custominfo": { + "pciBDF": "0000:83:00.0" + } + } +] ``` ## Running AMD vGPU Jobs @@ -110,6 +125,8 @@ Request AMD GPUs with `amd.com/gpu`, `amd.com/gpumem`, and `amd.com/gpucores`: - `amd.com/gpumem`: device memory quota per GPU, in MiB - `amd.com/gpucores`: CU quota percentage per GPU, range 0-100; for example `25` allocates about 76 CUs on a 304-CU device +Use a glibc workload image that meets the `GLIBC_2.34` requirement above (for example a recent `rocm/pytorch` tag): + ```yaml apiVersion: v1 kind: Pod @@ -162,7 +179,8 @@ AMD Instinct MI300X VF | `node unregistered` | Check that `amd-device-plugin` is running and `hami.io/node-amd-register` contains `devmem` and `devcore`. Restart the DaemonSet if needed. | | `CardInsufficientMemory` | The Pod requests more memory than the device has free. Lower `amd.com/gpumem` or wait for other workloads to finish. | | `insufficient free CUs` | Delete finished AMD vGPU test Pods and restart `amd-device-plugin` to clear stale allocations. | -| Memory inside the container still shows the full physical size | Check that the Pod env includes `LD_AUDIT` and `HIP_DEVICE_MEMORY_LIMIT`. | +| Memory inside the container still shows the full physical size | Check that the Pod env includes `LD_AUDIT` and `HIP_DEVICE_MEMORY_LIMIT`, and that the workload image has a compatible glibc. | +| Workload fails to start / cannot load `libamvgpu.so` | Switch to a glibc image with `GLIBC_2.34` or newer. musl/Alpine and older distros are not supported yet. | Clean up after testing: @@ -172,6 +190,7 @@ kubectl delete pod amd-vgpu-example ## Notes -1. Deploy [amd-device-plugin](https://github.com/Project-HAMi/amd-device-plugin); use an image that matches your HAMi version. +1. Deploy [amd-device-plugin](https://github.com/Project-HAMi/amd-device-plugin) `0.0.1` or newer (`ghcr.io/project-hami/amd-device-plugin`). 2. Keep the AMD GPU Operator native device-plugin disabled while using HAMi AMD soft vGPU sharing. 3. Omitting `amd.com/gpucores` allocates all CUs on each requested GPU. +4. The bundled `libamvgpu.so` delivery is temporary and will move to `amd-hami-core` once that project publishes a consumption pipeline. diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/userguide/amd-device/enable-amd-gpu-sharing.md b/i18n/zh/docusaurus-plugin-content-docs/current/userguide/amd-device/enable-amd-gpu-sharing.md index c9bcec367..c1af47ab2 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/userguide/amd-device/enable-amd-gpu-sharing.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/userguide/amd-device/enable-amd-gpu-sharing.md @@ -16,7 +16,9 @@ HAMi 支持共享 AMD Instinct/ROCm GPU。工作负载通过标准 Kubernetes :::caution -请使用与 HAMi 版本匹配的 [amd-device-plugin](https://github.com/Project-HAMi/amd-device-plugin) 镜像和部署清单。不要用上游 ROCm `k8s-device-plugin` 镜像来跑 HAMi soft vGPU。 +请使用 [amd-device-plugin](https://github.com/Project-HAMi/amd-device-plugin) 镜像和 Helm chart。不要用上游 ROCm `k8s-device-plugin` 镜像来跑 HAMi soft vGPU。 + +显存隔离依赖 glibc `LD_AUDIT` 加载 `libamvgpu.so`，当前需要 `GLIBC_2.34` 及以上符号。基于更旧 glibc（例如 Ubuntu 20.04、RHEL 8）或 musl/Alpine 的工作负载镜像暂不支持。详见 [HAMi#2265](https://github.com/Project-HAMi/HAMi/issues/2265)。 ::: @@ -28,9 +30,9 @@ HAMi 支持共享 AMD Instinct/ROCm GPU。工作负载通过标准 Kubernetes | --- | --- | --- | | HAMi | 调度、设备分配和准入 | scheduler 正常运行，并管理三个 AMD 资源 | | AMD GPU Operator（推荐） | 驱动和 ROCm 环境 | 禁用 Operator 原生 device-plugin | -| [amd-device-plugin](https://github.com/Project-HAMi/amd-device-plugin) | 注册 AMD 资源、分配 CU、注入容器运行时限制 | 部署 HAMi fork；通过 amd-smi/libdrm 发现显存与 CU | +| [amd-device-plugin](https://github.com/Project-HAMi/amd-device-plugin) | 注册 AMD 资源、分配 CU、注入容器运行时限制 | 部署 chart/镜像 `0.0.1` 或更新版本；通过 amd-smi/libdrm 发现显存与 CU | -节点还必须具备可用的 AMD 驱动和 ROCm。可用以下命令确认： +节点还必须具备可用的 AMD 驱动和 ROCm（已在 ROCm 7.0.2 上验证）。可用以下命令确认： ```bash amd-smi static --gpu 0 @@ -72,17 +74,16 @@ kubectl -n kube-amd-gpu patch deviceconfig default --type=merge -p \ ### 部署 amd-device-plugin -在所有 AMD GPU 节点上部署 [amd-device-plugin](https://github.com/Project-HAMi/amd-device-plugin)。推荐使用该仓库中的 Helm chart： +在所有 AMD GPU 节点上部署 [amd-device-plugin](https://github.com/Project-HAMi/amd-device-plugin)。Chart `0.0.1` 默认镜像为 `ghcr.io/project-hami/amd-device-plugin:0.0.1`，并通过 `postStart` lifecycle hook 将镜像内的 `libamvgpu.so` 安装到节点： ```bash -git clone https://github.com/Project-HAMi/amd-device-plugin.git -helm install amd-gpu ./amd-device-plugin/helm/amd-gpu -n kube-system \ - --dependency-update \ - --set dp.image.repository= \ - --set dp.image.tag= +helm upgrade --install amd-gpu \ + https://github.com/Project-HAMi/amd-device-plugin/releases/download/amd-gpu-helm-0.0.1/amd-gpu-0.0.1.tgz \ + --namespace kube-system \ + --create-namespace ``` -将 `` 和 `` 替换为与当前 HAMi 版本匹配的镜像。Chart 会挂载 `/var/lib/kubelet/device-plugins`、`/sys` 和 vGPU hook 路径，并将 `NODE_NAME` 设为 `spec.nodeName`。 +若环境中 GHCR 包为私有，请配置 `imagePullSecrets`。也可以 clone 仓库后从 `./helm/amd-gpu` 安装。 等待 DaemonSet 就绪： @@ -93,14 +94,28 @@ kubectl -n kube-system rollout status ds/amd-gpu-device-plugin-daemonset 确认 device-plugin 已将完整设备信息注册给 HAMi： ```bash -kubectl get node -o json | \ - jq -r '.metadata.annotations["hami.io/node-amd-register"]' +kubectl get node -o jsonpath='{.metadata.annotations.hami\.io/node-amd-register}' ``` 结果必须包含 `devmem` 和 `devcore`。例如 MI300X VF： ```json -[{ "count": 10, "devmem": 196608, "devcore": 304, "type": "AMD_Instinct_MI300X_VF" }] +[ + { + "id": "8eff74b5-0000-1000-801b-b56457addd1b", + "index": 0, + "count": 10, + "devmem": 196288, + "devcore": 304, + "type": "AMD Instinct MI300X VF", + "numa": 0, + "health": true, + "devicevendor": "amd", + "custominfo": { + "pciBDF": "0000:83:00.0" + } + } +] ``` ## 运行 AMD vGPU 任务 @@ -111,6 +126,8 @@ kubectl get node -o json | \ - `amd.com/gpumem`：每张 GPU 的显存配额，单位为 MiB - `amd.com/gpucores`：每张 GPU 的 CU 配额百分比，范围为 0-100；例如 `25` 在 304 CU 的设备上约分配 76 CU +请使用满足上述 `GLIBC_2.34` 要求的 glibc 工作负载镜像（例如较新的 `rocm/pytorch` 标签）： + ```yaml apiVersion: v1 kind: Pod @@ -163,7 +180,8 @@ AMD Instinct MI300X VF | `node unregistered` | 检查 `amd-device-plugin` 是否运行，以及 `hami.io/node-amd-register` 是否包含 `devmem`、`devcore`。必要时重启 DaemonSet。 | | `CardInsufficientMemory` | Pod 请求的显存超过设备剩余显存；降低 `amd.com/gpumem` 或等待其他工作负载结束。 | | `insufficient free CUs` | 删除已完成的 AMD vGPU 测试 Pod，并重启 `amd-device-plugin` 清理过期分配。 | -| 容器内显存仍为物理卡大小 | 检查 Pod 环境是否有 `LD_AUDIT` 和 `HIP_DEVICE_MEMORY_LIMIT`。 | +| 容器内显存仍为物理卡大小 | 检查 Pod 环境是否有 `LD_AUDIT` 和 `HIP_DEVICE_MEMORY_LIMIT`，以及工作负载镜像的 glibc 是否兼容。 | +| 工作负载无法启动 / 加载不了 `libamvgpu.so` | 换用 `GLIBC_2.34` 及以上的 glibc 镜像；musl/Alpine 和旧发行版暂不支持。 | 测试完成后删除工作负载： @@ -173,6 +191,7 @@ kubectl delete pod amd-vgpu-example ## 注意事项 -1. 请部署 [amd-device-plugin](https://github.com/Project-HAMi/amd-device-plugin)，并使用与 HAMi 版本匹配的镜像。 +1. 请部署 [amd-device-plugin](https://github.com/Project-HAMi/amd-device-plugin) `0.0.1` 或更新版本（`ghcr.io/project-hami/amd-device-plugin`）。 2. 使用 HAMi AMD soft vGPU 共享时，请保持 AMD GPU Operator 原生 device-plugin 关闭。 3. 未设置 `amd.com/gpucores` 时，容器会获得每张已分配 GPU 的全部 CU。 +4. 当前镜像内置的 `libamvgpu.so` 分发方式是临时方案，后续将迁移到 `amd-hami-core` 正式产物。