diff --git a/docs/contributor/roadmap.md b/docs/contributor/roadmap.md index aef0c7461..1eeafd24e 100644 --- a/docs/contributor/roadmap.md +++ b/docs/contributor/roadmap.md @@ -33,5 +33,5 @@ sidebar_label: Roadmap - [ ] Rich observability support - [x] DRA support - [ ] Support Intel GPU device -- [ ] Support AMD GPU device +- [x] Support AMD GPU device - [x] Support Enflame GCU device diff --git a/docs/userguide/amd-device/enable-amd-gpu-sharing.md b/docs/userguide/amd-device/enable-amd-gpu-sharing.md new file mode 100644 index 000000000..16b581cad --- /dev/null +++ b/docs/userguide/amd-device/enable-amd-gpu-sharing.md @@ -0,0 +1,196 @@ +--- +title: Enable AMD GPU Sharing +sidebar_label: GPU Sharing +--- + +## Introduction + +HAMi supports sharing AMD Instinct/ROCm GPUs. Workloads request device memory and compute-unit (CU) share through standard Kubernetes resources, without application code changes. + +**GPU sharing**: Multiple tasks can share one AMD GPU instead of occupying a whole card. + +**Device memory control**: Allocate a specific amount of device memory (MiB). HAMi enforces a hard limit so usage cannot exceed the allocation. + +**Device compute core limitation**: Allocate a percentage of compute units (`amd.com/gpucores: 25` means about 25% of the device CUs). + +:::caution + +Use the [amd-device-plugin](https://github.com/Project-HAMi/amd-device-plugin) image and Helm chart. Do not deploy the upstream ROCm `k8s-device-plugin` image for HAMi soft vGPU. + +Fractional memory isolation loads `libamvgpu.so` through glibc `LD_AUDIT` and currently requires glibc symbols through `GLIBC_2.34`. Workload images based on older glibc (for example Ubuntu 20.04 or RHEL 8) or musl/Alpine are not supported yet. See [HAMi#2265](https://github.com/Project-HAMi/HAMi/issues/2265). + +::: + +## Prerequisites + +Deploy these components: + +| Component | Role | Key requirement | +| --- | --- | --- | +| HAMi | Scheduling, allocation, and admission | Scheduler is running and manages the three AMD resources | +| AMD GPU Operator (recommended) | Driver and ROCm environment | Disable the Operator native device-plugin | +| [amd-device-plugin](https://github.com/Project-HAMi/amd-device-plugin) | Register AMD resources, allocate CUs, inject runtime limits | Deploy chart/image `0.0.1` or newer; discovers VRAM/CU via amd-smi/libdrm | + +Nodes also need a working AMD driver and ROCm (validated with ROCm 7.0.2). Verify with: + +```bash +amd-smi static --gpu 0 +``` + +The output should include the device model, VRAM, and `NUM_COMPUTE_UNITS`. + +## Enabling AMD GPU Sharing + +### Configure HAMi + +After installing HAMi, confirm the scheduler manages all AMD vGPU resources. The values file should include: + +```yaml +devices: + amd: + customresources: + - amd.com/gpu + - amd.com/gpumem + - amd.com/gpucores +``` + +Confirm the scheduler is running: + +```bash +kubectl -n kube-system get pods | grep hami-scheduler +``` + +### Disable the AMD GPU Operator device-plugin + +If you use the [AMD GPU Operator](https://github.com/ROCm/gpu-operator) for drivers and ROCm, disable its native device-plugin so it does not compete with HAMi `amd-device-plugin` for `amd.com/gpu`: + +```bash +kubectl -n kube-amd-gpu patch deviceconfig default --type=merge -p \ + '{"spec":{"devicePlugin":{"enableDevicePlugin":false}}}' +``` + +`amd-device-plugin` reads per-device VRAM, CU count, UUID, and product name through amd-smi/libdrm, so the Operator node-labeller is optional for HAMi soft vGPU. + +### Deploy amd-device-plugin + +Deploy [amd-device-plugin](https://github.com/Project-HAMi/amd-device-plugin) to all AMD GPU nodes. Chart `0.0.1` defaults to image `ghcr.io/project-hami/amd-device-plugin:0.0.1` and installs the bundled `libamvgpu.so` hook onto the node through a `postStart` lifecycle hook: + +```bash +helm upgrade --install amd-gpu \ + https://github.com/Project-HAMi/amd-device-plugin/releases/download/amd-gpu-helm-0.0.1/amd-gpu-0.0.1.tgz \ + --namespace kube-system \ + --create-namespace +``` + +If the GHCR package is private in your environment, configure `imagePullSecrets`. You can also clone the repository and install from `./helm/amd-gpu`. + +Wait for the DaemonSet: + +```bash +kubectl -n kube-system rollout status ds/amd-gpu-device-plugin-daemonset +``` + +Confirm the device-plugin registered full device info with HAMi: + +```bash +kubectl get node -o jsonpath='{.metadata.annotations.hami\.io/node-amd-register}' +``` + +The result must include `devmem` and `devcore`. Example for MI300X VF: + +```json +[ + { + "id": "8eff74b5-0000-1000-801b-b56457addd1b", + "index": 0, + "count": 10, + "devmem": 196288, + "devcore": 304, + "type": "AMD Instinct MI300X VF", + "numa": 0, + "health": true, + "devicevendor": "amd", + "custominfo": { + "pciBDF": "0000:83:00.0" + } + } +] +``` + +## Running AMD vGPU Jobs + +Request AMD GPUs with `amd.com/gpu`, `amd.com/gpumem`, and `amd.com/gpucores`: + +- `amd.com/gpu`: number of AMD GPUs +- `amd.com/gpumem`: device memory quota per GPU, in MiB +- `amd.com/gpucores`: CU quota percentage per GPU, range 0-100; for example `25` allocates about 76 CUs on a 304-CU device + +Use a glibc workload image that meets the `GLIBC_2.34` requirement above (for example a recent `rocm/pytorch` tag): + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: amd-vgpu-example +spec: + schedulerName: hami-scheduler + restartPolicy: Never + containers: + - name: pytorch + image: rocm/pytorch:latest + command: ["bash", "-c"] + args: + - | + env | grep -E 'LD_AUDIT|HIP_DEVICE_MEMORY_LIMIT' + python3 -c 'import torch; print(torch.cuda.mem_get_info(0)); print(torch.cuda.get_device_name(0))' + sleep 300 + resources: + requests: + amd.com/gpu: 1 + amd.com/gpumem: 49152 + amd.com/gpucores: 25 + limits: + amd.com/gpu: 1 + amd.com/gpumem: 49152 + amd.com/gpucores: 25 +``` + +```bash +kubectl apply -f amd-vgpu-example.yaml +kubectl get pod amd-vgpu-example -o wide +kubectl logs amd-vgpu-example +``` + +On success, logs look like: + +```text +LD_AUDIT=/usr/local/vgpu/libamvgpu.so +HIP_DEVICE_MEMORY_LIMIT=49152m +(51539607552, 51539607552) +AMD Instinct MI300X VF +``` + +`51539607552` is bytes, about 48 GiB. You can submit two identical 48 GiB / 25% CU workloads to verify sharing. + +## Troubleshooting + +| Symptom | Action | +| --- | --- | +| `node unregistered` | Check that `amd-device-plugin` is running and `hami.io/node-amd-register` contains `devmem` and `devcore`. Restart the DaemonSet if needed. | +| `CardInsufficientMemory` | The Pod requests more memory than the device has free. Lower `amd.com/gpumem` or wait for other workloads to finish. | +| `insufficient free CUs` | Delete finished AMD vGPU test Pods and restart `amd-device-plugin` to clear stale allocations. | +| Memory inside the container still shows the full physical size | Check that the Pod env includes `LD_AUDIT` and `HIP_DEVICE_MEMORY_LIMIT`, and that the workload image has a compatible glibc. | +| Workload fails to start / cannot load `libamvgpu.so` | Switch to a glibc image with `GLIBC_2.34` or newer. musl/Alpine and older distros are not supported yet. | + +Clean up after testing: + +```bash +kubectl delete pod amd-vgpu-example +``` + +## Notes + +1. Deploy [amd-device-plugin](https://github.com/Project-HAMi/amd-device-plugin) `0.0.1` or newer (`ghcr.io/project-hami/amd-device-plugin`). +2. Keep the AMD GPU Operator native device-plugin disabled while using HAMi AMD soft vGPU sharing. +3. Omitting `amd.com/gpucores` allocates all CUs on each requested GPU. +4. The bundled `libamvgpu.so` delivery is temporary and will move to `amd-hami-core` once that project publishes a consumption pipeline. diff --git a/docs/userguide/amd-device/examples/allocate-core-and-memory.md b/docs/userguide/amd-device/examples/allocate-core-and-memory.md new file mode 100644 index 000000000..47ccf392d --- /dev/null +++ b/docs/userguide/amd-device/examples/allocate-core-and-memory.md @@ -0,0 +1,31 @@ +--- +title: Allocate device core and memory resource +--- + +To allocate part of an AMD GPU, set `amd.com/gpucores` and `amd.com/gpumem` together with the number of GPUs in `amd.com/gpu`. + +The example below requests one GPU with 48 GiB device memory and 25% compute units: + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: amd-vgpu-example +spec: + schedulerName: hami-scheduler + restartPolicy: Never + containers: + - name: pytorch + image: rocm/pytorch:latest + command: ["bash", "-c"] + args: + - | + env | grep -E 'LD_AUDIT|HIP_DEVICE_MEMORY_LIMIT' + python3 -c 'import torch; print(torch.cuda.mem_get_info(0)); print(torch.cuda.get_device_name(0))' + sleep 300 + resources: + limits: + amd.com/gpu: 1 # requesting one AMD GPU + amd.com/gpumem: 49152 # each GPU requires 49152 MiB device memory + amd.com/gpucores: 25 # each GPU uses 25% of total compute units +``` diff --git a/docs/userguide/device-supported.md b/docs/userguide/device-supported.md index a6beae4d0..3b39d989b 100644 --- a/docs/userguide/device-supported.md +++ b/docs/userguide/device-supported.md @@ -18,4 +18,5 @@ The table below lists the devices supported by HAMi: | XPU | Kunlunxin | P800 | Yes | Yes | No | | GPU | Vastai | VA16 | Yes | Yes | No | | GPU | Biren | Biren166M | Yes | Yes | No | +| GPU | AMD | Instinct / ROCm | Yes | Yes | No | | DPU | Teco | Checking | In progress | In progress | No | diff --git a/i18n/zh/docusaurus-plugin-content-docs/current.json b/i18n/zh/docusaurus-plugin-content-docs/current.json index 4f107183e..85a18a02b 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current.json +++ b/i18n/zh/docusaurus-plugin-content-docs/current.json @@ -71,6 +71,14 @@ "message": "共享燧原 GCU 设备", "description": "The label for category 'Share Enflame GCU devices' in sidebar 'docs'" }, + "sidebar.docs.category.Share AMD GPU devices": { + "message": "共享 AMD GPU 设备", + "description": "The label for category 'Share AMD GPU devices' in sidebar 'docs'" + }, + "sidebar.docs.category.amd-examples": { + "message": "示例", + "description": "The label for category 'Examples' in sidebar 'docs'" + }, "sidebar.docs.category.Share MetaX GPU devices": { "message": "共享沐曦 GPU 设备", "description": "The label for category 'Share MetaX GPU devices' in sidebar 'docs'" diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/contributor/roadmap.md b/i18n/zh/docusaurus-plugin-content-docs/current/contributor/roadmap.md index 0622837f0..8482b0a89 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/contributor/roadmap.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/contributor/roadmap.md @@ -34,5 +34,5 @@ sidebar_label: 路线图 - [ ] 丰富的可观测性支持 - [x] 支持 DRA - [ ] 支持 Intel GPU 设备 -- [ ] 支持 AMD GPU 设备 +- [x] 支持 AMD GPU 设备 - [x] 支持 Enflame GCU 设备 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/userguide/amd-device/enable-amd-gpu-sharing.md b/i18n/zh/docusaurus-plugin-content-docs/current/userguide/amd-device/enable-amd-gpu-sharing.md new file mode 100644 index 000000000..c1af47ab2 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/userguide/amd-device/enable-amd-gpu-sharing.md @@ -0,0 +1,197 @@ +--- +title: 启用 AMD GPU 共享 +sidebar_label: GPU 共享 +translated: true +--- + +## 简介 + +HAMi 支持共享 AMD Instinct/ROCm GPU。工作负载通过标准 Kubernetes 资源请求显存和算力份额，无需修改应用代码。 + +**GPU 共享**: 每个任务可以只占用一部分显卡，多个任务可以共享一张显卡。 + +**可限制分配的显存大小**: 可以用显存值（MiB）分配 GPU，HAMi 会确保任务使用的显存不会超过分配数值。 + +**可限制计算单元数量**: 可以指定任务使用的算力比例（例如 `amd.com/gpucores: 25` 代表使用约 25% CU）。 + +:::caution + +请使用 [amd-device-plugin](https://github.com/Project-HAMi/amd-device-plugin) 镜像和 Helm chart。不要用上游 ROCm `k8s-device-plugin` 镜像来跑 HAMi soft vGPU。 + +显存隔离依赖 glibc `LD_AUDIT` 加载 `libamvgpu.so`，当前需要 `GLIBC_2.34` 及以上符号。基于更旧 glibc（例如 Ubuntu 20.04、RHEL 8）或 musl/Alpine 的工作负载镜像暂不支持。详见 [HAMi#2265](https://github.com/Project-HAMi/HAMi/issues/2265)。 + +::: + +## 节点需求 + +需要部署以下组件： + +| 组件 | 作用 | 关键要求 | +| --- | --- | --- | +| HAMi | 调度、设备分配和准入 | scheduler 正常运行，并管理三个 AMD 资源 | +| AMD GPU Operator（推荐） | 驱动和 ROCm 环境 | 禁用 Operator 原生 device-plugin | +| [amd-device-plugin](https://github.com/Project-HAMi/amd-device-plugin) | 注册 AMD 资源、分配 CU、注入容器运行时限制 | 部署 chart/镜像 `0.0.1` 或更新版本；通过 amd-smi/libdrm 发现显存与 CU | + +节点还必须具备可用的 AMD 驱动和 ROCm（已在 ROCm 7.0.2 上验证）。可用以下命令确认： + +```bash +amd-smi static --gpu 0 +``` + +输出应包含设备型号、VRAM 和 `NUM_COMPUTE_UNITS`。 + +## 开启 AMD GPU 共享 + +### 配置 HAMi + +安装 HAMi 后，确认 scheduler 管理所有 AMD vGPU 资源。values 文件中应包含： + +```yaml +devices: + amd: + customresources: + - amd.com/gpu + - amd.com/gpumem + - amd.com/gpucores +``` + +安装或升级后确认 scheduler 正常运行： + +```bash +kubectl -n kube-system get pods | grep hami-scheduler +``` + +### 禁用 AMD GPU Operator device-plugin + +若使用 [AMD GPU Operator](https://github.com/ROCm/gpu-operator) 管理驱动和 ROCm，请关闭其原生 device-plugin，避免与 HAMi `amd-device-plugin` 竞争 `amd.com/gpu`： + +```bash +kubectl -n kube-amd-gpu patch deviceconfig default --type=merge -p \ + '{"spec":{"devicePlugin":{"enableDevicePlugin":false}}}' +``` + +`amd-device-plugin` 通过 amd-smi/libdrm 读取每张卡的显存、CU、UUID 和产品名称，因此对 HAMi soft vGPU 而言，Operator node-labeller 是可选的。 + +### 部署 amd-device-plugin + +在所有 AMD GPU 节点上部署 [amd-device-plugin](https://github.com/Project-HAMi/amd-device-plugin)。Chart `0.0.1` 默认镜像为 `ghcr.io/project-hami/amd-device-plugin:0.0.1`，并通过 `postStart` lifecycle hook 将镜像内的 `libamvgpu.so` 安装到节点： + +```bash +helm upgrade --install amd-gpu \ + https://github.com/Project-HAMi/amd-device-plugin/releases/download/amd-gpu-helm-0.0.1/amd-gpu-0.0.1.tgz \ + --namespace kube-system \ + --create-namespace +``` + +若环境中 GHCR 包为私有，请配置 `imagePullSecrets`。也可以 clone 仓库后从 `./helm/amd-gpu` 安装。 + +等待 DaemonSet 就绪： + +```bash +kubectl -n kube-system rollout status ds/amd-gpu-device-plugin-daemonset +``` + +确认 device-plugin 已将完整设备信息注册给 HAMi： + +```bash +kubectl get node -o jsonpath='{.metadata.annotations.hami\.io/node-amd-register}' +``` + +结果必须包含 `devmem` 和 `devcore`。例如 MI300X VF： + +```json +[ + { + "id": "8eff74b5-0000-1000-801b-b56457addd1b", + "index": 0, + "count": 10, + "devmem": 196288, + "devcore": 304, + "type": "AMD Instinct MI300X VF", + "numa": 0, + "health": true, + "devicevendor": "amd", + "custominfo": { + "pciBDF": "0000:83:00.0" + } + } +] +``` + +## 运行 AMD vGPU 任务 + +通过 `amd.com/gpu`、`amd.com/gpumem` 和 `amd.com/gpucores` 请求 AMD GPU： + +- `amd.com/gpu`：Pod 需要的 AMD GPU 数量 +- `amd.com/gpumem`：每张 GPU 的显存配额，单位为 MiB +- `amd.com/gpucores`：每张 GPU 的 CU 配额百分比，范围为 0-100；例如 `25` 在 304 CU 的设备上约分配 76 CU + +请使用满足上述 `GLIBC_2.34` 要求的 glibc 工作负载镜像（例如较新的 `rocm/pytorch` 标签）： + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: amd-vgpu-example +spec: + schedulerName: hami-scheduler + restartPolicy: Never + containers: + - name: pytorch + image: rocm/pytorch:latest + command: ["bash", "-c"] + args: + - | + env | grep -E 'LD_AUDIT|HIP_DEVICE_MEMORY_LIMIT' + python3 -c 'import torch; print(torch.cuda.mem_get_info(0)); print(torch.cuda.get_device_name(0))' + sleep 300 + resources: + requests: + amd.com/gpu: 1 + amd.com/gpumem: 49152 + amd.com/gpucores: 25 + limits: + amd.com/gpu: 1 + amd.com/gpumem: 49152 + amd.com/gpucores: 25 +``` + +```bash +kubectl apply -f amd-vgpu-example.yaml +kubectl get pod amd-vgpu-example -o wide +kubectl logs amd-vgpu-example +``` + +成功时，日志应包含类似内容： + +```text +LD_AUDIT=/usr/local/vgpu/libamvgpu.so +HIP_DEVICE_MEMORY_LIMIT=49152m +(51539607552, 51539607552) +AMD Instinct MI300X VF +``` + +`51539607552` 为字节，约等于 48 GiB。可同时提交两个相同的 48 GiB / 25% CU 工作负载，验证共享和并发。 + +## 常见问题 + +| 症状 | 处理 | +| --- | --- | +| `node unregistered` | 检查 `amd-device-plugin` 是否运行，以及 `hami.io/node-amd-register` 是否包含 `devmem`、`devcore`。必要时重启 DaemonSet。 | +| `CardInsufficientMemory` | Pod 请求的显存超过设备剩余显存；降低 `amd.com/gpumem` 或等待其他工作负载结束。 | +| `insufficient free CUs` | 删除已完成的 AMD vGPU 测试 Pod，并重启 `amd-device-plugin` 清理过期分配。 | +| 容器内显存仍为物理卡大小 | 检查 Pod 环境是否有 `LD_AUDIT` 和 `HIP_DEVICE_MEMORY_LIMIT`，以及工作负载镜像的 glibc 是否兼容。 | +| 工作负载无法启动 / 加载不了 `libamvgpu.so` | 换用 `GLIBC_2.34` 及以上的 glibc 镜像；musl/Alpine 和旧发行版暂不支持。 | + +测试完成后删除工作负载： + +```bash +kubectl delete pod amd-vgpu-example +``` + +## 注意事项 + +1. 请部署 [amd-device-plugin](https://github.com/Project-HAMi/amd-device-plugin) `0.0.1` 或更新版本（`ghcr.io/project-hami/amd-device-plugin`）。 +2. 使用 HAMi AMD soft vGPU 共享时，请保持 AMD GPU Operator 原生 device-plugin 关闭。 +3. 未设置 `amd.com/gpucores` 时，容器会获得每张已分配 GPU 的全部 CU。 +4. 当前镜像内置的 `libamvgpu.so` 分发方式是临时方案，后续将迁移到 `amd-hami-core` 正式产物。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/userguide/amd-device/examples/allocate-core-and-memory.md b/i18n/zh/docusaurus-plugin-content-docs/current/userguide/amd-device/examples/allocate-core-and-memory.md new file mode 100644 index 000000000..258dc2b3f --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/userguide/amd-device/examples/allocate-core-and-memory.md @@ -0,0 +1,32 @@ +--- +title: 分配设备核心和显存资源 +translated: true +--- + +若要分配部分 AMD GPU 资源，只需在请求 GPU 数量 `amd.com/gpu` 的同时设置 `amd.com/gpucores` 和 `amd.com/gpumem`。 + +以下示例请求一张 GPU，并分配 48 GiB 显存和 25% 计算单元： + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: amd-vgpu-example +spec: + schedulerName: hami-scheduler + restartPolicy: Never + containers: + - name: pytorch + image: rocm/pytorch:latest + command: ["bash", "-c"] + args: + - | + env | grep -E 'LD_AUDIT|HIP_DEVICE_MEMORY_LIMIT' + python3 -c 'import torch; print(torch.cuda.mem_get_info(0)); print(torch.cuda.get_device_name(0))' + sleep 300 + resources: + limits: + amd.com/gpu: 1 # 请求一个 AMD GPU + amd.com/gpumem: 49152 # 每个 GPU 包含 49152 MiB 设备显存 + amd.com/gpucores: 25 # 每个 GPU 分配 25% 的计算单元 +``` diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/userguide/device-supported.md b/i18n/zh/docusaurus-plugin-content-docs/current/userguide/device-supported.md index 16ad1262b..ae0c03759 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/userguide/device-supported.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/userguide/device-supported.md @@ -18,4 +18,5 @@ HAMi 支持的设备如下表所示： | GCU | 燧原科技（Enflame） | S60 | 是 | 是 | 否 | | XPU | 昆仑芯（Kunlunxin） | P800 | 是 | 是 | 否 | | GPU | 瀚博（Vastai） | VA16 | 是 | 是 | 否 | +| GPU | AMD | Instinct / ROCm | 是 | 是 | 否 | | DPU | 太初元碁（Teco） | 检查中 | 进行中 | 进行中 | 否 | diff --git a/sidebars.js b/sidebars.js index 828ff4e7c..3487c0e9f 100644 --- a/sidebars.js +++ b/sidebars.js @@ -193,6 +193,19 @@ module.exports = { label: "Share Enflame GCU devices", items: ["userguide/enflame-device/enable-enflame-gcu-sharing"], }, + { + type: "category", + label: "Share AMD GPU devices", + items: [ + "userguide/amd-device/enable-amd-gpu-sharing", + { + type: "category", + label: "Examples", + key: "amd-examples", + items: ["userguide/amd-device/examples/allocate-core-and-memory"], + }, + ], + }, { type: "category", label: "Managing AWS Neuron devices",