diff --git a/docs/userguide/cambricon-device/specify-device-memory-usage.md b/docs/userguide/cambricon-device/specify-device-memory-usage.md index faf7b982a..90d073da8 100644 --- a/docs/userguide/cambricon-device/specify-device-memory-usage.md +++ b/docs/userguide/cambricon-device/specify-device-memory-usage.md @@ -13,6 +13,12 @@ resources: cambricon.com/mlu.smlu.vmemory: "20" # Each MLU contains 20% device memory ``` +:::warning + +Set `cambricon.com/mlu.smlu.vmemory` as a **plain integer** (here, a percentage from 1 to 100), not a Kubernetes quantity. A suffixed value such as `16Gi` is parsed as its byte count (about 17 billion), which overflows HAMi's 32-bit memory field and is silently truncated, often to `0`. A zero memory request passes the scheduler's memory check unconditionally, so the pod can be placed on an MLU that is already full and then fails with out-of-memory errors at runtime. Use the integer form shown above (for example `20`). + +::: + :::note Depending on the parameters of cambricon-device-plugin, resource name can be `cambricon.com/mlu370.smlu.vmemory` or other types. diff --git a/docs/userguide/hygon-device/specify-device-memory-usage.md b/docs/userguide/hygon-device/specify-device-memory-usage.md index adc0cb643..312b52140 100644 --- a/docs/userguide/hygon-device/specify-device-memory-usage.md +++ b/docs/userguide/hygon-device/specify-device-memory-usage.md @@ -2,11 +2,17 @@ title: Allocate device memory --- -Allocate a percentage size of device memory by specifying resources such as `hygon.com/dcumem`. Optional, each unit of `hygon.com/dcumem` equals 1 MiB of device memory. +Allocate device memory by specifying the `hygon.com/dcumem` resource. This field is optional. Each unit of `hygon.com/dcumem` represents 1 MiB of device memory. ```yaml resources: limits: hygon.com/dcunum: 1 # requesting 1 DCU - hygon.com/dcumem: 2000 # Each DCU contains 2000M device memory + hygon.com/dcumem: 2000 # 2000 units x 1 MiB = 2000 MiB of device memory ``` + +:::warning + +Set `hygon.com/dcumem` as a **plain integer** (a count of MiB), not a Kubernetes quantity. A suffixed value such as `16Gi` is parsed as its byte count (about 17 billion), which overflows HAMi's 32-bit memory field and is silently truncated, often to `0`. A zero memory request passes the scheduler's memory check unconditionally, so the pod can be placed on a DCU that is already full and then fails with out-of-memory errors at runtime. Use the integer form shown above (for example `2000`). + +::: diff --git a/docs/userguide/iluvatar-device/enable-iluvatar-gpu-sharing.md b/docs/userguide/iluvatar-device/enable-iluvatar-gpu-sharing.md index 028f7d9a4..cdd3a914a 100644 --- a/docs/userguide/iluvatar-device/enable-iluvatar-gpu-sharing.md +++ b/docs/userguide/iluvatar-device/enable-iluvatar-gpu-sharing.md @@ -121,6 +121,12 @@ Each unit of `iluvatar.ai/.vMem` represents 256 MB of device memory. ::: +:::warning + +Set `iluvatar.ai/.vMem` as a **plain integer** (a count of 256 MB units), not a Kubernetes quantity. A suffixed value such as `16Gi` is parsed as its byte count (about 17 billion), which overflows HAMi's 32-bit memory field and is silently truncated, often to `0`. A zero memory request passes the scheduler's memory check unconditionally, so the pod can be placed on a GPU that is already full and then fails with out-of-memory errors at runtime. Use the integer form shown above (for example `64`). + +::: + ## Device UUID Selection You can specify which GPU devices to use or exclude using annotations: diff --git a/docs/userguide/nvidia-device/specify-device-memory-usage.md b/docs/userguide/nvidia-device/specify-device-memory-usage.md index 2d2476fac..85fc91b4a 100644 --- a/docs/userguide/nvidia-device/specify-device-memory-usage.md +++ b/docs/userguide/nvidia-device/specify-device-memory-usage.md @@ -12,6 +12,12 @@ resources: nvidia.com/gpumem: 3000 # Each GPU contains 3000 MiB device memory ``` +:::warning + +Set `nvidia.com/gpumem` as a **plain integer** (a count of MiB), not a Kubernetes quantity. A suffixed value such as `16Gi` is parsed as its byte count (about 17 billion), which overflows HAMi's 32-bit memory field and is silently truncated, often to `0`. A zero memory request passes the scheduler's memory check unconditionally, so the pod can be placed on a GPU that is already full and then fails with out-of-memory errors at runtime. Use the integer form shown above (for example `3000`). + +::: + Allocate a percentage of device memory by specifying resource `nvidia.com/gpumem-percentage`. Optional, each unit of `nvidia.com/gpumem-percentage` equals 1% of device memory. ```yaml diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/userguide/cambricon-device/specify-device-memory-usage.md b/i18n/zh/docusaurus-plugin-content-docs/current/userguide/cambricon-device/specify-device-memory-usage.md index 5ed3d5898..6677c1e13 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/userguide/cambricon-device/specify-device-memory-usage.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/userguide/cambricon-device/specify-device-memory-usage.md @@ -13,6 +13,12 @@ resources: cambricon.com/mlu.smlu.vmemory: "20" # 每个 MLU 包含 20% 的设备显存 ``` +:::warning + +请将 `cambricon.com/mlu.smlu.vmemory` 设置为**纯整数**(此处为 1 到 100 的百分比),不要使用 Kubernetes 数量单位。带单位的值(例如 `16Gi`)会被解析为字节数(约 170 亿),从而超出 HAMi 的 32 位显存字段范围并被静默截断,通常截断为 `0`。显存请求为 0 时会无条件通过调度器的显存检查,导致 Pod 被调度到显存已满的设备上,并在运行时发生显存不足(OOM)。请使用上面示例中的整数形式(例如 `20`)。 + +::: + :::note 根据 cambricon-device-plugin 的参数,资源名称可以是 `cambricon.com/mlu370.smlu.vmemory` 或其他类型 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/userguide/hygon-device/specify-device-memory-usage.md b/i18n/zh/docusaurus-plugin-content-docs/current/userguide/hygon-device/specify-device-memory-usage.md index 65c003a2e..0115b1a93 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/userguide/hygon-device/specify-device-memory-usage.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/userguide/hygon-device/specify-device-memory-usage.md @@ -4,11 +4,17 @@ sidebar_label: 指定显存 translated: true --- -通过指定诸如 `hygon.com/dcumem` 之类的资源来分配设备显存的百分比大小。可选项,每个 `hygon.com/dcumem` 单位等于 1M 设备显存。 +通过指定 `hygon.com/dcumem` 资源来分配设备显存。此字段为可选项,每个 `hygon.com/dcumem` 单位代表 1 MiB 设备显存。 ```yaml resources: limits: hygon.com/dcunum: 1 # 请求 1 个 DCU - hygon.com/dcumem: 2000 # 每个 DCU 包含 2000M 设备显存 + hygon.com/dcumem: 2000 # 2000 个单位 x 1 MiB = 2000 MiB 设备显存 ``` + +:::warning + +请将 `hygon.com/dcumem` 设置为**纯整数**(以 MiB 为单位计数),不要使用 Kubernetes 数量单位。带单位的值(例如 `16Gi`)会被解析为字节数(约 170 亿),从而超出 HAMi 的 32 位显存字段范围并被静默截断,通常截断为 `0`。显存请求为 0 时会无条件通过调度器的显存检查,导致 Pod 被调度到显存已满的设备上,并在运行时发生显存不足(OOM)。请使用上面示例中的整数形式(例如 `2000`)。 + +::: diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/userguide/iluvatar-device/enable-iluvatar-gpu-sharing.md b/i18n/zh/docusaurus-plugin-content-docs/current/userguide/iluvatar-device/enable-iluvatar-gpu-sharing.md index e3add706d..25bd7e707 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/userguide/iluvatar-device/enable-iluvatar-gpu-sharing.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/userguide/iluvatar-device/enable-iluvatar-gpu-sharing.md @@ -117,6 +117,12 @@ spec: ::: +:::warning + +请将 `iluvatar.ai/.vMem` 设置为**纯整数**(以 256MB 为单位计数),不要使用 Kubernetes 数量单位。带单位的值(例如 `16Gi`)会被解析为字节数(约 170 亿),从而超出 HAMi 的 32 位显存字段范围并被静默截断,通常截断为 `0`。显存请求为 0 时会无条件通过调度器的显存检查,导致 Pod 被调度到显存已满的设备上,并在运行时发生显存不足(OOM)。请使用上面示例中的整数形式(例如 `64`)。 + +::: + ## 设备 UUID 选择 你可以通过 annotations 指定使用或排除哪些 GPU 设备: diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/userguide/nvidia-device/specify-device-memory-usage.md b/i18n/zh/docusaurus-plugin-content-docs/current/userguide/nvidia-device/specify-device-memory-usage.md index afd0b6cfb..fcd0cc50e 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/userguide/nvidia-device/specify-device-memory-usage.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/userguide/nvidia-device/specify-device-memory-usage.md @@ -13,6 +13,12 @@ resources: nvidia.com/gpumem: 3000 # 每个 GPU 包含 3000 MiB 设备显存 ``` +:::warning + +请将 `nvidia.com/gpumem` 设置为**纯整数**(以 MiB 为单位计数),不要使用 Kubernetes 数量单位。带单位的值(例如 `16Gi`)会被解析为字节数(约 170 亿),从而超出 HAMi 的 32 位显存字段范围并被静默截断,通常截断为 `0`。显存请求为 0 时会无条件通过调度器的显存检查,导致 Pod 被调度到显存已满的设备上,并在运行时发生显存不足(OOM)。请使用上面示例中的整数形式(例如 `3000`)。 + +::: + 通过指定资源 `nvidia.com/gpumem-percentage` 来分配设备显存的百分比。可选项,每个 `nvidia.com/gpumem-percentage` 单位等于设备显存的 1% 百分比。 ```yaml diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/userguide/cambricon-device/specify-device-memory-usage.md b/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/userguide/cambricon-device/specify-device-memory-usage.md index 5ed3d5898..6677c1e13 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/userguide/cambricon-device/specify-device-memory-usage.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/userguide/cambricon-device/specify-device-memory-usage.md @@ -13,6 +13,12 @@ resources: cambricon.com/mlu.smlu.vmemory: "20" # 每个 MLU 包含 20% 的设备显存 ``` +:::warning + +请将 `cambricon.com/mlu.smlu.vmemory` 设置为**纯整数**(此处为 1 到 100 的百分比),不要使用 Kubernetes 数量单位。带单位的值(例如 `16Gi`)会被解析为字节数(约 170 亿),从而超出 HAMi 的 32 位显存字段范围并被静默截断,通常截断为 `0`。显存请求为 0 时会无条件通过调度器的显存检查,导致 Pod 被调度到显存已满的设备上,并在运行时发生显存不足(OOM)。请使用上面示例中的整数形式(例如 `20`)。 + +::: + :::note 根据 cambricon-device-plugin 的参数,资源名称可以是 `cambricon.com/mlu370.smlu.vmemory` 或其他类型 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/userguide/hygon-device/specify-device-memory-usage.md b/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/userguide/hygon-device/specify-device-memory-usage.md index 65c003a2e..0115b1a93 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/userguide/hygon-device/specify-device-memory-usage.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/userguide/hygon-device/specify-device-memory-usage.md @@ -4,11 +4,17 @@ sidebar_label: 指定显存 translated: true --- -通过指定诸如 `hygon.com/dcumem` 之类的资源来分配设备显存的百分比大小。可选项,每个 `hygon.com/dcumem` 单位等于 1M 设备显存。 +通过指定 `hygon.com/dcumem` 资源来分配设备显存。此字段为可选项,每个 `hygon.com/dcumem` 单位代表 1 MiB 设备显存。 ```yaml resources: limits: hygon.com/dcunum: 1 # 请求 1 个 DCU - hygon.com/dcumem: 2000 # 每个 DCU 包含 2000M 设备显存 + hygon.com/dcumem: 2000 # 2000 个单位 x 1 MiB = 2000 MiB 设备显存 ``` + +:::warning + +请将 `hygon.com/dcumem` 设置为**纯整数**(以 MiB 为单位计数),不要使用 Kubernetes 数量单位。带单位的值(例如 `16Gi`)会被解析为字节数(约 170 亿),从而超出 HAMi 的 32 位显存字段范围并被静默截断,通常截断为 `0`。显存请求为 0 时会无条件通过调度器的显存检查,导致 Pod 被调度到显存已满的设备上,并在运行时发生显存不足(OOM)。请使用上面示例中的整数形式(例如 `2000`)。 + +::: diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/userguide/iluvatar-device/enable-iluvatar-gpu-sharing.md b/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/userguide/iluvatar-device/enable-iluvatar-gpu-sharing.md index 70c30b282..25bd7e707 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/userguide/iluvatar-device/enable-iluvatar-gpu-sharing.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/userguide/iluvatar-device/enable-iluvatar-gpu-sharing.md @@ -30,7 +30,7 @@ title: 启用天数智芯 GPU 共享 - 安装 HAMi 时设置 `devices.iluvatar.enabled=true` ```bash -helm install hami hami-charts/hami --set scheduler.kubeScheduler.imageTag={your kubernetes version} --set devices.iluvatar.enabled=true -n kube-system +helm install hami hami-charts/hami --set scheduler.kubeScheduler.image.tag={your kubernetes version} --set devices.iluvatar.enabled=true -n kube-system ``` **说明:** 当前支持的 GPU 型号及资源名称定义如下(位于 [https://github.com/Project-HAMi/HAMi/blob/master/charts/hami/templates/scheduler/device-configmap.yaml](https://github.com/Project-HAMi/HAMi/blob/master/charts/hami/templates/scheduler/device-configmap.yaml)): @@ -117,6 +117,12 @@ spec: ::: +:::warning + +请将 `iluvatar.ai/.vMem` 设置为**纯整数**(以 256MB 为单位计数),不要使用 Kubernetes 数量单位。带单位的值(例如 `16Gi`)会被解析为字节数(约 170 亿),从而超出 HAMi 的 32 位显存字段范围并被静默截断,通常截断为 `0`。显存请求为 0 时会无条件通过调度器的显存检查,导致 Pod 被调度到显存已满的设备上,并在运行时发生显存不足(OOM)。请使用上面示例中的整数形式(例如 `64`)。 + +::: + ## 设备 UUID 选择 你可以通过 annotations 指定使用或排除哪些 GPU 设备: diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/userguide/nvidia-device/specify-device-memory-usage.md b/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/userguide/nvidia-device/specify-device-memory-usage.md index 1cc258ca0..fcd0cc50e 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/userguide/nvidia-device/specify-device-memory-usage.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/userguide/nvidia-device/specify-device-memory-usage.md @@ -10,9 +10,15 @@ translated: true resources: limits: nvidia.com/gpu: 1 # 请求 1 个 GPU - nvidia.com/gpumem: 3000 # 每个 GPU 包含 3000m 设备显存 + nvidia.com/gpumem: 3000 # 每个 GPU 包含 3000 MiB 设备显存 ``` +:::warning + +请将 `nvidia.com/gpumem` 设置为**纯整数**(以 MiB 为单位计数),不要使用 Kubernetes 数量单位。带单位的值(例如 `16Gi`)会被解析为字节数(约 170 亿),从而超出 HAMi 的 32 位显存字段范围并被静默截断,通常截断为 `0`。显存请求为 0 时会无条件通过调度器的显存检查,导致 Pod 被调度到显存已满的设备上,并在运行时发生显存不足(OOM)。请使用上面示例中的整数形式(例如 `3000`)。 + +::: + 通过指定资源 `nvidia.com/gpumem-percentage` 来分配设备显存的百分比。可选项,每个 `nvidia.com/gpumem-percentage` 单位等于设备显存的 1% 百分比。 ```yaml diff --git a/versioned_docs/version-v2.9.0/userguide/cambricon-device/specify-device-memory-usage.md b/versioned_docs/version-v2.9.0/userguide/cambricon-device/specify-device-memory-usage.md index faf7b982a..90d073da8 100644 --- a/versioned_docs/version-v2.9.0/userguide/cambricon-device/specify-device-memory-usage.md +++ b/versioned_docs/version-v2.9.0/userguide/cambricon-device/specify-device-memory-usage.md @@ -13,6 +13,12 @@ resources: cambricon.com/mlu.smlu.vmemory: "20" # Each MLU contains 20% device memory ``` +:::warning + +Set `cambricon.com/mlu.smlu.vmemory` as a **plain integer** (here, a percentage from 1 to 100), not a Kubernetes quantity. A suffixed value such as `16Gi` is parsed as its byte count (about 17 billion), which overflows HAMi's 32-bit memory field and is silently truncated, often to `0`. A zero memory request passes the scheduler's memory check unconditionally, so the pod can be placed on an MLU that is already full and then fails with out-of-memory errors at runtime. Use the integer form shown above (for example `20`). + +::: + :::note Depending on the parameters of cambricon-device-plugin, resource name can be `cambricon.com/mlu370.smlu.vmemory` or other types. diff --git a/versioned_docs/version-v2.9.0/userguide/hygon-device/specify-device-memory-usage.md b/versioned_docs/version-v2.9.0/userguide/hygon-device/specify-device-memory-usage.md index adc0cb643..312b52140 100644 --- a/versioned_docs/version-v2.9.0/userguide/hygon-device/specify-device-memory-usage.md +++ b/versioned_docs/version-v2.9.0/userguide/hygon-device/specify-device-memory-usage.md @@ -2,11 +2,17 @@ title: Allocate device memory --- -Allocate a percentage size of device memory by specifying resources such as `hygon.com/dcumem`. Optional, each unit of `hygon.com/dcumem` equals 1 MiB of device memory. +Allocate device memory by specifying the `hygon.com/dcumem` resource. This field is optional. Each unit of `hygon.com/dcumem` represents 1 MiB of device memory. ```yaml resources: limits: hygon.com/dcunum: 1 # requesting 1 DCU - hygon.com/dcumem: 2000 # Each DCU contains 2000M device memory + hygon.com/dcumem: 2000 # 2000 units x 1 MiB = 2000 MiB of device memory ``` + +:::warning + +Set `hygon.com/dcumem` as a **plain integer** (a count of MiB), not a Kubernetes quantity. A suffixed value such as `16Gi` is parsed as its byte count (about 17 billion), which overflows HAMi's 32-bit memory field and is silently truncated, often to `0`. A zero memory request passes the scheduler's memory check unconditionally, so the pod can be placed on a DCU that is already full and then fails with out-of-memory errors at runtime. Use the integer form shown above (for example `2000`). + +::: diff --git a/versioned_docs/version-v2.9.0/userguide/iluvatar-device/enable-iluvatar-gpu-sharing.md b/versioned_docs/version-v2.9.0/userguide/iluvatar-device/enable-iluvatar-gpu-sharing.md index cd58909a0..cdd3a914a 100644 --- a/versioned_docs/version-v2.9.0/userguide/iluvatar-device/enable-iluvatar-gpu-sharing.md +++ b/versioned_docs/version-v2.9.0/userguide/iluvatar-device/enable-iluvatar-gpu-sharing.md @@ -34,7 +34,7 @@ Install only gpu-manager. Do not install the gpu-admission package. - Set `devices.iluvatar.enabled=true` when installing HAMi ```bash -helm install hami hami-charts/hami --set scheduler.kubeScheduler.imageTag={your kubernetes version} --set devices.iluvatar.enabled=true -n kube-system +helm install hami hami-charts/hami --set scheduler.kubeScheduler.image.tag={your kubernetes version} --set devices.iluvatar.enabled=true -n kube-system ``` The currently supported GPU models and resource names are defined in [device-configmap.yaml](https://github.com/Project-HAMi/HAMi/blob/master/charts/hami/templates/scheduler/device-configmap.yaml): @@ -121,6 +121,12 @@ Each unit of `iluvatar.ai/.vMem` represents 256 MB of device memory. ::: +:::warning + +Set `iluvatar.ai/.vMem` as a **plain integer** (a count of 256 MB units), not a Kubernetes quantity. A suffixed value such as `16Gi` is parsed as its byte count (about 17 billion), which overflows HAMi's 32-bit memory field and is silently truncated, often to `0`. A zero memory request passes the scheduler's memory check unconditionally, so the pod can be placed on a GPU that is already full and then fails with out-of-memory errors at runtime. Use the integer form shown above (for example `64`). + +::: + ## Device UUID Selection You can specify which GPU devices to use or exclude using annotations: diff --git a/versioned_docs/version-v2.9.0/userguide/nvidia-device/specify-device-memory-usage.md b/versioned_docs/version-v2.9.0/userguide/nvidia-device/specify-device-memory-usage.md index 1621c8167..85fc91b4a 100644 --- a/versioned_docs/version-v2.9.0/userguide/nvidia-device/specify-device-memory-usage.md +++ b/versioned_docs/version-v2.9.0/userguide/nvidia-device/specify-device-memory-usage.md @@ -9,9 +9,15 @@ Allocate a certain size of device memory by specifying resources such as `nvidia resources: limits: nvidia.com/gpu: 1 # requesting 1 GPU - nvidia.com/gpumem: 3000 # Each GPU contains 3000m device memory + nvidia.com/gpumem: 3000 # Each GPU contains 3000 MiB device memory ``` +:::warning + +Set `nvidia.com/gpumem` as a **plain integer** (a count of MiB), not a Kubernetes quantity. A suffixed value such as `16Gi` is parsed as its byte count (about 17 billion), which overflows HAMi's 32-bit memory field and is silently truncated, often to `0`. A zero memory request passes the scheduler's memory check unconditionally, so the pod can be placed on a GPU that is already full and then fails with out-of-memory errors at runtime. Use the integer form shown above (for example `3000`). + +::: + Allocate a percentage of device memory by specifying resource `nvidia.com/gpumem-percentage`. Optional, each unit of `nvidia.com/gpumem-percentage` equals 1% of device memory. ```yaml @@ -23,6 +29,6 @@ resources: :::note -`nvidia.com/gpumem` and `nvidia.com/gpumem-percentage` can't be assigned together +`nvidia.com/gpumem` and `nvidia.com/gpumem-percentage` cannot be assigned together :::