From af496ae76a2b3f5fed5f09ae0dc176baff9e982a Mon Sep 17 00:00:00 2001 From: lin121291 <4jp33f9e@gmail.com> Date: Fri, 7 Aug 2026 13:08:28 +0800 Subject: [PATCH] docs(userguide): add Coscheduling guide and node lock retry flag Add a user guide for running the scheduler-plugins Coscheduling plugin with HAMi, and document --node-lock-retry-timeout in the global config page. Both pages include a Chinese translation. Signed-off-by: lin121291 <4jp33f9e@gmail.com> --- docs/userguide/configure.md | 14 + .../coscheduling/how-to-use-coscheduling.md | 248 ++++++++++++++++++ .../current.json | 4 + .../current/userguide/configure.md | 14 + .../coscheduling/how-to-use-coscheduling.md | 248 ++++++++++++++++++ sidebars.js | 5 + 6 files changed, 533 insertions(+) create mode 100644 docs/userguide/coscheduling/how-to-use-coscheduling.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/userguide/coscheduling/how-to-use-coscheduling.md diff --git a/docs/userguide/configure.md b/docs/userguide/configure.md index e3ec92e6f..56d09aa19 100644 --- a/docs/userguide/configure.md +++ b/docs/userguide/configure.md @@ -71,6 +71,20 @@ helm install hami hami-charts/hami --set devicePlugin.deviceMemoryScaling=5 -n k | `scheduler.defaultSchedulerPolicy.nodeSchedulerPolicy` | String | GPU node scheduling policy: `"binpack"` allocates jobs to the same GPU node as much as possible. `"spread"` allocates jobs to different GPU nodes as much as possible. | `"binpack"` | | `scheduler.defaultSchedulerPolicy.gpuSchedulerPolicy` | String | GPU scheduling policy: `"binpack"` allocates jobs to the same GPU as much as possible. `"spread"` allocates jobs to different GPUs as much as possible. `"mutex"` allocates jobs only to GPUs with no other workloads. | `"spread"` | +## Scheduler Configs: extender arguments + +The scheduler extender reads flags from `scheduler.extender.extraArgs`. The chart ships `["--debug", "-v=4"]`, and setting the value replaces the whole list, so repeat the entries you want to keep: + +```bash +helm upgrade hami hami-charts/hami -n kube-system --reuse-values \ + --set-json 'scheduler.extender.extraArgs=["--debug","-v=4","--node-lock-retry-timeout=28s"]' +``` + +| Argument | Type | Description | Default | +| --- | --- | --- | --- | +| `--node-lock-retry-timeout` | Duration | How long `Bind` retries the node lock when it is held by another member of the same PodGroup. Applies only to pods carrying the `scheduling.x-k8s.io/pod-group` label; other pods fail fast as before. `0` disables the retry. Keep this below the extender `httpTimeout` in the KubeSchedulerConfiguration, which the chart sets to `30s`. See [How to use Coscheduling with HAMi](coscheduling/how-to-use-coscheduling.md). | `28s` | +| `--node-lock-timeout` | Duration | How long a node lock stays valid before another pod may take it over. Applies to every pod, not only PodGroup members. | `5m` | + ## Pod Configs: Annotations | Argument | Type | Description | Example | diff --git a/docs/userguide/coscheduling/how-to-use-coscheduling.md b/docs/userguide/coscheduling/how-to-use-coscheduling.md new file mode 100644 index 000000000..e23a88f08 --- /dev/null +++ b/docs/userguide/coscheduling/how-to-use-coscheduling.md @@ -0,0 +1,248 @@ +--- +title: How to use Coscheduling with HAMi +sidebar_label: How to use Coscheduling +--- + +[Coscheduling](https://github.com/kubernetes-sigs/scheduler-plugins/tree/master/pkg/coscheduling) is a scheduler plugin from [kubernetes-sigs/scheduler-plugins](https://github.com/kubernetes-sigs/scheduler-plugins) that provides gang scheduling. A group of Pods is admitted only when at least `minMember` of them can be placed at once, which is what distributed training needs: a job either gets all of its GPUs or none of them. + +This guide covers running Coscheduling inside the HAMi scheduler and tuning the node lock behavior that gang binding exercises. + +## How it works + +HAMi and Coscheduling operate at two different points of the scheduling cycle. + +Coscheduling works in the **Permit** phase. Each member Pod that passes filtering is parked in a waiting queue. Once `minMember` members are waiting, all of them are released into the bind phase at the same time. + +HAMi works in the **Bind** phase, through the extender. Before binding, the extender takes a per-node lock by writing the `hami.io/mutex.lock` annotation onto the Node object: + +```text +hami.io/mutex.lock: 2026-06-14T15:05:03Z,default,gang-pod-1 +``` + +The lock serializes device allocation on that node. Without it, two Pods bound at the same moment would both read the same device usage snapshot and could be handed overlapping slices of one GPU. The lock is released by the device plugin once `Allocate()` finishes and the Pod annotations are updated, which takes about 20 ms on a real GPU. A lock that is never released expires after the node lock timeout (5 minutes by default). + +These two mechanisms meet at gang release. Coscheduling releases every member in the same millisecond, so if several members target the same node, they contend on a lock that is held for only a few tens of milliseconds. The Pod that loses fails its bind, returns to Pending, and comes back through the default kube-scheduler backoff, which is measured in seconds. A five-member gang converges, but it takes several backoff rounds to do it. + +To close that gap, the extender retries the node lock for Pods that carry the Coscheduling group label: + +- A Pod with a non-empty `scheduling.x-k8s.io/pod-group` label polls the lock every 100 ms until `--node-lock-retry-timeout` expires. +- Any partially acquired lock is released before each retry, so a Pod requesting devices from more than one vendor cannot leave a stale lock behind. +- Errors that are not lock contention are returned immediately and are not retried. +- Pods without the label keep the original fail-fast behavior. + +:::note + +`--node-lock-retry-timeout` is available in builds newer than v2.9.0. + +::: + +## Prerequisites + +- A Kubernetes cluster with GPU nodes and HAMi installed. +- A [scheduler-plugins release](https://github.com/kubernetes-sigs/scheduler-plugins/releases) built against your Kubernetes minor version. The examples below use v0.34.7 on Kubernetes v1.35. +- Helm 3. +- `kubectl` with cluster-admin rights. + +## 1. Install HAMi with the scheduler-plugins kube-scheduler + +The HAMi scheduler Pod runs two containers: an upstream `kube-scheduler` and the HAMi `vgpu-scheduler-extender`. Coscheduling is compiled into the scheduler-plugins build of kube-scheduler, so point the chart at that image instead of the stock one: + +```bash +helm repo add hami-charts https://project-hami.github.io/HAMi/ +helm repo update + +helm install hami hami-charts/hami \ + --namespace hami-system --create-namespace \ + --set scheduler.kubeScheduler.image.registry=registry.k8s.io \ + --set scheduler.kubeScheduler.image.repository=scheduler-plugins/kube-scheduler \ + --set scheduler.kubeScheduler.image.tag=v0.34.7 \ + --wait --timeout 10m +``` + +On an existing installation, run the same three `--set` flags through `helm upgrade --reuse-values`. + +## 2. Install the PodGroup CRD + +Coscheduling reads `PodGroup` resources. Install the CRD from the same scheduler-plugins release: + +```bash +kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/scheduler-plugins/v0.34.7/config/crd/bases/scheduling.x-k8s.io_podgroups.yaml +``` + +## 3. Enable Coscheduling in the scheduler config + +The chart renders the KubeSchedulerConfiguration into the `hami-scheduler` ConfigMap. Add the plugin to the profile: + +```bash +kubectl edit configmap hami-scheduler -n hami-system +``` + +The `profiles` entry must look like this: + +```yaml +profiles: + - schedulerName: hami-scheduler + plugins: + multiPoint: + enabled: + - name: Coscheduling + queueSort: + disabled: + - name: PrioritySort + pluginConfig: + - name: Coscheduling + args: + permitWaitingTimeSeconds: 10 +``` + +:::warning + +`PrioritySort` must be disabled. Coscheduling registers its own queue sort plugin, and kube-scheduler refuses to start with two of them: + +```text +only one queue sort plugin required for profile with scheduler name "hami-scheduler", but got 2 +``` + +::: + +The ConfigMap is owned by the chart, so `helm upgrade` overwrites this edit. Re-apply it after every upgrade, or manage the ConfigMap outside the chart. + +Restart the scheduler to pick up the change: + +```bash +kubectl rollout restart deploy/hami-scheduler -n hami-system +kubectl rollout status deploy/hami-scheduler -n hami-system +``` + +## 4. Grant access to PodGroups + +The scheduler ServiceAccount installed by the chart cannot read `PodGroup` resources. Add the permission: + +```yaml +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: hami-podgroup-reader +rules: + - apiGroups: ["scheduling.x-k8s.io"] + resources: ["podgroups"] + verbs: ["get", "list", "watch", "create", "update", "patch"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: hami-podgroup-reader +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: hami-podgroup-reader +subjects: + - kind: ServiceAccount + name: hami-scheduler + namespace: hami-system +``` + +Without it, `PreFilter` fails for every member and the whole group stays Pending. + +## 5. Submit a gang + +Create a `PodGroup` and label every member with its name. Each member requests vGPU resources as usual: + +```yaml +apiVersion: scheduling.x-k8s.io/v1alpha1 +kind: PodGroup +metadata: + name: gang-training +spec: + minMember: 4 + scheduleTimeoutSeconds: 60 +--- +apiVersion: v1 +kind: Pod +metadata: + name: gang-worker-1 + labels: + scheduling.x-k8s.io/pod-group: gang-training +spec: + schedulerName: hami-scheduler + containers: + - name: worker + image: ubuntu:22.04 + command: ["sleep", "3600"] + resources: + limits: + nvidia.com/gpu: "1" + nvidia.com/gpumem: "3000" + nvidia.com/gpucores: "30" +``` + +The manifest above defines one member. Create `minMember` Pods from the same template with distinct names, otherwise the group never reaches its quorum and every member stays Pending. + +:::warning + +`scheduling.x-k8s.io/pod-group` must be under `metadata.labels`. Placing it under `metadata.annotations` bypasses gang logic without any error: the Pods schedule one by one regardless of `minMember`. + +::: + +Verify that the group was admitted together: + +```bash +kubectl get pods -l scheduling.x-k8s.io/pod-group=gang-training -o wide +``` + +When fewer than `minMember` members exist, `PreFilter` rejects the group before the HAMi extender is reached: + +```text +pre-filter pod gang-worker-1 cannot find enough sibling pods, +current pods number: 3, minMember of group: 5 +``` + +## Tune the node lock + +Two independent timeouts control node lock behavior. + +| Flag | Default | Description | +| --- | --- | --- | +| `--node-lock-retry-timeout` | `28s` | How long `Bind` retries the node lock for a Pod labeled with `scheduling.x-k8s.io/pod-group`. `0` disables retry and restores fail-fast behavior. Polling interval is 100 ms. | +| `--node-lock-timeout` | `5m` | How long a lock stays valid before another Pod may take it over. Applies to every Pod, not only gang members. | + +Set the retry timeout through the chart. `scheduler.extender.extraArgs` replaces the default list, so keep the existing entries: + +```bash +helm upgrade hami hami-charts/hami -n hami-system --reuse-values \ + --set-json 'scheduler.extender.extraArgs=["--debug","-v=4","--node-lock-retry-timeout=28s"]' +``` + +:::warning + +Keep `--node-lock-retry-timeout` below the extender `httpTimeout` in the KubeSchedulerConfiguration, which the chart sets to `30s`. If the retry outlives the HTTP call, kube-scheduler abandons the bind request while the extender is still waiting for the lock, and the Pod is retried from the top. + +::: + +The default of `28s` leaves 2 seconds of headroom under that `30s` timeout. + +## Troubleshooting + +**Pods stay Pending with `BindingFailed: node has been locked within 5m0s`** + +The retry is not active for these Pods. Check that the `scheduling.x-k8s.io/pod-group` label is on the Pod (not the PodGroup only, and not in annotations), and that `--node-lock-retry-timeout` is not set to `0`. + +**The scheduler container crash-loops on startup** + +Look for `only one queue sort plugin required` in the kube-scheduler logs. `PrioritySort` is still enabled alongside Coscheduling. See [step 3](#3-enable-coscheduling-in-the-scheduler-config). + +**All members of a group stay Pending and no node is ever selected** + +Either fewer than `minMember` members were created, or the scheduler cannot read `PodGroup` resources. Check the kube-scheduler logs for `PreFilter failed` and confirm the RBAC from [step 4](#4-grant-access-to-podgroups) is applied. + +**Containers fail with `libdl.so.2: cannot open shared object file`** + +HAMi injects `LD_PRELOAD` pointing at a glibc build of `libvgpu.so`. Images based on musl, such as `busybox` and `alpine`, cannot load it. Use a glibc image for GPU workloads. + +## Related links + +- [Coscheduling plugin](https://github.com/kubernetes-sigs/scheduler-plugins/tree/master/pkg/coscheduling) +- [scheduler-plugins releases](https://github.com/kubernetes-sigs/scheduler-plugins/releases) +- [Global Config](../configure.md) +- [Using HAMi with Kueue](../kueue/how-to-use-kueue.md) +- [Using HAMi with KAI Scheduler](../kai-scheduler/how-to-use-kai-scheduler.md) diff --git a/i18n/zh/docusaurus-plugin-content-docs/current.json b/i18n/zh/docusaurus-plugin-content-docs/current.json index 85a18a02b..ce04c7c97 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current.json +++ b/i18n/zh/docusaurus-plugin-content-docs/current.json @@ -103,6 +103,10 @@ "message": "在 Kueue 中使用 HAMi", "description": "The label for category 'Using HAMi with Kueue' in sidebar 'docs'" }, + "sidebar.docs.category.Using HAMi with Coscheduling": { + "message": "在 HAMi 中使用 Coscheduling", + "description": "The label for category 'Using HAMi with Coscheduling' in sidebar 'docs'" + }, "sidebar.docs.category.nvidia-examples": { "message": "示例", "description": "The label for category 'Examples' in sidebar 'docs'" diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/userguide/configure.md b/i18n/zh/docusaurus-plugin-content-docs/current/userguide/configure.md index 3c8816922..daadef300 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/userguide/configure.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/userguide/configure.md @@ -73,6 +73,20 @@ helm install hami hami-charts/hami --set devicePlugin.deviceMemoryScaling=5 -n k | `scheduler.defaultSchedulerPolicy.nodeSchedulerPolicy` | 字符串 | GPU 节点调度策略：`"binpack"` 表示尽可能将任务分配到同一个 GPU 节点；`"spread"` 表示尽可能将任务分配到不同的 GPU 节点。 | `"binpack"` | | `scheduler.defaultSchedulerPolicy.gpuSchedulerPolicy` | 字符串 | GPU 调度策略：`"binpack"` 表示尽可能将任务分配到同一个 GPU；`"spread"` 表示尽可能将任务分配到不同的 GPU。 | `"spread"` | +## 调度器配置：扩展器参数 + +调度器扩展器从 `scheduler.extender.extraArgs` 读取命令行参数。chart 默认值为 `["--debug", "-v=4"]`，设置该值会替换整个列表，因此需要保留想要沿用的条目： + +```bash +helm upgrade hami hami-charts/hami -n kube-system --reuse-values \ + --set-json 'scheduler.extender.extraArgs=["--debug","-v=4","--node-lock-retry-timeout=28s"]' +``` + +| 参数 | 类型 | 描述 | 默认值 | +| --- | --- | --- | --- | +| `--node-lock-retry-timeout` | 时长 | 当节点锁被同一个 PodGroup 的其他成员持有时，`Bind` 重试该锁的时长。仅对带有 `scheduling.x-k8s.io/pod-group` 标签的 Pod 生效，其他 Pod 保持原有的快速失败行为。设为 `0` 关闭重试。该值需小于 KubeSchedulerConfiguration 中扩展器的 `httpTimeout`（chart 设为 `30s`）。参见[如何在 HAMi 中使用 Coscheduling](coscheduling/how-to-use-coscheduling.md)。 | `28s` | +| `--node-lock-timeout` | 时长 | 一把节点锁在被其他 Pod 接管前的有效期。对所有 Pod 生效，不限于 PodGroup 成员。 | `5m` | + ## Pod 配置：注解 | 参数 | 类型 | 描述 | 示例 | diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/userguide/coscheduling/how-to-use-coscheduling.md b/i18n/zh/docusaurus-plugin-content-docs/current/userguide/coscheduling/how-to-use-coscheduling.md new file mode 100644 index 000000000..d3f144bfc --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/userguide/coscheduling/how-to-use-coscheduling.md @@ -0,0 +1,248 @@ +--- +title: 如何在 HAMi 中使用 Coscheduling +sidebar_label: 如何使用 Coscheduling +--- + +[Coscheduling](https://github.com/kubernetes-sigs/scheduler-plugins/tree/master/pkg/coscheduling) 是 [kubernetes-sigs/scheduler-plugins](https://github.com/kubernetes-sigs/scheduler-plugins) 提供的调度插件，用于实现 gang scheduling。只有当一组 Pod 中至少 `minMember` 个可以同时被放置时，这组 Pod 才会被准入。这正是分布式训练所需要的语义：一个作业要么拿到全部 GPU，要么一个都不拿。 + +本文介绍如何在 HAMi 调度器中运行 Coscheduling，以及如何调整 gang 绑定阶段会触发的节点锁行为。 + +## 工作原理 + +HAMi 与 Coscheduling 作用在调度周期的两个不同阶段。 + +Coscheduling 作用于 **Permit** 阶段。每个通过过滤的成员 Pod 会被放入等待队列，当等待中的成员达到 `minMember` 时，它们会被同时释放进入绑定阶段。 + +HAMi 通过扩展器作用于 **Bind** 阶段。绑定前，扩展器会在 Node 对象上写入 `hami.io/mutex.lock` 注解，从而获取该节点的锁： + +```text +hami.io/mutex.lock: 2026-06-14T15:05:03Z,default,gang-pod-1 +``` + +该锁用于串行化节点上的设备分配。如果没有它，同一时刻绑定的两个 Pod 会读到同一份设备使用快照，可能被分配到同一张 GPU 的重叠切片。锁由设备插件在 `Allocate()` 完成并更新 Pod 注解后释放，在真实 GPU 上耗时约 20 毫秒。若锁始终未被释放，则在节点锁超时（默认 5 分钟）后过期。 + +这两个机制在 gang 释放时相遇。Coscheduling 会在同一毫秒内释放全部成员，因此当多个成员落在同一节点时，它们会争抢一把只持有几十毫秒的锁。抢锁失败的 Pod 绑定失败并回到 Pending，随后经由 kube-scheduler 的退避重试回来，而退避是以秒为单位的。5 个成员的 gang 最终会收敛，但需要经过多轮退避。 + +为了消除这段空转，扩展器会为带有 Coscheduling 分组标签的 Pod 重试节点锁： + +- 带有非空 `scheduling.x-k8s.io/pod-group` 标签的 Pod 每 100 毫秒轮询一次锁，直到 `--node-lock-retry-timeout` 超时。 +- 每次重试前会释放已部分获取的锁，因此同时申请多个厂商设备的 Pod 不会残留过期锁。 +- 非锁争抢类错误会立即返回，不做重试。 +- 未带该标签的 Pod 保持原有的快速失败行为。 + +:::note + +`--node-lock-retry-timeout` 在高于 v2.9.0 的版本中可用。 + +::: + +## 前置条件 + +- 一个已安装 HAMi 且具备 GPU 节点的 Kubernetes 集群。 +- 与集群 Kubernetes 次版本匹配的 [scheduler-plugins 版本](https://github.com/kubernetes-sigs/scheduler-plugins/releases)。下文示例使用 Kubernetes v1.35 与 v0.34.7。 +- Helm 3。 +- 具备 cluster-admin 权限的 `kubectl`。 + +## 1. 使用 scheduler-plugins 的 kube-scheduler 安装 HAMi + +HAMi 调度器 Pod 中运行两个容器：上游 `kube-scheduler` 和 HAMi `vgpu-scheduler-extender`。Coscheduling 编译在 scheduler-plugins 版本的 kube-scheduler 中，因此需要把 chart 指向该镜像： + +```bash +helm repo add hami-charts https://project-hami.github.io/HAMi/ +helm repo update + +helm install hami hami-charts/hami \ + --namespace hami-system --create-namespace \ + --set scheduler.kubeScheduler.image.registry=registry.k8s.io \ + --set scheduler.kubeScheduler.image.repository=scheduler-plugins/kube-scheduler \ + --set scheduler.kubeScheduler.image.tag=v0.34.7 \ + --wait --timeout 10m +``` + +对已有安装，使用 `helm upgrade --reuse-values` 传入相同的三个 `--set` 参数。 + +## 2. 安装 PodGroup CRD + +Coscheduling 读取 `PodGroup` 资源。从同一个 scheduler-plugins 版本安装 CRD： + +```bash +kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/scheduler-plugins/v0.34.7/config/crd/bases/scheduling.x-k8s.io_podgroups.yaml +``` + +## 3. 在调度器配置中启用 Coscheduling + +chart 会把 KubeSchedulerConfiguration 渲染到 `hami-scheduler` ConfigMap 中。将插件加入 profile： + +```bash +kubectl edit configmap hami-scheduler -n hami-system +``` + +`profiles` 条目应如下所示： + +```yaml +profiles: + - schedulerName: hami-scheduler + plugins: + multiPoint: + enabled: + - name: Coscheduling + queueSort: + disabled: + - name: PrioritySort + pluginConfig: + - name: Coscheduling + args: + permitWaitingTimeSeconds: 10 +``` + +:::warning + +必须禁用 `PrioritySort`。Coscheduling 会注册自己的队列排序插件，而 kube-scheduler 在存在两个排序插件时拒绝启动： + +```text +only one queue sort plugin required for profile with scheduler name "hami-scheduler", but got 2 +``` + +::: + +该 ConfigMap 由 chart 管理，`helm upgrade` 会覆盖这次修改。每次升级后需要重新应用，或者把该 ConfigMap 移出 chart 自行管理。 + +重启调度器使配置生效： + +```bash +kubectl rollout restart deploy/hami-scheduler -n hami-system +kubectl rollout status deploy/hami-scheduler -n hami-system +``` + +## 4. 授予访问 PodGroup 的权限 + +chart 安装的调度器 ServiceAccount 无法读取 `PodGroup` 资源。补充权限： + +```yaml +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: hami-podgroup-reader +rules: + - apiGroups: ["scheduling.x-k8s.io"] + resources: ["podgroups"] + verbs: ["get", "list", "watch", "create", "update", "patch"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: hami-podgroup-reader +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: hami-podgroup-reader +subjects: + - kind: ServiceAccount + name: hami-scheduler + namespace: hami-system +``` + +缺少该权限时，每个成员的 `PreFilter` 都会失败，整组 Pod 会一直处于 Pending。 + +## 5. 提交一个 gang + +创建 `PodGroup`，并给每个成员打上分组名标签。成员照常申请 vGPU 资源： + +```yaml +apiVersion: scheduling.x-k8s.io/v1alpha1 +kind: PodGroup +metadata: + name: gang-training +spec: + minMember: 4 + scheduleTimeoutSeconds: 60 +--- +apiVersion: v1 +kind: Pod +metadata: + name: gang-worker-1 + labels: + scheduling.x-k8s.io/pod-group: gang-training +spec: + schedulerName: hami-scheduler + containers: + - name: worker + image: ubuntu:22.04 + command: ["sleep", "3600"] + resources: + limits: + nvidia.com/gpu: "1" + nvidia.com/gpumem: "3000" + nvidia.com/gpucores: "30" +``` + +上面的清单只定义了一个成员。请用同一份模板创建 `minMember` 个名称不同的 Pod，否则该组永远达不到法定成员数，所有成员都会停留在 Pending。 + +:::warning + +`scheduling.x-k8s.io/pod-group` 必须放在 `metadata.labels` 下。放在 `metadata.annotations` 下会静默绕过全部 gang 逻辑，不产生任何报错：Pod 会无视 `minMember` 逐个被调度。 + +::: + +确认整组被同时准入： + +```bash +kubectl get pods -l scheduling.x-k8s.io/pod-group=gang-training -o wide +``` + +当成员数量少于 `minMember` 时，`PreFilter` 会在请求到达 HAMi 扩展器之前拒绝该组： + +```text +pre-filter pod gang-worker-1 cannot find enough sibling pods, +current pods number: 3, minMember of group: 5 +``` + +## 调整节点锁 + +有两个相互独立的超时控制节点锁行为。 + +| 参数 | 默认值 | 说明 | +| --- | --- | --- | +| `--node-lock-retry-timeout` | `28s` | 对带有 `scheduling.x-k8s.io/pod-group` 标签的 Pod，`Bind` 重试节点锁的时长。设为 `0` 关闭重试，恢复快速失败行为。轮询间隔为 100 毫秒。 | +| `--node-lock-timeout` | `5m` | 一把锁在被其他 Pod 接管前的有效期。对所有 Pod 生效，不限于 gang 成员。 | + +通过 chart 设置重试超时。`scheduler.extender.extraArgs` 会替换整个默认列表，因此需要保留已有条目： + +```bash +helm upgrade hami hami-charts/hami -n hami-system --reuse-values \ + --set-json 'scheduler.extender.extraArgs=["--debug","-v=4","--node-lock-retry-timeout=28s"]' +``` + +:::warning + +`--node-lock-retry-timeout` 必须小于 KubeSchedulerConfiguration 中扩展器的 `httpTimeout`，chart 将其设为 `30s`。如果重试时间超过该 HTTP 调用时长，kube-scheduler 会在扩展器仍在等锁时放弃这次绑定请求，Pod 将从头重新调度。 + +::: + +默认值 `28s` 在 `30s` 超时之下预留了 2 秒余量。 + +## 故障排查 + +**Pod 持续 Pending，事件为 `BindingFailed: node has been locked within 5m0s`** + +这些 Pod 上的重试没有生效。确认 `scheduling.x-k8s.io/pod-group` 标签打在 Pod 上（不能只打在 PodGroup 上，也不能放在 annotations 里），并确认 `--node-lock-retry-timeout` 没有被设为 `0`。 + +**调度器容器启动后反复崩溃** + +在 kube-scheduler 日志中查找 `only one queue sort plugin required`。这表示 `PrioritySort` 仍与 Coscheduling 同时启用，见[步骤 3](#3-在调度器配置中启用-coscheduling)。 + +**整组 Pod 一直 Pending 且从未选中节点** + +要么创建的成员数少于 `minMember`，要么调度器读不到 `PodGroup` 资源。检查 kube-scheduler 日志中的 `PreFilter failed`，并确认[步骤 4](#4-授予访问-podgroup-的权限)的 RBAC 已应用。 + +**容器报错 `libdl.so.2: cannot open shared object file`** + +HAMi 注入的 `LD_PRELOAD` 指向基于 glibc 构建的 `libvgpu.so`。基于 musl 的镜像（如 `busybox`、`alpine`）无法加载它。GPU 工作负载请使用 glibc 镜像。 + +## 相关链接 + +- [Coscheduling 插件](https://github.com/kubernetes-sigs/scheduler-plugins/tree/master/pkg/coscheduling) +- [scheduler-plugins 版本列表](https://github.com/kubernetes-sigs/scheduler-plugins/releases) +- [全局配置](../configure.md) +- [在 Kueue 中使用 HAMi](../kueue/how-to-use-kueue.md) +- [如何在 KAI Scheduler 中使用 HAMi](../kai-scheduler/how-to-use-kai-scheduler.md) diff --git a/sidebars.js b/sidebars.js index 3487c0e9f..c040510ec 100644 --- a/sidebars.js +++ b/sidebars.js @@ -371,6 +371,11 @@ module.exports = { label: "Using HAMi with KAI Scheduler", items: ["userguide/kai-scheduler/how-to-use-kai-scheduler"], }, + { + type: "category", + label: "Using HAMi with Coscheduling", + items: ["userguide/coscheduling/how-to-use-coscheduling"], + }, ], }, ],