From 3e72faa8b5db9ea769e8ffb260c95a2e774e700c Mon Sep 17 00:00:00 2001 From: ipsitapp8 Date: Wed, 12 Aug 2026 14:34:25 +0530 Subject: [PATCH] docs(troubleshooting): add scheduler filter reason code reference A Pod that requests HAMi resources and stays Pending gets a `FilteringFailed` event carrying one of the reason codes defined in `pkg/device/common/common.go`, but the website documents none of them. The only page that mentions any of these codes, `developers/scheduler-event-log.md`, is a design proposal written in future tense and covers 7 of the 17 codes the scheduler emits today. Add a user-facing troubleshooting page that: - explains the two message formats a user actually sees: the aggregated event `N nodes (node-a,node-b)` and the per-node scheduler log `NodeUnfitPod ... reason="3/8 CardInsufficientMemory, ..."` - documents all 17 reason codes with the condition that triggers each one and the corresponding fix, grouped by the stage at which the scheduler rejects the card - flags three behaviours that are easy to misread: one event is emitted per reason code (so a single failure can produce several events), events appear only when no node fits at all, and `AllocatedCardsInsufficientRequest` inverts the numerator to count cards that fit rather than cards rejected - covers the messages emitted before per-device filtering runs, which carry no reason code at all - shows how to raise the extender to `-v=5` for device-level detail, with the cost of doing so and how to revert Verified against HAMi v2.9.0. Links added from the existing scheduler event log design page in both locales. Signed-off-by: ipsitapp8 --- docs/developers/scheduler-event-log.md | 2 + docs/troubleshooting/scheduling-failures.md | 176 +++++++++++++++++ .../current/developers/scheduler-event-log.md | 2 + .../troubleshooting/scheduling-failures.md | 177 ++++++++++++++++++ sidebars.js | 4 + 5 files changed, 361 insertions(+) create mode 100644 docs/troubleshooting/scheduling-failures.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/troubleshooting/scheduling-failures.md diff --git a/docs/developers/scheduler-event-log.md b/docs/developers/scheduler-event-log.md index 0facfc4b4..c565801fb 100644 --- a/docs/developers/scheduler-event-log.md +++ b/docs/developers/scheduler-event-log.md @@ -4,6 +4,8 @@ title: Scheduler Event Log sidebar_label: Scheduler Event Log --- +This page records the design of the scheduler's event and log format. If you are diagnosing a Pod that is stuck in `Pending` right now, read [Scheduling Failures](../troubleshooting/scheduling-failures.md) instead: it lists every reason code the scheduler emits and what to do about each one. + ## Current Status ### Ambiguous Event Descriptions Make Problem Diagnosis Difficult diff --git a/docs/troubleshooting/scheduling-failures.md b/docs/troubleshooting/scheduling-failures.md new file mode 100644 index 000000000..084e66471 --- /dev/null +++ b/docs/troubleshooting/scheduling-failures.md @@ -0,0 +1,176 @@ +--- +title: Scheduling Failures +sidebar_label: Scheduling Failures +--- + +When a Pod that requests HAMi resources stays `Pending`, the HAMi scheduler extender has already made a decision and told you why. It records that decision as `FilteringFailed` events on the Pod, using a fixed set of reason codes such as `CardInsufficientMemory` or `CardTimeSlicingExhausted`. + +This page explains how to read those messages and what each reason code means. + +:::info + +Reason codes are defined in `pkg/device/common/common.go` in the [HAMi repository](https://github.com/Project-HAMi/HAMi). The list below reflects HAMi v2.9.0. Older versions emit a subset of these codes; a Pod scheduled by an older scheduler may show free-form messages instead. + +::: + +## Step 1: Read the Pod events + +```bash +kubectl describe pod -n +``` + +The `Events` section holds two different kinds of message. Both matter: + +```plaintext +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Warning FailedScheduling 15s default-scheduler 0/3 nodes are available: 3 NodeUnfitPod. + Warning FilteringFailed 16s hami-scheduler 2 nodes CardInsufficientMemory(node-a,node-b) + Warning FilteringFailed 16s hami-scheduler 1 nodes CardTypeMismatch(node-c) +``` + +- `FailedScheduling` comes from the Kubernetes scheduler. It only tells you how many nodes were rejected. +- `FilteringFailed` comes from `hami-scheduler` and carries the actual reason. **This is the line to act on.** + +If you see no `FilteringFailed` event at all, the Pod never reached HAMi. Check that the Pod uses HAMi's scheduler and that the admission webhook is running. See [Verify HAMi](../get-started/verify-hami.md). + +## Step 2: Decode the message + +### The event format + +```plaintext + nodes (,,...) +``` + +Three properties of this format are easy to misread: + +- **One event per reason code.** A cluster where two nodes ran out of memory and one node had the wrong card type produces _two_ `FilteringFailed` events, not one. Read all of them before concluding anything. +- **These events appear only when no node fits.** If at least one node is viable, the Pod is scheduled and you get a single `FilteringSucceed` event instead, even though other nodes were rejected. +- **The node list is the set of nodes rejected for that specific reason.** A node can appear under more than one reason code, because different GPUs on the same node can fail for different reasons. + +### The scheduler log format + +The events aggregate per node. To see which _device_ failed and why, read the scheduler extender log: + +```bash +kubectl logs -n kube-system deploy/hami-scheduler -c vgpu-scheduler-extender --tail=200 +``` + +At the default verbosity (`-v=4`) each rejected node produces one `NodeUnfitPod` line: + +```plaintext +NodeUnfitPod pod="default/gpu-pod" node="node-a" reason="3/8 CardInsufficientMemory, 5/8 CardInsufficientCore" +``` + +Read the fraction as `/`. In the line above, node-a has 8 GPUs: 3 were short on memory and 5 were short on compute. Each rejected device is counted once, under the **first** check it failed, so a card that is short on both memory and compute appears only under `CardInsufficientMemory`. Fixing the reason with the largest count is not always the fastest route to a scheduled Pod. + +:::warning The one exception to the fraction rule + +`AllocatedCardsInsufficientRequest` inverts the numerator. There it counts the cards that **did** fit, not the ones that were rejected. `2/8 AllocatedCardsInsufficientRequest` means the node offered 2 usable cards for a request that needed more. + +::: + +Successful nodes log a matching `NodeFitPod` line with the score that decided the placement. + +## Step 3: Look up the reason code + +### Node-level rejections + +| Reason code | What the scheduler found | What to do | +| --- | --- | --- | +| `NodeInsufficientDevice` | The node has fewer devices of the requested type than the Pod asks for. Evaluated before any per-device check. | Lower the card count, or add nodes with more cards. A 4-GPU request never fits a 2-GPU node, regardless of how idle it is. | +| `NodeUnfitPod` | Summary line: this node was rejected. It is always accompanied by the per-device reasons. | Read the per-device reasons in the same log line. | +| `NodeFitPod` | Not a failure. The node passed filtering and was scored. | Nothing. | + +### The card was excluded before capacity was considered + +| Reason code | What the scheduler found | What to do | +| --- | --- | --- | +| `CardNotHealth` | The device plugin reported the device as unhealthy, so it is skipped entirely. | Check the device plugin logs and `nvidia-smi` on that node. An unhealthy card is a node problem, not a request problem. | +| `CardTypeMismatch` | The card model does not satisfy the Pod's type constraints. | Review the `nvidia.com/use-gputype` / `nvidia.com/nouse-gputype` annotations. Also triggered by `nvidia.com/vgpu-mode` when the card does not run the requested mode. See [Specify device type to use](../userguide/nvidia-device/specify-device-type-to-use.md). | +| `CardUuidMismatch` | The card's UUID is excluded by the Pod's UUID constraints. | Review `nvidia.com/use-gpuuuid` / `nvidia.com/nouse-gpuuuid`. A stale UUID pinned in a Deployment template survives node replacement and silently blocks every rescheduling attempt. See [Specify device UUID to use](../userguide/nvidia-device/specify-device-uuid-to-use.md). | +| `NumaNotFit` | The Pod requires all its cards on one NUMA node, and the candidate cards span a NUMA boundary. | Only applies when the Pod sets `nvidia.com/numa-bind: "true"`. Drop the annotation if NUMA locality is not required, or request a card count that a single NUMA node can serve. | +| `ModeNotFit` | The node cannot run the requested virtualization mode for that vendor. | Vendor-specific. On Ascend, it means HAMi-core sharing was requested on a node that does not support it; on Enflame, that no GCU profile matches the request. | +| `CardNotFoundCustomFilterRule` | A vendor-specific filter rule rejected the card. | Consult the guide for that vendor under [User Guide](../userguide/device-supported.md). NVIDIA cards outside MIG mode never produce this code. | +| `CardMigTopologyInfeasible` | The card is in MIG mode, but no allowed MIG profile with a free placement matches the requested memory. | The card may have free memory in total while still having no contiguous slot of the right shape. Align the request with a real MIG profile size, or drain the card. See [Dynamic MIG support](../userguide/nvidia-device/dynamic-mig-support.md). | + +### The card matched but had no room + +| Reason code | What the scheduler found | What to do | +| --- | --- | --- | +| `CardInsufficientMemory` | Free device memory is below the request: `total - used < requested`. | The most common code. Lower `nvidia.com/gpumem`, wait for a workload to finish, or add capacity. Remember that HAMi counts _allocated_ memory, not memory currently in use, so an idle-looking card can still be full. | +| `CardInsufficientCore` | Free compute percentage is below `nvidia.com/gpucores`. | Lower `gpucores`, or place the Pod on a card with fewer streaming workloads. | +| `CardTimeSlicingExhausted` | The card already hosts the maximum number of tasks. | Each card accepts `deviceSplitCount` tasks (default 10) regardless of remaining memory. A card with plenty of free memory still rejects task 11. Raise the split count if the workloads are small enough to justify it. See [Global configuration](../userguide/configure.md). | +| `CardComputeUnitsExhausted` | The Pod requests no cores at all, and the card's compute is fully committed. | A request that omits `gpucores` is not free to place: it still cannot land on a card at 100% committed compute. Give the Pod an explicit `gpucores` value, or free compute on the card. | +| `ExclusiveDeviceAllocateConflict` | Exclusive use was requested for a card that is already shared, or a shared request hit a card held exclusively. | Raised either when `nvidia.com/gpucores: 100` is requested on a card with existing tasks, or when the `mutex` GPU scheduler policy is in effect. See [Scheduling policy](../userguide/nvidia-device/scheduling-policy.md). | +| `ResourceQuotaNotFit` | The allocation would exceed the namespace's HAMi `ResourceQuota`. | A cluster-capacity problem in disguise: the cards are free, the namespace budget is not. See [Using ResourceQuota](../userguide/nvidia-device/using-resourcequota.md). | +| `AllocatedCardsInsufficientRequest` | Some cards on the node fit, but fewer than the requested count. | The node is partially usable. Reduce the card count, or free enough cards on one node. HAMi does not split one container's cards across nodes. | + +## Step 4: Get per-device detail when the node summary is not enough + +The `NodeUnfitPod` summary tells you _how many_ devices failed, not _which_ ones. Device identity is logged at `-v=5`: + +```bash +helm upgrade hami hami-charts/hami \ + --namespace kube-system \ + --reuse-values \ + --set-json 'scheduler.extender.extraArgs=["--debug","-v=5"]' + +kubectl rollout status deploy/hami-scheduler -n kube-system +``` + +Re-create the Pod, then read the log again. Each rejected device now names itself: + +```plaintext +CardInsufficientMemory pod="default/gpu-pod" node="node-a" device="GPU-62b7408e-edb2-41d1-bc91-f46165c61130" device total memory=40960 device used memory=39000 request memory=8000 +``` + +`-v=5` is verbose in proportion to cluster size: a 10-node cluster with 8 GPUs per node emits up to 80 lines for a single failed Pod. Return to the default once you have the answer: + +```bash +helm upgrade hami hami-charts/hami \ + --namespace kube-system \ + --reuse-values \ + --set-json 'scheduler.extender.extraArgs=["--debug","-v=4"]' +``` + +## Messages that carry no reason code + +Some failures are reported before per-device filtering runs, so they never produce a reason code. + +| Message | Meaning | +| --- | --- | +| `no available node, N nodes do not meet` | Every candidate node was rejected. The specific reasons are in the other `FilteringFailed` events on the same Pod, so do not stop reading at this one. | +| `no available node, all node scores do not meet` | Same situation, reported by older HAMi versions without per-reason breakdown. Fall back to the scheduler log. | +| `node unregistered` (log only, `-v=5`) | The node has no HAMi device registration. Either its device plugin is not running, or it genuinely has no supported accelerator. Check `kubectl get node -o jsonpath='{.metadata.annotations}'` for `hami.io/node-*-register`. | +| `Device type not found` | The Pod requests a device type the scheduler was not built or configured to handle, for example an Ascend request on a scheduler started without `--enable-ascend=true`. | + +:::note Why some reasons never reach the events + +The aggregated events are built by parsing the `/ ` fractions out of each node's reason string. Reason codes reported bare, without a fraction (`NodeInsufficientDevice` is the notable one), are recorded against the node but do not become their own `FilteringFailed` event. If the events look thinner than the failure, read the scheduler log. + +::: + +## Putting it together + +```mermaid +%% title: Diagnosing a Pending HAMi Pod +flowchart TD + A["Pod stays Pending"] --> B{"FilteringFailed
event present?"} + B -->|No| C["Pod never reached HAMi:
check schedulerName
and admission webhook"] + B -->|Yes| D["Read every FilteringFailed event
one per reason code"] + D --> E{"Reason code
recognised?"} + E -->|Yes| F["Look up the code
in the tables above"] + E -->|No| G["Read NodeUnfitPod lines
in the scheduler log"] + G --> H{"Need the
failing device?"} + H -->|Yes| I["Raise extender to -v=5,
reproduce, revert to -v=4"] + H -->|No| F +``` + +## Related Pages + +- [Troubleshooting](./troubleshooting.md): installation and runtime problems that are not scheduling decisions +- [Scheduler Event Log](../developers/scheduler-event-log.md): the design behind these events and logs +- [Scheduling Policy](../userguide/nvidia-device/scheduling-policy.md): node and GPU selection policies that change which cards are considered +- [FAQ](../faq/faq.md) diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/developers/scheduler-event-log.md b/i18n/zh/docusaurus-plugin-content-docs/current/developers/scheduler-event-log.md index b73f77131..3f5c94deb 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/developers/scheduler-event-log.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/developers/scheduler-event-log.md @@ -5,6 +5,8 @@ translated: true sidebar_label: 调度器事件日志 --- +本页记录的是调度器事件与日志格式的设计。如果你正在排查一个卡在 `Pending` 状态的 Pod，请改看[调度失败排查](../troubleshooting/scheduling-failures.md)：那里列出了调度器会输出的每一个原因码及其处理方式。 + ## 当前状态 ### 模糊的事件描述使问题诊断变得困难 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/troubleshooting/scheduling-failures.md b/i18n/zh/docusaurus-plugin-content-docs/current/troubleshooting/scheduling-failures.md new file mode 100644 index 000000000..1271795ba --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/troubleshooting/scheduling-failures.md @@ -0,0 +1,177 @@ +--- +title: 调度失败排查 +sidebar_label: 调度失败排查 +translated: true +--- + +当一个申请了 HAMi 资源的 Pod 一直处于 `Pending` 状态时，HAMi 调度器扩展器其实已经做出了决策，并告诉了你原因。它会在 Pod 上记录 `FilteringFailed` 事件，并使用一组固定的原因码，例如 `CardInsufficientMemory` 或 `CardTimeSlicingExhausted`。 + +本页说明如何读懂这些消息，以及每个原因码的含义。 + +:::info + +原因码定义在 [HAMi 仓库](https://github.com/Project-HAMi/HAMi) 的 `pkg/device/common/common.go` 中。下表基于 HAMi v2.9.0。更早的版本只会输出其中一部分原因码；由旧版调度器调度的 Pod 可能显示为自由格式的消息。 + +::: + +## 第一步：查看 Pod 事件 + +```bash +kubectl describe pod -n +``` + +`Events` 部分包含两类不同的消息，两者都很重要： + +```plaintext +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Warning FailedScheduling 15s default-scheduler 0/3 nodes are available: 3 NodeUnfitPod. + Warning FilteringFailed 16s hami-scheduler 2 nodes CardInsufficientMemory(node-a,node-b) + Warning FilteringFailed 16s hami-scheduler 1 nodes CardTypeMismatch(node-c) +``` + +- `FailedScheduling` 来自 Kubernetes 调度器，只告诉你有多少节点被拒绝。 +- `FilteringFailed` 来自 `hami-scheduler`，携带了真正的原因。**这才是需要处理的那一行。** + +如果完全看不到 `FilteringFailed` 事件，说明 Pod 根本没有走到 HAMi。请确认 Pod 使用了 HAMi 的调度器，并且准入 Webhook 正在运行，参见[验证 HAMi](../get-started/verify-hami.md)。 + +## 第二步：解读消息 + +### 事件格式 + +```plaintext + nodes (,,...) +``` + +这个格式有三个容易误读的地方： + +- **每个原因码一条事件。** 如果集群中两个节点显存不足、一个节点卡型不匹配，会产生**两条** `FilteringFailed` 事件，而不是一条。下结论之前请读完所有事件。 +- **只有在没有任何节点可用时才会出现这些事件。** 只要有一个节点可用，Pod 就会被调度，此时只会得到一条 `FilteringSucceed` 事件，即使其他节点确实被拒绝了。 +- **括号中的节点列表是因该原因被拒绝的节点集合。** 同一个节点可能出现在多个原因码下，因为同一节点上不同的 GPU 可能因不同原因被拒绝。 + +### 调度器日志格式 + +事件是按节点聚合的。要查看具体是哪块**设备**、因何失败，需要读调度器扩展器的日志： + +```bash +kubectl logs -n kube-system deploy/hami-scheduler -c vgpu-scheduler-extender --tail=200 +``` + +在默认日志级别（`-v=4`）下，每个被拒绝的节点会产生一行 `NodeUnfitPod`： + +```plaintext +NodeUnfitPod pod="default/gpu-pod" node="node-a" reason="3/8 CardInsufficientMemory, 5/8 CardInsufficientCore" +``` + +分数的含义是 `<因该原因被拒绝的设备数>/<节点上该类型设备总数>`。上面这行表示 node-a 有 8 块 GPU：3 块显存不足，5 块算力不足。每块被拒绝的设备只会被计入**第一个**未通过的检查，因此一块既缺显存又缺算力的卡只会出现在 `CardInsufficientMemory` 下。优先修复数量最多的那个原因，未必是让 Pod 最快调度成功的路径。 + +:::warning 唯一的例外 + +`AllocatedCardsInsufficientRequest` 的分子含义是相反的：它统计的是**通过**筛选的卡数，而不是被拒绝的卡数。`2/8 AllocatedCardsInsufficientRequest` 表示该节点只能提供 2 块可用的卡，而申请需要更多。 + +::: + +通过筛选的节点会输出对应的 `NodeFitPod` 日志，并带上决定最终放置的分数。 + +## 第三步：查阅原因码 + +### 节点级拒绝 + +| 原因码 | 调度器发现了什么 | 如何处理 | +| --- | --- | --- | +| `NodeInsufficientDevice` | 节点上该类型设备的数量少于 Pod 的申请数量。该检查在任何单卡检查之前执行。 | 减少申请卡数，或增加卡数更多的节点。申请 4 卡永远无法落在 2 卡节点上，无论这些卡多空闲。 | +| `NodeUnfitPod` | 汇总行：该节点被拒绝。总是伴随每块卡的具体原因一起出现。 | 阅读同一行日志中的单卡原因。 | +| `NodeFitPod` | 不是失败。该节点通过了筛选并参与打分。 | 无需处理。 | + +### 卡在容量检查之前就被排除 + +| 原因码 | 调度器发现了什么 | 如何处理 | +| --- | --- | --- | +| `CardNotHealth` | 设备插件上报该设备不健康，直接跳过。 | 检查设备插件日志以及该节点上的 `nvidia-smi`。这是节点问题，不是申请问题。 | +| `CardTypeMismatch` | 卡型不满足 Pod 的类型约束。 | 检查 `nvidia.com/use-gputype` / `nvidia.com/nouse-gputype` 注解。当卡不支持 `nvidia.com/vgpu-mode` 所要求的模式时也会触发。参见[指定使用的设备类型](../userguide/nvidia-device/specify-device-type-to-use.md)。 | +| `CardUuidMismatch` | 卡的 UUID 被 Pod 的 UUID 约束排除。 | 检查 `nvidia.com/use-gpuuuid` / `nvidia.com/nouse-gpuuuid`。写死在 Deployment 模板里的过期 UUID 会在节点更换后继续存在，并静默地阻止每一次重新调度。参见[指定使用的设备 UUID](../userguide/nvidia-device/specify-device-uuid-to-use.md)。 | +| `NumaNotFit` | Pod 要求所有卡位于同一 NUMA 节点，而候选卡跨越了 NUMA 边界。 | 仅在 Pod 设置了 `nvidia.com/numa-bind: "true"` 时出现。如果不需要 NUMA 亲和，去掉该注解；或者把卡数减少到单个 NUMA 节点能够满足的范围。 | +| `ModeNotFit` | 节点无法运行该厂商所要求的虚拟化模式。 | 与厂商相关。在 Ascend 上表示在不支持 HAMi-core 共享的节点上申请了该模式；在 Enflame 上表示没有匹配请求的 GCU 规格。 | +| `CardNotFoundCustomFilterRule` | 厂商自定义的过滤规则拒绝了该卡。 | 参见[用户指南](../userguide/device-supported.md)中对应厂商的文档。非 MIG 模式的 NVIDIA 卡不会产生该原因码。 | +| `CardMigTopologyInfeasible` | 卡处于 MIG 模式，但没有任何允许的 MIG 规格拥有与申请显存匹配的空闲位置。 | 卡的空闲显存总量可能是够的，却没有形状合适的连续切片。请将申请对齐到真实的 MIG 规格，或腾空该卡。参见[动态 MIG 支持](../userguide/nvidia-device/dynamic-mig-support.md)。 | + +### 卡匹配但容量不足 + +| 原因码 | 调度器发现了什么 | 如何处理 | +| --- | --- | --- | +| `CardInsufficientMemory` | 设备空闲显存低于申请量：`总量 - 已用 < 申请量`。 | 最常见的原因码。降低 `nvidia.com/gpumem`、等待其他任务结束，或扩容。注意 HAMi 统计的是**已分配**显存而非当前实际占用，因此一块看似空闲的卡也可能已经满了。 | +| `CardInsufficientCore` | 空闲算力百分比低于 `nvidia.com/gpucores`。 | 降低 `gpucores`，或把 Pod 放到流式负载更少的卡上。 | +| `CardTimeSlicingExhausted` | 该卡承载的任务数已达上限。 | 每块卡最多接受 `deviceSplitCount` 个任务（默认 10），与剩余显存无关。显存充足的卡依然会拒绝第 11 个任务。如果负载足够小，可以调高该值。参见[全局配置](../userguide/configure.md)。 | +| `CardComputeUnitsExhausted` | Pod 完全没有申请算力，而该卡的算力已被占满。 | 省略 `gpucores` 的申请并不是"随便放"：它同样无法落在算力已 100% 分配的卡上。请显式指定 `gpucores`，或释放该卡的算力。 | +| `ExclusiveDeviceAllocateConflict` | 对已被共享的卡申请了独占，或共享申请落到了被独占的卡上。 | 两种触发方式：在已有任务的卡上申请 `nvidia.com/gpucores: 100`，或启用了 `mutex` GPU 调度策略。参见[调度策略](../userguide/nvidia-device/scheduling-policy.md)。 | +| `ResourceQuotaNotFit` | 该次分配会超出命名空间的 HAMi `ResourceQuota`。 | 这是伪装成容量问题的配额问题：卡是空闲的，命名空间预算不是。参见[使用 ResourceQuota](../userguide/nvidia-device/using-resourcequota.md)。 | +| `AllocatedCardsInsufficientRequest` | 节点上有部分卡可用，但少于申请的数量。 | 该节点部分可用。请减少申请卡数，或在单个节点上腾出足够的卡：HAMi 不会把同一个容器的多张卡拆到不同节点上。 | + +## 第四步：当节点汇总不够时获取单卡细节 + +`NodeUnfitPod` 汇总只告诉你有**多少**块设备失败，不告诉你是**哪些**。设备标识需要 `-v=5` 才会记录： + +```bash +helm upgrade hami hami-charts/hami \ + --namespace kube-system \ + --reuse-values \ + --set-json 'scheduler.extender.extraArgs=["--debug","-v=5"]' + +kubectl rollout status deploy/hami-scheduler -n kube-system +``` + +重新创建 Pod，然后再次查看日志。此时每块被拒绝的设备都会自报家门： + +```plaintext +CardInsufficientMemory pod="default/gpu-pod" node="node-a" device="GPU-62b7408e-edb2-41d1-bc91-f46165c61130" device total memory=40960 device used memory=39000 request memory=8000 +``` + +`-v=5` 的日志量与集群规模成正比：一个 10 节点、每节点 8 卡的集群，单个失败的 Pod 最多会产生 80 行日志。拿到答案后请恢复默认级别： + +```bash +helm upgrade hami hami-charts/hami \ + --namespace kube-system \ + --reuse-values \ + --set-json 'scheduler.extender.extraArgs=["--debug","-v=4"]' +``` + +## 不带原因码的消息 + +有些失败发生在单卡筛选之前，因此不会产生原因码。 + +| 消息 | 含义 | +| --- | --- | +| `no available node, N nodes do not meet` | 所有候选节点都被拒绝。具体原因在同一个 Pod 的其他 `FilteringFailed` 事件中，**不要**只看这一条就停下。 | +| `no available node, all node scores do not meet` | 同样的情况，由不带原因码拆分的旧版 HAMi 输出。请改看调度器日志。 | +| `node unregistered`（仅日志，`-v=5`） | 该节点没有 HAMi 设备注册信息。要么设备插件没有运行，要么它确实没有受支持的加速卡。可用 `kubectl get node -o jsonpath='{.metadata.annotations}'` 检查是否存在 `hami.io/node-*-register`。 | +| `Device type not found` | Pod 申请的设备类型未被调度器构建或启用，例如在没有 `--enable-ascend=true` 的调度器上提交 Ascend 申请。 | + +:::note 为什么有些原因不会出现在事件里 + +聚合事件是通过解析每个节点原因字符串中的 `/ ` 分数得到的。以裸形式上报、不带分数的原因码（`NodeInsufficientDevice` 是典型例子）会记录在节点上，但不会形成独立的 `FilteringFailed` 事件。如果事件看起来比实际失败情况"少"，请查看调度器日志。 + +::: + +## 串起来看 + +```mermaid +%% title: 排查 Pending 状态的 HAMi Pod +flowchart TD + A["Pod 持续 Pending"] --> B{"是否有
FilteringFailed 事件？"} + B -->|否| C["Pod 未到达 HAMi：
检查 schedulerName
与准入 Webhook"] + B -->|是| D["读完每一条 FilteringFailed 事件
每个原因码一条"] + D --> E{"原因码
能否识别？"} + E -->|能| F["在上文表格中
查阅该原因码"] + E -->|否| G["查看调度器日志中的
NodeUnfitPod 行"] + G --> H{"是否需要定位
具体设备？"} + H -->|是| I["把扩展器调到 -v=5，
复现后再改回 -v=4"] + H -->|否| F +``` + +## 相关页面 + +- [排障手册](./troubleshooting.md)：安装与运行时问题，而非调度决策 +- [调度器事件日志](../developers/scheduler-event-log.md)：这些事件与日志背后的设计 +- [调度策略](../userguide/nvidia-device/scheduling-policy.md)：影响哪些卡会被纳入考虑的节点与 GPU 选择策略 +- [常见问题](../faq/faq.md) diff --git a/sidebars.js b/sidebars.js index 3487c0e9f..c64027bdd 100644 --- a/sidebars.js +++ b/sidebars.js @@ -422,6 +422,10 @@ module.exports = { type: "doc", id: "troubleshooting/troubleshooting", }, + { + type: "doc", + id: "troubleshooting/scheduling-failures", + }, { type: "doc", id: "faq/faq",