docs(troubleshooting): add scheduler filter reason code reference - #767
docs(troubleshooting): add scheduler filter reason code reference#767ipsitapp8 wants to merge 1 commit into
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: ipsitapp8 The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
✅ Deploy Preview for project-hami ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
Warning Review limit reached
Next review available in: 58 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughAdded English and Chinese documentation for diagnosing HAMi scheduling failures. The changes cover Pending Pods, ChangesScheduling Failure Documentation
Estimated code review effort: 2 (Simple) | ~10 minutes Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
coderabbitai
Bot
added
the
kind/documentation
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/troubleshooting/scheduling-failures.md`:
- Line 12: Align the HAMi reason-code documentation with v2.9.0: in
docs/troubleshooting/scheduling-failures.md at lines 12 and 95-96, update the
introduction to link to the v2.9.0 source and state 16 codes, then remove
CardMigTopologyInfeasible from the table. Apply the same introduction and table
changes in
i18n/zh/docusaurus-plugin-content-docs/current/troubleshooting/scheduling-failures.md
at lines 13 and 96-97.
- Around line 115-119: Update all four documented Helm upgrade commands in
docs/troubleshooting/scheduling-failures.md lines 115-119 and 131-136, and
i18n/zh/docusaurus-plugin-content-docs/current/troubleshooting/scheduling-failures.md
lines 115-119 and 132-137: pin each chart with the documented version and
preserve the complete existing scheduler.extender.extraArgs array while adding
the -v=5 or -v=4 flag, rather than replacing the array.
- Around line 106-107: Update the ExclusiveDeviceAllocateConflict description in
docs/troubleshooting/scheduling-failures.md lines 106-107 and
i18n/zh/docusaurus-plugin-content-docs/current/troubleshooting/scheduling-failures.md
lines 107-108 to describe only the v2.9.0 case where k.Coresreq == 100 is
requested on a card that already has tasks; remove the mutex scheduler trigger
and unsupported general shared-request wording from both locales.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: ad886529-4457-4b7c-adaa-7db0e2d3b81c
📒 Files selected for processing (5)
docs/developers/scheduler-event-log.mddocs/troubleshooting/scheduling-failures.mdi18n/zh/docusaurus-plugin-content-docs/current/developers/scheduler-event-log.mdi18n/zh/docusaurus-plugin-content-docs/current/troubleshooting/scheduling-failures.mdsidebars.js
|
|
||
| :::info | ||
|
|
||
| Reason codes are defined in `pkg/device/common/common.go` in the [HAMi repository](https://github.com/Project-HAMi/HAMi). The list below reflects HAMi v2.9.0. Older versions emit a subset of these codes; a Pod scheduled by an older scheduler may show free-form messages instead. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -e
printf '%s\n' '--- tracked documentation files ---'
git ls-files 'docs/troubleshooting/scheduling-failures.md' 'i18n/zh/docusaurus-plugin-content-docs/current/troubleshooting/scheduling-failures.md'
printf '%s\n' '--- English context ---'
sed -n '1,110p' docs/troubleshooting/scheduling-failures.md
printf '%s\n' '--- Chinese context ---'
sed -n '1,110p' i18n/zh/docusaurus-plugin-content-docs/current/troubleshooting/scheduling-failures.md
printf '%s\n' '--- HAMi v2.9.0 reason constants ---'
curl -fsSL https://raw.githubusercontent.com/Project-HAMi/HAMi/v2.9.0/pkg/device/common/common.go |
grep -n -E 'Reason|Card|Mig|const|iota' | head -120
printf '%s\n' '--- HAMi v2.9.0 source status ---'
curl -fsSL -o /tmp/hami-v2.9.0-common.go \
https://raw.githubusercontent.com/Project-HAMi/HAMi/v2.9.0/pkg/device/common/common.go
sha256sum /tmp/hami-v2.9.0-common.goRepository: Project-HAMi/website
Length of output: 15122
🏁 Script executed:
#!/bin/bash
set -e
printf '%s\n' '--- complete HAMi v2.9.0 declaration block ---'
sed -n '20,42p' /tmp/hami-v2.9.0-common.go
printf '%s\n' '--- parsed string-valued constants in the block ---'
python3 - <<'PY'
from pathlib import Path
import re
text = Path('/tmp/hami-v2.9.0-common.go').read_text()
block = re.search(r'const\s*\((.*?)\n\)', text, re.S).group(1)
items = re.findall(r'^\s*(\w+)\s*=\s*"([^"]+)"', block, re.M)
print(f'count={len(items)}')
for name, value in items:
print(f'{name}={value}')
print('CardMigTopologyInfeasible present:', 'CardMigTopologyInfeasible' in text)
PY
printf '%s\n' '--- referenced documentation targets ---'
for f in \
docs/userguide/device-supported.md \
docs/userguide/nvidia-device/dynamic-mig-support.md \
i18n/zh/docusaurus-plugin-content-docs/current/userguide/device-supported.md \
i18n/zh/docusaurus-plugin-content-docs/current/userguide/nvidia-device/dynamic-mig-support.md
do
if git ls-files --error-unmatch "$f" >/dev/null 2>&1; then
printf 'present %s\n' "$f"
else
printf 'missing %s\n' "$f"
fi
done
printf '%s\n' '--- matching reason-code uses in v2.9.0 source ---'
grep -R -n --exclude-dir=.git 'CardMigTopologyInfeasible' /tmp/hami-v2.9.0-common.go docs i18n/zh/docusaurus-plugin-content-docs/current 2>/dev/null || trueRepository: Project-HAMi/website
Length of output: 3039
Align the reason-code documentation with HAMi v2.9.0.
HAMi v2.9.0 defines 16 reason constants and does not define CardMigTopologyInfeasible. Update the English and Chinese introductions to link to the v2.9.0 source and state 16 codes. Remove the unsupported row from both tables or move it to a matching HAMi version.
🧰 Tools
🪛 LanguageTool
[grammar] ~12-~12: Ensure spelling is correct
Context: ...ommon/common.go` in the HAMi repository. The list below reflects HAMi v2.9.0. Ol...
(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)
📍 Affects 2 files
docs/troubleshooting/scheduling-failures.md#L12-L12(this comment)docs/troubleshooting/scheduling-failures.md#L95-L96i18n/zh/docusaurus-plugin-content-docs/current/troubleshooting/scheduling-failures.md#L13-L13i18n/zh/docusaurus-plugin-content-docs/current/troubleshooting/scheduling-failures.md#L96-L97
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/troubleshooting/scheduling-failures.md` at line 12, Align the HAMi
reason-code documentation with v2.9.0: in
docs/troubleshooting/scheduling-failures.md at lines 12 and 95-96, update the
introduction to link to the v2.9.0 source and state 16 codes, then remove
CardMigTopologyInfeasible from the table. Apply the same introduction and table
changes in
i18n/zh/docusaurus-plugin-content-docs/current/troubleshooting/scheduling-failures.md
at lines 13 and 96-97.
Source: MCP tools
| | `ExclusiveDeviceAllocateConflict` | Exclusive use was requested for a card that is already shared, or a shared request hit a card held exclusively. | Raised either when `nvidia.com/gpucores: 100` is requested on a card with existing tasks, or when the `mutex` GPU scheduler policy is in effect. See [Scheduling policy](../userguide/nvidia-device/scheduling-policy.md). | | ||
| | `ResourceQuotaNotFit` | The allocation would exceed the namespace's HAMi `ResourceQuota`. | A cluster-capacity problem in disguise: the cards are free, the namespace budget is not. See [Using ResourceQuota](../userguide/nvidia-device/using-resourcequota.md). | |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -eu
printf '%s\n' '--- candidate files ---'
git ls-files | rg '(^|/)(scheduling-failures\.md|device\.go)$|nvidia/.*/device\.go$' | head -80
printf '%s\n' '--- documentation context ---'
for f in docs/troubleshooting/scheduling-failures.md \
i18n/zh/docusaurus-plugin-content-docs/current/troubleshooting/scheduling-failures.md; do
if [ -f "$f" ]; then
printf '\n--- %s ---\n' "$f"
sed -n '1,145p' "$f"
fi
done
printf '%s\n' '--- local reason-code references ---'
rg -n -C 3 'ExclusiveDeviceAllocateConflict|ResourceQuotaNotFit|mutex|gpucores' --glob '*.go' --glob '*.md' --glob '*.yaml' --glob '*.yml' . | head -300Repository: Project-HAMi/website
Length of output: 48176
🏁 Script executed:
#!/bin/bash
set -eu
tmpdir="$(mktemp -d)"
trap 'rm -rf "$tmpdir"' EXIT
for spec in \
"device.go|https://raw.githubusercontent.com/Project-HAMi/HAMi/refs/tags/v2.9.0/pkg/device/nvidia/device.go" \
"score.go|https://raw.githubusercontent.com/Project-HAMi/HAMi/refs/tags/v2.9.0/pkg/scheduler/score.go" \
"common.go|https://raw.githubusercontent.com/Project-HAMi/HAMi/refs/tags/v2.9.0/pkg/device/common/common.go"; do
name="${spec%%|*}"
url="${spec#*|}"
if curl -fsSL "$url" -o "$tmpdir/$name"; then
printf '\n--- %s ---\n' "$url"
rg -n -C 8 'ExclusiveDeviceAllocateConflict|mutex|gpucores|used' "$tmpdir/$name" || true
else
printf '\n--- unavailable: %s ---\n' "$url"
fi
done
printf '\n--- all v2.9.0 source paths mentioning the reason ---\n'
api='https://api.github.com/repos/Project-HAMi/HAMi/git/trees/v2.9.0?recursive=1'
curl -fsSL "$api" |
jq -r '.tree[] | select(.type == "blob") | .path' |
while read -r path; do
case "$path" in
*.go)
url="https://raw.githubusercontent.com/Project-HAMi/HAMi/refs/tags/v2.9.0/$path"
body="$(curl -fsSL "$url" 2>/dev/null || true)"
if printf '%s' "$body" | grep -q 'ExclusiveDeviceAllocateConflict'; then
printf '%s\n' "$path"
printf '%s' "$body" | grep -n -C 8 'ExclusiveDeviceAllocateConflict'
fi
;;
esac
doneRepository: Project-HAMi/website
Length of output: 4657
🌐 Web query:
site:github.com/Project-HAMi/HAMi v2.9.0 "ExclusiveDeviceAllocateConflict" mutex
💡 Result:
In the context of the Project-HAMi (HAMi) project, ExclusiveDeviceAllocateConflict is a scheduling failure reason indicating that a container’s request for exclusive device access cannot be satisfied because the targeted device is already in use [1][2][3]. The mention of mutex in relation to this conflict refers to a specific GPU scheduling policy introduced to enforce exclusive access [4]. When a pod sets the annotation hami.io/gpu-scheduler-policy to mutex, the HAMi scheduler is instructed to allocate only those GPUs that have no existing users (where the usage count is 0) [4]. Key points regarding this functionality include: 1. Policy Enforcement: The mutex policy ensures that if a device is already occupied (Used > 0), the scheduler's Fit function will reject that device, triggering the ExclusiveDeviceAllocateConflict reason [4][3]. 2. Diagnostic Clarity: In earlier versions, this conflict was sometimes misreported as other errors (such as NumaNotFit) [4]. Updates were implemented to ensure that the ExclusiveDeviceAllocateConflict reason is explicitly reported when the mutex policy restricts device allocation, improving diagnostic clarity for users [4][5]. 3. Reporting: This reason appears in scheduler logs and event messages when the system determines that a node or device cannot fulfill the pod's requirement for exclusive access due to current device occupancy [1][6]. While v2.9.0 is a specific release version of HAMi [7], the ExclusiveDeviceAllocateConflict reason is a persistent part of the HAMi scheduling diagnostic framework, used across multiple versions to signify that a requested exclusive device is unavailable [1][8][2].
Citations:
- 1: https://github.com/Project-HAMi/HAMi/blob/master/docs/scheduler-event-log.md
- 2: https://github.com/Project-HAMi/HAMi/blob/c005cbd1/pkg/device/common/common.go
- 3: https://github.com/Project-HAMi/HAMi/blob/c005cbd1/pkg/device/enflame/gcu_test.go
- 4: Project-HAMi/HAMi@06d9b90
- 5: fix(device): GenReason uses len(reason) instead of len(devices) in kunlun/awsneuron HAMi#2291
- 6: Aggregated Scheduling Failure Events HAMi#1332
- 7: Project-HAMi/HAMi@v2.8.0...v2.9.0
- 8: https://github.com/Project-HAMi/HAMi/blob/v2.6.1/docs/scheduler-event-log.md
🏁 Script executed:
#!/bin/bash
set -eu
base='https://api.github.com/repos/Project-HAMi/HAMi'
commit='06d9b907f19e78551d616e106209d2a98ec20f01'
printf '%s\n' '--- cited commit metadata ---'
curl -fsSL "$base/commits/$commit" |
jq -r '[.sha, .commit.message, (.files | length | tostring)] | `@tsv`'
printf '%s\n' '--- cited commit file patches ---'
curl -fsSL "$base/commits/$commit" |
jq -r '.files[] | select((.patch // "") | test("ExclusiveDeviceAllocateConflict|gpu-scheduler-policy|mutex")) |
"\nFILE: \(.filename)\n\(.patch)"'
printf '%s\n' '--- v2.9.0 tag resolution ---'
curl -fsSL "$base/git/ref/tags/v2.9.0" |
jq -r '"type=\(.object.type) sha=\(.object.sha)"'Repository: Project-HAMi/website
Length of output: 399
🏁 Script executed:
#!/bin/bash
set -eu
tmp="$(mktemp)"
trap 'rm -f "$tmp"' EXIT
for url in \
'https://github.com/Project-HAMi/HAMi/commit/06d9b907f19e78551d616e106209d2a98ec20f01.patch' \
'https://github.com/Project-HAMi/HAMi/commit/06d9b907f19e78551d616e106209d2a98ec20f01.diff'; do
printf '\n--- %s ---\n' "$url"
if curl -fsSL "$url" -o "$tmp"; then
rg -n -C 12 'ExclusiveDeviceAllocateConflict|gpu-scheduler-policy|mutex|Fit\(' "$tmp" || cat "$tmp"
else
printf '%s\n' 'unavailable'
fi
done
printf '\n--- v2.9.0 NVIDIA policy-related source references ---\n'
for path in \
pkg/device/nvidia/device.go \
pkg/device/nvidia/nvidia.go \
pkg/device/nvidia/nvidia_test.go \
pkg/device/common/common.go \
pkg/device/device.go \
pkg/scheduler/scheduler.go \
pkg/scheduler/score.go; do
url="https://raw.githubusercontent.com/Project-HAMi/HAMi/refs/tags/v2.9.0/$path"
body="$(curl -fsSL "$url" 2>/dev/null || true)"
if [ -n "$body" ]; then
hits="$(printf '%s' "$body" | rg -n -C 6 'ExclusiveDeviceAllocateConflict|gpu-scheduler-policy|mutex|NodeLockNvidia' || true)"
if [ -n "$hits" ]; then
printf '\nFILE: %s\n%s\n' "$path" "$hits"
fi
fi
doneRepository: Project-HAMi/website
Length of output: 50377
🏁 Script executed:
#!/bin/bash
set -eu
python3 - <<'PY'
from urllib.request import urlopen
tag_url = "https://raw.githubusercontent.com/Project-HAMi/HAMi/refs/tags/v2.9.0/pkg/device/nvidia/device.go"
patch_url = "https://github.com/Project-HAMi/HAMi/commit/06d9b907f19e78551d616e106209d2a98ec20f01.patch"
tag = urlopen(tag_url).read().decode()
patch = urlopen(patch_url).read().decode()
tag_conflict = "if dev.Totalcore == 100 && k.Coresreq == 100 && dev.Used > 0"
tag_mutex_guard = "if isMutex && dev.Used > 0"
patch_nvidia_mutex = "+\t\tif isMutex && dev.Used > 0"
patch_reason = "+\t\t\treason[common.ExclusiveDeviceAllocateConflict]++"
print("v2.9.0 contains exclusive-used guard:", tag_conflict in tag)
print("v2.9.0 contains mutex guard:", tag_mutex_guard in tag)
print("later patch adds NVIDIA mutex guard:", patch_nvidia_mutex in patch)
print("later patch reports ExclusiveDeviceAllocateConflict for mutex:", patch_reason in patch)
print("v2.9.0 conflict-branch count:", tag.count("reason[common.ExclusiveDeviceAllocateConflict]++"))
PYRepository: Project-HAMi/website
Length of output: 2444
Remove the unsupported mutex trigger from both locales.
In HAMi v2.9.0, the NVIDIA implementation emits ExclusiveDeviceAllocateConflict only when k.Coresreq == 100 and the card already has tasks. The mutex rejection path was added later and is absent from v2.9.0. The general shared-request wording is also unsupported.
docs/troubleshooting/scheduling-failures.md#L106-L107: describe only the v2.9.0 exclusive request on a used card.i18n/zh/docusaurus-plugin-content-docs/current/troubleshooting/scheduling-failures.md#L107-L108: apply the same correction.
📍 Affects 2 files
docs/troubleshooting/scheduling-failures.md#L106-L107(this comment)i18n/zh/docusaurus-plugin-content-docs/current/troubleshooting/scheduling-failures.md#L107-L108
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/troubleshooting/scheduling-failures.md` around lines 106 - 107, Update
the ExclusiveDeviceAllocateConflict description in
docs/troubleshooting/scheduling-failures.md lines 106-107 and
i18n/zh/docusaurus-plugin-content-docs/current/troubleshooting/scheduling-failures.md
lines 107-108 to describe only the v2.9.0 case where k.Coresreq == 100 is
requested on a card that already has tasks; remove the mutex scheduler trigger
and unsupported general shared-request wording from both locales.
Source: MCP tools
| helm upgrade hami hami-charts/hami \ | ||
| --namespace kube-system \ | ||
| --reuse-values \ | ||
| --set-json 'scheduler.extender.extraArgs=["--debug","-v=5"]' | ||
|
|
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -eu
printf '%s\n' '--- target files ---'
for f in \
docs/troubleshooting/scheduling-failures.md \
i18n/zh/docusaurus-plugin-content-docs/current/troubleshooting/scheduling-failures.md
do
printf '\n### %s\n' "$f"
nl -ba "$f" | sed -n '1,155p'
done
printf '\n--- chart references and extraArgs definitions ---\n'
rg -n --hidden -S \
'scheduler\.extender\.extraArgs|extraArgs:|hami-charts/hami|version:' \
--glob '!node_modules' --glob '!dist' --glob '!build' .Repository: Project-HAMi/website
Length of output: 36028
🏁 Script executed:
#!/bin/bash
set -eu
for f in \
docs/troubleshooting/scheduling-failures.md \
i18n/zh/docusaurus-plugin-content-docs/current/troubleshooting/scheduling-failures.md \
docs/developers/profiling-scheduler.md \
i18n/zh/docusaurus-plugin-content-docs/current/developers/profiling-scheduler.md \
docs/installation/upgrade.md
do
printf '\n### %s\n' "$f"
sed -n '1,155p' "$f" | awk '{printf "%5d %s\n", NR, $0}'
done
printf '\n--- all repository definitions/usages of scheduler.extender.extraArgs ---\n'
rg -n -C 8 --hidden -S \
'scheduler\.extender\.extraArgs|extraArgs:' \
--glob '!node_modules' --glob '!dist' --glob '!build' .
printf '\n--- available Helm tooling ---\n'
command -v helm || trueRepository: Project-HAMi/website
Length of output: 50378
🌐 Web query:
Helm upgrade --reuse-values --set-json array override precedence official documentation
💡 Result:
When using helm upgrade with --reuse-values, the behavior regarding precedence and merging is defined by how Helm handles existing versus new values [1][2][3]. Precedence and Merging Behavior 1. --reuse-values takes the computed values from the previous release, rather than the default values.yaml from the new chart [3][4]. 2. Any values provided via --set, --set-json, --values, or other override flags are then merged into these existing values [1][2][5]. 3. In this merge, the new values from your command-line flags take precedence over the previously stored values [1][2]. Array Handling with --set-json When using --set-json or --set to override arrays, it is important to understand that Helm treats arrays as single entities during the merge process [6]. Unlike maps (which are merged key-by-key), arrays are typically overwritten entirely by the new value provided [6]. If you provide an array via --set-json, that array will replace the entire array currently stored in the release values [6]. It will not perform a partial merge or append items to an existing array unless you explicitly include the original items in your new JSON input [6]. Summary of Precedence - New overrides (--set, --set-json, --values) always take priority over the values reused from the previous release [1][2]. - When multiple override flags are used in the same command, the right-most (last) flag specified takes precedence [2]. - Because --reuse-values explicitly excludes the new chart's default values.yaml, any changes to default values in a new chart version will be ignored unless you explicitly apply those values via a -f or --set flag [3][4].
Citations:
- 1: https://helm.sh/docs/helm/helm_upgrade
- 2: https://github.com/helm/helm/blob/827a960e/pkg/cmd/upgrade.go
- 3: Upgrade to new chart version and --set a value at the same time with --reuse-values ignores changes from values.yaml helm/helm#3957
- 4: helm upgrade does not pick up NEW default values if --reuse-values is set helm/helm#9000
- 5: https://github.com/helm/helm-www/blob/main/docs/helm/helm_upgrade.md
- 6: --set with an array index delete all the previous content of the array helm/helm#11734
🏁 Script executed:
#!/bin/bash
set -eu
python3 - <<'PY'
import json
import re
from pathlib import Path
files = [
Path("docs/troubleshooting/scheduling-failures.md"),
Path("i18n/zh/docusaurus-plugin-content-docs/current/troubleshooting/scheduling-failures.md"),
]
found = []
for path in files:
text = path.read_text()
for match in re.finditer(
r"helm upgrade hami hami-charts/hami \\\n"
r"\s+--namespace kube-system \\\n"
r"\s+--reuse-values \\\n"
r"\s+--set-json 'scheduler\.extender\.extraArgs=(\[[^']+\])'",
text,
):
args = json.loads(match.group(1))
found.append((str(path), args, "--version" in match.group(0)))
assert len(found) == 4, found
assert {tuple(args) for _, args, _ in found} == {
("--debug", "-v=5"),
("--debug", "-v=4"),
}
assert all(not has_version for _, _, has_version in found)
for path, args, _ in found:
print(f"{path}: extraArgs={args}; chart_version_pinned=False")
PYRepository: Project-HAMi/website
Length of output: 654
Pin the chart version and preserve the complete extraArgs array.
Without --version, each command can upgrade to the latest chart. --set-json replaces the existing scheduler.extender.extraArgs array and can remove custom flags. Apply both safeguards to the -v=5 and -v=4 commands in the English and Chinese files.
📍 Affects 2 files
docs/troubleshooting/scheduling-failures.md#L115-L119(this comment)docs/troubleshooting/scheduling-failures.md#L131-L136i18n/zh/docusaurus-plugin-content-docs/current/troubleshooting/scheduling-failures.md#L115-L119i18n/zh/docusaurus-plugin-content-docs/current/troubleshooting/scheduling-failures.md#L132-L137
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/troubleshooting/scheduling-failures.md` around lines 115 - 119, Update
all four documented Helm upgrade commands in
docs/troubleshooting/scheduling-failures.md lines 115-119 and 131-136, and
i18n/zh/docusaurus-plugin-content-docs/current/troubleshooting/scheduling-failures.md
lines 115-119 and 132-137: pin each chart with the documented version and
preserve the complete existing scheduler.extender.extraArgs array while adding
the -v=5 or -v=4 flag, rather than replacing the array.
Source: MCP tools
A Pod that requests HAMi resources and stays Pending gets a `FilteringFailed` event carrying one of the reason codes defined in `pkg/device/common/common.go`, but the website documents none of them. The only page that mentions any of these codes, `developers/scheduler-event-log.md`, is a design proposal written in future tense and covers 7 of the 17 codes the scheduler emits today. Add a user-facing troubleshooting page that: - explains the two message formats a user actually sees: the aggregated event `N nodes <Code>(node-a,node-b)` and the per-node scheduler log `NodeUnfitPod ... reason="3/8 CardInsufficientMemory, ..."` - documents all 17 reason codes with the condition that triggers each one and the corresponding fix, grouped by the stage at which the scheduler rejects the card - flags three behaviours that are easy to misread: one event is emitted per reason code (so a single failure can produce several events), events appear only when no node fits at all, and `AllocatedCardsInsufficientRequest` inverts the numerator to count cards that fit rather than cards rejected - covers the messages emitted before per-device filtering runs, which carry no reason code at all - shows how to raise the extender to `-v=5` for device-level detail, with the cost of doing so and how to revert Verified against HAMi v2.9.0. Links added from the existing scheduler event log design page in both locales. Signed-off-by: ipsitapp8 <ipsitapp8@gmail.com>
b1f3b11 to
3e72faa
Compare
1 participant
What
Adds a troubleshooting page for the reason codes the HAMi scheduler puts in
FilteringFailedevents when a Pod won't schedule.Why
When a Pod that requests HAMi resources stays Pending, the scheduler has already said why:
There are 17 of these codes in
pkg/device/common/common.goand the site documents none of them. The closest page isdevelopers/scheduler-event-log.md, but that one is a design proposal written in future tense and it only lists 7. So if you hitCardTimeSlicingExhaustedorAllocatedCardsInsufficientRequest, there is nowhere to look it up.What's in the page
N nodes <Code>(node-a,node-b)) and the scheduler log line (NodeUnfitPod ... reason="3/8 CardInsufficientMemory").AllocatedCardsInsufficientRequestcounts the cards that fit, not the ones that didn't.-v=5for per-device detail, how much log volume that costs, and how to revert.Notes
Everything was checked against HAMi v2.9.0 source and the page says which version it reflects. Added as a new file instead of editing
troubleshooting.mdso it doesn't clash with #728. Chinese translation included. Not backported toversioned_docs/since it's new content.markdownlint,prettier --checkanddocusaurus build(en + zh) all pass locally.