-
Notifications
You must be signed in to change notification settings - Fork 85
docs(openshift): add openshift installation guide #765
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,211 @@ | ||
| --- | ||
| title: Deploy HAMi on OpenShift | ||
| sidebar_label: HAMi on OpenShift | ||
| translated: true | ||
| --- | ||
|
|
||
| This guide covers deploying HAMi on OpenShift, including SCC, random UIDs, non-privileged ports, and SELinux. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| - OpenShift cluster with the NVIDIA GPU Operator installed | ||
| - NVIDIA drivers and Container Toolkit ready | ||
| - NVIDIA RuntimeClass configured by the GPU Operator | ||
| - Permission to create cluster-scoped `SecurityContextConstraints` (SCC) | ||
|
|
||
| Verify the cluster: | ||
|
|
||
| ```bash | ||
| oc get runtimeclass | ||
| oc get nodes -L nvidia.com/gpu.present | ||
| oc describe node <gpu-node> | grep -A5 Taints | ||
| ``` | ||
|
|
||
| Example values in this guide assume the following GPU Operator layout. Adjust names and paths to match your cluster: | ||
|
|
||
| ```text | ||
| RuntimeClass: nvidia | ||
| node label: nvidia.com/gpu.present=true | ||
| taint: nvidia.com/gpu=true:NoSchedule | ||
| driver root: /run/nvidia/driver | ||
| toolkit validation: /run/nvidia/validations | ||
| ``` | ||
|
|
||
| ## Recommended configuration | ||
|
|
||
| Create a dedicated project for HAMi: | ||
|
|
||
| ```bash | ||
| oc new-project hami | ||
| ``` | ||
|
|
||
| Create `values-openshift.yaml`: | ||
|
|
||
| ```yaml | ||
| platform: | ||
| openshift: true | ||
|
|
||
| openshift: | ||
| securityContextConstraints: | ||
| create: true | ||
| name: hami-device-plugin | ||
|
|
||
| selinux: | ||
| enabled: true | ||
| type: container_file_t | ||
| level: s0 | ||
|
|
||
| scheduler: | ||
| service: | ||
| httpPort: 443 | ||
| httpTargetPort: 9443 | ||
|
Comment on lines
+59
to
+61
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -eu
printf '%s\n' '--- candidate files ---'
git ls-files | rg '(^|/)(docs/installation/openshift-installation\.md|i18n/zh/docusaurus-plugin-content-docs/current/installation/openshift-installation\.md|charts/hami/.*scheduler.*|charts/hami/.*values.*)$' || true
printf '%s\n' '--- English guide relevant sections ---'
sed -n '40,75p;150,175p;195,220p' docs/installation/openshift-installation.md
printf '%s\n' '--- Chinese guide relevant sections ---'
sed -n '40,75p;150,175p;195,220p' i18n/zh/docusaurus-plugin-content-docs/current/installation/openshift-installation.md
printf '%s\n' '--- chart scheduler references ---'
rg -n -S 'bind|targetPort|containerPort|scheduler.*port|httpPort|httpTargetPort|9443|scheduler.*url|scheduler.*URL' charts/hami charts 2>/dev/null | head -250
printf '%s\n' '--- chart tree ---'
git ls-files charts/hami | head -200Repository: Project-HAMi/website Length of output: 3611 🏁 Script executed: #!/bin/bash
set -eu
printf '%s\n' '--- repository references to the chart and scheduler ports ---'
rg -n -S 'hami-charts/hami|scheduler extender|httpTargetPort|httpPort|9443|scheduler.*bind|bind.*port|scheduler.*url|scheduler.*URL' . \
-g '!node_modules' -g '!build' -g '!dist' | head -300
printf '%s\n' '--- guide headings and links ---'
rg -n -S 'helm|chart|scheduler|Service|service|9443|443|GitHub|github' \
docs/installation/openshift-installation.md \
i18n/zh/docusaurus-plugin-content-docs/current/installation/openshift-installation.md
printf '%s\n' '--- upstream chart repository metadata (read-only remote queries) ---'
for url in \
'https://raw.githubusercontent.com/Project-HAMi/HAMi/master/charts/hami/values.yaml' \
'https://raw.githubusercontent.com/Project-HAMi/HAMi/master/charts/hami/templates/scheduler/deployment.yaml' \
'https://raw.githubusercontent.com/Project-HAMi/HAMi/master/charts/hami/templates/scheduler/service.yaml' \
'https://raw.githubusercontent.com/Project-HAMi/HAMi/master/charts/hami/templates/scheduler/configmap.yaml'
do
printf '\n--- %s ---\n' "$url"
curl -L --fail --silent --show-error "$url" | \
rg -n -C 4 '9443|httpPort|httpTargetPort|containerPort|bind|url|URL|scheduler' | head -160 || true
doneRepository: Project-HAMi/website Length of output: 50376 🏁 Script executed: #!/bin/bash
set -eu
log=/tmp/coderabbit-shell-logs/shell-output-WySvM9
printf '%s\n' '--- upstream query sections from the prior result ---'
rg -n -C 2 '^--- https://raw.githubusercontent.com/Project-HAMi/HAMi/master/charts/hami' "$log" || true
printf '%s\n' '--- exact guide context around the port contract ---'
sed -n '108,130p' docs/installation/openshift-installation.md
sed -n '108,130p' i18n/zh/docusaurus-plugin-content-docs/current/installation/openshift-installation.md
printf '%s\n' '--- current upstream chart files, reduced to relevant keys ---'
for file in values.yaml templates/scheduler/deployment.yaml templates/scheduler/service.yaml templates/scheduler/configmap.yaml; do
url="https://raw.githubusercontent.com/Project-HAMi/HAMi/master/charts/hami/$file"
printf '\n--- %s ---\n' "$file"
curl -L --fail --silent --show-error "$url" |
awk 'BEGIN{p=0} /9443|httpPort|httpTargetPort|containerPort|bind|url|URL|scheduler/{p=1} p{print} p && NR>0{c++} c>=80{exit}' |
head -100
doneRepository: Project-HAMi/website Length of output: 13215 🏁 Script executed: #!/bin/bash
set -eu
for file in values.yaml templates/scheduler/deployment.yaml templates/scheduler/service.yaml templates/scheduler/configmap.yaml; do
url="https://raw.githubusercontent.com/Project-HAMi/HAMi/master/charts/hami/$file"
tmp=$(mktemp)
curl -L --fail --silent --show-error "$url" > "$tmp"
printf '\n--- %s: matching lines ---\n' "$file"
rg -n -C 8 '9443|httpTargetPort|httpPort|containerPort|targetPort|urlPrefix|--port|port:|bind|listen|LISTEN|admissionWebhook' "$tmp" | head -240
rm -f "$tmp"
doneRepository: Project-HAMi/website Length of output: 10955 🏁 Script executed: #!/bin/bash
set -eu
printf '%s\n' '--- installation command and chart version context ---'
sed -n '82,100p' docs/installation/openshift-installation.md
sed -n '82,100p' i18n/zh/docusaurus-plugin-content-docs/current/installation/openshift-installation.md
printf '%s\n' '--- read-only behavioral verifier for the documented values ---'
python3 - <<'PY'
import re
import urllib.request
base = "https://raw.githubusercontent.com/Project-HAMi/HAMi/master/charts/hami/"
files = {}
for name in (
"templates/scheduler/deployment.yaml",
"templates/scheduler/service.yaml",
"templates/scheduler/configmap.yaml",
):
with urllib.request.urlopen(base + name) as response:
files[name] = response.read().decode()
deployment = files["templates/scheduler/deployment.yaml"]
service = files["templates/scheduler/service.yaml"]
configmap = files["templates/scheduler/configmap.yaml"]
# The documented values leave admissionWebhook.enabled at its chart default: true.
assert "admissionWebhook:\n enabled: true" in urllib.request.urlopen(
base + "values.yaml"
).read().decode()
bind = re.search(r"--http_bind=0\.0\.0\.0:(\d+)", deployment)
container = re.search(r"containerPort: (\d+)", deployment)
target = re.search(r"targetPort: \{\{ \.Values\.scheduler\.service\.httpTargetPort \| default 443 \}\}", service)
url = re.search(r'urlPrefix: "https://127\.0\.0\.1:(\d+)"', configmap)
print({
"scheduler_bind_port": bind.group(1) if bind else None,
"container_port": container.group(1) if container else None,
"service_target_port": "documented override: 9443",
"configmap_url_port": url.group(1) if url else None,
})
assert bind and bind.group(1) == "443"
assert container and container.group(1) == "443"
assert target
assert url and url.group(1) == "443"
print("Documented targetPort 9443 does not match the chart listener or extender URL.")
PYRepository: Project-HAMi/website Length of output: 3297 🏁 Script executed: #!/bin/bash
set -eu
tmpdir=$(mktemp -d)
trap 'rm -rf "$tmpdir"' EXIT
for file in values.yaml templates/scheduler/deployment.yaml templates/scheduler/service.yaml templates/scheduler/configmap.yaml; do
mkdir -p "$tmpdir/$(dirname "$file")"
curl -L --fail --silent --show-error \
"https://raw.githubusercontent.com/Project-HAMi/HAMi/master/charts/hami/$file" \
-o "$tmpdir/$file"
done
python3 - "$tmpdir" <<'PY'
import pathlib
import re
import sys
root = pathlib.Path(sys.argv[1])
values = (root / "values.yaml").read_text()
deployment = (root / "templates/scheduler/deployment.yaml").read_text()
service = (root / "templates/scheduler/service.yaml").read_text()
configmap = (root / "templates/scheduler/configmap.yaml").read_text()
assert re.search(r"admissionWebhook:\s*\n\s*enabled:\s*true", values)
bind = re.search(r"--http_bind=0\.0\.0\.0:(\d+)", deployment)
container = re.search(r"containerPort:\s*(\d+)", deployment)
target = re.search(r"targetPort:\s*\{\{\s*\.Values\.scheduler\.service\.httpTargetPort", service)
url = re.search(r'urlPrefix:\s*"https://127\.0\.0\.1:(\d+)"', configmap)
result = {
"documented_service_port": 443,
"documented_service_target_port": 9443,
"chart_scheduler_bind_port": bind.group(1) if bind else None,
"chart_container_port": container.group(1) if container else None,
"chart_configmap_url_port": url.group(1) if url else None,
"service_target_is_configurable": bool(target),
}
print(result)
assert result["chart_scheduler_bind_port"] == "443"
assert result["chart_container_port"] == "443"
assert result["chart_configmap_url_port"] == "443"
assert result["service_target_is_configurable"]
assert result["documented_service_target_port"] != int(result["chart_scheduler_bind_port"])
print("The documented 443 -> 9443 mapping is inconsistent with the chart's 443 listener and extender URL.")
PYRepository: Project-HAMi/website Length of output: 247 🏁 Script executed: #!/bin/bash
set -eu
tmpdir=$(mktemp -d)
trap 'rm -rf "$tmpdir"' EXIT
for file in values.yaml templates/scheduler/deployment.yaml templates/scheduler/service.yaml templates/scheduler/configmap.yaml; do
mkdir -p "$tmpdir/$(dirname "$file")"
curl -L --fail --silent --show-error \
"https://raw.githubusercontent.com/Project-HAMi/HAMi/master/charts/hami/$file" \
-o "$tmpdir/$file"
done
python3 - "$tmpdir" <<'PY'
import pathlib
import re
import sys
root = pathlib.Path(sys.argv[1])
values = (root / "values.yaml").read_text()
deployment = (root / "templates/scheduler/deployment.yaml").read_text()
service = (root / "templates/scheduler/service.yaml").read_text()
configmap = (root / "templates/scheduler/configmap.yaml").read_text()
# The chart default enables the TLS scheduler endpoint; comments may occur
# between the key and its value.
admission_enabled = bool(re.search(
r"admissionWebhook:\s*(?:\n\s*#.*)*\n\s*enabled:\s*true", values
))
bind = re.search(r"--http_bind=0\.0\.0\.0:(\d+)", deployment)
container = re.search(r"containerPort:\s*(\d+)", deployment)
target = re.search(
r"targetPort:\s*\{\{\s*\.Values\.scheduler\.service\.httpTargetPort", service
)
url = re.search(r'urlPrefix:\s*"https://127\.0\.0\.1:(\d+)"', configmap)
result = {
"admission_webhook_default_enabled": admission_enabled,
"documented_service_port": 443,
"documented_service_target_port": 9443,
"chart_scheduler_bind_port": bind.group(1) if bind else None,
"chart_container_port": container.group(1) if container else None,
"chart_configmap_url_port": url.group(1) if url else None,
"service_target_is_configurable": bool(target),
}
print(result)
assert admission_enabled
assert result["chart_scheduler_bind_port"] == "443"
assert result["chart_container_port"] == "443"
assert result["chart_configmap_url_port"] == "443"
assert result["service_target_is_configurable"]
assert result["documented_service_target_port"] != int(result["chart_scheduler_bind_port"])
print("The documented 443 -> 9443 mapping is inconsistent with the chart's 443 listener and extender URL.")
PYRepository: Project-HAMi/website Length of output: 515 Align the OpenShift scheduler port contract The chart binds the scheduler extender and container to 📍 Affects 2 files
🤖 Prompt for AI AgentsSource: MCP tools |
||
|
|
||
| devicePlugin: | ||
| runtimeClassName: nvidia | ||
| nvidiaDriverRoot: /run/nvidia/driver | ||
|
|
||
| gpuOperatorToolkitReady: | ||
| enabled: true | ||
| hostPath: /run/nvidia/validations | ||
|
|
||
| nvidiaNodeSelector: | ||
| nvidia.com/gpu.present: "true" | ||
| ``` | ||
|
|
||
| To use the built-in OpenShift `privileged` SCC instead of creating a custom SCC: | ||
|
|
||
| ```yaml | ||
| openshift: | ||
| securityContextConstraints: | ||
| create: false | ||
| name: privileged | ||
| ``` | ||
|
|
||
| Install HAMi: | ||
|
|
||
| ```bash | ||
| helm repo add hami-charts https://project-hami.github.io/HAMi/ | ||
| helm repo update | ||
|
|
||
| helm upgrade --install hami hami-charts/hami \ | ||
| --namespace hami \ | ||
| --create-namespace \ | ||
| -f values-openshift.yaml | ||
| ``` | ||
|
|
||
| ## Security model | ||
|
|
||
| ### Scheduler and admission | ||
|
|
||
| Scheduler and admission run under OpenShift `restricted-v2` or an equivalent restricted SCC. | ||
|
|
||
| Security context applied when OpenShift is enabled: | ||
|
|
||
| ```yaml | ||
| securityContext: | ||
| runAsNonRoot: true | ||
| allowPrivilegeEscalation: false | ||
| capabilities: | ||
| drop: | ||
| - ALL | ||
| seccompProfile: | ||
| type: RuntimeDefault | ||
| ``` | ||
|
|
||
| OpenShift assigns the container UID from the project UID range. On non-OpenShift clusters, the chart default remains `scheduler.patch.runAsUser: 2000`. | ||
|
|
||
| Port mapping: | ||
|
|
||
| ```text | ||
| Service port 443 -> targetPort 9443 -> containerPort 9443 | ||
| ``` | ||
|
|
||
| The Deployment, Service, and kube-scheduler extender ConfigMap use the same target port. | ||
|
|
||
| ### Device plugin | ||
|
|
||
| The chart creates the `hami-device-plugin` SCC and grants it to the device-plugin ServiceAccount. The SCC allows: | ||
|
|
||
| - privileged container | ||
| - host PID | ||
| - hostPath | ||
| - `SYS_ADMIN` capability | ||
| - `RunAsAny` for UID and SELinux context | ||
|
|
||
| Allowed volume types: `configMap`, `downwardAPI`, `emptyDir`, `hostPath`, `projected`, `secret`. Host IPC, host network, and host ports remain off. | ||
|
|
||
| Scheduler, admission, and workload ServiceAccounts continue to use the platform restricted SCC. | ||
|
|
||
| ## SELinux | ||
|
|
||
| Enable SELinux relabeling: | ||
|
|
||
| ```yaml | ||
| selinux: | ||
| enabled: true | ||
| type: container_file_t | ||
| level: s0 | ||
| ``` | ||
|
|
||
| The relabel initContainer applies `container_file_t` to HAMi shared directories: | ||
|
|
||
| ```text | ||
| /usr/local/vgpu | ||
| /usr/local/vgpu/containers | ||
| /tmp/vgpulock | ||
| ``` | ||
|
|
||
| Restricted workload containers can then access these paths under the standard SELinux container domain. | ||
|
|
||
| The NVIDIA driver root is managed by the GPU Operator and mounted read-only into the device plugin and monitor: | ||
|
|
||
| ```text | ||
| /run/nvidia/driver | ||
| ``` | ||
|
|
||
| SELinux labels on the driver root remain under GPU Operator ownership. After uninstall, restore host SELinux labels and directory permissions manually if the node requires it. | ||
|
|
||
| ## Verification | ||
|
|
||
| Render manifests: | ||
|
|
||
| ```bash | ||
| helm template hami hami-charts/hami \ | ||
| --namespace hami \ | ||
| -f values-openshift.yaml > /tmp/hami-openshift.yaml | ||
|
|
||
| grep -nE 'SecurityContextConstraints|system:openshift:scc|http_bind|targetPort|urlPrefix|runtimeClassName' \ | ||
| /tmp/hami-openshift.yaml | ||
| ``` | ||
|
|
||
| Check rollout status: | ||
|
|
||
| ```bash | ||
| oc rollout status deployment/hami-scheduler -n hami | ||
| oc rollout status daemonset/hami-device-plugin -n hami | ||
| ``` | ||
|
|
||
| Check assigned SCC: | ||
|
|
||
| ```bash | ||
| oc get pods -n hami \ | ||
| -o 'custom-columns=NAME:.metadata.name,SCC:.metadata.annotations.openshift\.io/scc' | ||
| ``` | ||
|
|
||
| Expected results: | ||
|
|
||
| - scheduler and admission: restricted SCC | ||
| - device-plugin: `hami-device-plugin` SCC | ||
| - scheduler extender listens on `9443` | ||
| - scheduler Service maps `443` to `9443` | ||
| - device-plugin uses the configured NVIDIA RuntimeClass | ||
| - SELinux relabel scope: HAMi shared directories | ||
|
|
||
| Check SELinux labels on the node: | ||
|
|
||
| ```bash | ||
| oc debug node/<gpu-node> -- chroot /host \ | ||
| ls -Zd /usr/local/vgpu /usr/local/vgpu/containers /tmp/vgpulock | ||
| ``` | ||
|
|
||
| HAMi shared directories should show the configured `container_file_t`. | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
Repository: Project-HAMi/website
Length of output: 5684
🏁 Script executed:
Repository: Project-HAMi/website
Length of output: 2453
🏁 Script executed:
Repository: Project-HAMi/website
Length of output: 509
🏁 Script executed:
Repository: Project-HAMi/website
Length of output: 7060
🏁 Script executed:
Repository: Project-HAMi/website
Length of output: 2444
🏁 Script executed:
Repository: Project-HAMi/website
Length of output: 2444
🏁 Script executed:
Repository: Project-HAMi/website
Length of output: 833
Do not document unsupported Helm values
Chart versions
2.4.0through2.9.0do not define these values or render SCC and SELinux resources. Helm therefore ignores this configuration.Update both guides at
#L47-L58. Add chart support and pin that release, or document the required OpenShift resources separately.📍 Affects 2 files
docs/installation/openshift-installation.md#L47-L58(this comment)i18n/zh/docusaurus-plugin-content-docs/current/installation/openshift-installation.md#L47-L58🤖 Prompt for AI Agents
Source: MCP tools