diff --git a/docs/installation/openshift-installation.md b/docs/installation/openshift-installation.md new file mode 100644 index 000000000..9e2a6c384 --- /dev/null +++ b/docs/installation/openshift-installation.md @@ -0,0 +1,211 @@ +--- +title: Deploy HAMi on OpenShift +sidebar_label: HAMi on OpenShift +translated: true +--- + +This guide covers deploying HAMi on OpenShift, including SCC, random UIDs, non-privileged ports, and SELinux. + +## Prerequisites + +- OpenShift cluster with the NVIDIA GPU Operator installed +- NVIDIA drivers and Container Toolkit ready +- NVIDIA RuntimeClass configured by the GPU Operator +- Permission to create cluster-scoped `SecurityContextConstraints` (SCC) + +Verify the cluster: + +```bash +oc get runtimeclass +oc get nodes -L nvidia.com/gpu.present +oc describe node | grep -A5 Taints +``` + +Example values in this guide assume the following GPU Operator layout. Adjust names and paths to match your cluster: + +```text +RuntimeClass: nvidia +node label: nvidia.com/gpu.present=true +taint: nvidia.com/gpu=true:NoSchedule +driver root: /run/nvidia/driver +toolkit validation: /run/nvidia/validations +``` + +## Recommended configuration + +Create a dedicated project for HAMi: + +```bash +oc new-project hami +``` + +Create `values-openshift.yaml`: + +```yaml +platform: + openshift: true + +openshift: + securityContextConstraints: + create: true + name: hami-device-plugin + +selinux: + enabled: true + type: container_file_t + level: s0 + +scheduler: + service: + httpPort: 443 + httpTargetPort: 9443 + +devicePlugin: + runtimeClassName: nvidia + nvidiaDriverRoot: /run/nvidia/driver + + gpuOperatorToolkitReady: + enabled: true + hostPath: /run/nvidia/validations + + nvidiaNodeSelector: + nvidia.com/gpu.present: "true" +``` + +To use the built-in OpenShift `privileged` SCC instead of creating a custom SCC: + +```yaml +openshift: + securityContextConstraints: + create: false + name: privileged +``` + +Install HAMi: + +```bash +helm repo add hami-charts https://project-hami.github.io/HAMi/ +helm repo update + +helm upgrade --install hami hami-charts/hami \ + --namespace hami \ + --create-namespace \ + -f values-openshift.yaml +``` + +## Security model + +### Scheduler and admission + +Scheduler and admission run under OpenShift `restricted-v2` or an equivalent restricted SCC. + +Security context applied when OpenShift is enabled: + +```yaml +securityContext: + runAsNonRoot: true + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL + seccompProfile: + type: RuntimeDefault +``` + +OpenShift assigns the container UID from the project UID range. On non-OpenShift clusters, the chart default remains `scheduler.patch.runAsUser: 2000`. + +Port mapping: + +```text +Service port 443 -> targetPort 9443 -> containerPort 9443 +``` + +The Deployment, Service, and kube-scheduler extender ConfigMap use the same target port. + +### Device plugin + +The chart creates the `hami-device-plugin` SCC and grants it to the device-plugin ServiceAccount. The SCC allows: + +- privileged container +- host PID +- hostPath +- `SYS_ADMIN` capability +- `RunAsAny` for UID and SELinux context + +Allowed volume types: `configMap`, `downwardAPI`, `emptyDir`, `hostPath`, `projected`, `secret`. Host IPC, host network, and host ports remain off. + +Scheduler, admission, and workload ServiceAccounts continue to use the platform restricted SCC. + +## SELinux + +Enable SELinux relabeling: + +```yaml +selinux: + enabled: true + type: container_file_t + level: s0 +``` + +The relabel initContainer applies `container_file_t` to HAMi shared directories: + +```text +/usr/local/vgpu +/usr/local/vgpu/containers +/tmp/vgpulock +``` + +Restricted workload containers can then access these paths under the standard SELinux container domain. + +The NVIDIA driver root is managed by the GPU Operator and mounted read-only into the device plugin and monitor: + +```text +/run/nvidia/driver +``` + +SELinux labels on the driver root remain under GPU Operator ownership. After uninstall, restore host SELinux labels and directory permissions manually if the node requires it. + +## Verification + +Render manifests: + +```bash +helm template hami hami-charts/hami \ + --namespace hami \ + -f values-openshift.yaml > /tmp/hami-openshift.yaml + +grep -nE 'SecurityContextConstraints|system:openshift:scc|http_bind|targetPort|urlPrefix|runtimeClassName' \ + /tmp/hami-openshift.yaml +``` + +Check rollout status: + +```bash +oc rollout status deployment/hami-scheduler -n hami +oc rollout status daemonset/hami-device-plugin -n hami +``` + +Check assigned SCC: + +```bash +oc get pods -n hami \ + -o 'custom-columns=NAME:.metadata.name,SCC:.metadata.annotations.openshift\.io/scc' +``` + +Expected results: + +- scheduler and admission: restricted SCC +- device-plugin: `hami-device-plugin` SCC +- scheduler extender listens on `9443` +- scheduler Service maps `443` to `9443` +- device-plugin uses the configured NVIDIA RuntimeClass +- SELinux relabel scope: HAMi shared directories + +Check SELinux labels on the node: + +```bash +oc debug node/ -- chroot /host \ + ls -Zd /usr/local/vgpu /usr/local/vgpu/containers /tmp/vgpulock +``` + +HAMi shared directories should show the configured `container_file_t`. diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/installation/openshift-installation.md b/i18n/zh/docusaurus-plugin-content-docs/current/installation/openshift-installation.md new file mode 100644 index 000000000..68c0b6434 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/installation/openshift-installation.md @@ -0,0 +1,211 @@ +--- +title: 在 OpenShift 上部署 HAMi +sidebar_label: OpenShift 上的 HAMi +translated: true +--- + +本文说明如何在 OpenShift 上部署 HAMi,涵盖 SCC、随机 UID、非特权端口和 SELinux。 + +## 前置条件 + +- OpenShift 集群已安装 NVIDIA GPU Operator +- NVIDIA 驱动和 Container Toolkit 已就绪 +- 集群中已有 GPU Operator 配置的 NVIDIA RuntimeClass +- 具备创建集群级 `SecurityContextConstraints`(SCC)的权限 + +确认集群配置: + +```bash +oc get runtimeclass +oc get nodes -L nvidia.com/gpu.present +oc describe node | grep -A5 Taints +``` + +下文示例基于常见的 GPU Operator 配置。若名称或路径不同,请按集群实际情况调整: + +```text +RuntimeClass: nvidia +node label: nvidia.com/gpu.present=true +taint: nvidia.com/gpu=true:NoSchedule +driver root: /run/nvidia/driver +toolkit validation: /run/nvidia/validations +``` + +## 推荐配置 + +为 HAMi 创建独立项目: + +```bash +oc new-project hami +``` + +创建 `values-openshift.yaml`: + +```yaml +platform: + openshift: true + +openshift: + securityContextConstraints: + create: true + name: hami-device-plugin + +selinux: + enabled: true + type: container_file_t + level: s0 + +scheduler: + service: + httpPort: 443 + httpTargetPort: 9443 + +devicePlugin: + runtimeClassName: nvidia + nvidiaDriverRoot: /run/nvidia/driver + + gpuOperatorToolkitReady: + enabled: true + hostPath: /run/nvidia/validations + + nvidiaNodeSelector: + nvidia.com/gpu.present: "true" +``` + +若使用 OpenShift 内置 `privileged` SCC: + +```yaml +openshift: + securityContextConstraints: + create: false + name: privileged +``` + +安装 HAMi: + +```bash +helm repo add hami-charts https://project-hami.github.io/HAMi/ +helm repo update + +helm upgrade --install hami hami-charts/hami \ + --namespace hami \ + --create-namespace \ + -f values-openshift.yaml +``` + +## 安全模型 + +### Scheduler 和 admission + +scheduler 与 admission 使用 OpenShift `restricted-v2` 或等价的 restricted SCC。 + +启用 OpenShift 时应用的安全上下文: + +```yaml +securityContext: + runAsNonRoot: true + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL + seccompProfile: + type: RuntimeDefault +``` + +容器 UID 由 OpenShift 从项目 UID range 分配。非 OpenShift 集群仍使用 Chart 默认值 `scheduler.patch.runAsUser: 2000`。 + +端口映射: + +```text +Service port 443 -> targetPort 9443 -> containerPort 9443 +``` + +Deployment、Service 与 kube-scheduler extender ConfigMap 使用相同的 target port。 + +### Device plugin + +Chart 创建 `hami-device-plugin` SCC,并授予 device-plugin ServiceAccount。该 SCC 允许: + +- privileged container +- host PID +- hostPath +- `SYS_ADMIN` capability +- UID 与 SELinux context 使用 `RunAsAny` + +允许的 volume 类型:`configMap`、`downwardAPI`、`emptyDir`、`hostPath`、`projected`、`secret`。host IPC、host network、host ports 保持关闭。 + +scheduler、admission 与业务工作负载继续使用平台 restricted SCC。 + +## SELinux + +启用 SELinux relabel: + +```yaml +selinux: + enabled: true + type: container_file_t + level: s0 +``` + +relabel initContainer 为 HAMi 共享目录设置 `container_file_t`: + +```text +/usr/local/vgpu +/usr/local/vgpu/containers +/tmp/vgpulock +``` + +受限业务容器即可在标准 SELinux container domain 下访问这些路径。 + +NVIDIA driver root 由 GPU Operator 管理,并以只读方式挂载到 device-plugin 和 monitor: + +```text +/run/nvidia/driver +``` + +driver root 的 SELinux label 由 GPU Operator 维护。卸载后如需恢复宿主机 SELinux label 与目录权限,请手动处理。 + +## 验证 + +渲染清单: + +```bash +helm template hami hami-charts/hami \ + --namespace hami \ + -f values-openshift.yaml > /tmp/hami-openshift.yaml + +grep -nE 'SecurityContextConstraints|system:openshift:scc|http_bind|targetPort|urlPrefix|runtimeClassName' \ + /tmp/hami-openshift.yaml +``` + +检查组件状态: + +```bash +oc rollout status deployment/hami-scheduler -n hami +oc rollout status daemonset/hami-device-plugin -n hami +``` + +检查 Pod 使用的 SCC: + +```bash +oc get pods -n hami \ + -o 'custom-columns=NAME:.metadata.name,SCC:.metadata.annotations.openshift\.io/scc' +``` + +预期结果: + +- scheduler 与 admission:restricted SCC +- device-plugin:`hami-device-plugin` SCC +- scheduler extender 监听 `9443` +- scheduler Service 将 `443` 映射到 `9443` +- device-plugin 使用配置的 NVIDIA RuntimeClass +- SELinux relabel 范围:HAMi 共享目录 + +检查节点 SELinux label: + +```bash +oc debug node/ -- chroot /host \ + ls -Zd /usr/local/vgpu /usr/local/vgpu/containers /tmp/vgpulock +``` + +HAMi 共享目录应显示配置的 `container_file_t`。 diff --git a/sidebars.js b/sidebars.js index 3487c0e9f..dbf482ef7 100644 --- a/sidebars.js +++ b/sidebars.js @@ -56,6 +56,7 @@ module.exports = { "installation/uninstall", "installation/webui-installation", "installation/aws-installation", + "installation/openshift-installation", "installation/how-to-use-hami-dra", "installation/how-to-use-volcano-vgpu", "installation/how-to-use-volcano-ascend",